Is there any guides for visualizing the attentions of ESM models

[Moeinh77]

Hi, I was trying to visualizae the attentions of ESMv2 with BertViz but I had a couple of questions. When I encode two inputs with encode_plus and the model adds the special tokens to the sequences, should I consider the separating token between the two inputs as a part of sequence 1 or 2? Also, can I expect to find any meaningful connections between two sequences if I visualize the attention of the model and look for seq1-> seq2 connections?

[tomsercu]

I don't know BertViz and encode_plus. Our LMs were not trained with concatenating two independent protein sequences so unclear what to expect in the contact maps. It's reasonable to try though if the sequences actually form a complex I'd expect it to show up int he contact maps as well - you could connect with a linker of 25 glycines as in the ESMFold paper:

To predict complexes shown in Figure 2D, we give a residue index break of 1000 to ESMFold and link chains with a 25-residue poly-glycine linker, which we remove before displaying. Note that this is using ESMFold out of distribution since single chains are used during training.