Pooling strategy to obtain Protein Level Embeddings from esm-2 and esm-if

[harshagrawal13]

I am trying to train a Siamese neural network to create joint embeddings for protein sequences (generated by esm2) and protein structures (generated by esm_if). My aim is that the unified embedding of an entire protein, both sequential and structural, should be similar.
I want to gain more insight into which pooling strategy to use for both esm_if and esm2 to go from residue level representations (Batch size * Num Residues * Embedding Size) to protein level representations (Batch size * Embedding Size).

I've tried mean pooling but that seems to eat away a lot of useful information. Is it wise to use BOS token embedding or any other pooling strategies?