Reproducing zero-shot variant prediction

[Turakar]

I am trying to reproduce the zero-shot results from the ESM paper by Meier et al. (2021). You can find my current efforts here. The main implementation is in the zeroshot.ipynb notebook which you can open in GitHub.

At the moment, I get vastly different results than the official values. While you report a spearman rho of > 0.6 for BLAT_ECOLX_Ranganathan2015, I only obtain a value of ~0.1. Thus, I kindly request you to review the notebook for any issues. I ensured that the following conditions are met:

• I use the 5-model ensemble of ESM-1v.
• I use the masked marginal strategy as in the paper (strategy (a) / Eq. (1)).
• I try to reproduce the values from Fig. 3.
• I use the same wildtype sequence as provided in examples/variant-prediction.
• I use the same mutations (from the raw_df.csv file linked there).

I am surprised by these differences. Could you clear up my confusion?

[Turakar]

I found an error in my implementation, I will report back if there are further issues.