First-hand information for everyone
Researchers found that protein language models (pLMs) do not keep accurate contact predictions when every amino‑acid letter is consistently renamed. The test, a global bijective relabeling that shuffles the alphabet but keeps sequence alignments unchanged, showed that all checked pLMs collapse their contact maps, whereas classical pseudolikelihood Potts models and inverse‑covariance estimators remain perfectly invariant.
The study introduced this diagnostic framework to probe whether contact readouts depend only on column statistics—what classic models use—or on explicit residue identities. Multiple architectures (ESM‑2, MSA Transformer, AMPLIFY) were tested at sizes from 8 million to 3 billion parameters. After relabeling, approximately 0.05 of the 38 contacts predicted by ESM‑2 650 M overlapped with the original set.
Results suggest that pLMs rely on “alphabet anchoring,” i.e., they use explicit, canonical residue identities rather than pure coevolutionary statistics. Even after increasing scale, performance still drops under relabeling.
However, the evaluation uses nine short, single‑domain proteins from four Pfam families and only three pLM families. It does not cover multi‑domain proteins or fine‑tuned models, and it tested only one grouping scheme based on BLOSUM62.
These findings show a gap between how pLMs and classical coevolutionary models compute contacts, calling for more diagnostics to understand what protein language models learn.