First-hand information for everyone
A new study suggests that some pieces of information inside large language models are “active” — they not only can be recalled, but also support learning and generalization — while others are “inert,” recalled but not usable for building new inferences. The work, by Faridoun Mehri and Hinrich Schütze (arXiv:2610.22300v1), introduces a simple experimental test and reports a striking finding: how linearly a model’s knowledge is encoded before fine‑tuning strongly predicts whether that knowledge will support generalization after fine‑tuning.
A clear toy example used in the paper makes the idea concrete. Suppose a pretrained model already encodes a relation such as “place → continent” (Paris → Europe, Munich → Europe). The authors create a novel task by reassigning the continents to arbitrary labels — for instance, Europe → “crab”, Asia → “zebra” — and fine‑tune the model on a handful of example pairs (Paris → crab). The question is whether the model, once fine‑tuned, will correctly label an unseen but related subject (Munich → crab). Generalization in that test requires using the model’s existing continent knowledge, but applied to an unfamiliar label mapping.
Two properties are measured. Recall is how well the pretrained model can already state the original relation (Paris → Europe). Pre‑linearity measures how well the relation’s answers can be read out from the model’s internal subject representations by a simple linear probe — in other words, whether the relevant structure is linearly separable in the model’s activations. Generalization is how accurately the fine‑tuned model assigns labels to held‑out subjects that share answers with the trained examples.
Across 11 open‑weight language models (from ~2B to 27B parameters) and 70 different relations, the main result is consistent: relations with higher Pre‑linearity before fine‑tuning lead to much better generalization after fine‑tuning. Within each model, the correlation between Pre‑linearity and Generalization across relations was very high (Pearson r ≈ 0.87–0.92). This relationship remained strong when the authors controlled for other plausible explanations — including how well the model recalled the relation beforehand, how many labels or examples the relation had, and how frequent the subjects were in pretraining data.
The authors also probed failure modes. When fine‑tuned models produced an incorrect label for a held‑out subject, that wrong label often corresponded to the answer the linear probe would predict for the subject (for example, if the model output “zebra” for Munich, the probe’s mapping would recover that Munich’s underlying continent label was Asia). In other words, fine‑tuning appears to exploit the linear structure that the probe detects; when that structure is absent, generalization fails even if the model could recall the fact.
Why this matters: the study separates two different senses of “knowing” in language models. A model can store a fact in a way that makes it easy to read out with a linear decoder — and those facts are the ones that support efficient transfer or learning new, related mappings. Conversely, facts that are memorized in a less linearly organized way may be accessible for direct recall but less useful when building new associations. This insight could inform work on knowledge editing, model interpretability, and techniques that aim to predict when a model can be taught or safely updated without extensive retraining.
Limitations and caveats
- The experimental design intentionally constructs synthetic new relations by randomly relabeling real answers (e.g., mapping continents to animal names) and evaluates single‑token answers under controlled prompts. This isolates the mechanism but is not the same as all real‑world knowledge updates. - Probes and fine‑tuning were applied with a specific recipe (layer selection, PCA to 16 dimensions, regularized linear classifiers, full fine‑tuning). Different probing or fine‑tuning setups could change numerical results. - The study uses a set of open‑weight models and a curated set of relations that meet data requirements; results may vary for other architectures, sizes, or relations outside the tested pool. - Correlation does not prove causation: strong linear readout predicts generalization, but further work is needed to show causal mechanisms and to discover how to modify a model’s representations to make inert facts active.
Takeaway
Mehri and Schütze’s experiments argue that not all remembered facts are equally useful for learning: it’s the ease with which a relation is read out linearly from internal representations — its “Pre‑linearity” — that best predicts whether that relation will support generalization after fine‑tuning. The result offers a practical diagnostic for when a model’s stored knowledge can be leveraged for new tasks, and points to representation geometry as an important factor in how language models learn and adapt.