This paper argues that small Transformer models fail on some compositional generalisation tests not because the models are fundamentally bad, but because the training data lack variety in the kinds of grammatical pieces they see. The authors call this property type diversity: how many different constructors produce phrases of a given kind, such as noun phrases or verb phrases. They find that when you increase that variety, Transformers generalise better — and this helps both word-level (lexical) and structure-level (structural) cases equally.

Type diversity is defined in a concrete way. The authors use Grammatical Framework (GF) to describe grammar as two levels: the abstract level that says which categories combine (for example that a sentence is a noun phrase plus a verb phrase), and the concrete level that turns those combinations into strings. A ‘‘constructor’’ is a function in the abstract grammar that builds one kind of phrase. For example, simple noun phrases like “a cat” and noun phrases with a preposition like “a cat on a mat” can come from different constructors. Counting how many different constructors exist for a type is what the paper calls type diversity.

To test the idea, the researchers changed the type diversity in existing benchmark datasets. They used GF to make more linguistically varied versions of the COGS and SLOG benchmarks by adding things like adjectives, plural forms, and verb tenses. They also generated over a hundred variants of the artificial SCAN task with different amounts of type diversity. Then they trained Transformers on these variants and measured how well the models generalised to held-out combinations of words and structures.

Their main finding is that higher type diversity in training data correlates with better compositional generalisation, and this effect is about the same for lexical and structural generalisation. This supports their view that the earlier observation — that structural generalisation is harder than lexical generalisation for Transformers — may have been driven by the original datasets having few structural constructors. They also report that this result conflicts with a previous claim that compound divergence, a measure used by distribution-based compositionality assessment (DBCA), explains generalisation difficulty. The authors further explore other dataset factors, such as the diversity of types besides the novel test structure and surface details like the order of terms in the logical output, and show those factors can matter too.