Researchers show that high internal similarity between visual and text representations inside Multimodal Large Language Models (MLLMs) can be misleading. They call this effect the “alignment illusion.” In many models the usual layer-by-layer similarity scores stay high even after the image input has been scrambled, while the model’s ability to answer image-based questions collapses.

To test this, the team ran controlled interventions on the visual stream of 13 MLLMs from five model families, ranging from about 0.5 billion to 72 billion parameters. They replaced the projector-output visual tokens with Gaussian noise and also tried replacing the input image with a structured but irrelevant image. Standard scalar similarity measures—such as CKA, SVCCA, MIR, and the leading principal-angle cosine—often failed to tell the corrupted cases apart from the original ones, even when task accuracy dropped sharply.

The authors trace the mismatch to the models’ shared language-model pathway. Visual and text tokens are processed together by the same Transformer blocks, including attention layers and multilayer perceptrons (MLPs). Because some MLP down-projections are anisotropic (they favor certain output directions), they can pull both visual and text vectors toward the same dominant direction. That weight-induced alignment looks like genuine cross-modal integration when measured by a single scalar, but it can arise even when the visual input carries no useful content.

To separate weight-induced alignment from genuine multi-directional visual structure, the paper introduces the principal-angle gap (PA gap). The PA gap is the difference between the largest and second-largest principal-angle cosines between visual and text subspaces. In the experiments, the PA gap tracked task accuracy more reliably under graded corruption than the standard scalar scores did. When a structured-but-irrelevant image was used, the PA gap also exposed situations where internal geometry and task performance diverged.