This paper checks whether the factors that large language models (LLMs) list as the top three reasons for a decision really line up with what would change the model’s output. The authors test two common interpretations of a cited factor: necessity (if you change that factor, does the output change?) and sufficiency (if you keep that factor but remove other information, does the output stay the same?). They run controlled, black‑box tests to compare what the model says influenced its answer with what actually influences it, as measured by input changes.

The team used two synthetic tasks. In the advisor recommendation task, they made 100 fictional client profiles with 18 named features (for example, goals, assets, income, and communication preferences) and a fixed pool of 13 advisors. The model picked the best advisor and named the top three client features that influenced its choice. In the prompt‑risk monitoring task, they made 100 prompts that combined a normal request with three or four stored “risk” segments drawn from eight categories (for example, personally identifiable information, jailbreak attempts, illegal advice requests). The model gave a risk score from 1 (benign) to 5 (critical) and named the top three risk features.

To test necessity, the researchers changed only one named feature and saw whether the model’s decision changed. For advisor recommendation they substituted a contrasting value; for prompt monitoring they deleted the associated risk segment. To test sufficiency, they kept one feature and removed other changeable parts. Each modified input was submitted in three API trials and models were queried at temperature 0. The study evaluated eight models from the Claude, GPT, and Gemini families, including Claude Opus 4.6, Sonnet 5, Haiku 4.5, several GPT‑5.4 variants, and Gemini 3.5 Flash. The authors compared the model’s stated ranking with empirical necessity and sufficiency scores using Spearman correlations.