This paper introduces SWE-Flux, a new test suite that checks whether large language models (LLMs) can reason about how real programs run. The authors built 480 execution-grounded questions drawn from 12 real Python repositories. Instead of asking people or other models to write answers, SWE-Flux gathers gold answers automatically by running instrumented tests and recording the actual program behavior.

The benchmark focuses on repository-level reasoning, not just small snippets or single functions. It includes both single-test and multi-test questions that probe control flow, loops, program state, dataflow, exceptions, and program invariants (properties that hold across runs). The use of instrumented test executions means the correct answers reflect real runtime outcomes rather than human judgement or another model’s opinion.

To see how well current LLMs handle this task, the team evaluated five models. The best model reached only 37% accuracy on the benchmark, which shows the task is still hard. The models tended to do better on localized behavior such as simple loops, exceptions, intra-procedural control flow (flow inside a single function), and invariant questions. They struggled more with dataflow that crosses functions, inter-procedural execution (how data moves between functions), precise reasoning about program state, and aggregating results across a whole test suite.

The authors also built an oracle-harvesting pipeline that can create fresh benchmark variants by perturbing inputs. That pipeline successfully produced valid variants for almost 90% of selected instances. Those new variants were notably harder for the evaluated models, suggesting the benchmark can be expanded and made more challenging automatically.

Why this matters: many tools now use LLMs to help write and diagnose code, but SWE-Flux shows those models have important blind spots when asked to reason about actual program runs. Important caveats include the benchmark’s scope: it covers 12 Python repositories and 480 instances, so it is not a complete picture of all programming tasks. The harvested answers depend on the quality and coverage of the instrumented tests, which can bias what is measured. Overall, the work provides a reproducible, execution-grounded way to measure runtime reasoning, while showing that current models are far from perfect at this skill.