This paper proposes a structured way to study how so-called coding agents — large language models (LLMs) that write and run code — behave when left to discover models from data. The authors argue that these agents are stochastic (random) and adaptive, so a single benchmark run does not capture their behavior. To address this, they treat an LLM-based agent as a model-discovery operator that maps discovery data and an optimization goal to a fitted model, and then study that operator’s variability under controlled conditions.

The team ran a controlled, full-factorial experiment. They tested two commercial coding agents, Codex and Claude Code, across several factors: the agent itself, the task (two types: a predictive task and a generative task), the optimization metric, the composition of training data, and three levels of requested reasoning effort (how much work the agent is asked to do). For each run they recorded a full trace: candidate models the agent explored, the final script it returned, token use, money spent, wall-clock time, and measures of process complexity. They scored performance on held-out data and analyzed multiple responses together using regression and multivariate inference.

A key analytic tool they introduce is the Utility-Aligned Canonical Decomposition (UACD). In plain terms, UACD finds the main direction in which increasing the agent’s requested effort moves the joint outcomes (for example, quality, cost, and complexity). It then checks whether that direction lines up with a user’s utility — a chosen trade-off between performance and cost. This lets researchers see not just whether more effort changes outcomes, but whether those changes match what a user would want.

The experiments use a concrete testbed: a networked word-forming (anagram) game. In that game, small teams of players repeatedly take actions such as forming words, requesting letters, or sharing letters; group payoffs reward cooperation, and each session produces a sequence of interdependent actions. The authors chose this setting because modeling it requires iterative data exploration and feature construction — the same kinds of steps an LLM coding agent would perform.