Language models that act as judges now decide what training data to keep, score generated answers, and rank systems on leaderboards. These judging models are treated as measurement instruments. The study tests a simple assumption behind that treatment: that asking the same model the same question at different times yields the same answer. The authors ran two preregistered audits with all thresholds fixed in advance to see if that assumption holds.

The teams made 52,988 audited request attempts to black‑box observers (large language models, or LLMs). They measured agreement in two ways. Same‑window repeats — asking again within the same browser window — produced a Spearman rank correlation (a standard measure of how similar two rankings are) of 0.400 when the preregistered gate required 0.90. Byte‑identical next‑day replays agreed at 0.78 against a required 0.99. At the same time, execution records showed near‑perfect delivery and logging, so the failures are not due to dropped or missing calls.

The paper identifies three main causes. First, a label‑to‑meaning mapping bias pushed the readouts as strongly as the real signal did. Second, the true differences between candidate answers were often tiny — the gaps between candidates sat seven orders of magnitude below the instrument’s own noise floor — so the judge could not reliably detect them. Third, identical inputs sometimes returned different rankings on replay, and exact‑permutation readouts made that noise worse. The authors also tested metric substitution and sampling strategies and found they did not repair the instability on their test grid.

The authors ran preregistered follow‑ups to bound the problem. Waiting a day did not improve replay stability on the days sampled (0.805 versus 0.800, replicated over five more days). Switching among four providers did not fix it; per‑provider medians ranged from 0.74 to 0.88 and none reached the 0.99 replay gate. Self‑hosting on batch‑invariant kernels helped only when the server was quiet. In experiments with constructed errors, the judge’s separation tracked the type of error rather than the size of the gap.