The paper asks a simple but important question: when a language model gives a probability for an event, do its probabilities hang together in a logically consistent way? This matters because people increasingly use model probabilities to guide decisions about things like investments, disasters, or other risks.

The authors use a classic idea from probability theory called de Finetti’s theorem. It says a set of probability statements is consistent exactly when no gambler can place bets against those statements and make a sure profit. The researchers treat the model’s stated probabilities as betting prices. They then solve a linear program — a type of optimization — to find the largest profit an arbitrageur could guarantee. That guaranteed profit is their measure of “incoherence.” An important advantage of this test is that it needs no real-world outcomes. The check uses only the model’s probabilities and the logical relationships among the events.

To test models in practice, the team asked models to forecast whether future daily stock returns would fall into certain return bins. They built event sets from return bins and combinations of those bins so the logical structure of events was known. The data came from CRSP for returns and Refinitiv for matched news headlines, with an anchor window from August 15, 2025 to December 23, 2025. From 10,000 candidate stock-days they sampled 100 stock-days (50 dates, two stocks per date). They ran 365,100 total probability queries across 15 models. One model, GPT-OSS-120B, served as their main example because it was released before the sample period and has open weights. Prompts were held constant across models and providers’ default settings were used.

The main finding is that incoherence is common and sometimes large. The optimization often finds a positive guaranteed profit, meaning the model’s probabilities violate basic consistency constraints. Incoherence grows when the events have richer logical relationships (for example, when many unions, intersections, and complements are included). The authors also find that adding irrelevant context or changing how questions are posed can raise incoherence by about an order of magnitude. They emphasize that the “profits” here are a diagnostic measure of inconsistency, not real trading returns.