This paper shows how a simple idea from machine learning can give choice modellers a clearer view of where their models work and where they fail. Choice modelling has long relied on aggregate fit measures such as log-likelihood, Akaike or Bayesian Information Criteria, and rho-squared. Those measures summarise overall fit but can hide systematic weaknesses at the level of individual alternatives—for example when some options are chosen much less often than others.

The authors introduce a probabilistic version of the confusion matrix. A standard confusion matrix counts how often a model’s single best guess matches the observed choice. The probabilistic version instead records, for each actually chosen alternative, the average probability the model assigned to every alternative across all choice tasks. In other words, it shows how probability mass is spread across competitors when a particular option was chosen.

Using this matrix, the paper derives probabilistic performance metrics that can be applied to both traditional discrete choice models (including the long-standing Random Utility Maximisation framework) and machine learning classifiers. This lets researchers compare models using overall fit (like log-likelihood) and alternative-level prediction patterns at the same time. The authors report that models with similar log-likelihood values can have very different confusion matrices, meaning they make different kinds of mistakes even if their aggregate fit looks the same.

Why this matters: the probabilistic confusion matrix reveals which alternatives a model systematically “confuses” and where probability is being misallocated. That information can point to specific improvements—adding explanatory variables, interaction terms, or changing functional forms for the alternatives that the model struggles with. It also helps explain why some models predict better out-of-sample: the paper finds that evaluating these matrices out-of-sample exposes shifts in alternative-level predictions that materially affect forecasting performance.