This paper presents a practical way to get more accurate estimates of how an AI system performs on small slices of data. The authors point out that reporting a single overall score can hide big differences across task types or user groups. But measuring performance for every slice can be costly because each measurement requires running the system and grading its output. To address this, they build a workflow that improves estimates when only a few labeled examples are available for a domain, and a validation method for choosing when to trust the improved estimates.

The researchers treat the whole evaluation set as a finite population and focus on estimating the mean outcome within each reporting domain. They note that ‘‘direct’’ methods, including prediction-powered inference (PPI), only use labels from a single domain and therefore can be noisy when labels are few. To reduce that noise they introduce prediction-powered smoothing (PP-S). PP-S is a Bayesian smoothing step fit to each domain’s PPI estimate. They also offer an extension called PP-TS that ‘‘borrows strength’’ across a reporting taxonomy — in other words, it uses data from related domains to improve estimates for each one. For validation, they derive a new design-based cross-validation score that is approximately unbiased and helps choose between direct and smoothed estimators.

At a high level, prediction-powered approaches use model predictions to supplement scarce labels and lower variance. The smoothing step then shrinks very noisy domain estimates toward a more stable center, with PP-TS allowing domains that are grouped in a taxonomy to pull information from one another. The new validation score operates under survey-sampling ideas: it evaluates candidate estimators using the same labeled budget, rather than requiring a separate, large holdout set. This makes it possible to pick and assess estimators without paying for a second set of labels.