This paper introduces Comparison-based Preference Optimization, or ComPO, a new way to align large language models (LLMs) with human preferences. The method avoids directly optimizing the usual likelihood-based loss on human preference pairs. Instead, it uses simple comparisons between model outputs to extract the direction in which the model should change. The authors argue this can help when small differences in model likelihoods—what they call “likelihood displacement”—make direct likelihood-based learning uninformative.

ComPO is a “zeroth-order” method. That means it does not need gradients of a differentiable preference loss on each pair. It relies on a comparison oracle, a procedure that tells which of two outputs is better. From many such pairwise comparisons, ComPO estimates which way to move the model’s behavior without taking gradient steps with respect to a numeric loss on each pair. The paper presents both an offline version and an online version of the algorithm.

The authors give theoretical support for their offline scheme. They prove a convergence guarantee under three assumptions: smoothness (the underlying objective changes in a controlled way), gradient sparsity (only a few directions matter at a time), and compatibility between the comparison oracle and a hidden objective the oracle is implicitly ranking. For online ComPO, the method keeps the same comparison mechanism but also uses unlabeled model outputs to control a reverse Kullback–Leibler (reverse-KL) divergence relative to a reference policy. Reverse-KL control is a way to keep the updated model close to a baseline behavior while changing it in the preferred direction.

Why this matters: direct preference alignment methods are popular because they are fast and memory-efficient, but they can struggle when preference pairs have only small likelihood differences. ComPO offers an alternative that pulls information from comparisons rather than raw likelihoods. The authors report experimental improvements over existing direct alignment methods on several models, including Mistral, Llama, Gemma-2, Qwen3, and Gemma-3. They also show length-controlled win rates and pair-level diagnostics the authors say are consistent with reducing the problems caused by likelihood displacement.