Video encoders usually pick settings by minimizing the sum of squared errors (SSE). SSE is easy to use inside an encoder because it breaks the image into independent blocks. But SSE does not always match what people see. Full-reference image quality metrics (FR-IQA) such as SSIM, MS-SSIM, LPIPS, DISTS and a Wasserstein-based metric tend to agree better with human judgment. The problem is that these metrics need the whole decoded image and do not split into block-wise terms, so they are hard to use “in-loop” inside the encoder.

This paper shows how to approximate many FR-IQA metrics with a block-wise quadratic distortion that the encoder can use directly. The idea starts from a Taylor expansion of the metric around the original image. That expansion gives an input-dependent quadratic form whose curvature is the Hessian matrix of the metric at the source image. The authors call this approximation the input-dependent quadratic distortion (IDQD). Because the full Hessian is huge and couples pixels across the image, they introduce two practical simplifications: keep only the block-diagonal of the Hessian, or keep only the diagonal (per-pixel) entries.

To get those Hessian pieces without forming the full matrix, the team estimates them stochastically. Their estimators use Hessian-vector products computed by automatic differentiation. For metrics that are squared norms of a residual (for example LPIPS and DISTS), the Hessian has a Gauss–Newton structure and the estimator can use vector–Jacobian products, which lower variance. They also derive a way to set the encoder’s Lagrange multiplier (the rate–distortion balance) for the IDQD using the trace of the Hessian, which simplifies rate control.

The authors put this idea into a modern codec, replacing SSE with IDQD inside rate–distortion optimization (RDO) for Versatile Video Coding (VVC) intra-coding. They tested five FR-IQA targets on the Kodak and CLIC image sets. The block-diagonal estimator reduced BD-rate (a common measure of bitrate savings for a fixed quality target) by 15.5–23.6% on Kodak and 14.2–36.7% on CLIC for the target metric. These gains come with no decoder changes. The encoder runs slower: the reported encoder runtime overhead is about 10–30%. The block-diagonal estimator generally outperformed the diagonal estimator, and it beat a prior per-block QP method (PerceptQPA) and another recent approach that transfers bit allocation from a learned codec.