This paper introduces DreamStream, a new type of driving simulator built to test end-to-end driving policies more faithfully. End-to-end (E2E) driving policies take raw sensor inputs like video and output steering and speed decisions. The authors argue that typical simulators look photo-realistic but change the visual details that these policies use, which can corrupt how the policy perceives a scene and hide real failure modes.

The core of DreamStream is a simulator-grounded autoregressive video model. In plain terms, the model predicts video frames one after another, using both the simulator’s scene layout and previous frames. The video model is distilled from a large pretrained video model and guided by traffic layouts. This lets the simulator vary lighting and appearance while preserving the scene layout and the timing of moving objects — features that matter for a policy’s decisions.

The paper also points out that common perceptual scores, such as the Frechet Inception Distance (FID), do not reliably measure whether a simulator preserves the features a driving policy uses. To address this, the authors propose FDπ (FD-pi), a new metric that measures the sim-to-real gap using the Fréchet distance between feature distributions. Those features come from several public end-to-end policies, so the metric focuses on how the simulator affects representations that actual driving systems use.

Measured by FDπ, DreamStream outperforms the strongest prior closed-loop simulator by 1.6× on the nuScenes dataset and 4.7× on the NAVSIM dataset. The authors also report that DreamStream causes the least perturbation to a policy’s perceptual observability compared with alternatives. Using DreamStream, they build Navhard-CL, a benchmark that converts the non-reactive NAVSIM recordings into interactive test scenes. Navhard-CL adds adversarial driving behaviors and weather variations and exposes many failure modes that earlier closed-loop benchmarks missed, such as scorer bias and poor recovery behaviors.