This paper introduces PhiZero, a new way to model how the physical world unfolds in videos. Instead of directly predicting pixels for future frames, the authors train a small discrete representation they call a physical language. The model first reasons about future changes as a short sequence of these discrete tokens, and then renders those inferred changes back into a realistic video. The result is a separation between inferring dynamics (what will happen) and generating appearance (what it looks like).

To build this representation the team trains two main components from unlabeled, in‑the‑wild videos. A Physical Language Tokenizer compresses the frame‑to‑frame transitions into a compact discrete sequence. Technically, it uses a spatio‑temporal encoder followed by a transition‑level Q‑Former that looks at pairs of adjacent latent states, and then applies finite scalar quantization (FSQ) to produce discrete tokens. A diffusion decoder — a type of image/video generator that learns to turn noise into images — is trained to reconstruct the original video given the first frame and the token sequence. The discrete bottleneck encourages the tokenizer to focus on changes over time instead of static appearance.

A second component, the Physical Language Reasoner, is initialized from a pretrained visual‑language model (VLM). Given the current first frame and an optional text action intent (for example, “pour the wine into the glass”), the reasoner autoregressively predicts a physical‑language sequence describing the future transition. The diffusion decoder then renders that sequence into a video. This “reason‑then‑render” design makes the model’s prediction of dynamics explicit and separate from pixel synthesis.

Why this matters: prior world models typically predict pixels directly and can hide inconsistent or physically implausible dynamics inside complex visual predictors. PhiZero’s explicit transition tokens make it easier to reason about, control, and transfer state changes. The authors report that the approach works across both generation and video understanding benchmarks and that it enables use cases such as interactive rollouts under sequential controls, fine‑grained action‑conditioned simulation, and zero‑shot motion transfer.