This paper introduces VideoRAE, a method that converts the internal features of large, frozen video encoders into compact representations that are easy for generative models to use. Instead of training the usual 3D variational autoencoders (3D-VAEs) that focus on reconstructing pixels, the authors compress higher-level features from video foundation models. The goal is to keep more of the video’s meaning and motion while producing latents that work well for different generation methods.

Concretely, the team takes multi-scale, hierarchical features from a frozen video foundation encoder (examples named in the paper include V-JEPA 2 and VideoMAEv2). Those features are passed through a lightweight 1D self-attention projector to make a compact representation. VideoRAE can output either continuous latents for diffusion-based generators (Diffusion Transformers) or discrete tokens for autoregressive models by using multi-codebook high-dimensional quantization. At decode time the model trains with a local-and-global representation alignment objective that compares the decoded output to the frozen encoder’s features. This helps preserve semantics and lets the authors avoid the usual KL (Kullback–Leibler) regularization used in many VAEs.

The authors report strong reconstruction performance in both the continuous and discrete settings. On the UCF-101 action video dataset they achieve state-of-the-art class-to-video gFVD scores of 40 and 93 when paired with autoregressive (AR) and Diffusion Transformer (DiT) generators, respectively. They also find that VideoRAE converges about five times faster than competing autoencoder baselines. In a controlled text-to-video experiment at roughly 2-billion-parameter scale, replacing a prior VAE (LTX-VAE) with VideoRAE led to faster convergence under comparable training conditions.

Why this matters: many current video generators work from a latent space rather than raw pixels. If those latents better capture semantics and motion, generative models can learn faster and produce higher-quality videos. VideoRAE shows that frozen video foundation models, which were trained for understanding, can be repurposed as sources of such latents. The approach supports multiple downstream generators and reduces the need for large, pixel-focused autoencoders.