First-hand information for everyone
Main discovery VioLA is a generalist humanoid policy that predicts compact motion codes (latents) for body and hands. A frozen, pretrained body controller and a frozen hand decoder execute those latents on the robot. Trained mainly on human recordings, VioLA controls a Unitree G1 in the authors’ lab without task‑specific fine‑tuning (zero‑shot). It reached 100% success on the evaluated locomotion tasks and 88.6% manipulation success in the reported trials.
Method and how the data were prepared The policy predicts body+hand motion latents rather than raw joint commands. These latents are a compressed motion representation the decoders understand. Human and robot motions are encoded into the same latent spaces so human recordings supervise the robot action space. A latent is a compact code representing a short motion window.
VioLA reuses SONIC’s pretrained body encoders and controller for the body module. A separately pretrained hand encoder–decoder implements the hand module. All encoders and decoders remain frozen while the generalist policy is trained to predict their latents.
At each prediction step the model conditions on an egocentric image, a language instruction, and the robot’s proprioception. It predicts a chunk of H successive body+hand latents. During deployment the frozen body controller and hand decoder convert predicted latents into joint and finger commands. Commands are updated at 50 Hz.
To label human recordings with robot‑action targets, VioLA rolls the frozen body controller in simulation to track the annotated human motion. That closed‑loop rollout yields simulated robot proprioception and body‑latent targets aligned with the robot’s heading. Hand targets are encoded from annotated hand motion.
VioLA applies real‑time chunking (RTC) in latent space. Each new chunk prediction is conditioned on the latents that will still be executing. RTC reduces abrupt changes when chunks are replaced. In example recordings the largest body‑latent step change fell from 0.332 to 0.064 for Stand Still and from 0.455 to 0.070 for Raise Arm when RTC was used.
Training data and schedule The converted training demonstration pool contains 140.6 million frames. 93.2% of those frames came from human demonstrations. The pool combined human recordings (EgoSuite / EgoStandard) with four robot corpora (Humanoid Everyday, PSI, UnifoLM‑WBT, LeVERB). The reported GR00T‑backbone VioLA was trained 150k steps on human demonstrations, then 50k steps sampling uniformly across all five corpora.
Concrete quantitative results on the real robot Evaluation was done on a Unitree G1 with five‑fingered Inspire hands in the authors’ lab. Six locomotion/posture tasks used instructions from LeVERB. Seven manipulation tasks used instructions from Humanoid Everyday. Zero‑shot denotes deployment without task‑specific fine‑tuning. Each policy was tested five times per task. Results report one standard error across trials.
On these tests the reported VioLA (GR00T backbone) succeeded on all locomotion trials: 100% success where the released GR00T N1.7 baseline reached 16.7% (5/30) and Ψ0 reached 0% (0/30). For manipulation, VioLA reached 88.6% success (31/35 successful trials). The released GR00T N1.7 completed 0/35 manipulation trials in the reported comparison; Ψ0 completed 3/35 (8.6%).
A VioLA variant trained only on human demonstrations (human‑only) shows that encoded human motion can supervise robot behavior zero‑shot. That human‑only variant achieved 12/30 locomotion successes (40%) on the reported locomotion tasks. Robot demonstrations still improved overall performance under the reported training schedule.
Backbone comparisons The authors evaluated three backbone types with the same data and budget: GR00T N1.7, a vision‑language agent π0.5, and a world‑action model DiT4DiT. GR00T and DiT4DiT reached 100% locomotion success in the reported tests. π0.5 reached 66.7% locomotion success (20/30).
Manipulation success varied by backbone: GR00T 88.6%, π0.5 45.7%, DiT4DiT 25.7%. Reported prediction delays were about 160.2 ms for GR00T, 208.6 ms for π0.5, and 248.5 ms for DiT4DiT. The frozen body controller and hand decoder were shared across these variants; differences arose from the generalist policy’s predictions.
Significance and supported caveats This work shows that large human demonstration sets can supervise robot motion when human and robot motion are encoded into a shared latent action space. Using frozen pretrained body and hand modules lets the policy learn to predict motions that robot controllers can execute. The approach enabled strong zero‑shot locomotion and high manipulation success on the tested tasks and robot.
There are important limitations. VioLA’s capabilities depend on what the pretrained body and hand modules can represent and execute. The hand decoder outputs finger‑position commands without contact‑force feedback. Reported manipulation failures include difficulties acquiring and retaining objects (for example, failures on coat folding and shampoo tasks). A separate bottle‑carrying task remained unsolved in all trials. Extending VioLA to additional videos requires reliable human body and hand motion annotations.
The main evaluation covers thirteen tasks on a single Unitree G1 with five trials per task. Broader testing across tasks, environments, and robot embodiments is needed to assess reliability beyond this study.
Overall, VioLA demonstrates a path to generalist humanoid control that leverages large‑scale human motion data and frozen robot motion modules. The reported results apply to the described training data, backbones, controllers, and the Unitree G1 tests in the authors’ laboratory.