This paper introduces VLM-IE3D, a way to give vision-language models a stronger sense of 3D space using only ordinary RGB video. The authors point out that many current models learn a rough, compressed sense of 3D layout from images. Those “implicit” representations capture coarse scene structure but often miss the fine geometric details needed for precise spatial reasoning.

VLM-IE3D adds two complementary geometry representations. Implicit Geometry Tokens (IGTs) come from a 3D geometry encoder and encode high‑level, global 3D priors. Explicit Geometry Tokens (EGTs) are derived from reconstructed 3D attributes such as depth maps and point clouds and retain local, fine-grained geometric details. A 3D‑aware adapter fuses these geometry tokens with standard 2D visual tokens, producing geometry-rich features for the language model.

At a high level, the system works with three streams on RGB video frames: a 2D visual encoder that produces image tokens, a 3D encoder that yields implicit geometry tokens, and an explicit embedding module that converts reconstructed depth or point‑cloud attributes into explicit geometry tokens. The adapter merges these streams so the language model can reason with both coarse layout cues and concrete geometric structure—without needing extra 3D sensors.

The authors report that VLM-IE3D improves performance across several 3D tasks compared with other RGB-only methods. They evaluate on tasks that need spatial understanding and precise localization, including 3D video detection, 3D visual grounding, 3D dense captioning, and spatial reasoning. According to the paper, this combination of implicit and explicit geometry gives a practical way to inject strong 3D inductive bias into vision-language models. Code and models are available from the authors’ GitHub repository.