This paper introduces TrackEverything, a system that can track every visible 3D point in a video for very long time spans. Previous trackers faced a hard choice: follow a few points for a long time, or follow many points but only for a short clip. TrackEverything avoids that trade-off by keeping a persistent 3D scene representation in world coordinates. In plain terms, the method treats the video as repeated 2D views of the same 3D world and stores tracks in that shared 3D space.

The authors bring three main technical ideas. First, they use a voxel-based de-duplication step at sliding-window boundaries. This detects when different observations actually correspond to the same physical location and merges those tracks. That prevents the tracker from redundantly storing the same surface many times as the video moves on. Second, they split the tracking task into two stages: an endpoint refiner that predicts where each point will end up and whether it is static or moving, and then a lightweight trajectory refiner that only decodes full dense trajectories for the points classified as dynamic. Third, they replace a memory-heavy 4D correlation volume with a new component called 3D WAFT, which samples features efficiently from the scene cloud instead of building huge correlation structures.

Together these choices let TrackEverything decouple model size from video length. Instead of growing with the number of frames, the method scales with the amount of unique physical geometry in the scene. That design lets the authors run tracking across more than 1,000 frames while keeping the peak GPU use within 40 GB, which they report as a practical memory bound for their experiments.

On the TAPVid-3D benchmark, the paper reports strong results. TrackEverything outperforms all open-source all-frame dense 3D trackers by more than 20% on the APD metric for short clips. At the same time, it remains competitive with state-of-the-art sparse trackers on long sequences, despite tracking many more points. These numbers suggest the method can give dense, long-range tracks without the usual memory explosion.