Researchers have assembled the largest private‑access Egocentric Tennis Video (ETV) collection yet. 85 rally sequences were recorded with a camera worn by the serving player, producing 53,155 short video frames. Each rally is synchronized with five external cameras, giving a full 3‑D view of the ball and player positions.

The dataset is fully annotated. Every frame includes the 3‑D ball trajectory, the 2‑D court‑frame positions of both players, the type of stroke (forehand, backhand, etc.), and the precise time when the player contacts the ball.

To assess how well vision models can interpret this data, the team trained and tested seven spatiotemporal tasks: current‑frame 2‑D player location, 3‑D ball location, short‑term trajectory forecasts for both, stroke‑type prediction, time to the next ball contact, and the ball’s 3‑D position at contact.

Fine‑tuned image encoders, video encoders, and vision‑language models (VLMs) were used in each case, with lightweight cross‑attention heads added after a shared encoder. The best systems achieved an average 2‑D player error of 0.14 m, a 3‑D ball error of 0.54 m, and forecast both targets within 0.34 m (player) and 0.91 m (ball) over 1.1 s. Contact‑time predictions averaged 0.14 s.

Adding extra depth information improved 3‑D ball localization, but longer image histories and frozen backbone features did not further reduce errors, indicating the importance of fine‑tuning.

Finally, the authors demonstrated a conditional diffusion model that consumes the encoder outputs and generates realistic whole‑body kinematics for the player in real time, showing the potential for computer‑vision–driven robotic training tools.