Track4Action: Distilling World-Centric 3D Tracker into
Vision-Language-Action Policies

Chenyi Wang^1,3* · Xinkai Wang^3* · Bokai Lin^2,3 · Jialin Tian^2,3 · Fucheng Zhang³
Cewu Lu^2,3,4 · Lixin Yang^2,3†

¹Zhejiang University, China · ²Shanghai Jiao Tong University, China
³Shanghai Innovation Institute, China · ⁴Noematrix, China
chenyiwang@sii.edu.cn · siriusyang@sjtu.edu.cn
* Equal contribution · † Corresponding author

📄 Paper 🤗 Checkpoints 🎬 Demo 💻 Code

Track4Action distills action-aligned 3D world transitions from demonstration videos into a vision-language-action policy.

Model Architecture

Track4Action aligns a frozen world-centric 3D tracker with the VLA hidden states and action head during training.

Abstract

Vision-language-action (VLA) policies are strong at instruction grounding and broad-task generalization, but they often rely on weakly structured 2D observations and under-utilize explicit 3D motion cues. In contrast, world-centric trackers can recover geometry, visibility, and temporal correspondence, yet they are not directly executable as robot policies.

We present Track4Action, a framework for turning world-centric tracking into robot action prediction. A frozen Track4World teacher produces geometry-aware features from aligned demonstration clips. A VLA policy learns tracker-aligned query tokens from its own multimodal hidden states, and a shared flow-matching action head turns those features into action chunks. Crucially, the tracker is used only as training-time privileged supervision; future video and the tracker are removed during deployment.

Method

World-centric teacher

Track4World exposes scene geometry, 3D motion, visibility, and camera cues from short aligned video clips.

Track-token alignment

Learnable query tokens read from the VLA hidden states and align with the frozen tracker representation.

Tracker-free deployment

The teacher is needed only during training. At inference, the policy uses the current observation, language, and learned aligned bottleneck.

Demo Video

Real-World Experiments

Real-world experimental platform and task settings — Real-world bimanual evaluation across task and environment shifts.

Experiments

Track4Action improves both in-distribution manipulation and robustness to unseen visual changes.

97.0%LIBERO

82.3%LIBERO-PLUS · ZERO-SHOT

80.44 / 81.48%ROBOTWIN · CLEAN / RANDOMIZED

+25 ptsREAL-WORLD GAIN

Takeaway

Demonstration videos can teach a policy both what to do and how the world should change, while enabling tracker-free deployment.

Citation

If you find this work useful, please consider citing:

@article{track4action,
  title={Track4Action: Distilling World-Centric 3D Tracker into Vision-Language-Action Policies},
  author={Wang, Chenyi and Wang, Xinkai and Lin, Bokai and Tian, Jialin and Zhang, Fucheng and Lu, Cewu and Yang, Lixin}
}