PointZero: 3D Point Track Completion for Learning Transferable 3D Dynamics

Abstract

World models endow perceptual systems with the ability to predict how scenes evolve under interaction. They are most beneficial when trained on diverse volumes of data, to instill a rich prior into downstream applications. Existing methods for capturing 3D world models are typically scene-specific dynamics models, or require robot action labels — which excludes web video data from the training pool. We study 3D point track completion as a pre-training objective for learning transferable 3D dynamics without robot data. Given a single RGB-D observation and sparse partial 3D trajectories (tracks), we predict future 3D tracks of all observed points. We show this objective provides a rich dynamics prior, without requiring robot action labels. We contribute a diverse dataset of 2.9 million synthetic frames spanning deformable, articulated, and rigid objects, and use it to train PointZero. We show that a flexible and expressive transformer, PointZero, outperforms prior methods on the same data. We demonstrate the utility of our pre-training objective by fine-tuning PointZero for two downstream applications: (1) action-conditioned dynamics and (2) robot manipulation. When fine-tuned to condition on end-effector pose, PointZero outperforms the baselines on a recent action-conditioned 3D dynamics benchmark. When fine-tuned to predict robot actions, PointZero outperforms or matches the baselines on 6/7 simulated and real-world robot manipulation tasks. We furthermore evaluate training PointZero from scratch to isolate the benefits of our proposed architecture from those of our proposed pre-training objective and dataset. We release the dataset, model checkpoints, and full training recipe.

Method

PointZero takes in a single RGB-D image and a small number of complete point tracks, and predicts dense 3D point trajectories for all observed scene points. The architecture combines a Perceiver-IO visual encoder over DINOv2 features with a denoising transformer that alternates point-level and global attention. We train it with a JiT-style x-prediction objective. Because both the labels and the conditioning are just 3D point tracks — available from simulation, and in principle from any video via point tracking — supervision is robot-free and interaction-agnostic. We can then fine-tune the same model to condition on robot end-effector pose, or to predict robot action chunks in addition to point tracks.

Interactive method figure — select a panel for details. The static paper figure is here.

A 4D dataset of diverse dynamics

We contribute a new synthetic dataset of 2.9 million frames with dense per-point trajectory annotations, spanning deformable, articulated, and rigid objects with motions driven by randomized interactions. For evaluation of zero-shot sim-to-real transfer, we also collect a real-world set of 14 objects and 124 interactions, with trajectory labels from FoundationStereo and CoTracker3. We release the datasets, code, and checkpoints.

Model architecture ablation

We ablate PointZero, a simple DiT, against existing neural dynamics models trained on the same data. On held-out samples of our synthetic dataset, PointZero outperforms every baseline on every metric and object category. Flow matching and JiT outperform direct regression. With sufficient training and matched sampling protocols, JiT does not consistently outperform flow matching.

Method	Deformable			Articulated			Rigid
	MDE↓	CD↓	EMD↓	MDE↓	CD↓	EMD↓	MDE↓	CD↓	EMD↓
GBND	17.38	7.45	16.47	11.15	7.38	11.11	155.72	93.30	154.44
ParticleFormer	17.35	7.42	16.44	10.86	7.11	10.83	155.04	92.76	153.77
PGND	9.01	4.10	7.86	8.70	5.40	8.29	67.58	33.56	64.02
PTv3	9.20	3.91	8.28	8.25	5.25	8.00	61.97	29.87	58.57
PointZero-FM-mean-10	3.59	1.55	3.18	2.40	1.77	2.34	27.26	14.17	25.41
PointZero-FM-oracle-10	2.80	1.24	2.41	1.96	1.48	1.91	19.91	10.39	18.43
PointZero-Regression	4.08	1.74	3.63	3.85	2.57	3.68	32.14	16.43	29.78
PointZero-JiT-mean-10	3.62	1.56	3.22	2.36	1.70	2.30	28.06	14.65	26.35
PointZero-JiT-oracle-10	2.85	1.26	2.47	1.95	1.44	1.89	19.46	10.30	17.99

Dynamics prediction errors on held-out samples of our synthetic dataset. We train all baselines on the proposed dataset. MDE, CD and EMD are in centimetres. Mean-10 averages each error metric over ten seeded predictions; oracle-10 uses ground truth to select the lowest error independently for each scene and metric from ten predictions. Bold marks the best result per column.

Post-training

We evaluate the utility of the PointZero pre-training objective by post-training it for robotic manipulation.

Action-conditioned dynamics prediction

Post-trained to condition on robot end-effector state, PointZero convincingly outperforms strong scene-specific baselines on 4 of 6 real robot–object interaction scenes in the PGND benchmark. Pre-training is critical: the identical architecture, trained from scratch (PointZero-Scratch), falls far behind.

PointZero predicts object dynamics inside the original 4D capture, conditioned on the robot end-effector.

Method	Bread			Paperbag			Cloth			Box			Rope			Sloth
	MDE↓	CD↓	EMD↓	MDE↓	CD↓	EMD↓	MDE↓	CD↓	EMD↓	MDE↓	CD↓	EMD↓	MDE↓	CD↓	EMD↓	MDE↓	CD↓	EMD↓
GBND	3.1	3.1	1.6	3.0	4.2	1.6	7.7	8.3	3.5	4.5	6.2	3.2	6.2	7.3	3.6	7.8	6.4	3.2
PGND	2.0	1.8	1.0	1.6	2.1	0.9	4.5	4.3	2.2	2.2	1.5	1.6	3.9	3.8	2.1	4.3	3.3	1.7
PointZero-Scratch	4.2	3.9	1.9	4.2	4.4	1.9	5.6	4.7	2.1	3.7	3.9	2.1	9.3	10.8	5.0	7.3	5.9	2.8
PointZero-FT	1.5	1.1	0.6	1.6	2.0	0.8	4.0	3.4	1.5	2.4	2.7	1.4	3.3	3.3	1.6	3.9	3.0	1.5

Prediction errors in centimetres on the PGND benchmark. Both PointZero rows report per-scene, per-metric best-of-10 oracle results, selected using ground truth. Bold marks the best result per column.

Robot manipulation

With a lightweight action head, post-trained PointZero drives imitation-learning policies using 20 action-labeled demonstrations and 100 actionless demonstration videos per task. It achieves the highest or joint-highest success rate on 6 of 7 simulated and real-world manipulation tasks, and outperforms DP3 on all seven.

Simulation

Block stacking

Open microwave

Grasp glass

Real robot

Open drawer

Grasp cup

Paper into cup

Fold sock

Successful policy rollouts under the paper's evaluation protocol. Real-robot clips play faster than real time.

Task	DP3	ATM	3PoinTr	PointZero-FT
Blockstack (sim)	44.4	40.0	90.9	99.8
Microwave (sim)	31.4	30.4	80.8	93.1
Glass (sim)	74.9	3.9	95.2	95.9
Drawer (real)	70.0	30.0	90.0	100.0
Cup (real)	80.0	30.0	100.0	100.0
Paper (real)	10.0	0.0	90.0	70.0
Sock (real)	20.0	0.0	90.0	90.0

Imitation-learning success rates (%). PointZero uses 20 action-labeled demonstrations and 100 actionless demonstration videos per task. Bold marks the best result per task. We omit DP: it has no real-world results.

Sampling diverse dynamics

The JiT objective learns to sample from a distribution of point trajectories given a single RGB-D observation. Here we show how different seeds produce plausible yet diverse dynamics. With the same input and conditioning track, different random seeds predict different motions of the notebook.

BibTeX

@article{duisterhof2026pointzero,
  title   = {PointZero: 3D Point Track Completion for Learning Transferable 3D Dynamics},
  author  = {Duisterhof, Bardienus P. and Zhang, Kaifeng and Hung, Adam and Wen, Bowen and
             Birchfield, Stan and Li, Yunzhu and Ramanan, Deva and Ichnowski, Jeffrey},
  journal = {arXiv preprint arXiv:XXXX.XXXXX},
  year    = {2026}
}