T2-zTest-Time Optimization of Environment Latents
for Robotic World Models

Chenyuan Zhou1*, Yining Hong1*, Huang Huang1♡, Yunfan Jiang1♡, Wenlong Huang1♡, David Held2,
Vitor Campagnolo Guizilini3, Yue Wang4, Li Fei-Fei1, Jiajun Wu1†, Leonidas Guibas1†, Yejin Choi1†

1 Stanford   2 CMU
3 Toyota Research Institute   4 University of Southern California

* Co-First Contribution♡ Co-Second Contribution† Equal Advising

Stanford UniversityCarnegie Mellon UniversityToyota Research InstituteUniversity of Southern California
PaperCodeDatasets
Explore T2-z

Overview

T2-z overview: latent optimization, emergent physical structure and real-world tasks
T2-z enables robotic world models to adapt to unobserved environmental factors, from physical properties to interaction rules, through a compact environment latent. The latent is learned from world prediction without labels for the underlying factors and optimized from a small number of interactions at test time, while the world model remains frozen. Across seven simulated settings and four real-world tasks, this representation supports environment-adapted prediction and action selection, and exhibits structured variation across physical factors and visual scenes.

Abstract

Predicting how the world responds to an action requires more than what can be seen: robots must account for environmental factors that are neither directly observable nor explicitly specified, from physical properties such as friction and mass to human-defined environmental rules. We introduce T2-z for test-time optimization of environment latents, which augments a robotic world model with a learnable environment latent to condition world predictions. This compact latent captures underlying environmental variation without supervision of what environmental factors exist or what values they take.

During training, T2-z alternates between two stages: a world-model optimization stage that fixes the latents and learns general action-conditioned dynamics across environments; and an environment-latent optimization stage that freezes the world model and optimizes the conditioning latents, allowing latent structure to emerge purely through world prediction. At test time, the world model remains frozen and a newly initialized environment latent is optimized from one or more test-time interactions, adapting the model to previously unseen environmental configurations.

Across 7 simulated settings and 4 real-world robotic tasks, T2-z reduces object prediction error metric from 15.20 to 7.00 and raises action success from 33.73% to 72.96%. Its emergent latents capture structured representations of individual and compounded factors across diverse visual scenes. Ablations further show that T2-z can efficiently infer environment latents from as little as a single test-time interaction trajectory.

Method

Alternating world-model and environment-latent training, followed by test-time latent optimization

Training. The 32-dimensional environment latents are randomly initialized and used to condition the world model. Training alternates between optimizing the world model with the latents fixed and optimizing the latents with the world model frozen, using the same flow-matching objective.Test-time adaptation. A newly initialized latent is optimized from observed test-time interaction trajectories, while all world-model weights remain frozen.Action-conditioned prediction. The adapted latent is reused with a new action and an initial observation to predict the resulting video in the same environment.

Qualitative Results

Explore predictions across seven simulated settings and four real-world tasks.

Ball Friction

Same environment and action across methods.

Emergent Latent Structure

Learned latents organize around individual and compounded factors.

PCA visualizations of environment latents showing physical-factor structure
PCA visualizations from the paper. Physical-factor labels are used for analysis, not as latent supervision.

Quantitative Results

Compare T2-z with Standard World Model, LoRA TTA, DINOv2 Context and TTT-KQV.

Task-specific action success (%) reported in Tables 1 and 2. Hover or focus a bar for its exact value.

View the numerical results
Ablations of training and the number of test-time interaction trajectories
(a) Learning-rate strategy. Ours uses a coarse-to-fine schedule of 3.0, 1.5, 0.5, and 0.15, with 10 updates at each rate. Const. Max. and Const. Min. keep the rate fixed at 3.0 and 0.15, respectively. Ours ×10 and Ours ÷10 multiply or divide the entire schedule by ten.(b) Test-time optimization steps. We vary the number of environment-latent updates from 10 to 60, with the world model frozen. The default is 40 steps. Panels (a) and (b) show relative metric quality, with the default setting at 100%.(c) Interaction trajectories and prediction error. We use 1, 2, or 4 test-time interaction trajectories to optimize the latent and report mean physical prediction error. Lower values indicate better predictions.(d) Interaction trajectories and action success. Using the same trajectory counts, we measure task-specific action success; higher values are better. In (c) and (d), circles, squares, triangles, and diamonds denote Mass Collision, Light Switch, Door Close, and Soft Pull, respectively. Outlined markers indicate the trajectory counts used in our main experiments.

BibTeX

@misc{zhou2026t2z,
  title  = {{T2-z}: Test-Time Optimization of Environment Latents for Robotic World Models},
  author = {Zhou, Chenyuan and Hong, Yining and Huang, Huang and Jiang, Yunfan
            and Huang, Wenlong and Held, David and Guizilini, Vitor Campagnolo
            and Wang, Yue and Li, Fei-Fei and Wu, Jiajun and Guibas, Leonidas
            and Choi, Yejin},
  year   = {2026},
  eprint = {2610.22364},
  archivePrefix = {arXiv},
  primaryClass = {cs.RO},
  url    = {https://arxiv.org/abs/2610.22364}
}