September 2026 | Research
ZYT-World: A Real-Time Controllable World Model for Closed-Loop Autonomous-Driving Simulation
What It Really Takes to Close the Loop for End-to-End Driving
An end-to-end driver can only be judged by how it behaves, and neither road testing nor open-loop replay measures behavior at scale. Real miles are expensive and the critical ones unrepeatable; a replayed log has already decided the future — it can tell you what the model would predict, never what the vehicle would do.
Closing the loop gives the future back to the model: each ego action shapes the next observation, with generated video standing in for the vehicle's camera streams. To stand in for reality, the world model has to meet four requirements.
Sensor equivalence. Interchangeable with the real sensor stream at the policy's input: cameras, projection models, intrinsics, extrinsics and per-view resolution matching the production rig, pixel statistics adding no domain shift.
Action response and scene control. Every step conditioned on the current action; ego trajectory and scene layout controllable — staging counterfactual and long-tail scenarios real data never contains.
Consistency across views, time and trajectories. One scene geometry across cameras; no error compounding over minute-scale rollouts; revisit the same road by a different path, and the static world comes back the same.
Real time on deployable hardware. The policy waits online: generation plus decoding inside the decision cycle, on a single machine or even a single GPU.
Introducing ZYT-World
ZYT-World is built to meet all four at once: a real-time, controllable world model for the camera rig a production vehicle actually ships with. Seven views — four cylindrical fisheyes above 180°, three pinholes — are generated jointly, each at its native resolution and aspect ratio: to our knowledge, the first driving world model trained on a mixed fisheye-and-pinhole rig without homogenizing it. Camera geometry, ego-motion and scene layout enter through one dense interface, sourced entirely from on-vehicle recordings and the production perception stack, with no human annotation anywhere. A four-stage distillation turns a 40-step bidirectional teacher into a frame-wise causal generator: one denoising step per latent, streaming seven views at 4 FPS on two GPUs, coherent over minute-scale rollouts. A pluggable memory keeps revisits to the same location consistent.
The single-step student matching its 40-step teacher frame for frame, crossing an urban intersection through moving traffic.
A snowy turn at an intersection, with cars parked along the curb.
A lane change in the rain.
A turn at night, with the ego cutting into traffic.
Cruising on the highway.
ZYT-World System Architecture
Dual-Path Camera Control: Rays + Ego-Motion
Two complementary paths: Plücker rays give every pixel its viewing direction, fisheye and pinhole in one representation; ego-motion modulates every layer with whole-frame movement. Together they outperform a full-resolution ray field at a fraction of the memory, with per-frame action control for free.
Trajectory control: the same intersection under three commanded futures — left, straight and right — with pedestrians crossing throughout.
Trajectory control: the same fork under two commanded futures — a U-turn, and a straight approach braked to a stop — with e-bikes weaving across.
Pixel-Aligned Layout Control at Near-Zero Cost
The wireframe never touches a VAE encoder. One lightweight adapter, shared by all seven views, lifts it onto the token grid in a single injection. And the video follows what the wireframe draws: agent boxes place the traffic, lane lines lay down the road geometry, and the rendered scene tracks both. The control is also robust to imperfect input — under jittered boxes, generation stays coherent and on-layout, so upstream layouts need not be pixel-perfect.
Layout control: road structure and traffic following the injected wireframe, editable scene by scene.
Traffic-signal intervention: green, yellow, and red phases with the ego vehicle stopping on red.
One Step Is Enough
One step per latent, 107.7× faster inference, and an action that can change at every timestep. ZYT-World generates each frame in a single denoising step, streaming seven 720P views in real time at 4 FPS. Getting there means crossing a gap: video diffusion priors are bidirectional and multi-step, while the loop needs causal and single-step — causality costs quality, and few steps compound rollout error. Four distillation stages bridge it.
Causal adaptation. Training visibility matched exactly to the deployed KV-cache.
Consistency distillation. A few-step student learned from adjacent noise levels — no full-trajectory regression.
Self-rollout DMD. The student trains on its own predictions, corrected by the frozen bidirectional teacher.
Perceptual and adversarial refinement. High-frequency detail anchored to ground truth; mode coverage restored.
A turn at night, then a straight stretch — quality holding all the way down to one step.
Driving straight through rain, past a row of parked cars — one step, no visible gap.
TinyVAE Boosting Latent Decode
A 19M decoder that approaches the 555M Wan decoder in reconstruction and generation quality, while running 59.8× faster at ~1/27 the memory.
Decoder latency used to be the bottleneck in single-step closed-loop simulation, making real-time execution impractical. TinyVAE removes that bottleneck while keeping both latent reconstruction quality and end-to-end DiT generation quality close to the original Wan decoder, enabling high-fidelity video generation to run inside a real-time simulation loop.
Same latents, 29× fewer parameters — compare the 555M Wan decoder with our 19M TinyVAE side by side.
Same Place, Same World
Generation can imagine any street — what it can't do is come back to one. Memory adds the missing ability: reconstruction. ZYT-World is the first driving world model to build its paired supervision from 4DGS re-renders of real scenes along new trajectories, and to add recall as a pluggable, implicit pathway — past observations enter as latents, no explicit 3D pipeline, no sim-to-real gap.
Learned on the front views, propagated to all seven by cross-view attention. On revisit, static geometry and traffic layout come back as remembered.
Front-view memory: road signs and the key traffic participants restored exactly on revisit.
Minute-Scale Rollout, Constant Cost
However long it drives, the cost per frame stays the same: a bounded KV-cache with attention sinks decouples compute and memory from elapsed time.
Closed-loop evaluation runs for minutes, not clips. ZYT-World advances frame by frame under a bounded cache — per-frame cost never grows — and holds quality over the full run through self-rollout training and memory anchoring, with no long-horizon training stage at all.
Minute-scale rollout: dense urban traffic by day and by night — no drift, no collapse, across all seven views from first frame to last.
Minute-scale rollout: dense urban traffic by day and by night — no drift, no collapse, across all seven views from first frame to last.
Acknowledgements
Built through shared research, engineering, and evaluation across world-model, simulation, safety, and autonomous-driving teams.
Citation
If you find ZYT-World useful in your research, please cite the technical report:
@article{zytworld2026,
title = {ZYT-World: A Real-Time Controllable World Model for Closed-Loop Autonomous-Driving Simulation},
author = {{ZYT AI Team}},
journal = {arXiv preprint arXiv:2609.21712},
year = {2026}
}