Mundo turns a few photos into a world robots can rehearse in
ACE Robotics and NTU S-Lab built a world model that rebuilds a room in 3D, simulates it with physics, and renders robot practice runs as video.
ACE Robotics and Nanyang Technological University's S-Lab have released Mundo, a world model that turns a handful of photographs into a three-dimensional environment robots can explore and practice in. Given one to twelve images of a real place, Mundo reconstructs the scene as an explicit 3D proxy of boxes that stand in for objects, then lets users edit that world through code, simulate what happens in it with a physics engine, and render the result as video. The aim is a shortcut for robot training: build the training ground from a site visit instead of modeling it by hand.
The pipeline has three layers. A 3D foundation model recovers camera poses and geometry from the input views, a 3D detector places initial bounding boxes around the main objects, and a vision-language-model agent refines the proxy by rendering it, comparing the renders with the original photos, and correcting the boxes through successive rounds, checked by a separate reviewer agent. The scene is then stored as a code program that records each object's pose and physical properties such as mass and friction. A simulator, usually MuJoCo, computes motion, contact, and collisions under gravity, and a video model renders the outcome using the input images as visual context.
For robots, the reconstructed world is a place to rehearse rather than a picture to look at. A path planner can route a robot between a start and a goal described in words, using the proxy's record of obstacles and traversable space. Manipulation tasks such as placing an apple in a bowl or stacking blocks can be attempted by robots of different shapes in the same scene, with head and wrist camera views generated alongside external views, and trajectories and physical interaction states retained as training data. A hotel-floor demonstration reconstructs a lobby, rooms, and a laundry area, including a robot opening a washing-machine door, and multi-camera recordings can be replayed from new angles at a chosen instant. Mundo sits inside ACE Robotics' Kairos-centered world-model stack, adding environment modeling and simulation training to the company's spatial-understanding work.
The authors are open about the limits. Reconstructing a scene usually takes ten to twenty minutes, and video rendering runs at roughly 1.5 frames per second on eight A800 GPUs. Small objects tend to float in the reconstructed scene, and the video model can hallucinate when rendering fine-grained handling. Mundo also has no long-horizon memory mechanism yet, so scene details can drift after extended dynamic changes or when a previously hidden area comes back into view. The renderer was trained on about 40,000 pairs of real or rendered video and matching proxy video, fine-tuning the MiniMax-H3 Ref2VA model on clips of 124 frames at 640 by 480 resolution. The project page and technical write-up are public.