> ## Content Index > Fetch the complete content index at: https://globalfeed.ai/llms.txt > Use this file to discover other available public pages before exploring further. # Why do robots get lost on a new table? Ai2's open ToyBlocks world tests it with wooden blocks - URL: https://globalfeed.ai/en/why-do-robots-get-lost-on-a-new-table-ai2s-open-toyblocks-world-tests-it-with-wooden-blocks/ - Published: 2026-11-07T23:20:20.000Z - Updated: 2026-11-07T23:20:20.000Z - Description: Ai2 opened ToyBlocks to study robot learning in a controlled setting, like a petri dish: automatically generated data, a two-layer robot model and results tested on a real arm. We explain how it works, what the first experiments found and how to try it. - Author: GlobalFeed Editor - Tags: Ai2, ToyBlocks, robotics, robot learning, simulation, Molmo2, world model, open source, x-Ai2, dil-en, elle Videos of robots folding laundry or sorting parts arrive every week. Most of them are still filmed in a narrow setting and leave two questions open: what did the robot actually learn, and why did it fail when it failed? Biologists studying a complex organism often start with something small and controllable, like a petri dish. The Allen Institute for AI (Ai2) proposes the same approach for robotics, and on 6 November it opened the first example: [ToyBlocks](https://allenai.org/blog/toyblocks?ref=globalfeed.ai). ToyBlocks is a simulated world made of a table, a robot arm and wooden blocks with letters and pictures on their faces. The scene is deliberately simple, but the robot still has to push, flip and stack blocks, and turn a block to see a hidden face. Ai2 calls such environments FoundationWorlds, and ToyBlocks is the first of them. Simulation assets, task and demonstration generators, 85,336 ready demonstrations, two trained models, and training and evaluation code are open under Apache 2.0 on [GitHub](https://github.com/allenai/toyblocks?ref=globalfeed.ai) and Hugging Face. ## Who makes the data that teaches the robot The most expensive part of robot learning is data. Usually a person drives the robot by remote control and repeats the same job thousands of times, and those recordings become training data. ToyBlocks automates this. First, short skills (pushing a block across the table, tipping it onto another face without grasping it) are produced with reinforcement learning and motion planning inside Isaac Lab, NVIDIA's open-source simulation framework. It uses a simulation advantage: the block's exact position and angle can be read straight from the simulator instead of being estimated from a camera. Then a task-and-motion planner picks which skills to run in which order to reach a goal and plans the arm motions between them. Each plan is played in simulation and the recording is added to the dataset with labels describing the task. Hundreds of plans run in parallel on one GPU, reaching up to 24 times real time. Recordings are stored in the LeRobot format, so they can train vision-language-action models directly. Because every task goal is written in the PDDL planning language, success and partial progress are measured without task-specific code. Researchers can change the data mix freely. They can test whether rotation and pick-and-place demonstrations improve flipping, or train the robot only on the left half of the table and test it on the right. Measurements go beyond success rate to include execution time, force on the blocks and how close the robot got to the goal. ## How it was built: a two-layer robot brain Ai2 released two models with the framework. The upper layer, Molmo2-4B-tb, decides what to do. Derived from Ai2's Molmo2-4B vision-language model, this 4.5-billion-parameter model looks at a 640x360 front-camera image and a written goal, states how much of the goal is done, names the next step and marks the target block with a point on the image. It makes about 6 decisions per second. It was trained for 6,000 steps at batch size 256 on ToyBlocks three-block stacking episodes, with 25% of the data from PixMo and CoSyn pointing sets so it keeps its general pointing skill. The lower layer, ManiFlow-tb, moves the arm. This 322.5-million-parameter model takes front and wrist camera images, the robot's state, the task name and the target marked by the upper model, and outputs chunks of 25 actions. It was trained with flow matching for 200,000 steps on 8 GPUs. When it finishes a skill it pauses and asks the upper model for the next instruction. The framework can replace either layer with an oracle that knows the right answer, so when something fails you can tell whether planning or execution was at fault. ## What the first experiments showed Ai2 first used the framework to measure how well popular action models generalize. ManiFlow, π0.5 and MolmoAct 2 were tested on stacking, unstacking, flipping and rotating in regions and tower heights they had not seen in training. According to the team, despite large-scale pretraining these models generalized poorly in the initial results. The team tried generalization techniques one by one and writes that the problem remains far from solved. The most striking result concerns world models. The team tested three models that predict how a scene changes in response to the robot's actions: the 5-billion-parameter video model Wan 2.2, Astra, which builds its own physics simulator, and LeWM, which works in a latent space. For each skill they generated 900 recordings, half successful and half failing because of small position changes. Wan 2.2 predicted block motion and outcomes best. Adding 100 synthetic runs from that model to training raised the robot's success in an unseen region from 19% to 78%. The team also connected an agent running Claude Opus 5.5 to the skills through MCP. Using text descriptions it solved all 30 classic stacking tasks, and again 30 of 30 when working step by step from camera images. In "search and spell" tasks, where some letters were missing, it solved or correctly declared impossible 63% of tasks. Most errors came from confusing look-alike letters (13%) and placing a letter sideways (15%). When the agent learned how its skills behave through its own small experiments and wrote its own skill descriptions, success rose from 24% to 63%; descriptions written by a human expert reached 36%. ## From simulation to a real table The real test is whether models trained only in simulation work on a real robot. Ai2 ran experiments with a Franka FR3 arm with a Robotiq gripper, a front ZED camera and a wrist-mounted ZED Mini, across thirty scenes with 4 to 10 blocks. ManiFlow's single-skill success was 80.0% in simulation and 81.1% in the real world. On long three-letter spelling tasks run by Molmo2 and ManiFlow together, success was 55.5% in simulation and 50.0% on the real robot; the team notes that the real-world sample was 30 episodes and more trials are needed. Three steps made the transfer possible. Before generating data, joint friction and motor inertia in the simulator were tuned to match the real robot's motions. During generation, block size, friction, table height, lighting, backgrounds and camera settings were randomized. At run time, clean depth maps were computed from two camera images with FastFoundationStereo instead of using a noisy depth sensor. ## How to try it ToyBlocks is built on NVIDIA Isaac Sim and Isaac Lab and needs a CUDA GPU. Installation and a first small dataset take a few commands: ``` git clone --recursive https://github.com/allenai/toyblocks.git && cd toyblocks uv venv --python 3.11 uv pip install -e ".[dev,isaac,torch-cu128,vecgen]" get-toyblocks-assets --asset-manifest asset_manifest.json # generate two small three-block stacking tasks on one GPU python scripts/datagen/run_datagen_pipeline.py \ --experiment-id my_first_run --output-root data \ --task-gen-script scripts/task_gen/stack_three.py --task-gen-count 2 \ --env-per-problem 8 --num-envs 8 --exec-num-envs 8 --vecgen-problems-per-env 1 \ --render-pass src/toyblocks/domain_randomization/flags/render_flags_plain.yaml ``` These commands plan and run two stacking problems, render the recordings and merge them into one LeRobot dataset. The [documentation](https://allenai.github.io/ToyBlocks/docs/user%5Fguide/index.html?ref=globalfeed.ai) walks through data generation, ManiFlow training and evaluation. Molmo2-4B-tb needs about 20 GB of GPU memory in bfloat16\. The ready dataset is [on Hugging Face](https://huggingface.co/datasets/allenai/toyblocks-data?ref=globalfeed.ai); the first release holds nearly 1,600 hours of robot motion and 144 million frames. A web interface also lets you connect Claude or another frontier model and give the robot tasks in plain language, so someone without robotics expertise can watch how a model plans and acts. Ai2 is aiming for a second release within weeks with a larger dataset, more skills and built-in reinforcement learning training; two-arm robots and hands with touch sensors are among the next steps. The technical report has not been published yet, and the results shared are Ai2's own experiments. Sources: [Ai2 blog post](https://allenai.org/blog/toyblocks?ref=globalfeed.ai), [GitHub](https://github.com/allenai/toyblocks?ref=globalfeed.ai), [Molmo2-4B-tb](https://huggingface.co/allenai/Molmo2-4B-tb?ref=globalfeed.ai), [ManiFlow-tb](https://huggingface.co/allenai/maniflow-tb?ref=globalfeed.ai), [Dataset](https://huggingface.co/datasets/allenai/toyblocks-data?ref=globalfeed.ai), [Documentation](https://allenai.github.io/ToyBlocks/docs/user%5Fguide/index.html?ref=globalfeed.ai)