the premise

Yesterday I recorded over 300 episodes of data, then I tried to train over 10 different policies (VLAs, JEPAs, MultiTask-DIT, etc) and benchmark them by measuring task success rate.

Mathematically, to have statistically significant comparisons, for a projected difference of 5% on task success rate between policies, I would need at least 1550 rollouts per policy. For readers who don't know what a rollout is, it's simply deployment of a policy you train: a successful rollout would be a policy completing a task that it was trained on i.e. cleaning a dish.

If I were to faithfully compare 10 policies, I would need to accumulate over 15550 rollouts. At 2 minutes per rollout, which in itself is unrealistic for complex tasks and with different hardware failure model, I would cost 22 days, or upward to a month and a half considering real life constraints.

Reality is much harsher though. Research teams often move faster than just 10 policies per day, just simple tuning can give more than a dozen model variations. Ablations, evaluations and comparisons become much harder problem than they seem to be. In fact, most recent robotics papers I've read have completely thrown statistics out of the window, which results in unfaithful or non existent improvements over priors.

So we need better way to do rollouts. This is one of the reasons why world models are so valuable, aside from learning a task, it’s also a key driver in evaluations. You can’t realistically evaluate a policy trained on real world data in simulations: I don't believe this is a term, this is a real2sim gap lol. As seen in recent rigorous papers i.e ABC, research teams would rather collect data in sims and evals in sims for ease of operations.

having fun

This is obviously the future. Now my question is whether I can reproduce the premise with a budget of a hobbyist. After all, I still have a ton of policies to eval, and I would like to uphold my principle of doing things the right way.

So I set out to train an action-conditioned world model to see how fun it would be. Now, I already did this before, but I did everything from scratch, and it was unsupervised training. This time I'm going to just use an existing code base and train it using my labeled video-action dataset. Pardon for not sharing the dataset in this article, I have another project in which I will share the dataset soon.

My dataset has more than 300 episodes of a robot doing pick-and-place task, a total of 115,368 frames at 30fps, which is roughly 64 minutes, very small compared to others. Using it, I trained NanoWM-B/2 with the following settings:

• Model: ~242M params BF16, diffusion-forcing DiT, trained in SD-VAE latent space, actioned conditioned through simple addition
• Input/Output: 4 frames in, predict one ahead
• Batch size 16 * 30k steps = ~4.8 epochs

The training took place on a RTX5090, which costs about 2 hours total. After that, to evaluate the world model, I use held-out episodes from my dataset and replay the trajectories inside the trained world model side-by-side. The result is as following.

Task 1: Put the purple bar into the orange bin

Task 2: Put the red bar into the yellow bin

Task 3: Put the blue bar into the blue bin

Knowing that this took 2 hours of compute and 1 hour of data, the result is impressive. While colored blocks are spawning all over the place, we have fairly accurate representations of the robot arm over time. Remember, this is purely autoregressive, only the first 4 frames in the world model generation are from ground truth, the world model did not drift too far from reality even after hundreds of generations.

In 5 hours, we have produced the intuition of world model rollouts. I cost more time writing this article than training the model lol.

Some fun notes:

• World model training can be heavily decode bound. Decoding frames with CPU initially took a significant chunk out of my training time, which I solved by caching frame latents since the model use a frozen VAE. One can use hardware-accelerated decoding for videos on the fly (as most teams in reality) but in my case it’s faster to do caching considering the dataset size.
• Sampling strategy can also hurt performance, if you randomly sample frames from compressed video (codecs like AV1 or H.264), your decoding process has to search and reconstruct frames from key frames, this adds unnecessary fixed overhead per frame. Sequential reads were ~50× faster than random in my tests. Caching also helped close this gap.

what this means

I did this for fun, it costs me like 5 hours, but it did gave me some good takeaways.

The first is the sheer potential of building a world model eval company. The need for world model inference is going to skyrocket as robots get deploy in the wild. If Physical Intelligence can hand off eval to a third party, I think they would gladly do, here is their research with Nvidia on exactly this btw. You already saw the math. If I need more than a month of evals per week, how much would that translate to a high-caliber team, especially with stuff like auto research.

On an adjacent perspective, it's good to note that robotics is highly mirroring LLM development. LLM eval companies are very hot at the moment, not only for the reason that they do evals, but also that they are aligned to build great RL environments. If you know how important RL is for post-training, you just know. The same curve is going to happen in robotics at much larger scale, arguably, as, RL exists at every training stage here.

The final take away is another notion on scale. If a hobbyist like me can do rough calculations and proof of concepts with only 2 hours of compute and 1 hour of data, how far can a lab with millions in funding go? By the end of this year, we'll start to see unfathomable progress in world modeling through data scaling. In fact, you'll most likely be very surprised by this world model (which is proudly served through LiveKit -- cough cough shameless plug): https://decart.ai/oasis.