the intuition of planning

it has been shown that VLAs improve with fine-grained data labeling.

π0.5 first showed traces of this through a chain of thought process, in which the model produces a subtask of a task it is instructed to do before generating the action. concretely, it predicts a subtask label as text, then conditions its action generation on that label, all within the same network.

π0.7 further reinforces this by introducing subgoal images along with subtasks. rather than retrieving a recording, it conditions on a generated image of what the scene should look like a moment into the future, produced at runtime by a lightweight world model, essentially giving the policy a visual target to steer toward.

it's fun to reflect on why such information is even helpful to a model, especially when given explicitly. is this steering behavior essentially a shortcut before we can truly achieve data scaling in robotics? or is it desirable and actually reflects the way we plan ourselves?

in psychology, there is this notion of system 1 and system 2 thinking where system 1 is like our subconscious driver of actions and motor planning while system 2 is the strategic conscious planner that is engaged when solving complex tasks.

while in principle the purpose of subtasks is to reflect this hierarchy of thoughts, where a model would dissect tasks into smaller tasks, it's counterintuitive to think a command like "pour me a cup of wine" would engage system 2 in our brain for complex planning: "locate the wine bottle", "hold it", "unscrew it", "hold it up", "pour it", "put back the wine bottle".

yet, by the looks of it, we are optimizing for more and more fine-grained task labeling in datasets, essentially creating primitives of actions, or atomic actions that can't be further divided into smaller actions.

this weekend, i did a small experiment to see whether this actually helps increase task success rate within SmolVLA: one trained on coarse tasks and the other on fine-grained tasks.

in this short article, i'll go through how i did subtask labelling, its failure modes, compare performance empirically, then discuss other applications of subtask labelling.

subtasking

the inspiration of this article comes from Macrodata's subtask labelling benchmark. in their blog, they showcase an automated pipeline to turn a LeRobot dataset with coarse task into one with subtask labeling, and create a labeling benchmark with it.

i recently recorded a ~300-episode dataset for sorting colored bars into colored bins. it was for benchmarking whether models would generalize to different color combinations of sorting, so i thought, why don't i benchmark the impact of subtasking on model's performance.

the environment has 4 colored bins. the task instruction for each episode is in the following format: "put the {1st_color} bar into the {2nd_color} bin". i was very curious whether Gemini 3.5 Flash would:

• segment the boundary of the subtask within an episode correctly.
• label it correctly into 2 subtasks: pick and put.

so i tried Macrodata's platform (long story short: they currently have a bug that made it not usable for my dataset, i have since then pushed a pr to their codebase to fix this bug, but it might just be some quirks on my side that produces this). you can follow their example here. they reached out when they saw that i was trying the platform and was very helpful.

on the first count of segmenting subtask, Gemini did this wonderfully, I think this was because the boundary between pick and place was fairly easy to spot in my dataset. Macrodata has documented their method to do this, which essentially renders frames of the video at intervals then passes it to a VLM:

the image itself is what Macrodata calls a contact sheet: a grid of sampled frames packed into one tiled image. the idea is inspired by Scale.ai's dense video captioning work, though Scale used past/present/future "hand-collages" (hand-cropped frame grids) rather than a single sheet.

on the second count of labelling subtask, Gemini did not succeed with flying colors. out of ~300 episodes in my dataset, roughly 30-40 episodes were wrongly labeled.

since the task is just bin sorting, every episode should have yielded a pick segment "pick up the {1st_color} bar" and a put segment "put the {1st_color} bar into the {2nd_color} bin". this makes it fairly easy to scan for failure modes or wrongly labeled subtasks, as well as fixing them, as shown by the mismatched row below.

some of the failure modes are:

• hallucinating non-existent or inconsistent steps: "pick up the red bar and place it into the blue bin" -> "from the blue bin and place it into the yellow bin"
• duplicating non-atomic actions in a single step: "put the purple bar into the orange bin, then pick up another purple bar and put it into the orange bin"
• wrong color: "pick up the orange bar" while the episode was for a blue bar

none of this surprised me since this was mentioned in Macrodata's own results on their benchmark. fixing it in my case was pretty simple too, so this wasn't much of an obstacle.

empirical impact

to compare the impact of subtask labeling, i trained 2 variants of SmolVLA, one trained on coarse task labelling and the other trained on fine-grained task labelling.

for simplification, the coarse task dataset contains task A, which is "put the {1st_color} bar into the {2nd_color} bin". the fine-grained task dataset directs task A into task B and task C: "pick up the {1st_color} bar" and "put the {1st_color} bar into the {2nd_color} bin".

in the last article, i mentioned the rigor of benchmarking in robotics. in this article, i'll instead retardmax for the simple purpose that i'm constrained on compute and time (this is my weekend project).

to benchmark the models, after playing with the prompt for a while, i realized the subtask SmolVLA, while given only the prompt of "putting the … bar into the … bin", would actually also do the pick up process. so i decided to roll out 40 episodes between the two policies to see if i can measure the difference in task success rate.

the eval task will be "put the purple bar into the {color} bin", we'll cycle the color every 10 rollouts, totaling 40 rollouts since we have 4 different-color bins. below is a sped up video of the rollouts which takes nearly an hour. for success criteria, i simply let the model run until it finishes. if the model can't solve the scene in less than 2 minutes, i consider that a failure (this is very generous).

performance wise, subtask SmolVLA has 82.5% success rate while normal SmolVLA has 75% success rate. beyond the raw number, two observations stood out:

• subtask SmolVLA has better spatial awareness: it can pick up bars in positions that normal VLA can't pick up.
• subtask SmolVLA has better color awareness: it puts the bar into the correct color bin, unlike normal VLA which sometimes mistakes blue with white and orange with yellow.

now the disclaimers:

• first, please understand i truly have an ick when stating this result, for there were too few rollouts per policy to statistically conclude that subtask SmolVLA is better. empirically, subtask SmolVLA were better for the reasons mentioned above.
• second, the subtask policy is prompted slightly out of its training distribution, since it was trained on the split "pick" and "put" instructions but evaluated with the composite "put the … bar into the … bin".
• third, on why it also does the pick when only told to put: i believe this is a collapse mode in training for smolvla in which the image has more contributions to the action than the prompt. if that's true and the prompt barely steers the policy anyway, then "out of distribution" may not be costing it much at all, which is worth keeping in mind against the fairness disclaimer above.

as i did this, i also wondered whether the subtask SmolVLA actually learned to generalize to the position of the bin. it didn't: when i switch the position of the color bin, it still puts the bar into where the bin used to be, so it's more language-spatial association for the bin location than any real spatial grounding.

the implications

it's no surprise subtasking is going to be more prominent. the biggest draw is reusability. almost every task decomposes into a few primitive actions. break a task into primitives and the model gets a library of skills, each trained on way more data than any full task, which is where the generalization comes from.

aside from boosting performance of general robotics models, subtasks also enable training multiple single-task policies like ACT. this is helpful if you have a small amount of data. for example, instead of training a VLA the task of "pick this cube and place it into the container" which requires thousands of episodes, you can train 2 ACT policies, one for "pick this cube" and another for "place it into the container", each with just a hundred of episodes, then use a VLM to orchestrate them.

furthermore, automatic subtasking saves time. if you plan to record a fine-grained dataset, you can now do a single continuous large task, then let a pipeline segment and label it for you, instead of changing labels manually as you do the task.

remarks

this was a rather fun exploration path, which makes me wonder what the real impact of subtask labeling is, had i spent more time evaluating (forgive me the fathers of science for i have disappointed you this weekend).

the evaluation harness, inference, and observation were built entirely with LiveKit Portal. the SmolVLA policies were trained using single H100 on Nebius, then inferred on a RTX5090 in LiveKit office in SF, the physical rig is with me in Vietnam, they were tethered using LiveKit Portal. this allowed me to spawn 2 warm SmolVLAs processes side-by-side and switch between them with just a click.