For a long time I've wondered why open-loop policies and action chunks dominate modern robotics. It's counter-intuitive for me that an action model would work best when executing actions blindly, with a small context and large action chunks.

So last week, inspired by two recent writings, Behavioral cloning mystery and Revisiting Open-Loop Execution in Robotics, I took a stab at understanding the problem space myself and simply did some testing.

In this article I'll try to answer one big question by answering a series of smaller ones: in which case do closed-loop policies win?

Why this question matters to me: I've been interested in doing RL for robotics recently, and along the way I've seen the number of hacks the field has derived to accommodate action chunks. I'm a big fan of being exceptionally naive and stupid in answering questions (cough, the bitter lesson), so I take it personally when the answer to a problem is not elegant and requires hacks. We should just be able to RL modern robotics the naive way.

My test environment

What we have in this picture is an aluminum enclosure with a ReBot arm in the center. That's quite it.

The task is a simple: stack the given pieces onto a peg. The goal is to put the blue piece on first, then the yellow piece, then the orange piece. I designed the pieces and the peg myself to be on brand (sorry, I'm obsessed with things I design), but more importantly, I designed it myself so that I can later RL the task in a sim environment and measure the transfer rate.

A nice design quirk is that the peg is removable, because I learned how easily a 3D-printed peg breaks while collecting data for this task (on average I break a peg every 40 episodes). Another design quirk is since the pieces shape is irregular, the policy has to learn to grasp at the correct orientation, or else the pieces will literally fly out of the gripper. This contributes largely to the demise of open-loop policies with long horizons in later experiments.

At first I wanted to do Tower of Hanoi, and I still have a version I designed, but after several attempts at collecting data for it I realized my time collecting data would be greater than my time doing experiments, so I stopped.
You can find the designs in the sim2real folder of SO-Frame on OnShape.

After designing the task, I collected 200 episodes of playing the game over the course of 3 days (the whole thing should have cost me 5 hours total, but I re-judge my life decisions every 2 hours lol, so it took 3 days). I also collected 200 episodes of resetting the environment, so I can later RL the task and do automatic rollouts. The dataset can be found here. I divide it into a 50-episode, a 100-episode, and a 200-episode variant.

For the data-collection and inference setup, I'll write a separate article, since I think it's a nice and simple architecture that deserves its own praise.

My test subject

For the simplicity of the experiment, I chose the subject of questioning to be one that has already been rigorously tested on: Diffusion Policy. Throughout my experiments, I’ll sweep the following parameters:

• Context: how many past frames the policy looks at before deciding. I trained 2, 10, 18, and 34 frames, which at 30 fps corresponds to approximately 0.07 s to 1.13 s of frames.
• Execution horizon: how many actions it executes blind before looking again. Horizon 1 is fully closed loop (re-plan every tick). Horizon 31 runs the whole predicted chunk before looking again. The default preset in LeRobot is context 2, horizon 8.

To ensure my experiment is isolated from latency and task dynamics concerns, I optimized a single Diffusion Policy inference pass to take less than 15 ms for every context length, including the 34-frame policy. This is comfortably below the 33.3 ms control period of a 30Hz control loop, so even horizon-1 policies can replan every frame without starving the control loop.

Question 1: Are action chunks long because the policy can’t see enough?

This isn't really my question. It comes from the two articles I mentioned at the beginning. I mostly wanted to see if I could reproduce their results.

Park's post shows that a fully closed-loop policy can fail badly while a long open-loop chunk succeeds. One explanation he proposes is that demonstrations can be non-Markovian from the policy’s perspective: the demonstrator’s next action depends on history that the policy cannot infer from its current observation.

Repeatedly replanning from an ambiguous observation can produce individually plausible actions that do not form a coherent sequence. Executing a chunk can preserve commitment to one behavior.

But Park also tests history conditioning, and it does not solve the problem in his experiment. It makes rollout performance worse despite improving imitation error, even with effectively infinite data. He suggests causal confusion or greater sensitivity to distribution shift as possible explanations.

Zeng et al. reach a different result: sufficiently long observation context can reduce the need for open-loop commitment and make reactive policies perform best. Both works identify missing temporal information as a problem, but they differ on how successfully observation history resolves it.

That tension is what I wanted to test.

In my task, a 2-frame policy may not have enough information to infer the current stage of manipulation, while a 34-frame policy receives over one second of history. History could also help when a piece briefly leaves the camera’s view, by preserving information about where it was moving.

Following Zeng’s result, I want to see two predictions:

• Increasing observation context should reduce prediction error.
• The cost of increasing execution horizon should grow with observation context.

Let's test them, woohoo.

Evaluation protocol

All results in Question 1 are computed using offline evaluation on a held-out set of 15 demonstrations that were never used for training. In Question 2, we’ll use real rollouts. I mention this here in case you read this section and preemptively be like why tf I didn't roll out in real or in sim to validate.

For each demonstration, the policy receives the recorded observation sequence and predicts an action chunk of length 31. Predicted actions are compared against the corresponding human actions using mean squared error.

For an execution horizon H, I score only the first H actions of each predicted chunk, since those are the actions that would be executed before replanning.

This metric measures held-out imitation error under logged observations. It does not measure real-world task success, recovery from policy-induced errors, or compounding state-distribution shift during rollout.

Measurement 1: Full-chunk evaluation

As a first diagnostic, I evaluate the mean action-prediction error over all 31 predicted actions.

Under this metric, increasing context produces only a modest improvement, and the trend is not monotonic.

However, this metric does not correspond to the actions actually used when the execution horizon is shorter than 31.

At horizon 8, for example, only the first eight predicted actions are executed. Actions 9 through 31 are discarded when the policy replans.

Since prediction error increases with prediction offset in these evaluations, averaging over the full chunk places substantial weight on actions that never affect execution.

Measurement 2: Execution-matched evaluation

I therefore recompute the metric using only the first H actions for each execution horizon.

Under execution-matched evaluation, every longer-context model outperforms the 2-frame baseline at all three reported horizons.

The benefit is strongest at short execution horizons. Relative to 2 frames, the 34-frame model reduces MSE by:

• 37.4% at horizon 1
• 30.2% at horizon 8
• 8.9% at horizon 31

The benefit of additional observation history therefore decreases as execution becomes more open loop.

Measurement 3: Open-loop penalty

The stronger claim is not simply that more context improves prediction.

If long execution horizons mainly compensate for missing temporal context, then the cost of increasing execution horizon should itself grow as more context is provided.

I define the open-loop penalty as:

The ratio increases monotonically with observation context.

For the 2-frame model, expanding the scored execution window from horizon 1 to horizon 31 increases error by a factor of 2.04. For the 34-frame model, it increases error by a factor of 2.96.

I use "open-loop penalty" here as shorthand for the increase in held-out action-prediction error as the execution window expands. Because this is an offline evaluation on logged human trajectories, it does not include state-distribution shift or error compounding caused by the policy's own actions during a real rollout.

The result is nevertheless consistent with the proposed mechanism: additional temporal context is most useful when the policy can repeatedly incorporate new observations. As execution becomes more open loop, that advantage is progressively lost.

What the results suggest so far

The three measurements support a consistent offline pattern: longer context improves prediction most at short execution horizons, and its advantage shrinks as the scored action window expands. This is consistent with history reducing temporal ambiguity, but logged-observation MSE does not establish the cause or demonstrate successful feedback control.

A longer context also only helps if the policy learns how to use it. That leads to the next question: how much data and optimization does long context need before the advantage appears?

Measurement 4: Data and optimization regime

I compared 100 episodes trained for 20k steps with 200 episodes trained for 40k steps, keeping the epoch count approximately matched. This comparison changes both dataset size and the number of optimization steps, it does not isolate the effect of additional data.

On the 100-episode dataset and horizon-8 policy, the result initially went in the opposite direction: longer context appeared to hurt.

At 100 episodes and 20k steps, the 34-frame model was 22.2% worse than the 2-frame model. At 200 episodes and 40k steps, it was 30.2% better.

My first interpretation was overfitting. The train/evaluation gap grew from roughly 3.5× at 2 frames to 7× at 34 frames, which was especially interesting given Park's result that conventional overfitting signals in behavioral cloning do not necessarily predict worse rollout performance.

But matched epochs do not mean matched optimization difficulty. To separate data from optimization, I trained the same 100-episode dataset for 40k steps. That single control recovered approximately 94% of the improvement seen in the 200-episode, 40k-step condition.

So the reversal was mostly an optimization effect, not a sharp data threshold. More data still helped, particularly for long context, but the main lesson was that long-context policies required more optimization to learn how to use the additional history.

Question 2: Does this hold on hardware?

Everything in Question 1 was measured offline. That let me compare context and execution horizon, but it removed the main reason feedback should matter: consequences.

So I ran the policies on the real robot.

Evaluation Protocol

The hardware evaluation used the 200-demonstration checkpoints trained for 40k steps. Five configurations were evaluated. The outcome is binary task success.

BASE is the default context-2, horizon-8 configuration in LeRobot. The predefined headline comparison was whether LC-closed exceeded BASE with a 95% confidence interval for the success-rate difference excluding zero.

Measurement 1: Overall success

LC-closed achieved the highest observed success rate: 84%, compared with 72% for BASE. The improvement over BASE is not statistically significant at this number of rollouts (this is why I increase the rollout count for BASE and LC-closed, but we would need around 200 for us to record the 12% difference reliably, so I make do).

The pattern across horizons is clearer. With long context, horizon 1 reached 84% success while horizon 31 reached 10%. With short context, horizon 8 performed best, beating both horizon 1 and horizon 31. The offline MSE ranking did not predict this intermediate optimum.

Qualitatively, LC-closed runs much more reliably than others. It still fails some trials, but I do not see the frequent problems of short-context closed-loop or open-loop policies like:

• The short-context closed-loop policy sometimes does not start moving, or reaches the pole and gets stuck. I still don't know why this is the case. My guess is that it can't push itself into distribution through noise since it only has one action.
• The open-loop policies sometimes release before alignment is complete or keep moving after a bad grasp, pushing the piece away. This makes the task unachievable in my setup. This was by design, as I mentioned before, thanks to how the pieces are designed, the policy needs to react and recourse quickly or else it will fail.

So why? Why does long context with small action horizon work? A possible explanation is that short context needs some action commitment to stay consistent, while long context provides enough history to replan without getting stuck. Frequent feedback then gives the policy a chance to correct a bad grasp or alignment. The short-context stalls resemble the failures discussed by Park. However, adding history worked well here, which aligns more closely with Zeng’s results than with Park’s history-conditioning experiment.

What we learned

I started with a fairly naive question: when do closed-loop policies win?

My answer is fairly naive too: when the task needs the policy to react quickly. We have seen that closed-loop policy can match performance with open-loop policy once trained with enough data and with long enough context. Therefore, at the end of the day, the choice comes down to the nature of the task and whether reactiveness is desirable. In my task, as you have learned, it is designed so that open-loop fails miserably, and closed-loop is necessary.

Maybe that tradeoff will change as world models give action models better predictions of what will happen next.

A few caveats:

• Short context may be easier to deploy reliably because it depends less on past observations. Long context introduces more ways for delays, stalled motion, and execution errors to produce unfamiliar histories, potentially requiring more varied data or augmentation. This may require the training dataset for long context to be augmented in a way that would resemble deployment distribution.
• Short context is fast. Since long context requires you to encode and attend to a good amount of images, this often makes it several times slower than short context policies. Of course, since the policies themselves are small, this encoding time takes a majority. However, as the policy grows larger, this is negligible.