teaching ob15: what I’ve learned about robot policies

OB15 sorting coloured blocks into matching bowls on a lab table

This is about how OB15 learns: collecting demonstrations, training policies, what the results taught me, and what I’m working on now. The hardware has its own post.

OB15 sorting coloured blocks into matching bowls on a lab table
sorting blocks by colour
OB15 v3 standing in a lab: two orange arms on a lift, omni-wheel base and orange head camera
OB15 v3

collecting the data

I teleoperate OB15 in VR, controlling both arms, the grippers, the head, the base and the lift at once, and record each demonstration through LeRobot. For the four test tasks (passing a block between grippers, putting a tool in a box, sorting blocks by colour, and mobile trash pickup) that came to about 8 hours of data, with 150–250 demonstrations per task. All of my datasets are public on Hugging Face.

what ACT taught me

I trained ACT (Action Chunking with Transformers) policies on each task. Across all four, the overall success rate was 43%, against roughly 90% for Mobile ALOHA, a $30,000 robot with a team of engineers behind it. The interesting part was why the failures happened:

  • Out-of-distribution views: colour sorting failed at first because the robot saw angles and lighting it had never been shown. Collecting targeted data to cover those gaps fixed it.
  • Transparent objects: a clear plastic bottle on a featureless black floor was nearly invisible to the policy, and with no landmarks it drove straight past.
  • Settings matter: on block transfer, a chunk threshold of about 0.7 (replanning when 70% of the current action chunk has run) gave the best success, falling sharply above it. Weighted-average blending of old and new chunks beat the “latest only”, “average” and “conservative” options.
end-to-end: driving to the table and putting the pliers in the box

better labels: stage-aware rewards

I added SARM (stage-aware reward modelling) to the datasets: a vision-language model watches each demonstration and scores progress from 0 to 1 through the task’s sub-stages. That gives the policy a much clearer picture of where it is in a task than “success” or “failure” alone.

the lesson

More data builds a better foundation, but data quality is what makes fine-tuning work. Clean, varied demonstrations that cover the situations the robot will actually see beat a bigger pile of similar ones.

one-shot learning: instant policy

Most imitation-learning policies need dozens of demonstrations per task. Instant Policy treats a few demonstrations as context instead, so the robot can attempt a new task right after being shown it. I’m building a demo-conditioned, one-shot policy for OB15 inspired by that idea, on top of the SmolVLA vision-language-action model.

world models: jepa

JEPA (joint-embedding predictive architecture) models learn by predicting what happens next in an abstract representation rather than pixel by pixel. I’m experimenting with JEPA-based models as a way for OB15 to learn more from the data it already collects.

next: whole-body policies

The plan is for one policy to control all 18 of OB15’s joints at once: six per arm, head pan and tilt, three for the base and one for the lift. To get enough training data, I’ll co-train on my own recordings alongside large public datasets from stationary arms, masking out the joints those robots don’t have. I’m also integrating MolmoAct2, a newer open vision-language-action model, into my LeRobot fork, and the tactile sensors will add touch as another input.

Code: my LeRobot fork · OB15 on GitHub