This is about how OB15 learns: collecting demonstrations, training policies, what the results taught me, and what I’m working on now. The hardware has its own post.


collecting the data
I teleoperate OB15 in VR, controlling both arms, the grippers, the head, the base and the lift at once, and record each demonstration through LeRobot. For the four test tasks (passing a block between grippers, putting a tool in a box, sorting blocks by colour, and mobile trash pickup) that came to about 8 hours of data, with 150–250 demonstrations per task. All of my datasets are public on Hugging Face.
what ACT taught me
I trained ACT (Action Chunking with Transformers) policies on each task. Across all four, the overall success rate was 43%, against roughly 90% for Mobile ALOHA, a $30,000 robot with a team of engineers behind it. The interesting part was why the failures happened:
- Out-of-distribution views: colour sorting failed at first because the robot saw angles and lighting it had never been shown. Collecting targeted data to cover those gaps fixed it.
- Transparent objects: a clear plastic bottle on a featureless black floor was nearly invisible to the policy, and with no landmarks it drove straight past.
- Settings matter: on block transfer, a chunk threshold of about 0.7 (replanning when 70% of the current action chunk has run) gave the best success, falling sharply above it. Weighted-average blending of old and new chunks beat the “latest only”, “average” and “conservative” options.
better labels: stage-aware rewards
I added SARM (stage-aware reward modelling) to the datasets: a vision-language model watches each demonstration and scores progress from 0 to 1 through the task’s sub-stages. That gives the policy a much clearer picture of where it is in a task than “success” or “failure” alone.
the lesson
More data builds a better foundation, but data quality is what makes fine-tuning work. Clean, varied demonstrations that cover the situations the robot will actually see beat a bigger pile of similar ones.
one-shot learning: instant policy
Most imitation-learning policies need dozens of demonstrations per task. Instant Policy treats a few demonstrations as context instead, so the robot can attempt a new task right after being shown it. I’m building a demo-conditioned, one-shot policy for OB15 inspired by that idea, on top of the SmolVLA vision-language-action model.
world models: jepa
JEPA (joint-embedding predictive architecture) models learn by predicting what happens next in an abstract representation rather than pixel by pixel. I’m experimenting with JEPA-based models as a way for OB15 to learn more from the data it already collects.
next: whole-body policies
The plan is for one policy to control all 18 of OB15’s joints at once: six per arm, head pan and tilt, three for the base and one for the lift. To get enough training data, I’ll co-train on my own recordings alongside large public datasets from stationary arms, masking out the joints those robots don’t have. I’m also integrating MolmoAct2, a newer open vision-language-action model, into my LeRobot fork, and the tactile sensors will add touch as another input.
Code: my LeRobot fork · OB15 on GitHub
