A Parol 6 arm with a camera on its wrist that recognises objects and acts on them from text input: my introduction to robot vision, embedded AI and arm control.
01 · the problem
Could a robot arm act on plain text?
When I arrived in the UK for my degree I wanted a project I could build in a small space. Tools like ChatGPT had just shown what AI could do with text, and I wanted to know how far that could reach into the physical world.
02 · the hypothesis
Small, on-board vision is enough
After talking with experts in the field, the goal became an arm that interacts with its surroundings based on text input. The bet: a small embedded vision module that recognises known objects, mounted on a self-built open-source arm, could find things and act on them without a big computer.
03 · planning
Open-source parts and a small team
- Researched open-source arms that could be built from off-the-shelf parts, and learned the fundamentals of neural networks.
- Chose the Parol 6 by Source Robotics, built from scratch from its GitHub project, with an OpenMV camera on the end.
- Assembled a small team to build it.
04 · executing
From camera to coordinates
The camera mapped a known object and sent its x and y position as JSON to an ESP32, which talked to the arm’s control software over WiFi. The OpenMV is a single camera and can’t see depth, so we used AprilTags to get the third coordinate.
05 · iteration
Adding a tool changer
Once the arm could find objects, we designed and built a tool changer. The arm located it automatically and swapped to a new tool based on predefined AprilTags.
06 · latest version
It worked, and showed the next step
It worked decently but not perfectly, and it taught me everything from embedded vision to arm kinematics. It also showed that fixed object detectors weren’t enough: vision-language models, like those Physical Intelligence was building, were the better route. That led to my open-source ML arm work.
| stack | OpenMV · AprilTags · ESP32 |
