text-controlled robotic arm

robotics

text-controlled robotic arm

A Parol 6 arm with a camera on its wrist that recognises objects and acts on them from text input: my introduction to robot vision, embedded AI and arm control.

01 · the problem

Could a robot arm act on plain text?

When I arrived in the UK for my degree I wanted a project I could build in a small space. Tools like ChatGPT had just shown what AI could do with text, and I wanted to know how far that could reach into the physical world.

02 · the hypothesis

Small, on-board vision is enough

After talking with experts in the field, the goal became an arm that interacts with its surroundings based on text input. The bet: a small embedded vision module that recognises known objects, mounted on a self-built open-source arm, could find things and act on them without a big computer.

03 · planning

Open-source parts and a small team

  • Researched open-source arms that could be built from off-the-shelf parts, and learned the fundamentals of neural networks.
  • Chose the Parol 6 by Source Robotics, built from scratch from its GitHub project, with an OpenMV camera on the end.
  • Assembled a small team to build it.

04 · executing

From camera to coordinates

The camera mapped a known object and sent its x and y position as JSON to an ESP32, which talked to the arm’s control software over WiFi. The OpenMV is a single camera and can’t see depth, so we used AprilTags to get the third coordinate.

05 · iteration

Adding a tool changer

Once the arm could find objects, we designed and built a tool changer. The arm located it automatically and swapped to a new tool based on predefined AprilTags.

06 · latest version

It worked, and showed the next step

It worked decently but not perfectly, and it taught me everything from embedded vision to arm kinematics. It also showed that fixed object detectors weren’t enough: vision-language models, like those Physical Intelligence was building, were the better route. That led to my open-source ML arm work.

stackOpenMV · AprilTags · ESP32