Manipulation
Bimanual VLA manipulation on OpenArm
Full teleoperation-to-policy pipeline bringing a generalist Vision-Language-Action model to a new bimanual arm with very little data.
LeRobotOpenPIX-VLAPyTorchikpyDepthAICANJetson AGX Orin
Problem
Generalist VLA policies are trained on other embodiments. Bringing one to a new bimanual arm with only a handful of demonstrations requires the whole stack: motors, CAN bus, cameras, teleoperation, data format, and fine-tuning.
Approach
- Built a 4-channel CAN bus (PEAK PCAN-USB Pro FD) driving two follower and two leader arms.
- Added Damiao motor support in LeRobot and wired three OAK-D cameras over DepthAI.
- Ran ExoArm leader-follower teleoperation on a Jetson AGX Orin.
- Collected 40 teleoperated episodes across two tasks ("pick bottle", "pick box") in LeRobot format.
Result
- OpenPI pi-0.5 (DROID checkpoint) failed zero-shot due to embodiment mismatch — trained on Franka Panda, outputs Cartesian end-effector deltas requiring an IK layer.
- Pivoted to fine-tuning X-VLA on the collected data; pipeline built and partially validated.
- [ADD SUCCESS RATES WHEN AVAILABLE]
System breakdown
Hardware
- OpenArm bimanual system: 2 leader + 2 follower arms
- Damiao actuators over 4-channel CAN (PEAK PCAN-USB Pro FD)
- 3 × OAK-D depth cameras
- NVIDIA Jetson AGX Orin compute
Firmware / embedded
- CAN motor driver integration and bus arbitration across 4 channels
- Patched a motor-driver handshake hang caused by continuous motor broadcasts, using a time-bounded drain loop
- Leader-follower torque/position streaming for ExoArm teleoperation
Software
- LeRobot dataset recording and replay
- OpenPI pi-0.5 zero-shot evaluation, X-VLA fine-tuning
- ikpy inverse-kinematics layer for Cartesian delta actions
- DepthAI multi-camera synchronisation
Project demo video slot