HexaShell: A Thin Harness for Exploring the Limits of VLM Agents as Robot Coding Agents

HexaShell is a thin harness that enables vision language model agents to control a two-armed robot through a command line interface. The system uses three RGB cameras with known calibration to provide visual input, and the agent writes or fixes small programs called tools to perform tasks based on triangulation and camera observations. After each failed attempt, the agent reads logs to identify causes and improve its tools.