✦
Summary
Show-Harness lets vision-language models control robots through a compact set of semantic action symbols. It connects high-level visual reasoning with fine-grained physical decisions, using one interface across different embodiments rather than requiring every model to directly predict continuous motor commands.
Key Features
It selects discrete semantic action units interpreted as local robot motion.
Action symbols are parameter-free; the embodiment interpreter supplies metric magnitudes.
The action vocabulary is shared across real robot embodiments and simulation.
The project demonstrates direct robot control with closed-source frontier VLMs.
Open VLMs can be fine-tuned through the same semantic interface.
GUMI is a graphical manipulation interface for collecting robot demonstrations.
The demonstrated collection workflow uses keyboard actions through a GUI.
The project links public code, trained models, and demonstration data.
About Show-Harness
The VLM chooses actions such as moving, rotating, grasping, releasing, or finishing. An embodiment-specific interpreter deterministically translates each symbol into local motion. The same interface supports zero-shot use of frontier VLMs and lightweight adaptation of smaller open models for faster deployment.
The project is useful for robot-control research and demonstration collection without specialized teleoperation hardware. Its GUMI interface allows humans or GUI agents to drive the robot by keyboard while recording training pairs. The release links code, models, and datasets for reproducing the approach.