Why HUD for robotics
We have built robotics evaluations internally for months, for VLAs, world action models, and now coding agents. The framework is what that work settled on:- Port any simulator. Wrap a simulator in a bridge and a contract, and it becomes an eval. We have done this for MuJoCo, robosuite, ManiSkill, Newton, and Isaac.
- One environment, any model type. The simulator, tasks, and grading stay the same. Only the
interface changes: a policy streams actions over the
robotwire, and a tool-calling agent calls motion tools such asmove_to. - Graded and replayable. Every template grades its episode, and every episode replays with camera video and state and action plots.
Robot support in the SDK is in beta. See the robots guide.
Two ways to control a robot
Featured environment
robot-libero is a quick way to try robot control from the web app. It serves LIBERO, a Franka Panda tabletop benchmark on MuJoCo, with one template per suite:libero_spatial, libero_object,
libero_goal, libero_10, and libero_90. It serves both interfaces, so a policy and a
tool-calling agent run the same tasks, and its traces use the episode replay.
It is open source in the SDK’s
environments/featured
directory, so you can clone it and add tasks of your own.
Build a robot environment
Start from the Robotics template, for MuJoCo and LIBERO: on Environments, select New Environment, then See all templates. You implement a bridge around your simulator withreset, step, and get_observation, and
describe the embodiment in its contract; the SDK handles the connection and sends telemetry to HUD.
See the robots guide.
Set the contract’s frame rate and per-dimension names. The trace viewer uses them to play video at
real speed and to label each state and action plot.
Replay episodes
Environments built on the SDK’s robot support stream every control tick to the trace. Its Robotics view plays the episode back:- Cameras shows each camera as video, with play, pause, and playback speed.
- Plots charts each action and observation dimension over time, named from the contract.
- Split shows the cameras and the plots together.
Robots guide
Bridges, contracts, policies, and batching.
Robot benchmark
Run pi0.5 on LIBERO end to end.
Evaluations
Run tasks across models and read the results.