Skip to main content
For teams evaluating robot policies, such as vision-language-action models (VLAs), or LLM agents, including coding agents, that control a robot. A robot environment wraps a simulator and describes its cameras, state, and actions in a contract. Each episode becomes a trace you can replay: every camera as video, with the robot’s state and actions plotted over time.

Why HUD for robotics

We have built robotics evaluations internally for months, for VLAs, world action models, and now coding agents. The framework is what that work settled on:
  • Port any simulator. Wrap a simulator in a bridge and a contract, and it becomes an eval. We have done this for MuJoCo, robosuite, ManiSkill, Newton, and Isaac.
  • One environment, any model type. The simulator, tasks, and grading stay the same. Only the interface changes: a policy streams actions over the robot wire, and a tool-calling agent calls motion tools such as move_to.
  • Graded and replayable. Every template grades its episode, and every episode replays with camera video and state and action plots.
Robot support in the SDK is in beta. See the robots guide.

Two ways to control a robot

robot-libero is a quick way to try robot control from the web app. It serves LIBERO, a Franka Panda tabletop benchmark on MuJoCo, with one template per suite: libero_spatial, libero_object, libero_goal, libero_10, and libero_90. It serves both interfaces, so a policy and a tool-calling agent run the same tasks, and its traces use the episode replay. It is open source in the SDK’s environments/featured directory, so you can clone it and add tasks of your own.

Build a robot environment

Start from the Robotics template, for MuJoCo and LIBERO: on Environments, select New Environment, then See all templates. You implement a bridge around your simulator with reset, step, and get_observation, and describe the embodiment in its contract; the SDK handles the connection and sends telemetry to HUD. See the robots guide. Set the contract’s frame rate and per-dimension names. The trace viewer uses them to play video at real speed and to label each state and action plot.

Replay episodes

Environments built on the SDK’s robot support stream every control tick to the trace. Its Robotics view plays the episode back:
  • Cameras shows each camera as video, with play, pause, and playback speed.
  • Plots charts each action and observation dimension over time, named from the contract.
  • Split shows the cameras and the plots together.
Each template grades its episode, for example with the simulator’s own success check, so a trace’s reward says whether the episode succeeded. Run each task with several seeds or repetitions to measure a success rate. See traces.

Robots guide

Bridges, contracts, policies, and batching.

Robot benchmark

Run pi0.5 on LIBERO end to end.

Evaluations

Run tasks across models and read the results.