Robotics

For teams evaluating robot policies, such as vision-language-action models, or LLM agents that control a robot. A robot environment wraps a simulator and describes its cameras, state, and actions in a contract. Each episode becomes a trace you can replay: every camera as video, with the robot’s state and actions plotted over time.

​

Two ways to control a robot

The featured robot-gripper environment is a quick way to try tool-calling control from the web app: the agent drives a simulated gripper on a tabletop through tools, in the open-drawer, pick-object, move-object, and force-grasp templates, and each is graded from the simulator’s state. It serves the simulator as a tool server rather than through a contract, so its traces show the tool calls and the camera images the agent requests with render, not the episode replay.

​

Build a robot environment

Start from the Robotics template, for MuJoCo and LIBERO, or Worldsim Robotics, for Newton physics: on Environments, select New Environment, then See all templates. You implement a bridge around your simulator with reset, step, and get_observation, and describe the embodiment in its contract; the SDK handles the connection and sends telemetry to HUD. See the robots guide. Set the contract’s frame rate and per-dimension names. The trace viewer uses them to play video at real speed and to label each state and action plot.

​

Replay episodes

Environments built on the SDK’s robot support stream every control tick to the trace. Its Robotics view plays the episode back:

• Cameras shows each camera as video, with play, pause, and playback speed.
• Plots charts each action and observation dimension over time, named from the contract.
• Split shows the cameras and the plots together.

Each template grades its episode, for example with the simulator’s own success check, so a trace’s reward says whether the episode succeeded. Run each task with several seeds or repetitions to measure a success rate. See traces.