Skip to main content
Pick a taskset and models, then launch the batch. An evaluation job is the batch; a trace is one model attempting one task within it. HUD gives every trace its own environment instance and records the full trajectory. The trace viewer replays it with the model conversation and the environment state aligned at every step.

​
Before you run

A platform evaluation needs:
  • a taskset with at least one runnable task
  • an environment with a successful build for each task
  • one or more models with an available checkpoint
  • any environment variables required by the selected task templates

​
Launch a job

Start from Run Taskset on a taskset page, or select Evaluation job from the add menu in the platform sidebar.
  1. Select the taskset.
  2. Select one or more models.
  3. Choose the group size, which controls repeated attempts for every task and model pair.
  4. Set the maximum number of agent steps per trace.
  5. Select Start Evaluation.
HUD creates the job and its traces immediately, then schedules each trace as capacity becomes available. A queued trace stays pending until a slot opens.

​
Statuses

The job status rolls up its traces. A job completes when every trace is terminal. If any trace errors, the job is marked error; otherwise a cancelled trace makes the job cancelled.

​
Read the job

The job page has three views: The overview shows reward distributions, task coverage, errors, and tool usage. Use the task grid to find weak tasks or inconsistent models, then open the corresponding traces. A trace can be marked invalid when it should not contribute to the evaluation, such as an infrastructure failure that produced a misleading reward. Invalidating a trace preserves the record but excludes it from valid reward aggregates.

​
Inspect a trace

The available inspector views depend on the environment: The final reward is only the verdict. Read the trajectory and grading result together to determine whether the agent solved the intended task or found a shortcut. For workspace reconstruction and redaction behavior, see file tracking.

​
Cancel, export, and share

  • Cancel stops pending and active traces, then cleans up their instances.
  • Export downloads the job’s hud eval configuration so the run can be reproduced or adapted.
  • Share grants scoped access or publishes a public job URL.
To launch the same workflow from the terminal or Python, see evaluating agents. To automate it over HTTP, see the rollout endpoints.