Skip to main content
This guide outlines how to run an evaluation: pointing an agent at tasks, choosing where each rollout runs, and reading the reward. The hud eval command is the central tool for evals: it drives every eval and works the same whether your environment is deployed or a local env.py. Every rollout traces to your platform. There are two workflows, and they differ only in where the environment lives:

​
Running a deployed environment

A deployed environment has been built and published to the platform with hud deploy. It runs entirely on hosted infra, so there’s no local process to manage - you start a run from the terminal or from the platform, and both trace to the same place. From the terminal, pass the taskset name or id to hud eval. The tasks are fetched from the platform and each rollout runs remotely on hosted infra.
From the platform, open hud.ai, pick the environment, choose a taskset and a model, and launch - no CLI required. See evaluations on the platform for that flow.

​
Choosing an agent

The agent name (claude, openai, gemini) selects a provider harness. It calls the provider with its own key when one is set, and otherwise the HUD gateway with HUD_API_KEY; --gateway forces the gateway. The claude_cli and codex_cli harnesses run an installed CLI inside the environment; --gateway routes their model calls through HUD too. Switching models is a single flag, and hud models list shows every model the gateway knows.
For a custom loop - a fine-tuned model, a framework you already use - see bring your own harness.

​
Reading traces and results

Each rollout is a trace: a replayable timeline of everything the agent did and the reward it earned. Traces are grouped into a job and shown on the platform. You can also read them from the terminal:

​
Running a local environment

A local environment is an env.py on your disk - the usual case while you’re still developing it. hud eval is the way to run it; alternatively you can write your own run script that calls Taskset.run the same way when you want programmatic control. Both take a runtime, the one argument that decides where each rollout runs. The environment definition never changes - only the runtime does.
Where the environment runs is separate from how the agent reaches its model. A local run needs no HUD_API_KEY, but the claude, openai and gemini agents need a provider key or HUD_API_KEY. With HUD_API_KEY set, local runs also trace to the platform. To check a task without a model, run it with an agent that returns a fixed answer.

​
With hud eval

hud eval starts the environment for you, so a local run needs no hud serve or Docker. Point it at your task source and set --runtime to choose where the environment runs.
By default each rollout runs in a child process from your env.py. The --runtime flag moves that placement elsewhere without touching the environment. See hud eval for the full flag set.

terminal

​
With a run script

When you want programmatic control - looping over agents, feeding a training pipeline, routing tasks to different infra - call Taskset.run directly and hand it a runtime object. This is the same eval hud eval runs, written out in Python. create_agent always uses the gateway and needs HUD_API_KEY; to call a provider directly, construct its agent.
run.py
The runtime is where you set placement. Swap it for another and the env runs somewhere else, with no change to env.py or the tasks: The runtime reference covers each constructor and how to bring your own.

​
Serving an environment directly

hud eval starts and stops the env subprocess for you. hud serve instead exposes the same control channel as a standalone, long-lived process - useful for talking to a running env from a script, testing a packaged image by hand, or driving hud task start and hud task grade yourself.
Attach to it from a script with Runtime("tcp://localhost:8765"), or from hud eval with --runtime tcp://localhost:8765. The control channel describes every message the channel speaks. With runs in hand, turn the reward spread into model updates - covered in training agents.