HUD Documentation - Evaluations and RL Environments.

Install the CLI with uv tool install hud --python 3.12.

Authentication

Get your API key from HUD settings, then save it locally:

hud set HUD_API_KEY=your-key-here

This stores the key in ~/.hud/.env. For CI, set HUD_API_KEY in the process environment.

Build & iterate

`hud init`

Start a new environment package in a fresh <name> directory. HUD provides example environments from the SDK repository; a named non-interactive run uses coding by default. In a source checkout, hud init copies the matching environments/ directory. An installed release fetches the matching SDK tag and extracts that same directory, so the example always matches the installed SDK rather than main.

hud init                              # choose an example interactively
hud init my-env                       # coding → ./my-env
hud init my-env --template cua        # computer use → ./my-env
hud init my-env --template blank      # minimal scaffold → ./my-env
hud init my-env --dir envs            # coding → ./envs/my-env

Option	Description
`--template`, `-t`	Example environment to use: `coding`, `cua`, or `blank`. Omit for the interactive picker; a named non-interactive run uses `coding`. `--preset`/`-p` are aliases.
`--dir`, `-d`	Parent directory (default `.`).
`--force`, `-f`	Overwrite existing files.

`hud serve`

Serve an environment’s control channel locally (tcp JSON-RPC).

hud serve               # auto-detect env.py
hud serve env:env       # explicit module:attribute
hud serve env.py -p 9000

Option	Default	Description
`--port`, `-p`	`8765`	Port to serve on.
`--host`	`127.0.0.1`	Interface to bind (use `0.0.0.0` inside containers).
`--verbose`, `-v`	-	Detailed logs.

`hud deploy`

Build and publish to HUD infra in one step. The environment’s name comes from the Environment(...) declaration in code; deploying the same name again rebuilds that environment.

hud deploy
hud deploy --project browser-evals

Option	Description
`--all`, `-a`	Deploy all environments in the directory.
`--env`, `-e`	Env var `KEY=VALUE` (repeatable).
`--env-file`	Path to a `.env` file.
`--project`	One-command Project override by name or ID. Falls back to the directory Project, `HUD_DEFAULT_PROJECT`, then the team default.

Evaluate

`hud eval`

The primary local iteration loop: run an agent over a task source (.py, directory, or JSON/JSONL), grade the result, and print the reward. Each rollout gets a fresh subprocess for the env - no shared state between tasks.

# Split layout (hud init): templates in env.py, task rows in tasks.py
hud eval tasks.py claude
hud eval tasks.py claude --full --group 3

# Single-file layout: env + tasks list in one file
hud eval env.py claude
hud eval env.py claude --model claude-haiku-4-5   # cheaper model for fast iteration
hud eval env.py claude --max-steps 30
hud eval env.py claude --all                      # every task, not just the first
hud eval env.py claude --full                     # every task, auto-respond, 100 steps

hud eval loads tasks from the path you pass. In a split project, point it at tasks.py (or . to scan the directory). It spawns env.py for the control channel automatically - you don’t pass both files.

What you don’t need for a local run:

A HUD API key - local evals don’t hit the platform
hud serve running - hud eval spawns the env subprocess for you
Docker - unless your env explicitly uses DockerRuntime
An SSH connection - the gateway timeout only applies when env.workspace() is declared

For a platform taskset, pass its name or id directly: hud eval "My Tasks" claude. The tasks are fetched from the platform and the rollouts run remotely by default, since the env source is not on disk. Single-task runs show step-by-step progress (step number + tool calls). Multi-task batches are silent unless --verbose is passed.

Option	Description
`--full`	Run the whole dataset (`--all --auto-respond --max-steps 100`).
`--all`	Run every task instead of just the first.
`--model`, `-m`	Model id.
`--gateway`, `-g`	Force LLM calls through the HUD gateway. Implied when only `HUD_API_KEY` is set (no provider key); pass it to force the gateway when a provider key is also present.
`--group` (alias `--group-size`)	Runs per task - a group of repeats whose reward spread you can inspect.
`--max-concurrent`	Cap parallel rollouts.
`--max-steps`	Cap steps per task (default 10).
`--task-ids`	Comma-separated slugs or 0-based indices.
`--config`, `-c`	Agent config `key=value` (repeatable).
`--verbose`, `-v`	Show agent logs (step progress, tool calls) for batch runs too.
`--very-verbose`, `-vv`	Debug-level logs.
`--runtime`	Placement: `local` (a child process per rollout, serving the env source — `SubprocessRuntime`), `hud` (HUD runtime tunnel), or `tcp://host:port`. Defaults to `local` for a tasks file; platform tasksets default to remote hosted execution.
`--remote`	Run the whole rollout remotely on the HUD platform.
`--yes`, `-y`	Skip confirmation prompt.

Run a packaged image

hud task start / hud task grade resolve authored task slugs and arguments from --source. Without --url, HUD also spawns that source locally. With --url, it runs the resolved task against the already-served environment instead. Omit --source to call a raw template id with explicit --args. hud task list always reads from source (default .) - it doesn’t attach.

hud task list
hud task start fix_bug --source tasks.py --url tcp://127.0.0.1:8765
hud task grade fix_bug --source tasks.py --url tcp://127.0.0.1:8765 --answer "..."

Command	Key options
`hud task start <task>`	`--source`/`-s`, `--args` (JSON), `--url`/`-u`, `--out`/`-o`
`hud task grade <task>`	`--answer`, `--answer-file`, `--source`, `--args`, `--url`, `--out`
`hud task list`	`--source`/`-s`

Platform

`hud project`

Show and choose the HUD Project where new environments and tasksets are created.

hud project                              # resolved Project for the current directory
hud project list                         # visible Projects and access
hud project create browser-evals         # create and use a Project (team admins)
hud project create browser-evals --no-use
hud project use browser-evals            # save projectId to .hud/config.json
hud project use browser-evals -C ./my-env

Command or option	Description
`hud project [--directory PATH]`	Show where new resources from a directory will be created. `-C` is an alias.
`hud project list`	List visible Projects, marking the team default and read-only entries.
`hud project create <name>`	Create and pin a Project. Accepts `--description`, `--directory`/`-C`, and `--no-use`.
`hud project use <name-or-id>`	Pin a directory to an existing writable Project. Accepts `--directory`/`-C`.

Project selection

hud deploy and hud sync tasks resolve their destination in this order:

Priority	Source	Scope
1	`--project <name-or-id>`	Current command
2	`.hud/config.json` → `projectId`	Current directory
3	`HUD_DEFAULT_PROJECT`	Machine-wide fallback
4	Team default Project	Zero-configuration fallback

--project does not change .hud/config.json. Use hud project use <name-or-id> when later deploys and task syncs from the directory should use that Project. Set a fallback for unconfigured directories with hud set HUD_DEFAULT_PROJECT=<name-or-id>. hud project list marks the team default and read-only Projects. Pinning a directory, deploying, or syncing to a Project requires create access. Only team admins can create Projects. An existing environment or taskset stays in its original Project. Selecting a different Project for an existing resource fails rather than moving it; use its current Project or create a separately named resource in the new one.

`hud sync`

Publish task definitions or link a local directory to a deployed environment:

hud sync tasks my-taskset                          # publish using resolved Project
hud sync tasks my-taskset --project browser-evals # override by Project name or ID
hud sync env                                       # link environment metadata

hud sync tasks --project is a one-command override, like hud deploy --project. Existing resources remain in their original Project. See Projects for the conceptual overview. External benchmark formats can be adapted into runnable Tasksets through the experimental Harbor integration.

Platform QA

hud qa
hud qa run <agent-id> <trace-id>
hud qa results <trace-id>
hud qa results <trace-id> --rollout

hud qa lists trace agents (name and UUID), including public ones. Copy an id into hud qa run. Reuses matching evidence by default and waits for every requested trace. Pass --overwrite to create a fresh attempt, --no-wait to return after launch, or --json for machine-readable output. Exit code 1 means a completed QA result failed or the SDK could not complete the command. hud qa results prints the analysis verdict (passed or failed), not the job status. Failure Analysis shows a cause and numbered findings when present. Boolean agents (false negative, false positive, reward hacking, prompt alignment) show yes/no and a summary, without a findings list. Pass --rollout for the sanitized analysis trajectory. --json still prints the raw result list and does not fetch the trajectory.

Inspect

`hud jobs [<id>]`

Without an id, lists your most recent jobs. With an id, lists every trace in that job.

hud jobs               # recent jobs - id, name, taskset, status, created
hud jobs <job-id>      # traces for one job - trace id, status, reward, error
hud jobs --json        # raw JSON
hud jobs -n 50         # show up to 50 rows (default 20)

`hud trace <trace_id>`

Inspect a single rollout. Reads from the local telemetry JSONL file first (HUD_TELEMETRY_LOCAL_DIR/<trace_id>.jsonl) and falls back to the platform API.

hud trace <trace-id>          # rendered view: agent turns, tool calls, tool results
hud trace <trace-id> --json   # raw event list

Set HUD_TELEMETRY_LOCAL_DIR in your environment to write spans to disk during a rollout so hud trace can read them offline without an API call.

Other commands

Command	Description
`hud set KEY=VALUE`	Persist credentials/vars to `~/.hud/.env`.
`hud models list`	List gateway models.
`hud models fork <model> --name <slug>`	Fork a trainable model from an existing one.
`hud models checkpoints <model>`	List a model’s checkpoint tree.
`hud models head <model> [--set <checkpoint-id>]`	Show - or set (rollback/select) - a model’s active checkpoint.
`hud cancel`	Cancel a running job.
`hud version`	Show the CLI version.

Start here

Guides

Reference

Advanced

Experimental

Cookbooks

Internals

More

CLI

Authentication

Build & iterate

`hud init`

`hud serve`

`hud deploy`

Evaluate

`hud eval`

Run a packaged image

Platform

`hud project`

Project selection

`hud sync`

Platform QA

Inspect

`hud jobs [<id>]`

`hud trace <trace_id>`

Other commands

See also

Quickstart

Run & deploy

​Authentication

​Build & iterate

​hud init

​hud serve

​hud deploy

​Evaluate

​hud eval

​Run a packaged image

​Platform

​hud project

​Project selection

​hud sync

​Platform QA

​Inspect

​hud jobs [<id>]

​hud trace <trace_id>

​Other commands

​See also

Quickstart

Run & deploy

Authentication

Build & iterate

`hud init`

`hud serve`

`hud deploy`

Evaluate

`hud eval`

Run a packaged image

Platform

`hud project`

Project selection

`hud sync`

Platform QA

Inspect

`hud jobs [<id>]`

`hud trace <trace_id>`

Other commands

See also