HUD Documentation - Evaluations and RL Environments.

Install the CLI with uv tool install hud --python 3.12.

Authentication

Get your API key from HUD settings, then save it locally:

hud set HUD_API_KEY=your-key-here

This stores the key in ~/.hud/.env. For CI, set HUD_API_KEY in the process environment.

Flags and output

List, get, create, and status commands accept --json. JSON is written to stdout; progress and warnings go to stderr. With --json, a failure is one JSON object on stdout: {"error": "not_found", "message": "...", "input": {...}, "suggestion": "..."}. List commands also accept --quiet / -q for one identifier per line.

Exit code	Meaning
`0`	Success
`1`	Failure
`2`	Usage error (bad arguments, or a prompt needed in a non-interactive terminal)

Use --dry-run to preview supported commands without performing the action or prompting. Pass --yes / -y to skip confirmation where supported; a required prompt exits 2 in a non-interactive terminal. For deployment, select local secrets with --env-file or skip them with --no-env. Resource commands use noun-verb grammar (hud jobs list, hud models fork, hud task start). Bare hud jobs and hud qa list. hud cancel is a hidden, deprecated alias for hud jobs cancel.

Build & iterate

`hud init`

Start a new environment package in a fresh <name> directory. HUD provides example environments from the SDK repository; a named non-interactive run uses coding by default. In a source checkout, hud init copies the matching environments/ directory. An installed release fetches the matching SDK tag and extracts that same directory, so the example always matches the installed SDK rather than main.

hud init                              # choose an example interactively
hud init my-env                       # coding → ./my-env
hud init my-env --template cua        # computer use → ./my-env
hud init my-env --template blank      # minimal scaffold → ./my-env
hud init my-env --dir envs            # coding → ./envs/my-env

Option	Description
`--template`, `-t`	Example environment to use: `coding`, `cua`, `blank`, or `argument-hints`. Omit for the interactive picker; a named non-interactive run uses `coding`. `--preset`/`-p` are aliases.
`--dir`, `-d`	Parent directory (default `.`).
`--force`, `-f`	Overwrite existing files.
`--json`	Structured result on stdout.
`--dry-run`	Print the planned scaffold without writing files.

`hud serve`

Serve an environment’s control channel locally (tcp JSON-RPC).

hud serve               # auto-detect env.py
hud serve env:env       # explicit module:attribute
hud serve env.py -p 9000

Option	Default	Description
`--port`, `-p`	`8765`	Port to serve on.
`--host`	`127.0.0.1`	Interface to bind (use `0.0.0.0` inside containers).
`--verbose`, `-v`	-	Detailed logs.

`hud deploy`

Build and publish to HUD infra in one step. The environment’s name comes from the Environment(...) declaration in code; deploying the same name again rebuilds that environment. Use --name to select the registry identity. Separately configure the image’s entrypoint to serve that environment.

hud deploy
hud deploy --name judge
hud deploy --project <name-or-id>

Option	Description
`--name`	Which `Environment(...)` name to register when the directory’s `.py` files (searched recursively, skipping `.venv`, `node_modules`, caches) declare more than one.
`--env`, `-e`	Env var `KEY=VALUE` (repeatable). Upserts registry secrets.
`--env-file`	Upsert registry secrets from this file. Keys omitted from the file are kept.
`--no-env`	Skip seeding from a local `.env` on first deploy. Rebuilds already skip `.env` unless `--env-file` is passed.
`--build-arg`	Docker build arg `KEY=VALUE` (repeatable). Omit on a rebuild to keep the registry’s stored args.
`--secret`	Docker build secret for this build only, e.g. `--secret id=GITHUB_TOKEN,env=GITHUB_TOKEN` (`src=` also works).
`--no-cache`	Disable the remote build cache.
`--project`	One-command Project override by name or ID. Falls back to the directory Project, `HUD_DEFAULT_PROJECT`, then the team default.
`--runtime`	Persist a registry default runtime for tasks that do not specify one: `hud` or `modal`.
`--runtime-config`	JSON `RuntimeConfig` to persist as the registry default (compose, resources, limits). Must set at least one field.
`--json`	Structured result on stdout.
`--dry-run`	Print the deploy plan without uploading or prompting. A first deploy with an unresolved `.env` reports `dotenv_pending`.

--env / --env-file upsert onto the registry’s existing secrets. Omitting them on a rebuild keeps whatever is already stored. A first deploy that finds a local .env prompts once for that invocation (or needs --env-file / --no-env when stdin is not a TTY).

Evaluate

`hud eval`

Run and grade tasks from a Python file, a directory, JSON/JSONL, or a platform taskset.

# Split layout (hud init): templates in env.py, task rows in tasks.py
hud eval tasks.py claude
hud eval tasks.py claude --full --group 3

# Single-file layout: env + tasks list in one file
hud eval env.py claude
hud eval env.py claude --model claude-haiku-4-5   # cheaper model for fast iteration
hud eval env.py claude --max-steps 30
hud eval env.py claude --all                      # every task, not just the first
hud eval env.py claude --full                     # every task, auto-respond, 100 steps

Local Python environments run in a new process for each rollout:

Task source	Environment served
Python tasks created by `@env.template`	The original task file or directory
JSON/JSONL or Python rows that only name an environment	The task source’s directory

Python sources can export environments directly or through their tasks. Import all templates and hooks needed by those tasks. Environment names must be unique within the source being loaded; re-exporting the same environment is fine. To choose another source, use hud serve path with --runtime tcp://host:port, or SubprocessRuntime(path) in Python. Each process runs from its source directory and shares host files and ports. Use separate workspaces and ports for concurrent runs. Tasks configured with an image or Compose project run in Docker. Use a HUD API key for gateway inference and platform reporting, or a direct provider key with HUD telemetry uploads disabled. Platform tasksets run remotely by default: hud eval "My Tasks" claude. To run them locally, pass --runtime local; each task and any verifier running in its own environment must include an image or Compose configuration. Single-task runs show step-by-step progress (step number + tool calls). Multi-task batches are silent unless --verbose is passed.

Option	Description
`--full`	Run the whole dataset (`--all --auto-respond --max-steps 100`).
`--all`	Run every task instead of just the first.
`--model`, `-m`	Model id.
`--gateway`, `-g`	Force LLM calls through the HUD gateway. Agents use a provider key when one is set and the gateway otherwise; pass this to use the gateway even when a provider key is present.
`--group` (alias `--group-size`)	Runs per task - a group of repeats whose reward spread you can inspect.
`--max-concurrent`	Cap parallel rollouts.
`--max-steps`	Cap steps per task (default 10).
`--task-ids`	Comma-separated slugs or 0-based indices.
`--config`, `-c`	Agent config `key=value` (repeatable).
`--verbose`, `-v`	Show agent logs (step progress, tool calls) for batch runs too.
`--very-verbose`, `-vv`	Debug-level logs.
`--runtime`	`local`: a Python process or Docker container per rollout. `hud`: local agent, deployed environment. `hosted`: agent and environment on HUD. `docker`, `modal`, `daytona`: run using the task’s container configuration. `tcp://host:port`: connect to a running environment. Defaults to `local` for files and `hosted` for platform tasksets.
`--remote`	Alias for `--runtime hosted`.
`--yes`, `-y`	Skip confirmation prompt (required when stdin is not a TTY).
`--json`	Structured results on stdout.
`--dry-run`	Print the resolved eval plan without running.

`.hud_eval.toml`

Defaults for the directory you run hud eval from. Precedence is CLI arguments, then this file, then built-in defaults. hud init drops a fully commented template into new projects; hud eval never writes one itself, and a missing file just means defaults.

[eval]
source = "tasks.py"           # tasks file, or a platform taskset name / id
agent = "claude"              # claude, claude_cli, codex_cli, openai, gemini, openai_compatible
model = "claude-sonnet-4-6"
all = false                   # run every task instead of just the first
task_ids = ["fix_bug", "0"]   # slugs or 0-based indices
max_steps = 10
max_concurrent = 30
group_size = 1                # runs per task
auto_respond = false
gateway = false               # set true to force LLM calls through the HUD gateway
runtime = "local"             # local, hud, hosted, docker, modal, daytona, or tcp://host:port
verbose = false
very_verbose = false

[claude]                      # one section per agent; keys go to that agent's config
max_tokens = 16384

[openai_compatible]
model = "my-model"
base_url = "http://localhost:8000/v1"
api_key = "${MY_PROVIDER_API_KEY}"

Every key is optional. ${VAR} expands from the environment and HUD settings (${HUD_API_KEY}, ${api_key}); an unset variable is an error, as are unknown [eval] keys, unknown sections, and a file that does not parse. Agent-specific keys are validated by the selected agent’s configuration model. --config key=value layers onto the selected agent’s section. Use runtime = "hosted" for hosted execution.

Run a packaged image

hud task start / hud task grade resolve authored task slugs and arguments from --source (default .). Without --url, HUD serves the bound environment’s template file in a subprocess. With both --source and --url, it runs the resolved task against the already-served environment. Pass --url without --source to call a raw template id with explicit --args. hud task list always reads from source.

hud task list
hud task start fix_bug --source tasks.py --url tcp://127.0.0.1:8765
hud task grade fix_bug --source tasks.py --url tcp://127.0.0.1:8765 --answer "..."

Command	Key options
`hud task start <task>`	`--source`/`-s`, `--args` (JSON), `--url`/`-u`, `--json`
`hud task grade <task>`	`--answer`, `--answer-file` (`-` reads stdin), `--source`, `--args`, `--url`, `--json`
`hud task list`	`--source`/`-s`, `--json`, `--quiet`

Platform

`hud project`

Show and choose the HUD Project where new environments and tasksets are created.

hud project                              # resolved Project for the current directory
hud project list                         # visible Projects and access
hud project create browser-evals         # create and use a Project (team admins)
hud project create browser-evals --no-use
hud project use browser-evals           # name or ID; save project_id to .hud/config.json
hud project use browser-evals -C ./my-env

Command or option	Description
`hud project [--directory PATH]`	Show where new resources from a directory will be created. `-C` is an alias.
`hud project list`	List visible Projects, marking the team default and read-only entries.
`hud project create <name>`	Create and pin a Project. Accepts `--description`, `--directory`/`-C`, and `--no-use`.
`hud project use <name-or-id>`	Pin a directory to an existing writable Project. Names are normalized (`My Project` = `my-project`). Accepts `--directory`/`-C`.

Project selection

hud deploy and hud sync tasks resolve their destination in this order:

Priority	Source	Scope
1	`--project <name-or-id>`	Current command
2	`.hud/config.json` → `project_id`	Current directory
3	`HUD_DEFAULT_PROJECT`	Machine-wide fallback
4	Team default Project	Zero-configuration fallback

--project does not change .hud/config.json. Use hud project use <name-or-id> when later deploys and task syncs from the directory should use that Project. Set a fallback for unconfigured directories with hud set HUD_DEFAULT_PROJECT=<name-or-id>. .hud/config.json is CLI-owned workspace state (gitignored). It stores the Project, environment, and taskset ids for this working copy, plus the API origin, user, and team that created the link. A worktree or copy of the directory keeps the file; credentials that do not match the stored scope fail instead of using the ids. Existing resources remain in their original Project. Legacy .hud/deploy.json is left unread. A .hud/config.json written by an earlier release (camelCase registryId/tasksetId, no credential scope) still links the directory and is rewritten in the current schema, with the scope recorded, by the next deploy, project, or sync command. A config written under different credentials fails instead of being imported; switch credentials or delete the file. The first successful task upload records its canonical ID; later explicit targets only replace it with --link. Credentials remain in ~/.hud/.env, cache in ~/.hud/.cache, and authored source and .hud_eval.toml remain in the project. Missing evaluation configuration uses defaults without creating a file; malformed or invalid configuration fails instead of falling back to defaults.

`hud sync`

Publish task definitions or link a local directory to a deployed environment:

hud sync tasks my-taskset                          # publish using resolved Project
hud sync tasks my-taskset --project browser-evals  # override the Project (name or ID)
hud sync tasks my-taskset --link                   # also make my-taskset this directory's default
hud sync env                                       # pick a deployed environment to link
hud sync env my-env                                # link by environment name or ID

hud sync tasks --project is a one-command override, like hud deploy --project. Existing resources remain in their original Project. The first upload from a directory records the taskset in .hud/config.json; later explicit targets only replace it with --link. See Projects for the conceptual overview. External benchmark formats can be adapted into runnable Tasksets through the experimental Harbor integration.

QA checks

hud qa list
hud qa
hud qa run failure_analysis <trace-id>
hud qa run reward_hacking,false_positive <trace-id> <trace-id>
hud qa results <trace-id>

hud qa list lists HUD’s QA checks: title, key, and the question each one answers. hud qa is an alias for list. hud qa run runs one check, or several separated by commas, on up to 100 completed traces as a single run, then waits until every check has finished on every trace. Where a check’s current result on a trace is completed or still running, it is kept rather than run again; pass --overwrite to run it anyway. Pass --no-wait to return after launch, or --json for machine-readable output. Exit code 1 means a check did not pass or the SDK could not complete the command. hud qa results prints the result in effect for every check on the given traces: its verdict (passed, failed, or unknown), summary, and numbered findings. A check passed on reward alone shows source skipped; a reviewer’s verdict shows source manual with its note. --json prints the raw results.

Inspect

`hud jobs`

hud jobs list                 # recent jobs
hud jobs list --json          # raw JSON array on stdout
hud jobs list --quiet         # job ids, one per line
hud jobs get <job-id>         # traces for one job
hud jobs cancel <job-id> --yes
hud jobs                      # alias for list

Option	Description
`--json`	Structured stdout.
`--quiet`, `-q`	Bare ids, one per line.
`--limit`, `-n`	Max rows (default 20).
`--yes`, `-y`	Skip cancel confirmation.
`--dry-run`	Print the cancel plan without calling the API.

`hud trace`

Inspect a single rollout. Reads from the local telemetry JSONL file first (HUD_TELEMETRY_LOCAL_DIR or ~/.hud/spans when uploads are disabled) and falls back to the platform API.

hud trace get <trace-id>          # rendered view: agent turns, tool calls, tool results
hud trace get <trace-id> --json   # raw event list

Set HUD_TELEMETRY_ENABLED=0 to keep spans on disk at ~/.hud/spans, or set HUD_TELEMETRY_LOCAL_DIR to choose a directory. hud trace reads those files without an API call.

Other commands

Command	Description
`hud set KEY=VALUE`	Persist credentials/vars to `~/.hud/.env`.
`hud models list`	List gateway models (`--json`, `--quiet`).
`hud models fork <model> --name <slug>`	Fork a trainable model (`--json`, `--dry-run`, `--if-not-exists`).
`hud models checkpoints <model>`	List a model’s checkpoint tree.
`hud models head <model> [--set <checkpoint-id>]`	Show - or set - a model’s active checkpoint.
`hud jobs cancel`	Cancel a running job (`--yes`, `--dry-run`, `--json`).
`hud version`	Show the CLI version (`--json`).

Start here

Guides

Reference

Advanced

Experimental

Cookbooks

Internals

More

CLI

Authentication

Flags and output

Build & iterate

`hud init`

`hud serve`

`hud deploy`

Evaluate

`hud eval`

`.hud_eval.toml`

Run a packaged image

Platform

`hud project`

Project selection

`hud sync`

QA checks

Inspect

`hud jobs`

`hud trace`

Other commands

See also

Quickstart

Run & deploy

​Authentication

​Flags and output

​Build & iterate

​hud init

​hud serve

​hud deploy

​Evaluate

​hud eval

​.hud_eval.toml

​Run a packaged image

​Platform

​hud project

​Project selection

​hud sync

​QA checks

​Inspect

​hud jobs

​hud trace

​Other commands

​See also

Quickstart

Run & deploy

Authentication

Flags and output

Build & iterate

`hud init`

`hud serve`

`hud deploy`

Evaluate

`hud eval`

`.hud_eval.toml`

Run a packaged image

Platform

`hud project`

Project selection

`hud sync`

QA checks

Inspect

`hud jobs`

`hud trace`

Other commands

See also