uv tool install hud --python 3.12.
Authentication
Get your API key from HUD settings, then save it locally:~/.hud/.env. For CI, set HUD_API_KEY in the process environment.
Flags and output
List, get, create, and status commands accept--json. JSON is written to stdout; progress and warnings go to stderr. With --json, a failure is one JSON object on stdout: {"error": "not_found", "message": "...", "input": {...}, "suggestion": "..."}. List commands also accept --quiet / -q for one identifier per line.
Every command that changes platform state takes
--dry-run, which previews the action without sending or writing anything, and asks for confirmation before it acts. Pass --yes / -y to skip the confirmation; without it, the command exits 2 in a non-interactive terminal. For deployment, select local secrets with --env-file or skip them with --no-env.
If the HUD API deprecates a route that hud calls, hud prints a warning to stderr with the route’s sunset date and replacement. Upgrade hud when you see one. SDK code receives it as hud.utils.exceptions.HudDeprecationWarning, a FutureWarning.
Resource commands use noun-verb grammar (hud jobs list, hud models fork, hud task start). Bare hud jobs and hud qa list. hud cancel is a hidden, deprecated alias for hud jobs cancel.
Build & iterate
hud init
Start a new environment package in a fresh <name> directory. HUD provides example environments from the SDK
repository; a named non-interactive run uses coding by default. In a source checkout, hud init
copies the matching environments/ directory. An installed release
fetches the matching SDK tag and extracts that same directory, so the example always matches the
installed SDK rather than main.
hud serve
Serve an environment’s control channel locally (tcp JSON-RPC).
hud deploy
Build and publish to HUD infra in one step. The environment’s name comes
from the Environment(...) declaration in code; deploying the same name again
rebuilds that environment. Use --name to select the registry identity.
Separately configure the image’s entrypoint to serve that environment.
--env / --env-file upsert onto the registry’s existing secrets. Omitting them on a rebuild keeps whatever is already stored. A first deploy that finds a local .env prompts once for that invocation (or needs --env-file / --no-env when stdin is not a TTY).
Evaluate
hud eval
Run and grade tasks from a Python file, a directory, JSON/JSONL, or a platform taskset.
Python sources can export environments directly or through their tasks. Import
all templates and hooks needed by those tasks. Environment names must be unique
within the source being loaded; re-exporting the same environment is fine.
To choose another source, use
hud serve path with --runtime tcp://host:port, or
SubprocessRuntime(path) in Python.
Each process runs from its source directory and shares host files and ports.
Use separate workspaces and ports for concurrent runs. Tasks configured with an
image or Compose project run in Docker.
Use a HUD API key for gateway inference and platform reporting, or a direct
provider key with HUD telemetry uploads disabled.
Platform tasksets run remotely by default: hud eval "My Tasks" claude. To run
them locally, pass --runtime local; each task and any verifier running in its
own environment must include an image or Compose configuration.
Single-task runs show step-by-step progress (step number + tool calls). Multi-task batches are silent unless --verbose is passed.
hud eval exits 1 when no run ended with a valid grade (each hit a grading
error, a grader exception, or a failure before grading), and 0 otherwise,
whatever the rewards. With --json, the result document is printed either way:
error_count and each run’s is_error mark the errored runs, and mean_reward
excludes them. A run that failed mid-run, such as on a model provider error or a
timeout, is still graded when it can be; its trace_error says what failed.
.hud_eval.toml
Defaults for the directory you run hud eval from. Precedence is CLI arguments, then this file, then built-in defaults. hud init drops a fully commented template into new projects; hud eval never writes one itself, and a missing file just means defaults.
${VAR} expands from the environment and HUD settings (${HUD_API_KEY}, ${api_key}); an unset variable is an error, as are unknown [eval] keys, unknown sections, and a file that does not parse. Agent-specific keys are validated by the selected agent’s configuration model. --config key=value layers onto the selected agent’s section. Use runtime = "hosted" for hosted execution.
Run a packaged image
hud task start / hud task grade resolve authored task slugs and arguments from --source (default .). Without --url, HUD serves the bound environment’s template file in a subprocess. With both --source and --url, it runs the resolved task against the already-served environment. Pass --url without --source to call a raw template id with explicit --args. hud task list always reads from source.
Platform
hud project
Show and choose the HUD Project where new environments and tasksets are
created.
Project selection
hud deploy, hud sync tasks, and hud harbor import (for a taskset it creates) resolve their
destination in this order:
--project does not change .hud/config.json. Use hud project use <name-or-id> when later deploys
and task syncs from the directory should use that Project. Set a fallback for unconfigured
directories with hud set HUD_DEFAULT_PROJECT=<name-or-id>.
.hud/config.json is CLI-owned workspace state (gitignored). It stores the Project, environment, and
taskset ids for this working copy, plus the API origin, user, and team that created the link.
A worktree or copy of the directory keeps the file; credentials that do not match the stored
scope fail instead of using the ids. Existing resources remain in their original Project.
Legacy .hud/deploy.json is left unread. A .hud/config.json written by an earlier release (camelCase
registryId/tasksetId, no credential scope) still links the directory and is rewritten in the current
schema, with the scope recorded, by the next deploy, project, or sync command. A config written under
different credentials fails instead of being imported; switch credentials or delete the file. The first
successful task upload records its canonical ID; later explicit targets only replace it with --link.
Credentials remain in ~/.hud/.env, cache in ~/.hud/.cache, and authored source and
.hud_eval.toml remain in the project. Missing evaluation configuration uses defaults without
creating a file; malformed or invalid configuration fails instead of falling back to defaults.
hud sync
Publish task definitions or link a local directory to a deployed environment:
hud sync tasks --project is a one-command override, like hud deploy --project. Existing resources
remain in their original Project. The first upload from a directory records the taskset in
.hud/config.json; later explicit targets only replace it with --link. See
Projects for the conceptual overview.
External benchmark formats can be adapted into runnable Tasksets through the
experimental Harbor integration.
hud harbor
Import Harbor task folders and the job runs Harbor recorded into a taskset:
hud harbor import zips the folders, uploads the zip, imports it into the taskset, and prints what became of each task, job, and trial. --taskset takes a name or ID; a name
is normalized as HUD normalizes taskset names (TB2 Sample is tb2-sample), and a name no
taskset has creates one, while an ID has to name an existing taskset. Without it, the directory’s
taskset is used, then a taskset named after the first folder. A created taskset goes to the Project
chosen as in
Project selection. --link saves the taskset as the directory’s default.
Exit code 1 means the import did not complete; tasks or trials it skipped are listed in the report
without failing the command. See Harbor import for the zip layout, what
an import creates, the report, and limits.
QA checks
trace checks judge one completed evaluation
trace, and task checks judge one task’s current version.
hud qa list lists HUD’s QA checks: key, level, title, and the question each
one answers. --level trace or --level task lists only the checks at that
level. hud qa is an alias for list.
hud qa run runs one check, or several separated by commas, on up to 100
traces or tasks as a single run, then waits until every check has finished on
every one. The checks decide what the IDs are: trace-level checks take
completed trace IDs, task-level checks take task IDs. Trace-level and
task-level checks can’t be mixed in one run. Where a check’s current result is
completed or still running, it is kept rather than run again; pass
--overwrite to run it anyway. Pass --no-wait to return after launch, --yes
to skip the confirmation, or --json for machine-readable output. Exit code 1 means a check did not pass
or the SDK could not complete the command.
hud qa results prints the result in effect for every check on the given
traces, or on tasks with --level task: its verdict (passed, failed, or
unknown), summary, and numbered findings. A task’s result in effect is the
latest one on its current version. --history lists every result, newest
first, including results on a task’s earlier versions. A check passed on reward
alone shows source skipped; a reviewer’s verdict shows source manual with
its note. --json prints the raw results.
Inspect
hud jobs
hud trace
Inspect a single rollout. Reads from the local telemetry JSONL file first
(HUD_TELEMETRY_LOCAL_DIR or ~/.hud/spans when uploads are disabled) and falls
back to the platform API.
HUD_TELEMETRY_ENABLED=0 to keep spans on disk at ~/.hud/spans, or set
HUD_TELEMETRY_LOCAL_DIR to choose a directory. hud trace reads those files
without an API call.