HUD Documentation - Evaluations and RL Environments.

Its behavior is part of the SDK contract; its documentation path, naming, and organization may change without a major version. The Harbor adapter (hud/integrations/harbor/) translates between Harbor task directories and HUD’s native Environment, Task, and Taskset contracts. adapt() packages Harbor tasks without publishing images. export() materializes HUD tasks as self-contained Harbor folders. hud harbor import brings Harbor tasks and the runs Harbor recorded into a HUD taskset, to browse and compare them on the platform without running them again.

Adapt Harbor tasks

A Harbor source is either one task directory or a dataset directory containing task directories. Each task contains task.toml and instruction.md, plus an environment image or build recipe and a tests/ verifier. Multi-container sources use Harbor’s environment/docker-compose.yaml recipe name; other Compose-like filenames remain ordinary environment files.

from hud.integrations import harbor
from hud.eval import DockerRuntime

result = harbor.adapt("./terminal-bench")
job = await result.taskset.run(agent, runtime=DockerRuntime())

Each returned row carries its instruction and grading configuration in args, Harbor metadata in columns, resource requirements in runtime_config, and, when declared, a Task.verifier. The row’s Compose path points into a generated project under .hud-adapt/. Tasks that cannot be adapted appear in result.failures, with every detectable finding classified by a stable code. Other tasks in the dataset are still packaged. AdaptResult.taskset contains the successfully adapted rows. AdaptResult.failures is a tuple of AdaptFailure(task, path, findings) records, where each finding is an AdaptFinding(code, kind, message) and kind is contract or invalid. adapt() raises ValueError when the source contains no Harbor task directories.

.hud-adapt/<environment>/
├── compose.yaml               # the Compose project the task rows and `hud deploy` run
├── env.py                     # deploy identity
├── tasks.json
└── compose-project/
    ├── environment/           # authored build context
    ├── main/                  # HUD build payload
    ├── tests/                 # inline-verifier task tests
    └── verifier/              # verifier-environment build context, when declared

Adaptation requires a Docker-compatible builder when an authored image must be built or pulled. Images build for the builder’s own platform, and a verifier environment builds for the same platform as the task’s image. Tasks authored for linux/amd64 often pin dependencies that publish no wheels elsewhere, so adapt them with DOCKER_DEFAULT_PLATFORM=linux/amd64 on an arm64 host. The artifact is self-contained. Authored build contexts and Compose-relative paths are rebased beneath compose-project/; task instructions remain in tasks.json; inline tests remain outside the environment image and are mounted read-only for the selected task. Build-only base and verifier services stay in the Compose model with scale: 0 and feed the main build through named service: contexts. The full recipe is covered in Compose environments. Tasks with identical environment and phase configuration share one generated environment. A task with a verifier environment forms its own group because its verifier image and artifact contract are task-specific. Pass name to name the environment of a single task folder.

Run locally

DockerRuntime() reads the Compose project from each task row, starts the main service and its sidecars, and applies the workspace security profile automatically. The generated task data is independent from the environment build. Editing an instruction, verifier timeout, or inline tests/ content updates tasks.json or the mounted test tree without changing the environment image identity.

Deploy and sync

Save the Compose configuration as runtime.json in the generated project root:

{"compose": {"document": "compose.yaml", "root": "."}}

Each generated environment is deployed separately, followed by its task rows:

hud deploy .hud-adapt/<environment>/env.py --runtime-config .hud-adapt/<environment>/runtime.json
hud sync tasks my-harbor-tasks .hud-adapt/<environment>/tasks.json

Hosted deploys build the project privately from the uploaded root; any registry image the project uses must be pullable by the hosted builder. The generated project can also be built with docker compose build and published on owned infrastructure; see Build on owned infrastructure.

Verifier environments

Harbor’s environment_mode = "separate" and [verifier.environment] declarations map to an authoritative verifier task. The adapter requires tests/Dockerfile, builds its filesystem as a build-only service, and preserves the agent/verifier phase boundary. Artifact collection, phase ordering, and runtime isolation are described in Verifier environments.

Supported Harbor surface

The adapter handles:

an environment/Dockerfile, an environment.docker_image, or a Compose project; when authored Compose omits main, the environment recipe supplies that agent service;
the Compose variables Harbor defines (CONTEXT_DIR, MAIN_IMAGE_NAME, CPUS, MEMORY, PREBUILT_IMAGE_NAME) and literal [environment.env] values, which take precedence over the project .env;
Compose sidecars, including ones that expose no TCP ports;
HTTP MCP servers (sse and streamable-http), healthchecks, network modes, allowlists, phase users, environment variables, and placement requirements;
inline verifier scripts and separate verifier Dockerfiles;
verifier collect hooks and declared artifact paths from main or a sidecar, including directory exclusions and host-side destinations.

Placement requirements are copied to the task that declares them: the main environment configures the actor task, and a separate verifier environment can configure its verifier task independently. The selected runtime decides whether it can provision those requirements. stdio MCP servers and skills directories configure Harbor’s own agents; the adapter accepts them and does not expose them to HUD agents. Unsupported declarations fail during adaptation with a finding of kind contract: multi-step tasks (harbor.unsupported.multi_step), Compose variables only the host environment could supply (harbor.unsupported.host_compose_variable), and Compose include and extends (harbor.unsupported.compose_include_extends). Malformed tasks fail with findings of kind invalid, such as a project whose main service has neither an image nor a valid build recipe.

Import into HUD

hud harbor import copies Harbor task folders and job runs into a taskset, the way harbor publish and harbor upload send them to Harbor Hub:

Harbor	HUD
`harbor publish ./tasks`	`hud harbor import ./tasks --taskset <name>`
`harbor upload ./jobs/<job>`	`hud harbor import ./jobs/<job> --taskset <name>`

One import can take both:

hud harbor import ./terminal-bench ./jobs/2026-10-01__12-00 --taskset tb2

The command zips the folders it is given (or takes one prepared zip), checks the zip against the 1 GiB limit, uploads it to your team’s data store, and imports it into the taskset. It follows the import until it ends and prints a report of every task, job, and trial, with the reason for anything skipped or failed. --no-wait returns once the import starts; hud harbor imports lists imports, hud harbor report <import-id> prints a report later, and hud harbor cancel <import-id> stops a running import. The hud harbor reference covers taskset selection and every option. hud harbor import exits 1 when the import did not complete (stopped, failed, or cancelled). Skipped and failed items are listed in the report and do not fail the command; --json returns the import and the full report for scripts. Harbor import describes the zip layout, what an import creates, the report, reruns, and limits. The web app’s Import from Harbor takes the same zip, and the taskset’s Imports tab lists the same imports. Imported tasks are for browsing and comparing runs; HUD cannot run them. To run Harbor tasks on HUD, adapt and deploy them.

Export HUD tasks to Harbor

export(source, out_dir) accepts a Python task source or a JSON/JSONL taskset beside its environment source and Dockerfile:

from hud.integrations.harbor import export

created = await export("tasks.py", "harbor_tasks")

harbor_tasks/
└── <slug>/
    ├── task.toml
    ├── instruction.md
    ├── environment/
    │   ├── Dockerfile
    │   └── hud_entrypoint.sh
    └── tests/test.sh

The exported image serves the HUD control channel and parks the task after setup. Harbor runs the agent in that container, then tests/test.sh reconnects to grade the parked task and writes the reward. This mapping requires Harbor’s same-container verifier; exported task.toml files do not declare [verifier.environment]. Only environments exposing ssh or mcp capabilities are exportable. The task’s setup executes once while exporting instruction.md and once when the container starts the graded task, so setup must be deterministic for its arguments. Randomized state belongs in task arguments.

Validate the mapping

Run each Harbor task’s solution/solve.sh against the adapted artifact before evaluating a model. The reference solution should exercise the same environment and verifier path and produce the expected reward. The generated prompt, artifact declarations, and verifier should also be reviewed for answer leakage and reward-hacking paths; Designing tasks for signal covers that review.

API surface

Function	Contract
`harbor.adapt(path, *, hud_requirement=f"hud=={hud.__version__}") -> AdaptResult`	Package valid tasks as generated Compose environments and native task rows, and return structured failures for the rest. `hud_requirement` installs the controller each image serves (`hud.integrations.harbor.env`); pass a wheel path when adapting from an unreleased checkout. Raises `ValueError` when no task directories exist.
`await harbor.export(source, out_dir, *, answer_file=..., timeout_sec=600) -> list[Path]`	Write HUD task rows as Harbor task directories.

Start here

Guides

Reference

Advanced

Experimental

Cookbooks

Internals

More

Harbor interoperability

Adapt Harbor tasks

Run locally

Deploy and sync

Verifier environments

Supported Harbor surface

Import into HUD

Export HUD tasks to Harbor

Validate the mapping

API surface

Compose environments

Verifier environments

Runtime reference

Task reference

​Adapt Harbor tasks

​Run locally

​Deploy and sync

​Verifier environments

​Supported Harbor surface

​Import into HUD

​Export HUD tasks to Harbor

​Validate the mapping

​API surface

Compose environments

Verifier environments

Runtime reference

Task reference

Adapt Harbor tasks

Run locally

Deploy and sync

Verifier environments

Supported Harbor surface

Import into HUD

Export HUD tasks to Harbor

Validate the mapping

API surface