HUD Documentation - Evaluations and RL Environments.

Its behavior is part of the SDK contract; its documentation path, naming, and organization may change without a major version. The Harbor adapter (hud/integrations/harbor/) translates between Harbor task directories and HUD’s native Environment, Task, and Taskset contracts. adapt() packages Harbor tasks without building or publishing images. export() materializes HUD tasks as self-contained Harbor folders.

Adapt Harbor tasks

A Harbor source is either one task directory or a dataset directory containing task directories. Each task contains task.toml and instruction.md, plus an environment image or build recipe and a tests/ verifier. Multi-container sources use Harbor’s environment/docker-compose.yaml recipe name; other Compose-like filenames remain ordinary environment files.

from hud.integrations import harbor
from hud.eval import DockerRuntime

result = harbor.adapt("./terminal-bench")
job = await result.taskset.run(agent, runtime=DockerRuntime())

Each returned row carries its instruction and grading configuration in args, Harbor metadata in columns, resource requirements in runtime_config, and, when declared, a Task.verifier. The row’s Compose path points into a generated project under .hud-adapt/. Tasks that cannot be adapted appear in result.failures, with every detectable finding classified by a stable code. Other tasks in the dataset are still packaged. AdaptResult.taskset contains the successfully adapted rows. AdaptResult.failures is a tuple of AdaptFailure(task, path, findings) records, where each finding is an AdaptFinding(code, kind, message) and kind is contract or invalid. adapt() raises ValueError when the source contains no Harbor task directories.

.hud-adapt/<environment>/
├── build.sh
├── compose.yaml
├── env.py
├── tasks.json
└── compose-project/
    ├── build.sh
    ├── compose.json
    ├── environment/
    ├── hud/ or main/           # HUD build payload (depends on source shape)
    ├── tests/                 # inline-verifier task tests
    └── verifier/              # verifier-environment build context, when declared

The artifact is self-contained. Authored build contexts and Compose-relative paths are rebased beneath compose-project/; task instructions remain in tasks.json; inline tests remain outside the environment image and are mounted read-only for the selected task. Build-only base and verifier services stay in the Compose model with scale: 0 and feed the main build through named service: contexts. The full recipe is covered in Compose environments. Tasks with identical environment and phase configuration share one generated environment. A task with a verifier environment forms its own group because its verifier image and artifact contract are task-specific.

Run locally

DockerRuntime() reads the Compose project from each task row, runs the adjacent build.sh preparation hook, starts the main service and its sidecars, and applies the workspace security profile automatically. The generated task data is independent from the environment build. Editing an instruction, verifier timeout, or inline tests/ content updates tasks.json or the mounted test tree without changing the environment image identity.

Deploy and sync

hud deploy recognizes the generated root compose.yaml as the build recipe. Each generated environment is deployed separately, followed by the task rows that reference it:

hud deploy .hud-adapt/<environment>
hud sync tasks my-harbor-tasks .hud-adapt/<environment>/tasks.json

Hosted deploys build the project privately from the uploaded root; any registry image the project uses must be pullable by the hosted builder. The generated project can also be built and published on owned infrastructure; the commands and BuildKit context requirements are documented under Build on owned infrastructure.

Verifier environments

Harbor’s environment_mode = "separate" and [verifier.environment] declarations map to an authoritative verifier task. The adapter requires tests/Dockerfile, builds its filesystem as a build-only service, and preserves the agent/verifier phase boundary. Artifact collection, phase ordering, and runtime isolation are described in Verifier environments.

Supported Harbor surface

The adapter handles:

an environment/Dockerfile, an environment.docker_image, or a Compose project; when authored Compose omits main, the environment recipe supplies that agent service;
Compose sidecars with any number of declared TCP endpoints;
HTTP MCP servers (sse and streamable-http), healthchecks, network modes, allowlists, phase users, environment variables, and placement requirements;
inline verifier scripts and separate verifier Dockerfiles;
verifier collect hooks and declared artifact paths from main or a sidecar, including directory exclusions and host-side destinations.

Placement requirements are copied to the task that declares them: the main environment configures the actor task, and a separate verifier environment can configure its verifier task independently. The selected runtime decides whether it can provision those requirements. Unsupported declarations fail during adaptation. These include stdio MCP servers, skills directories, multi-step tasks, Compose variables not bound by the project .env or document defaults, include, and extends. A project also fails if its main service has neither an image nor a valid build recipe.

Export HUD tasks to Harbor

export(source, out_dir) accepts a Python task source or a JSON/JSONL taskset beside its environment source and Dockerfile:

from hud.integrations.harbor import export

created = await export("tasks.py", "harbor_tasks")

harbor_tasks/
└── <slug>/
    ├── task.toml
    ├── instruction.md
    ├── environment/
    │   ├── Dockerfile
    │   └── hud_entrypoint.sh
    └── tests/test.sh

The exported image serves the HUD control channel and parks the task after setup. Harbor runs the agent in that container, then tests/test.sh reconnects to grade the parked task and writes the reward. This mapping requires Harbor’s same-container verifier; exported task.toml files do not declare [verifier.environment]. Only environments exposing ssh or mcp capabilities are exportable. The task’s setup executes once while exporting instruction.md and once when the container starts the graded task, so setup must be deterministic for its arguments. Randomized state belongs in task arguments.

Validate the mapping

Run each Harbor task’s solution/solve.sh against the adapted artifact before evaluating a model. The reference solution should exercise the same environment and verifier path and produce the expected reward. The generated prompt, artifact declarations, and verifier should also be reviewed for answer leakage and reward-hacking paths; Designing tasks for signal covers that review.

API surface

Function	Contract
`harbor.adapt(path, *, hud_requirement="hud") -> AdaptResult`	Package valid tasks as generated Compose environments and native task rows, and return structured failures for the rest. Raises `ValueError` when no task directories exist.
`await harbor.export(source, out_dir, *, answer_file=..., timeout_sec=600) -> list[Path]`	Write HUD task rows as Harbor task directories.

Start here

Guides

Reference

Advanced

Experimental

Cookbooks

Internals

More

Harbor interoperability

Adapt Harbor tasks

Run locally

Deploy and sync

Verifier environments

Supported Harbor surface

Export HUD tasks to Harbor

Validate the mapping

API surface

Compose environments

Verifier environments

Runtime reference

Task reference

​Adapt Harbor tasks

​Run locally

​Deploy and sync

​Verifier environments

​Supported Harbor surface

​Export HUD tasks to Harbor

​Validate the mapping

​API surface

Compose environments

Verifier environments

Runtime reference

Task reference

Adapt Harbor tasks

Run locally

Deploy and sync

Verifier environments

Supported Harbor surface

Export HUD tasks to Harbor

Validate the mapping

API surface