HUD Documentation - Evaluations and RL Environments.

Its behavior is part of the SDK contract; its documentation path, naming, and organization may change without a major version. A grader is sometimes an environment of its own - a judge that needs services, golden data, or credentials the agent must never see. A verifier environment makes that structural: the actor task owns the environment the agent changes, and its verifier field names an agent-less task whose grade becomes the run’s final reward. The rollout engine (hud/eval/run.py) owns the phase boundary. The two tasks can share an environment or name different ones. Different environment names produce a two-substrate rollout: the engine exits the actor acquisition before opening the verifier’s. Whether that yields a genuinely separate substrate is the provider’s property - fresh-per-rollout providers (DockerRuntime, LocalRuntime) tear down the actor world and provision a new one, so grading material and agent state never coexist; a borrowed Runtime(url) or a Shared scope can hand the verifier the same live server.

Declaring the phases

1 · Two ordinary tasks

Both phases are ordinary environment templates and Task rows - there is no separate verifier API. Task.verifier accepts another Task; nested verifier tasks are rejected. When the verifier requires its own acquisition, the same provider receives the verifier row and can route its distinct env name and runtime_config to different infrastructure; on the same-environment reuse path below, the provider is called only once, with the actor row.

env.py · actor and judge

from hud import Environment

actor = Environment("workspace")
verifier = Environment("judge")

@actor.template(id="solve")
async def solve():
    answer = yield "Write the secret to the target system."
    yield {"score": 0.0, "answer": answer}

@verifier.template(id="verify")
async def verify(expected: str):
    actor_result = yield ""
    yield 1.0 if actor_result["answer"] == expected else 0.0

task = solve()
task.verifier = verify(expected="secret")

Provisioning order

The actor task is graded to complete its generator lifecycle. Its full result frame is sent to the verifier, whose evaluation replaces it as the run’s grade of record. An actor result must include the numeric score required of every task result plus whatever fields its verifier consumes. If actor grading fails, the engine supplies an error result containing the submitted answer so an authoritative verifier can still run. A verifier provisioning or grading failure leaves the run errored and ungraded. If both rows name the same environment and the verifier has no row-level runtime_config, the engine keeps the actor connection and substrate alive and starts the verifier task on that control channel immediately after the actor task completes. A different environment name or verifier runtime configuration forces actor cleanup followed by a fresh provider acquisition. HostedRuntime accepts a Task.verifier row and serializes it with the actor submission, including the verifier’s environment, arguments, and runtime_config. LocalRuntime, DockerRuntime, and custom providers execute verifier acquisitions through the client-side engine described above.

What runs where

Phase	Code that runs	Agent access	Grade status
Actor	Actor environment setup, the agent loop, actor task teardown	Actor capabilities and state	Provisional; retained only when no verifier exists
Verifier	Verifier setup and grading; no agent loop	None through HUD’s agent interface	Authoritative

The engine forwards the actor task’s result to the verifier unchanged. Files, processes, sockets, and environment memory do not cross between distinct substrates automatically - the actor result must carry any artifact reference, object-store key, or shared service endpoint the verifier needs.

Harbor verifier environments

Harbor declares a verifier environment with either form:

[verifier]
environment_mode = "separate"

[verifier.environment]
workdir = "/judge"
network_mode = "no-network"

The adapter requires tests/Dockerfile and packages its root filesystem as the build-only hud-verifier service. The generated actor row points to a verifier task in the same HUD environment, allowing one outer Compose runtime to stay alive across both phases. Inside that runtime, the phases remain isolated:

the actor works in the environment image’s Workspace sandbox;
actor sessions are terminated before artifact collection;
declared collect hooks run against main or named Compose services;
agent-produced state crosses into the verifier filesystem only through declared absolute artifact paths and /logs;
/tests/test.sh runs from the verifier image with its own user, workdir, environment, network mode, allowlist, and credentials directory; and
verifier output is read from /logs/verifier/reward.json or /logs/verifier/reward.txt.

RuntimeConfig.compose.service_access=True mounts a Docker socket in the generated main service.

Under local DockerRuntime, the mounted socket is the host’s own Docker daemon (remote Docker endpoints are rejected). Code in main can then control every container on the machine - grant this only to environment code you trust. Under hosted execution, the daemon is private to the rollout.

Start here

Guides

Reference

Advanced

Experimental

Cookbooks

Internals

More

Verifier environments

Declaring the phases

Provisioning order

What runs where

Harbor verifier environments

See also

Task API

Graders

Harbor interoperability

Compose environments

​Declaring the phases

​Provisioning order

​What runs where

​Harbor verifier environments

​See also

Task API

Graders

Harbor interoperability

Compose environments

Declaring the phases

Provisioning order

What runs where

Harbor verifier environments

See also