HUD Documentation - Evaluations and RL Environments.

An environment is where the agent acts. A task is the work you measure there: the unit of evaluation, one prompt paired with one graded outcome that resolves to a single number, the reward. That one number is what an evaluation reports and what training learns from. A task has two representations. You author it as an async generator decorated with @env.template, the shape that prompts the agent and scores what happens. Calling that template mints a Task: a plain data row (an env name, a template id, bound args) that travels over the wire and runs anywhere. The agent loop and grading happen in the gap between the generator’s two yields, and a call to run() executes one row and returns a Job.

Concept	What it is
Template	The async generator you author with `@env.template`. Callable; calling it mints a task without running anything.
Task	A filled-in template: one set of arguments bound into a single runnable row.
Taskset	A named, ordered collection of tasks - a table of those rows.
Job	The receipt a run produces: the graded runs and their mean reward.

Defining a task

A task is an async generator with exactly two yields. The first yield is the prompt; the generator pauses while the agent works; the agent’s answer comes back into the generator; the second yield is the reward (0.0-1.0).

tasks.py

from hud import Environment

env = Environment(name="letter-count")

@env.template()
async def count_letter(word: str = "strawberry", letter: str = "r"):
    answer = yield f"How many '{letter}'s are in '{word}'?"   # 1st yield: the prompt
    yield 1.0 if answer == str(word.count(letter)) else 0.0   # 2nd yield: the reward

This shape is deliberate. Everything the agent does - every step, tool call, and observation - happens in the gap between the two yields, so you describe a task as plain Python: ask on one line, score on the next. The agent loop in the middle is HUD’s job, not yours. The second yield can be a bare float, an EvaluationResult, or a dict with a numeric score; the env layer normalizes any of them to the reward.

Template

@env.template() registers the generator as a template: a callable that mints a concrete Task without running anything.

task = count_letter(word="raspberry")   # a Task row, not yet run

It takes four optional arguments:

Parameter	Type	Description
`id`	`str \| None`	Task id registered on the environment. Defaults to the function name.
`description`	`str`	Human-readable summary surfaced in the manifest. Defaults to `""`.
`input`	`Any`	Type declaring the template’s arguments; surfaced in the manifest as a JSON schema.
`returns`	`Any`	Type declaring the agent’s answer; surfaced in the manifest and parses the answer into a typed `Answer[T]`.

Declare returns=T and the answer arrives as a parsed Answer[T] (.content parsed, .raw the original string); without it, answer is the raw string the agent submitted. One template spans a space of tasks. Call it with different arguments and a single function becomes a whole dataset, with no separate artifact per task:

tasks.py

tasks = [count_letter(word=w) for w in ("strawberry", "raspberry", "blueberry")]

Parameterize across difficulties, inputs, or seeds and one definition becomes an entire benchmark.

Task

A Task is a Pydantic model - one portable, validated row of data. It holds no live environment: env is a name, the join key between the row and whatever brings that environment up at run time. So a task runs anywhere without an env object in-process - the prompt and reward arrive over the wire from the substrate that placement brings up.

Field	Type	Description
`env`	`str`	Name of the environment the row belongs to.
`id`	`str`	Task id registered on the environment.
`args`	`dict`	Bound arguments (what the template was called with).
`slug`	`str \| None`	Stable id for sync, filtering, and lookup.
`columns`	`dict \| None`	Metadata surfaced as filter/leaderboard facets.
`validation`	`list[dict] \| None`	Platform/sync metadata.
`agent_config`	`dict \| None`	Per-task agent overrides (e.g. `{"max_steps": 50, "timeout_seconds": 900}`).
`runtime_config`	`RuntimeConfig \| None`	Per-row launch requirements; the runtime applies what it supports.
`verifier`	`Task \| None`	Agent-less task whose evaluation replaces the actor task’s grade. Nested verifiers are rejected.

When you don’t have the template in hand (data pipelines, generated rows), build the model directly - the model is the row, so task.model_dump() and Task.model_validate(data) are the whole codec:

from hud import Task

task = Task(env="letter-count", id="count_letter", args={"word": "strawberry"}, slug="count-straw")

Grading

The second yield is the reward. Return a float inline, or reach for the comparison helpers, async graders, and composed grades documented in graders. For how to design a reward that separates good work from bad and resists hacking, see designing tasks. When authoritative grading belongs in another environment, set Task.verifier to a second task row. Its declaration, placement order, and phase boundary are documented in Verifier environments.

Tasksets

A Taskset is a named collection of task rows. Build one in code, or load it from a source:

from hud import Taskset

# in code - the authoring case
ts = Taskset("letters", [count_letter(word="strawberry"), count_letter(word="raspberry")])

# from a Python source (.py file or directory) - scans it for Task / Taskset objects
ts = Taskset.from_file("tasks.py")

# from a data file (.json / .jsonl) - portable rows, no source needed
ts = Taskset.from_file("tasks.jsonl")

# from the platform - by taskset name or id (uses HUD_API_KEY)
ts = Taskset.from_api("SheetBench-50")

Write rows back out with ts.to_file("tasks.json") (or .jsonl). Tasksets are also ordered collections:

Operation	Description
`len(ts)` / `iter(ts)`	Count / iterate tasks in order.
`ts["slug"]`	Look up one task by slug.
`ts.filter(slugs)` / `ts.exclude(slugs)`	Keep / drop matching slugs (returns a new taskset).

Running

taskset.run(agent, ...) executes every task and returns a Job. task.run(...) is the same call over a taskset of one, with identical semantics.

from hud import LocalRuntime

job = await ts.run(agent, runtime=LocalRuntime("env.py"), group=8, max_concurrent=10)

Parameter	Type	Description
`agent`	`Agent`	The shared, stateless agent driven across every rollout.
`runtime`	`Provider \| HostedRuntime \| None`	Where each rollout runs; omit it and placement is inferred (local source serves itself, else the HUD tunnel by env name). See runtime.
`group`	`int \| None`	Repeats each task N times for GRPO.
`max_concurrent`	`int \| None`	Caps how many rollouts run in parallel.
`job`	`Job \| None`	An open `Job` to accumulate into; defaults to a fresh job per call.
`rollout_timeout`	`float \| None`	Optional execution deadline (seconds) through grading in every placement; omitted means no SDK-wide deadline.

agent_config.timeout_seconds bounds only the agent phase. If it expires, the run is marked as timed out but grading still runs against any completed work. rollout_timeout instead bounds submission, queueing, startup, the agent, and grading together. It is not a wall-clock bound on Task.run or Taskset.run returning: after expiry, local execution may wait up to two seconds for cancellation and still reports the trace, and Taskset.run can spend up to 120 seconds flushing telemetry. Runtime cleanup may finish in the background. When both are set, the agent timeout must be less than the rollout timeout and any explicit actor run_timeout_s. Limits imposed by the selected runtime still apply when rollout_timeout is omitted. Grader-specific limits, such as BashGrader.timeout_seconds, remain local to that grader. A crashed rollout comes back as a failed Run inside the job rather than raising, so one bad rollout never collapses a batch. For the end-to-end workflow see running an eval.

Syncing

Sync publishes a locally-authored taskset to hud.ai so you can run it there, compare models on it, and browse its traces; local runs never need it. The hud sync tasks <name> command uploads a taskset and only what changed, a workflow covered in creating an environment. In code, diff(local, remote) returns a SyncPlan describing the comparison: To create the taskset outside the team default Project, select a directory Project first or pass an explicit destination:

hud project use browser-evals
hud sync tasks my-taskset

# Or select it on the sync command
hud sync tasks my-taskset --project browser-evals

See Projects for placement precedence and existing-resource behavior.

from hud.eval.sync import diff

plan = diff(Taskset.from_file("tasks.py"), Taskset.from_api("SheetBench-50"))
print(plan.summary())

Field	Description
`to_create`	Local tasks not present remotely.
`to_update`	Local tasks whose content differs from remote.
`unchanged`	Local tasks that match remote.
`remote_only`	Remote tasks with no local counterpart.

Once a reward separates good work from bad, designing tasks covers the difficulty and spread that make a taskset worth training on, and graders the scoring that produces each reward.

Start here

Guides

Reference

Advanced

Experimental

Cookbooks

Internals

More

Tasks & Tasksets

Defining a task

Template

Task

Grading

Tasksets

Running

Syncing

​Defining a task

​Template

​Task

​Grading

​Tasksets

​Running

​Syncing

Defining a task

Template

Task

Grading

Tasksets

Running

Syncing