HUD Documentation - Evaluations and RL Environments.

A task is one concrete set of arguments for a task template. The template, written once by an environment author, decides what the agent can do and how it is graded. The task supplies whatever the template asks for, such as a prompt, input files, and a rubric. Most tasks need no code.

Ways to add tasks

Method	Where	Best for
Create task	Taskset header, Task	One task from any template you can use.
Create Task on a template	Environment Templates tab	Starting from an environment you just deployed.
Duplicate	Task selection bar	Variants of an existing task, in the same or another taskset.
Upload Tasks	Taskset More actions menu	Many tasks at once from JSON.
`hud sync tasks`	CLI	Tasks defined in Python. See tasks and tasksets.
Import from Harbor	Tasksets page or taskset header	Existing Harbor task folders. See Harbor import.

Featured environments

Featured environments are public environments HUD maintains and keeps deployed, so you can create tasks on them without deploying anything:

Environment	What the agent does	Task templates
file-analysis	Reads files in a shell and writes a report, graded against your rubric by an LLM judge.	`review_files` (your uploads) `reconcile_supplier_payments` (bundled supplier records)
browser-use	Researches a rental listing in a real Chromium browser; the answer must cite the screenshots it relied on.	`rental-kitchen-chairs`
flask-coding	Modifies a repository; hidden tests grade the change.	`coding-task`
libbet	Plays a Game Boy game turn by turn through button tools, scored by the distinct screens it reaches.	`play-game`
robot-gripper	Controls a simulated gripper on a tabletop scene through tools.	`open-drawer` `pick-object` `move-object` `force-grasp`

Choose Public when you create a task and search for the template. The quickstart walks through reconcile_supplier_payments end to end. Browse them under Environments with the Featured filter; each one’s page has View example run, which opens a sample run HUD has published. Read their source in the SDK repository.

Create a task from a template

Open a taskset and select Task in its header.
In Create task, choose where to look for templates:
- This taskset: templates its tasks already use.
- This project: templates of every environment in the current project.
- Public: templates of public environments, including the featured environments.
Search, select a template, and select Create task.

The task opens on Parameters. Your filter choice is remembered for next time.

Fill in parameters

Parameters shows one editor per template argument. The template decides which editor each argument gets:

Argument kind	Editor
Prompt	A text editor for the instructions the agent receives.
Data files	File tiles. Add files or Choose file picks from Data or uploads a new file.
Grading criteria	The rubric editor, with Add criterion and Import.
Anything else	A field for the argument’s type, or a JSON editor for structured values.

Edits are pending until you save. Save shows how many arguments changed. Revert discards one argument’s edit, and Revert all discards them all.

Write a rubric

A rubric is a list of criteria, each a requirement and a weight. Each criterion is judged on its own: met counts as 1 times its weight, not met as 0. A negative weight marks behavior to penalize; write its requirement as the behavior itself (“Claims to have moved funds”), and it subtracts when the behavior happens. With HUD’s LLMJudgeGrader, which the featured file-analysis templates use, the reward is the weight of the met criteria divided by the total positive weight, clamped to the range 0 to 1. The trace shows each criterion’s verdict and the judge’s reason. To paste an existing rubric, select Import and drop a .csv file or a Markdown table, up to 2 MB. The file needs a requirement column and can have a weight column, plus an id column when the template gives criteria IDs:

requirement,weight
"Identifies the duplicate EUR 900 payment on NORTH-1048",2
"Claims to have moved funds or contacted a supplier",-1

Versions

Every save creates a new task version. Traces keep the version they ran, so earlier results stay interpretable after the task changes, and the task’s QA checks start over on the new version.

Read the task page

Tab	What it shows
Overview	Description, properties, status and review actions, the latest trace, QA, and activity.
Parameters	The arguments of the current version.
Files	The source of the environment build this version runs.
Workspace	The files the agent starts with. Shown when the environment declares a workspace.
Traces	Every run of the task, across versions. Select traces here to run QA on them.

The Properties panel shows the task’s Status, Owner, the taskset’s Reviewers, Template, Version, Average reward, and ID, plus any custom taskset columns. Change the owner or description from the Overview; set status from the task lifecycle actions. Activity is the task’s timeline: comments, new versions, argument changes, and status changes. Anyone with access to the taskset can comment and reply, one level deep; authors can edit and delete their own comments.

Upload tasks as JSON

From a taskset’s More actions menu, select Upload Tasks and paste a JSON array. Each task names its template, its arguments, and its environment:

[
  {
    "scenario": "reconcile_supplier_payments",
    "env": "file-analysis",
    "args": {
      "prompt": "Reconcile the September supplier invoices...",
      "attachments": [],
      "criteria": [{ "requirement": "Flags the duplicate payment", "weight": 2 }]
    }
  }
]

The dialog finds the environment from the first task’s env, so one upload targets one environment. It checks every task’s arguments against its template and previews each task before Upload. Each task is named after the first 80 characters of its prompt field, or task-1, task-2, and so on without one. A task with the name of an existing task in the taskset updates that task with a new version instead of adding a duplicate. Export JSON in the same menu downloads every task’s latest version, with its name, environment, template, and arguments. To create or update tasks from code with names you choose, use hud sync tasks or the tasks endpoints.

Make a template editable in the web app

Environment authors choose the editors above by annotating template arguments. The annotations are exported by hud.environment:

from pydantic import BaseModel
from hud.environment import DataFileRef, DataFilesArg, GradingArg, PromptArg


class Criterion(BaseModel):
    requirement: str
    weight: float = 1.0


@env.template(id="review_files", description="Answer a prompt about uploaded files")
async def review_files(
    prompt: PromptArg,
    attachments: DataFilesArg[DataFileRef],
    criteria: GradingArg[Criterion],
):
    ...

Annotation	Editor
`PromptArg`	Prompt editor.
`DataFileArg[DataFileRef]`	One-file picker.
`DataFilesArg[DataFileRef]`	Multi-file picker.
`GradingArg[T]`	Rubric editor. `T` needs a string `requirement` and a numeric `weight`; add a string `id` for named criteria.

The template still owns staging and grading: download picked files in setup (see use a file in a task) and grade against the criteria with a grader such as LLMJudgeGrader. The featured file-analysis environment is a complete example.

Get started

Create tasks

Evaluate and train

QA agents

Use cases

Collaborate

Manage your team

API

Create tasks

Ways to add tasks

Featured environments

Create a task from a template

Fill in parameters

Write a rubric

Versions

Read the task page

Upload tasks as JSON

Make a template editable in the web app

​Ways to add tasks

​Featured environments

​Create a task from a template

​Fill in parameters

​Write a rubric

​Versions

​Read the task page

​Upload tasks as JSON

​Make a template editable in the web app

Ways to add tasks

Featured environments

Create a task from a template

Fill in parameters

Write a rubric

Versions

Read the task page

Upload tasks as JSON

Make a template editable in the web app