Skip to main content
For analysts, consultants, lawyers, scientists, and other domain experts who know what good work looks like. You write a task in the web app from three parts: the files the agent works from, a prompt, and a rubric. HUD runs models on it, and QA agents check that the rubric grades what you meant. New to HUD? The quickstart builds one on bundled records first.

​
What a task is made of

Tasks like this use the review_files template of the featured file-analysis environment. For example, a lease review: The agent reads the files in a sandboxed shell with no network access and writes a report. An LLM judge then checks the report against each criterion. The judge reads the prompt and the report, not your files, so a criterion has to say what the right answer is. A missing or empty report scores 0. The agent has Python’s standard library and no way to install more, so it works best with text: Markdown, CSV, JSON, or plain text. Export spreadsheets to CSV and documents to Markdown or text before you upload them. For other formats or tools, an engineer can build an environment from the Argument Hints template (hud init my-env --template argument-hints); its tasks get the same editors in the web app.

​
Build and check a task

1

Upload your files

Upload them on Data and keep them team-shared, so everyone who runs the task can use them. See upload files.
2

Create the task

Open a taskset, select Task, choose Public, and pick review_files under file-analysis.
3

Write the prompt, files, and rubric

Write the prompt, select Add files under attachments to pick your uploads, and add one criterion per requirement. To reuse a rubric, import it from a CSV file or a Markdown table. See write a rubric.
4

Run it

Run the task on a few models, with several repetitions each. Each trace shows what the agent read and wrote, and each criterion’s verdict with the judge’s reason. See evaluations.
5

Check it with QA

Run QA on the traces. Prompt Alignment finds criteria the prompt never asks for, False Positive finds passing answers that are wrong, and False Negative finds correct answers the rubric failed. Fix the task and run it again. See run QA.
6

Send it for review

Mark the task ready. A reviewer verifies it or asks for a revision. See task lifecycle.
Every save creates a new task version, and traces keep the version they ran, so you can improve a task without losing earlier results.

​
Write rubrics that hold up

  • Put the answer in the criterion. The judge never sees your files. “Identifies the duplicate EUR 900 payment on invoice NORTH-1048” can be checked; “Finds the duplicate payment” cannot.
  • Put one requirement in each criterion. Each criterion is met or not, so a combined one loses partial credit.
  • Say in the prompt what the rubric checks. A requirement only the rubric knows about grades the agent on something it was never asked to do. Prompt Alignment flags these.
  • Penalize with negative weights. Write the behavior itself, such as “Claims to have contacted the supplier”, and give it a negative weight.
  • Weight what matters. The reward is the weight of the met criteria divided by the total positive weight.
  • Expect variance. Run several repetitions before trusting a score, and read what QA says about the failures.

​
Work as a team

At scale, a few reviewers check the work of many authors:
  • Authors. Invite contract authors as External members, so they see only their own work and what is shared with them.
  • Projects. Organize the work in projects and grant access per project.
  • Review. Add review stages, such as an expert check, to a taskset, and reviewers work from one review queue.

Quickstart

Build and grade a first task on bundled records.

Create tasks

Every editor, versions, and bulk upload.

Fix what QA finds

Turn findings into task fixes.