Skip to main content
In about ten minutes you will create a taskset, add a task to it, run the task on a model, read the graded trace, and review it with QA. The task uses file-analysis, a featured public environment: the agent gets a folder of files and a shell, writes a report, and an LLM judge grades the report against your rubric. Prefer code? The SDK quickstart installs the HUD SDK, writes a task in Python, and runs it against a model.

​
Before you begin

  • An account. Sign in at hud.ai.
  • Credits. Runs and QA reviews are paid from your team’s credit balance, and a new team starts with none. Open Settings → Billing and select Add credits, or Upgrade to Pro for 50 credits each month. See billing and usage.

​
1. Create a taskset

Open Tasksets and select New Taskset. Name it quickstart and select Create. The taskset opens on its Tasks tab.

​
2. Create a task

  1. Select Task in the taskset header. The Create task dialog lists task templates.
  2. Select the Public filter and search for reconcile.
  3. Select reconcile_supplier_payments under file-analysis, then Create task.
The task opens on Parameters. This template ships with a set of supplier records: four invoices, the bank payments, signed vendor terms, and a receiving log. You only write the prompt and the rubric.

​
3. Write the prompt and rubric

In prompt, paste:
Leave attachments empty; the template stages its bundled records for the agent. To work on your own files instead, see expert tasks without code. In criteria, select Add criterion for each row below and fill in the requirement and weight: Each criterion is judged met or not met. The reward is the weight of the met criteria divided by the total positive weight, so this rubric scores out of 7, and the last row subtracts 1 when the report breaks the rule. To paste a longer rubric, select Import and choose a .csv or .md table with requirement and weight columns. Select Save. Every save creates a new task version.

​
4. Run the task

  1. Back on the taskset’s Tasks tab, select the task’s checkbox, then Run task in the selection bar.
  2. In Run tasks, pick a model and keep Repetitions at 1.
  3. Select the run button. It shows the number of traces it will start.
The evaluation job opens in a new tab. Its trace is done once the agent has written its report and the judge has graded it.

​
5. Read the trace

Open the trace from the job’s Traces tab. The conversation shows every command the agent ran, and Workspace shows the files it read and the report it wrote. The conversation ends with the evaluation result: the reward and each criterion with its weight, whether it was met, and the judge’s reason. See traces.

​
6. Review it with QA

Open the trace’s QA view and select Run QA. Keep the QA agents that are selected and select Run. Each QA agent returns a verdict with findings. For example, Prompt Alignment reports if a criterion checks something the prompt never asked for, and False Positive reports if a passing report is wrong in a way the rubric missed. QA agents read the task, the trajectory, and the grading together, so they catch rubric problems that are easy to miss by reading the reward alone. See QA agents.

​
Next steps

Use your own files

Upload files to Data and build tasks on them with the review_files template.

Scale up the taskset

Duplicate tasks, upload task files, and run the whole set across models.

Bring in authors and reviewers

Invite experts, add review stages, and verify tasks before they ship.