Skip to main content
Run QA agents wherever you read results: on one trace, on a task, or on a selection across a taskset. Turn on automatic QA, and every new trace in a taskset is reviewed when it completes. To run QA you need:
  • Something to review. Trace checks run on completed evaluation traces. Task checks run on a task’s current version.
  • Edit access to the trace or task.
  • Credit to start work. QA is refused when your team cannot start new work. See when work can start.

​
On a trace

Select Run QA in the trace header, or open the inspector’s QA view and select Run QA. Choose QA agents, then select Run. The QA agents that start selected are the ones without a passing or in-progress result on the trace. Once any check has a result, the header shows the trace’s QA mark instead, and its menu opens the QA view.

​
On a task

A task page runs both kinds of check:
  • Task checks. The Quality assurance section of the task’s Overview tab shows each task check’s latest result on the current version and has Run QA. On the other tabs, the task header has the same action until a check has a result.
  • Trace checks on the task’s traces. The Traces tab’s Run QA offers Latest, the newest completed trace on the current version, or All, every completed trace on the current version that you can edit. You can also select traces in the list and run QA on just those.

​
From a taskset

Select tasks in the task grid and select Run QA in the selection bar, or right-click a task and select Run QA…. The Run QA on selected traces dialog runs task checks on the selected tasks and trace checks on the traces you pick from them: The trace list starts from the grid’s current filters, leaves out invalidated traces, and lets you add or remove individual traces. The run button counts both groups, such as Run on 10 traces × 5 checks + 2 tasks × 2 checks.

​
Automatically

Open the taskset’s QA tab and turn on the trace checks to run under Automatic QA. From then on, each check runs once on every evaluation trace of the taskset’s tasks when the trace completes, whether it was started from the web app or from code. Turning a check on does not review traces that already exist; run those from the task grid. Automatic QA does not run on traces of a cancelled job, and does not run a check again on a trace that already has a result of it. The review is billed to whoever ran the trace. When their team cannot start new work, the trace is not reviewed, and you can run QA on it later.

​
Running a check again

Run QA on a trace, a task, or a job runs every QA agent you select on every subject, including subjects that already have a result of it, and each analysis is billed. The agents that start selected are the ones some subject still lacks a passing or in-progress result from. To review only what is new across a taskset, use the dialog’s Run new only mode. A new result never deletes an earlier one: earlier results stay in the subject’s history.

​
From code

The hud CLI waits for the checks to finish and prints each verdict. It skips a check on a subject whose current result of it is completed or still running, unless you pass --overwrite:
See the CLI reference. Over HTTP, POST /v2/qa/runs takes check_keys and either trace_ids or task_ids, up to 100 subjects per request:
The request returns at once. Poll GET /v2/qa/results until each result is completed, error, or cancelled. Over HTTP, overwrite defaults to true; with "overwrite": false, a check whose current result is queued, running, or completed is not run again. See QA checks in the API reference.

​
What QA costs

Each analysis is billed from your team’s credits like a short run: the model usage of the QA agent, shown as HUD QA on the Usage page, and the runtime of the environment it runs in. A check that the reward decides passes without an analysis and costs nothing. QA started from the web app runs in batches of 100 traces; if a batch is refused, for example because credit ran out, the message says how many traces had started.