QA agents

A reward is only useful if the task and its grader measure what the task asks for. QA agents are reviewers that HUD runs for you. Each one answers one question about a completed trace or a task, and returns a verdict with findings that point at the steps, files, or grading involved. They catch what reading rewards alone misses: graders that pass broken work, prompts that never state what is graded, and agents that game the evaluation. Run them on every task you write or import, before its rewards reach a benchmark or a training run. A review is billed like a short run, and checks that the reward already decides cost nothing. See run QA for every way to start one.

​

Trace checks

Trace checks review one completed evaluation trace.

​

Task checks

Task checks review a task’s current version together with the traces run on it. The QA Agents page in the platform sidebar shows each check with a worked example. The REST API and the hud qa CLI call them QA checks and identify each by key: reward_hacking, false_positive, false_negative, prompt_alignment, and failure_analysis for traces, and task_reward_exploit and task_failure_analysis for tasks.

​

Checks the reward decides

Some trace checks are settled by the reward alone. A trace with zero reward cannot be a false positive, and a trace with full reward has no failure to explain and cannot be a false negative. In those cases the check passes at once, without an analysis and at no cost, and its result says why. Every other check, including both task checks, always runs an analysis.

​

What a QA agent reads

Before a QA agent starts, HUD copies what it reviews into a private workspace. For a trace, that includes:

• the task prompt, arguments, reward, and status
• every message, tool call, and tool result, plus a readable summary of the steps
• the evaluation result: the reward, subscores, and each grader’s verdict
• the template’s setup and grading code, when available
• environment and worker logs
• before-and-after diffs of the files the agent changed
• screenshots from computer-use and browser steps
• the environment’s source code, only when your team owns the environment

A task check reads the task’s current version, the traces run on that version, and the source of the build the task runs on. The QA agent explores this copy with shell tools and can view each screenshot. It cannot reach the live environment, and nothing it does changes the trace or the task. Failure Analysis can ask a second agent to verify each failure it reports before it records the result.

​

Reviewers, not graders

QA agents never change a reward. A failed review flags a reward you should not trust, and you decide what to do with it: fix the task and run it again, or invalidate the trace so it stays out of leaderboards and reward summaries.