> ## Documentation Index
> Fetch the complete documentation index at: https://docs.hud.ai/llms.txt
> Use this file to discover all available pages before exploring further.

# Graders

> Native HUD graders, comparison helpers, and the native grade combiner for scoring agent runs, with patterns for partial credit and multi-step tasks.

Graders turn an agent's answer into a reward. HUD ships reusable ones so you don't hand-build common scoring logic. Yield the result (a `float` or an `EvaluationResult`) as the task's second yield.

```python theme={"dark"}
from hud.graders import (
    BashGrader, LLMJudgeGrader, Grader,
    SubScore, EvaluationResult,
    combine, combine_any, combine_all,
    exact_match, contains, contains_any, contains_all,
    numeric_match, f1_score, normalize,
)
```

## What grading scores

Grading decides *what* the reward measures. Two framings cover most tasks:

* **Grade the answer** - compute a float from what the agent said.
* **Grade the world** - score the state the agent left behind: tests passing, a file written, a
  service responding. Because the agent acts on a real system through its
  [capabilities](/v6/reference/capabilities), this is **outcome verification**: the rigor of a test
  suite with no fixed protocol the agent has to follow.

## Approaches

Three approaches cover grading, in increasing power. Each produces the `float` or
[`EvaluationResult`](#subscore-and-evaluationresult) yielded as the task's second yield.

| Approach         | When to reach for it                                                     | Example                                                                 |
| ---------------- | ------------------------------------------------------------------------ | ----------------------------------------------------------------------- |
| **Plain Python** | A reward you compute from the answer, optionally via a comparison helper | `yield numeric_match(answer, 3)`                                        |
| **Async grader** | Score a command's exit code or an LLM's judgment of the answer           | `yield (await BashGrader.grade(weight=1.0, command="pytest -q")).value` |
| **Composed**     | Weight several graders into one reward, each subscore kept in the trace  | `yield await combine(BashGrader.grade(...), LLMJudgeGrader.grade(...))` |

## Comparison helpers

Each returns a `float` (`0.0`-`1.0`) you can yield directly or wrap in a `SubScore`.

| Helper          | Signature                                                   | Returns                                   |
| --------------- | ----------------------------------------------------------- | ----------------------------------------- |
| `exact_match`   | `exact_match(answer, expected, *, normalize_text=True)`     | `1.0` if equal (normalized)               |
| `contains`      | `contains(answer, substring, *, case_sensitive=False)`      | `1.0` if substring present                |
| `contains_any`  | `contains_any(answer, substrings, *, case_sensitive=False)` | `1.0` if any present                      |
| `contains_all`  | `contains_all(answer, substrings, *, case_sensitive=False)` | `1.0` if all present                      |
| `numeric_match` | `numeric_match(answer, expected, *, tolerance=0.0)`         | `1.0` if first number matches             |
| `f1_score`      | `f1_score(answer, reference)`                               | token-level F1                            |
| `normalize`     | `normalize(text) -> str`                                    | lowercased, punctuation/articles stripped |

```python theme={"dark"}
@env.template()
async def capital(country: str = "France"):
    answer = yield f"What is the capital of {country}?"
    yield exact_match(answer, "Paris")
```

## `BashGrader`

Runs a shell command via `/bin/bash -lc` and scores by exit code (`1.0` if it exits `0`). Async; returns a `SubScore`. Needs bash - macOS, Linux, WSL, or a built image; on native Windows it scores `0.0` with a `/bin/bash not found` error.

```python theme={"dark"}
@env.template()
async def fix_tests():
    answer = yield "Make the tests pass."
    result = await BashGrader.grade(weight=1.0, command="pytest -q", cwd="/workspace")
    yield result.value
```

`cwd` is the host directory to run in - for a workspace-backed task, pass the workspace root so the grader sees the same files the agent edited.

| Parameter         | Default | Description                    |
| ----------------- | ------- | ------------------------------ |
| `weight`          | -       | Weight in a composed grade.    |
| `command`         | -       | Shell command to run.          |
| `cwd`             | `None`  | Working directory.             |
| `timeout_seconds` | `600`   | Kill + score `0.0` on timeout. |

## `LLMJudgeGrader`

Scores an answer against weighted criteria with an LLM judge (uses the HUD gateway). Each criterion is graded `MET`/`UNMET` in parallel and combined by weight; no extra install needed.

```python theme={"dark"}
result = await LLMJudgeGrader.grade(
    weight=1.0,
    answer=answer,
    criteria=["Correct", ("Well-reasoned", 2.0)],
    question=prompt,
    model="claude-haiku-4-5",
)
```

`criteria` items are strings, or `(requirement, weight)` tuples. A negative weight marks an error to
avoid: the criterion scores as a penalty, MET when the answer actually makes the mistake.

The returned `SubScore` has one child per criterion: `name` is the requirement, `value` is
`1.0`/`0.0` for the verdict (on a negative-weight criterion, `1.0` means the error is present),
and `info["reason"]` is the judge's justification. The rubric value is the weighted
pass-fraction: `sum(value * weight)` over the positive weights, clamped to 0-1.

## `combine` - compose multiple graders

`combine` resolves `SubScore`s and grader coroutines in parallel and combines them into a weighted `EvaluationResult`. Positive weights are normalized to sum to `1.0`; negative weights are penalties.

```python theme={"dark"}
@env.template()
async def composed(answer: str = ""):
    answer = yield "Solve the task."
    yield await combine(
        BashGrader.grade(weight=0.5, command="pytest -q"),
        LLMJudgeGrader.grade(weight=0.3, answer=answer, criteria=["Matches the spec"]),
        SubScore(name="format", value=exact_match(answer, "42"), weight=0.2),
    )
```

| Function                         | Description                                                                  |
| -------------------------------- | ---------------------------------------------------------------------------- |
| `await combine(*items)`          | Resolve `SubScore` / `Awaitable[SubScore]` in parallel → `EvaluationResult`. |
| `combine_any(weight, subscores)` | Boolean OR: a `SubScore` that passes if any input passes (max).              |
| `combine_all(weight, subscores)` | Boolean AND: a `SubScore` that passes only if all inputs pass (min).         |

The subscores appear in the trace, so a partial reward is legible. `combine_any`/`combine_all` fold alternatives into a single component you can feed to `combine` - e.g. "tests pass via `pytest` OR via `make test`" as one 0/1 subscore. The combined `SubScore` keeps its inputs on `children`, so the full grading tree survives into the trace. Combinators nest: a `combine_all` over `combine_any` nodes produces a deeper tree.

## Custom graders

Subclass `Grader` and implement async `compute_score`, returning a `SubScore`:

```python theme={"dark"}
class LengthGrader(Grader):
    name = "length"

    @classmethod
    async def compute_score(cls, answer: str = "", target: int = 100, **kwargs):
        return SubScore(name=cls.name, value=1.0 if len(answer) >= target else 0.0)

result = await LengthGrader.grade(weight=1.0, answer=answer, target=200)
```

Use `info` for grader-specific details and `children` for nested results such as rubric
verdicts. `grade` applies the caller's weight and optional name override, then records the call
parameters under `_parameters`. Legacy float and `(float, info)` returns are still coerced to
`SubScore` for backwards compatibility.

## `SubScore` and `EvaluationResult`

A `SubScore` is one node in the grade breakdown tree. A leaf is a single measurement; a parent
preserves the measurements used to derive its value in `children`:

| Field      | Default | Description                                                             |
| ---------- | ------- | ----------------------------------------------------------------------- |
| `name`     | -       | Name of this component (for a rubric verdict: the requirement checked). |
| `value`    | -       | Measurement, 0-1.                                                       |
| `weight`   | `1.0`   | Weight in the parent's combination; negative = penalty.                 |
| `children` | `None`  | Input subscores this node was combined from.                            |
| `info`     | `None`  | Grader-specific details recorded on the evaluation span.                |

An `EvaluationResult` is the combined grade payload you can yield from a task:

| Field       | Default | Description                              |
| ----------- | ------- | ---------------------------------------- |
| `reward`    | `0.0`   | Final score.                             |
| `done`      | `True`  | Episode complete.                        |
| `subscores` | `None`  | Optional breakdown (shown in the trace). |
| `info`      | `{}`    | Extra metadata.                          |
| `content`   | `None`  | Human-readable explanation.              |
| `isError`   | `False` | Whether grading itself failed.           |

`EvaluationResult.from_float(value)` wraps a bare reward.

Graders run as the second yield of a [task](/v6/reference/tasks); for choosing a reward that separates
good work from bad, see [designing tasks](/v6/reference/advice).