Tasksets

Documentation Index

Fetch the complete documentation index at: https://docs.hud.ai/llms.txt

Use this file to discover all available pages before exploring further.

​

Overview

• Leaderboards — Public benchmarks with ranked results
• My Tasksets — Your personal task collections

​

Leaderboards

• Dataset cards — Each card shows a benchmark with ranked entries
• Metrics — Average score, Best@3, Best@5
• Filter by organization — Focus on specific providers
• Search — Find specific benchmarks

​

Creating a Taskset

1. Enter a name for your taskset
2. Click Create
3. You’re taken to your new (empty) taskset

​

Adding Tasks

1. Go to an environment’s Scenarios tab
2. Click on a scenario
3. Create tasks with specific arguments
4. Select your taskset as the destination

1. Open your taskset
2. Click Upload Tasks in the header
3. Paste a JSON array of task configurations
4. The modal validates your tasks before upload

​

Taskset Details

​

Leaderboard Tab

• Agent rankings — Performance by agent/model
• Metrics — Success rate, average score
• Trends — Performance over time

​

Tasks Tab

• Task name — Click to open the task detail panel
• Progress — Clickable progress pips for review stages
• Tr (Traces) — Trace count; click to open the detail panel directly to the Traces tab
• Reward — Average reward score
• Distribution — Mini histogram of reward values
• Updated — Relative timestamp
• Custom columns — Data columns and QA agent result columns

​

Task Detail Panel

​

Task Tab

• Task name with a play-on-hover icon to run the task
• Progress pips — interactive, labeled stages (e.g. “ready”, “verified”) that you can toggle by clicking
• Scores — reward value with a histogram and x-axis to the right

• Environment badge — links to the environment page
• Scenario arguments — editable inline. Simple strings shorter than 30 characters render as single-line inputs; longer strings use a textarea. JSON objects and arrays use a JSON editor. Changes save immediately and create a new task version.
• Custom columns — editable task-level data columns appear under a COLUMNS heading (hidden if none exist)

​

Traces Tab

• Each attached QA agent appears as a compact pill with a play button
• Click the play button for options: Run for this task or Run for all tasks (N)
• Add QA Agent button to attach new agents

• Version label (v1, v2, etc.) — older versions show an amber warning badge
• Model name — the agent/model used
• Reward — evaluation score
• Wallclock time, steps, and tokens — compact metrics next to the trace name
• QA analysis results — collapsible rows below each trace showing verdicts from attached agents; agents without results show a Run button

​

Comments Tab

• Comments — post messages with Enter to send, Shift+Enter for new lines
• Task update history — when scenario arguments change or versions are created
• Progress status changes — every progress update (completion/un-completion) is recorded

​

History Tab

• Summary — which arguments changed between versions, who made the change
• Word-level diffs — GitHub-style collapsible diff view showing exactly what was added, removed, or modified in each argument
• Optimized for large inputs with performance guards and fallback rendering for very long values

​

Running a Taskset

1. Open your taskset
2. Click Run Taskset in the header
3. Select model and configuration
4. Jobs are queued and run in parallel

hud eval "My Tasks" claude --full --group-size 10

​

Task Configuration

{
  "scenario": "checkout",
  "args": {
    "product_name": "Laptop"
  },
  "env": {
    "name": "my-store-env"
  },
  "prompt": "Optional custom prompt override"
}

• scenario — The scenario name to run
• args — Arguments passed to the scenario
• env.name — The environment containing the scenario
• prompt — (Optional) Override the scenario’s default prompt

​

Next Steps