Skip to main content
A taskset is a named collection of concrete tasks, used as an evaluation suite. Jobs that run the same taskset answer the same question, even when the models or the number of attempts change. If your tasks already live in Python, publish them from the project that defines them:
The taskset then appears under Tasksets.

​
From template to taskset

Tasks are mutable containers, but their versions are not. Editing a task creates a new version without changing the task version attached to an existing trace. This keeps old jobs interpretable after the suite changes.

​
Create a taskset

There are two supported entry points:
  1. Sync from the SDK. Define tasks in Python and run hud sync tasks <name>. HUD uploads only what changed.
  2. Create in the platform. Select New taskset, then create tasks individually or upload a task file from the taskset page.
A runnable task version must point to a task template from a deployed environment. If no runnable versions exist, the platform will not start the batch. For task authoring patterns and graders, see tasks and tasksets and designing tasks.

​
Run the suite

Select Run Taskset and configure: The platform expands the batch as:
A taskset with 20 tasks, 2 models, and a group size of 3 creates 120 traces in one evaluation job. Group size is also how evaluation connects to training: training compares groups of attempts at the same task, so a run with groups produces its input directly.

​
Read the taskset page

The available tabs depend on your access. Public viewers see the published summary; editors can change tasks, launch jobs, and manage access.

​
Compare without changing the suite

Keep the taskset fixed when comparing models or repeated runs. Change the models, group size, or step limit at job launch instead. Results across jobs then stay comparable, because the suite itself never moved. When the suite itself changes, the new task versions apply to future jobs. Existing traces continue to reference the versions they actually ran.

​
Share and export

Use Share to grant viewer or editor access. Use Export JSON to download the runnable task definitions. Public tasksets can be viewed through their shared URL without signing in. A runnable suite is the input to an evaluation job.