Annotations

Label Sets and Annotations

A Label Set is an annotation contract: a name, a kind (categorical or numeric), and either the fixed labels an annotator can choose from—each with its own criteria description—or a numeric range. An Annotation is one row's recorded labels or value under one Label Set, along with an optional description and, when generation or import supplied one, a suggested value it can be checked against.

A Dataset can carry more than one Label Set—for example a primary quality judgment plus a secondary sentiment check—each tracked and annotated independently. The stats header on a Dataset's detail page reflects its primary Label Set; open Annotate from there, or start from the Annotations panel, to reach any of them.

Creating and managing Label Sets

Select Annotations, choose a Dataset, then New label set. Name it, choose Categorical or Numeric, and either add labels—each with the criteria text shown to annotators as reference during review—or set a minimum and maximum.

Providing a label schema while creating a Dataset (see Adding datasets) creates its first Label Set for you. Add further Label Sets here at any time; each starts with every row unannotated.

The Annotation Queue

Opening a Label Set from the Annotations panel loads a table over its Dataset's rows, one hundred at a time, with columns for the row data, current labels, any suggestion, its source, and a description field. Expand a row to see its full column values, edit a cell inline, or delete the row entirely.

Keyboard shortcuts

The queue is built for keyboard-first review: j/k move the focused row, 1–9 toggle the Nth categorical label on the focused row (multi-select, not replace), a accepts its suggestion, u clears its annotation, Enter opens it in the single-row view, and ? shows this list in the app. Every change saves immediately with an optimistic update that rolls back if the request fails.

Suggestions and acceptance

Rows from a generated or imported Dataset can carry a suggested label or value alongside the confirmed one, shown in the Suggested column. Press a or select Accept to copy it into the row's Annotation; the Source column then reads accepted rather than manual or generated.

The single-row view

Press Enter on a focused row, or open a row directly, for a full-page view of its data, labels, and description with Prev/Next buttons over the same loaded page—useful for careful, one-row-at-a-time review rather than the denser queue table.

Feeding Validation runs

A validation Experiment Run needs a Label Set on the chosen Dataset whose kind and, for categorical sets, whose exact labels match the Agent Evaluator's score contract. Valcore reports how many of the Dataset's rows that matching Label Set has annotated, and Validation is unavailable until every row is covered. See Validation Experiment Runs for how those labels are compared against Agent Evaluator scores.