Datasets

Dataset panel tour

The Datasets panel is both a catalog and a readiness check. The list shows how much evaluation material exists and whether it is labeled enough for validation; opening a dataset switches to the row-level authoring surface.

1. Use the summary before opening anything. Dataset count and total rows show coverage; the labeled percentage tells you how close the workspace is to supporting validation.
2. Choose the creation path that matches your source. Blank is for hand-authored cases, Upload accepts CSV/JSON, Generate creates synthetic cases, and Logfire imports hosted datasets or queried traces.
3. Read labeled status as a run constraint. “Complete” means every row has ground truth and can be used for validation. Partially labeled datasets remain usable for ordinary evaluation runs.
4. Work row data in the detail view, labels in Annotate. The detail grid edits case fields inline, adds rows, and deletes cases that do not belong. Select Annotate to apply or correct labels—see Annotations.

The detail panel's stats header shows total/labeled/unlabeled counts and label distribution for the dataset's primary Label Set. Generated and Logfire-sourced datasets retain collapsible provenance panels so you can see the instructions or SQL that created them.

Adding datasets

Start from a blank table, upload existing examples, query real traces, sync a hosted dataset, or generate synthetic cases.

Blank entries

Select New dataset → Blank, enter a name, add the columns the Agent Evaluator will receive, and optionally define a label schema. Providing one here creates the dataset's first Label Set immediately, ready to annotate. After creation, use Add row in the grid and edit each cell inline. This is the most direct route for a small golden set or a handful of deliberate edge cases.

CSV or JSON upload

Select New dataset → Upload. Valcore accepts CSV, JSONL, or a Valcore eval-package JSON file, infers the data columns, and previews the result before creation. Keep one case per CSV row or JSONL object. If labels are present, identify the label column and provide the matching categorical or numeric schema—Valcore creates a Label Set from it and seeds each row's annotation.

Logfire traces query

Select New dataset → Logfire and use the query mode to run SQL against the configured agent project. Choose a sample size, seed, time bounds, and optional label column. When matching child spans exist, Valcore preserves them in a children JSON column for richer evaluation context.

Logfire dataset syncing

The same Logfire tab can fetch a named hosted dataset from the agent project. From the CLI:

valcore logfire fetch <dataset-name>

Generating synthetic datasets

Creating a dataset from specs

Describe the behavior to test, choose the exact columns, and add per-column notes. You can request suggested labels and, for categorical schemas, opt into a target label mix. Review the editable draft before saving.

Creating datasets from evaluators

From an evaluator version, select Generate dataset. Valcore locks in the version's required columns so the resulting cases are compatible, while still allowing optional context columns and suggested labels. Generation settings and evaluator-version provenance are saved, so Generate more rows can extend the dataset consistently later.

Labeling datasets

Evaluation runs can use unlabeled data. Validation runs require every row to have a ground-truth label from a Label Set whose kind and labels match the evaluator's score contract.

Defining a label schema when creating a dataset—Blank, Upload, Generate, or Logfire—creates that dataset's first Label Set automatically and, for generated or imported rows, seeds each row's suggested label. From there, confirming suggestions, correcting them, and adding further Label Sets to the same dataset all happen in Annotations, not in the Dataset grid.

Writing datasets

Sync to the Logfire Valcore project

Once the cases and labels are ready, publish the dataset with the scoped write key:

valcore logfire push <dataset-name-or-id>

The local dataset remains the editable source; the Logfire copy makes the curated set available to your wider evaluation workflow.