Datasets

Dataset panel tour

The Datasets panel is both a catalog and a readiness check. The list shows how much evaluation material exists and whether it is labeled enough for validation; opening a dataset switches to the row-level authoring and labeling surface.

1. Use the summary before opening anything. Dataset count and total rows show coverage; the labeled percentage tells you how close the workspace is to supporting validation.
2. Choose the creation path that matches your source. Blank is for hand-authored cases, Upload accepts CSV/JSON, Generate creates synthetic cases, and Logfire imports hosted datasets or queried traces.
3. Read labeled status as a run constraint. “Complete” means every row has ground truth and can be used for validation. Partially labeled datasets remain usable for ordinary evaluation runs.
4. Work row by row in the detail view. Edit case fields inline, apply or correct labels, accept suggestions deliberately, add rows, and delete cases that do not belong.

Above the labeling grid, the detail panel also shows total/labeled/unlabeled counts and label distribution. Generated and Logfire-sourced datasets retain collapsible provenance panels so you can see the instructions or SQL that created them.

Adding datasets

Start from a blank table, upload existing examples, query real traces, sync a hosted dataset, or generate synthetic cases.

Blank entries

Select New dataset → Blank, enter a name, add the columns the agent judge will receive, and optionally define a label schema. After creation, use Add row in the grid and edit each cell inline. This is the most direct route for a small golden set or a handful of deliberate edge cases.

CSV or JSON upload

Select New dataset → Upload. Valcore accepts CSV, JSONL, or a Valcore eval-package JSON file, infers the data columns, and previews the result before creation. Keep one case per CSV row or JSONL object. If labels are present, identify the label column and provide the matching categorical or numeric schema.

Logfire traces query

Select New dataset → Logfire and use the query mode to run SQL against the configured agent project. Choose a sample size, seed, time bounds, and optional label column. When matching child spans exist, Valcore preserves them in a children JSON column for richer evaluation context.

Logfire dataset syncing

The same Logfire tab can fetch a named hosted dataset from the agent project. From the CLI:

valcore logfire fetch <dataset-name>

Generating synthetic datasets

Creating a dataset from specs

Describe the behavior to test, choose the exact columns, and add per-column notes. You can request suggested labels and, for categorical schemas, opt into a target label mix. Review the editable draft before saving.

Creating datasets from evaluators

From an evaluator version, select Generate dataset. Valcore locks in the version's required columns so the resulting cases are compatible, while still allowing optional context columns and suggested labels. Generation settings and evaluator-version provenance are saved, so Generate more rows can extend the dataset consistently later.

Labeling datasets

Evaluation runs can use unlabeled data. Validation runs require every row to have a ground-truth label compatible with the evaluator.

Numeric labels

Enter a score within the dataset's configured minimum and maximum. Validation summarizes error with metrics such as MAE and RMSE.

Categorical labels

Choose one of the fixed labels defined by the dataset. The labeling view supports keyboard-first review, including number shortcuts, accept suggestion, clear, and next/previous navigation.

Writing datasets

Sync to the Logfire Valcore project

Once the cases and labels are ready, publish the dataset with the scoped write key:

valcore logfire push <dataset-name-or-id>

The local dataset remains the editable source; the Logfire copy makes the curated set available to your wider evaluation workflow.