Experiment Runs
Use validation to measure agreement with human labels, evaluation to score new data, and comparisons to see what changed.
Experiment Runs panel tour
The Experiment Runs panel is the execution history for every Agent Evaluator version and Dataset pairing. The list surfaces kind, status, headline metric, and start time; opening an Experiment Run reveals progress or the completed metrics and row-level evidence.
Validation results
- Bind exact inputs. A run records one version and one data contract: a Dataset, or a Dataset plus a saved Agent derivation. That immutable pairing makes results attributable later.
- Choose engine behavior deliberately. Run kind controls whether labels are compared; the Logfire experiment checkbox changes the execution engine and sync destination, not the kind.
- Read the headline before drilling down. Categorical validation starts with Accuracy and Cohen's κ; numeric validation starts with MAE, RMSE, and correlations.
- Inspect the evidence. Filter to disagreements or errors, compare score with label, read structured output, and retry only failed rows when available.
Starting an Experiment Run
Select Runs → New run, then choose an Agent Evaluator. Valcore loads its versions and selects the active one by default. Choose the Data, run kind, and concurrency between 1 and 64.
The Data menu lists each Dataset and every saved derivation of it, with the columns each one provides. Entries missing a column the Agent Evaluator requires are disabled and name what is missing.
Eval
Scores every row and records Agent Evaluator output. Labels are optional and agreement metrics are not required.
Validation
Compares every score with human ground truth. The option is disabled when the selected Dataset contains any unlabeled rows.
Logfire experiment
An execution option for either run kind. It runs through pydantic-evals and appears in Logfire's experiments view.
Agent runs and responses
A Derive run executes an Agent version over every row of a Dataset instead of scoring it. Its detail page lists each row's inputs, responses, errors, and latency, with Save to keep them as a numbered derivation or Discard to drop them.
To score those responses, start an Eval run and choose the derivation as its Data. Rows without a usable response are skipped and reported in the run's metrics rather than silently dropped.
# Run an Agent over a Dataset, then score its saved responses
valcore run agent support-agent --dataset support-cases --save
valcore run evaluator reply-quality --dataset support-cases \
--derivation support-agent/v2/0Logfire Experiment Runs
Run kind and Run as a Logfire experiment are independent choices. Select either Eval or Validation, then enable the Logfire option to execute that run through pydantic_evals.Dataset.evaluate and send it to the configured Logfire project.
Eval + Logfire
Scores labeled or unlabeled rows and records each case in Logfire. It does not compare judgments with labels or calculate agreement metrics.
Validation + Logfire
Requires every row to have a confirmed label, sends labels as expected outputs, and calculates Valcore's agreement statistics.
Same experiment surface
Both kinds appear in Logfire with experiment, case, task, and evaluator spans. Neither can be cancelled or retried row by row.
The CLI's run experiment command currently creates a Validation run. Use Runs → New run in the interface when you want an Eval run backed by the Logfire experiment engine.
# Validation through the Logfire experiment engine
valcore run experiment <evaluator> --dataset <dataset>See the Logfire integration guide for the required tracing token and the data attached to each experiment case.
Validation Experiment Runs
A validation Experiment Run compares Agent Evaluator scores with a fully labeled Dataset. Use it while developing an Agent Evaluator or as a release gate.
Interpreting results
Categorical Experiment Runs report Accuracy, Cohen's κ, sample count, per-label Precision/Recall/F1/Support, and a confusion matrix. Matrix rows are human labels and columns are Agent Evaluator scores; off-diagonal cells are the mistakes to investigate.
Numeric Experiment Runs report MAE and RMSE, where lower is better, plus Pearson and Spearman correlation. Correlation can be unavailable when either the labels or scores have zero variance.
Thresholds API
For categorical validation, turn minimum accuracy into a CI gate. Valcore exits with status 2 when the result misses the threshold.
valcore run evaluator <evaluator> --dataset <dataset> \
--kind validation \
--min-accuracy 0.90Use --json when another tool needs the structured run result. Accuracy thresholds do not apply to numeric labels.
Unlabeled Experiment Runs
These Experiment Runs record Agent Evaluator outputs without comparing them to ground truth, so labels are optional. Use them to score fresh cases, inspect reasoning, and find examples that should join a labeled validation Dataset.
Interpreting results
The detail table shows the original row, structured output, score, optional label, and agreement when available. Use Errors only to isolate execution failures. An Experiment Run completed with errors exposes Retry failed rows, which preserves successful results and reruns only failures.
An evaluation score is a measurement from the configured Agent Evaluator—not a human-verified answer.
Comparisons
Select Runs → Compare and choose Run A and Run B. Valcore requires both runs to use the same dataset so each row has a stable basis for comparison.
The comparison first shows deltas for shared numeric metrics. Higher is better for agreement metrics; lower is better for MAE and RMSE. The row table places A and B outputs side by side and marks differing scores, with disagreements ordered first so regressions are easy to find.