Runs & experiments
Use validation to measure agreement with human labels, evaluation to score new data, and comparisons to see what changed.
Runs panel tour
The Runs panel is the execution history for every evaluator-version and dataset pairing. The list surfaces kind, status, headline metric, and start time; opening a run reveals progress or the completed metrics and row-level evidence.
Validation results
- Bind exact inputs. A run records one evaluator version and one dataset. That immutable pairing makes results attributable later.
- Choose engine behavior deliberately. Run kind controls whether labels are compared; the Logfire experiment checkbox changes the execution engine and sync destination, not the kind.
- Read the headline before drilling down. Categorical validation starts with Accuracy and Cohen's κ; numeric validation starts with MAE, RMSE, and correlations.
- Inspect the evidence. Filter to disagreements or errors, compare score with label, read structured output, and retry only failed rows when available.
Starting a run
Select Runs → New run, then choose an evaluator. Valcore loads its versions and selects the active one by default. Choose a dataset, run kind, and concurrency between 1 and 64.
Eval
Scores every row and records agent judge output. Labels are optional and agreement metrics are not required.
Validation
Compares every score with human ground truth. The option is disabled when the selected dataset contains any unlabeled rows.
Logfire experiment
Runs through pydantic-evals and appears in Logfire's experiments view. It cannot be cancelled and individual rows cannot be rerun.
Validation runs
A validation run compares evaluator scores with a fully labeled dataset. Use it while developing an agent judge or as a release gate.
Interpreting results
Categorical runs report Accuracy, Cohen's κ, sample count, per-label Precision/Recall/F1/Support, and a confusion matrix. Matrix rows are human labels and columns are agent judge scores; off-diagonal cells are the mistakes to investigate.
Numeric runs report MAE and RMSE, where lower is better, plus Pearson and Spearman correlation. Correlation can be unavailable when either the labels or scores have zero variance.
Synced to Logfire
Run the experiment command to evaluate through pydantic_evals.Dataset.evaluate. The experiment and its spans appear in the configured Logfire project.
valcore experiment <evaluator> <dataset>Thresholds API
For categorical validation, turn minimum accuracy into a CI gate. Valcore exits with status 2 when the result misses the threshold.
valcore run <evaluator> <dataset> \
--kind validation \
--min-accuracy 0.90Use --json when another tool needs the structured run result. Accuracy thresholds do not apply to numeric labels.
Evaluation runs
Evaluation runs record agent judge outputs without comparing them to ground truth, so labels are optional. Use them to score fresh cases, inspect reasoning, and find examples that should join a labeled validation set.
Interpreting results
The detail table shows the original row, structured output, score, optional label, and agreement when available. Use Errors only to isolate execution failures. A run completed with errors exposes Retry failed rows, which preserves successful results and reruns only failures.
An evaluation score is a measurement from the configured agent judge—not a human-verified answer.
Comparisons
Select Runs → Compare and choose Run A and Run B. Valcore requires both runs to use the same dataset so each row has a stable basis for comparison.
The comparison first shows deltas for shared numeric metrics. Higher is better for agreement metrics; lower is better for MAE and RMSE. The row table places A and B outputs side by side and marks differing scores, with disagreements ordered first so regressions are easy to find.