The workbench for Agents and their evaluators

Agentic
experimentation
done the right way.

Valcore connects the Agent you are improving, flexible Datasets, fully harnessed Agent Evaluators, and statistical Experiment Runs in one workflow. Run your Agent over real cases, score what it said on the same data contract, then version and compare every result.

$brew install duncankmckinnon/tap/valcore

Build your first eval

Download Skills

Open source · Apache 2.0 · Python 3.11+ · Local CLIs or Pydantic AI Gateway

Experiment Runevaluator_v4 × support_set

94.2%label agreement

128 labeled cases✓ threshold passed

One connected system—not a pile of eval utilities.

01 Define the Agent→02 Build Agent Evaluator + evidence→03 Measure and compare

Local by design

Runs on your machine.
Syncs to Logfire.

Storage, authoring, and execution stay local. The browser, CLI, and coding agents operate the same persistent workspace, while every core surface can sync directly to its counterpart in Pydantic Logfire.

Interactive

Open the visual workbench

Take full control in the interface: define Agents, author contracts, label rows, compare results, and inspect the evidence behind every score.

See and control every part of the workflow.

Headless

Hand the workflow to an agent

Install the Valcore skill and let Claude, Codex, or Cursor create, version, run, inspect, and export through the CLI—even while the interface is closed.

Use your agent to build and operate the system with you.

On your machine

One workspace. Every interface.

Agents, Datasets, Agent Evaluator versions, Experiment Runs, configuration, and results share one active local state.

AgentDatasetAgent EvaluatorExperiment Run

Persistent and shared

Pydantic Logfire is the other half

Valcore keeps the Pydantic objects intact. Sync to Logfire without translating your work into a second, proprietary data model.

Agent promptsDatasetAgent EvaluatorExperiment Run

Pydantic native

One typed stack.
From contract to trace.

Valcore turns the Pydantic ecosystem into a complete experimentation workflow, without hiding the portable primitives underneath.

pydantic-ai-harness

Agents and evaluators with real capabilities

Portable AgentSpec definitions, typed inputs and outputs, structured scoring, MCP, tools, CodeMode, planning, filesystem, and shell.

pydantic-evals

Experiment Runs you can take with you

Run the same Dataset through native Pydantic experiments, with portable JSON and Python exports.

logfire

Every result connected to its trace

Trace every Agent and Agent Evaluator call, sync Experiment Runs automatically, keep Agent prompts in managed variables, and move Datasets in either direction.

Use what is already on your machine

Run locally with Claude, Codex, or Cursor

Generate Agents, Datasets, and Agent Evaluators, then run Agents and Experiment Runs through an authenticated coding-agent CLI. No new model key required.

Or choose a hosted model

Route through Pydantic AI Gateway

Use a consistent hosted route across providers when you need pinned models, harness capabilities, tools, or portable Python exports.

The connected workflow

Start from any side.
Keep the contract intact.

Your Agent runs on a Dataset and adds its responses as new columns on the same rows. It can also generate the Agent Evaluator that scores it, and Datasets and Agent Evaluators generate each other. Every step shares one contract, so nothing needs rewiring.

System under test

Agent

inputsquestion

outputresponse

Evidence

Dataset

questionstring

responseAgent v2 · run 0

Agent program

Agent Evaluator

inputsquestion · response

outputreasoning · score

scorepass · fail · review

Release evidence

Experiment Run

94.2%

label agreement · κ · F1 · a score for every Agent response

traced and synced to Logfire

Ground truth

Annotations

labelpass · fail · review

rationalestring · reviewer note

Flexible where it matters

Bring what you have.
Generate what you need.

Every entry point lands in the same versioned, portable evaluation workflow.

The system under test

Agents measured where they live

Store the Agent you are improving as a versioned Pydantic AI spec—written by hand, imported, or drafted from a prompt—then run it over any Dataset and keep each run's responses as a numbered overlay on the same rows.

Explore Agents

DefineModel, instructions, and inputs as a portable AgentSpec

GenerateDescribe the Agent and review the first version a model drafts

ImportLoad the YAML or JSON spec you already run

Try an inputCheck a response before committing to a full run

Run a DatasetStage every row's response, then save or discard

Score the responsesPoint an Agent Evaluator at the saved run

SyncLogfire variablesAgentSpec YAML

Representative evidence

Datasets that fit the work

Define real input columns, a fixed label set, and the distribution you expect in production—or use a refreshable SQL query over your Logfire traces.

Explore datasets

Hand-authorShape a small golden set row by row

ImportUpload an existing CSV or JSON test set

Query LogfireDefine a dataset in SQL and re-sync new matching traces at any time

GenerateControl columns, characteristics, labels, and distributions

From an Agent EvaluatorCreate compatible edge cases from its existing contract

ExportPydantic LogfirePython

More than a prompt

Agent Evaluators that act like agents

Each Agent Evaluator is a fully specified agent: typed inputs, structured outputs, model, prompt, tools, and harness capabilities in one versioned unit.

Explore Agent Evaluators

IdentityMODELgateway/anthropic:claude-sonnet-5

JudgmentINSTRUCTIONS

You are a domain reviewer. Judge whether the response is accurate, safe, and complete.

PROMPT TEMPLATE

Review the response to {customer_request}

InputsREQUIRED COLUMNS

customer_requestresponse

Output contractOUTPUT FIELDS

reasoningstr

judgmentenum · score

⌄Capabilities & tools

Run onClaudeCodexCursorGateway

Evidence, not vibes

Experiment Runs that prove performance

Run an Agent Evaluator against a human-labeled Dataset, inspect every disagreement, and compare Agent, prompt, capability, or model changes on the exact same cases.

Explore Experiment Runs

Accuracy94.2%+3.8

Cohen's κ0.91strong

Macro F10.93+0.04

Agent Evaluator v3same DatasetAgent Evaluator v4

Full traces and Pydantic Experiment Runs synced to Logfire

Your judgment, made repeatable

Build the Dataset and Agent Evaluator as one system.

Open the guide

Agenticexperimentationdone the right way.