Pydantic-native observability

Logfire integration

Connect production evidence to the full Valcore loop: build Datasets from traces, observe Agent Evaluators, and open Pydantic Experiment Runs with every case attached.

Choose a project layout

For most teams, use two Logfire projects. This keeps production access separate from evaluation output while still giving Valcore one connected workflow.

Source Agent ProjectProduction traces and source Datasets

Valcore reads from this project with the read API key.

Local Valcore workspaceAgent Evaluators, Datasets, Experiment Runs

Your authoring and results remain available locally.

Valcore ProjectEvaluation traces, experiments, published Datasets

The tracing token and write API key point here.

Set up Logfire

Create an account and organization. Sign in at Logfire, then create the projects you chose above.
Create the tracing token. Open the Valcore Project, then Settings → Write tokens → New write token. Copy it when it is shown; Logfire only displays the full value once.
Create the read API key. In the Source Agent Project, open Settings → API Keys. Grant project:read and project:read_datasets.
Create the write API key. In the Valcore Project, create an API key with project:read_datasets and project:write_datasets.
Add the credentials to Valcore. Open Settings → Logfire and paste each value into its matching field, or use the CLI commands below.

valcore config set-logfire-token
valcore config set-logfire-read-key
valcore config set-logfire-write-key
valcore config get

Valcore stores secrets in ~/.valcore/config.toml with file mode 0600. Settings and config get report whether a secret is configured without returning its value.

Set up the Pydantic AI Gateway

The Gateway is managed from your Logfire organization, but its API key has a different job from Logfire's tracing token and Dataset API keys: it authorizes hosted model requests.

Enable the Gateway. In Logfire, open your organization and go to AI Engineering → Gateway. Activate the Gateway if it is not already enabled.
Choose how models are funded. Use Pydantic's built-in providers with a Gateway balance, or add your own upstream provider credentials under Providers. Built-in providers may require a payment method and prepaid balance.
Create a Gateway key. Open the Gateway's API Keys tab, create a project-scoped key for Valcore, and apply the spending limits appropriate for evaluation runs.
Add the key to Valcore. Open Settings → Pydantic AI Gateway and paste the key, or configure it from the terminal:

valcore config set-key
valcore config get

Next, open Settings → Model Selection, clear the local CLI default, and use a gateway/<provider>:<model> route. One Gateway key can reach models from multiple configured providers.

gateway/anthropic:claude-sonnet-5
gateway/openai:gpt-5
gateway/google:gemini-2.5-pro

Dataset generation

Generate rows, suggested labels, controlled label distributions, and Dataset contracts with a hosted model.

Agent Evaluator authoring

Generate or refine prompts and contracts, then run a pinned hosted model with Pydantic AI tools and harness capabilities.

Experiment Runs

Execute the Agent Evaluator across a Dataset without depending on a locally installed and authenticated coding CLI.

See Pydantic's official AI Gateway setup guide for provider configuration, project and user keys, regions, endpoints, spending limits, and usage telemetry.

Credentials and scopes

Credential	Logfire location and scope	What it enables
Gateway API key`gateway_api_key`	Organization → AI Engineering → Gateway Project-scoped Gateway key	Hosted model calls for Dataset generation, Agent Evaluator authoring, and Experiment Runs.
Tracing token`logfire_token`	Valcore Project Project write token	Valcore service telemetry, Pydantic AI traces, and Pydantic Experiment Runs.
Read API key`logfire_read_key`	Source Agent Project `project:read` `project:read_datasets`	SQL trace queries, trace import, and listing or fetching hosted Datasets.
Write API key`logfire_write_key`	Valcore Project `project:read_datasets` `project:write_datasets`	Publishing local Datasets to Logfire's hosted Dataset store.

The tracing token does not query traces or publish Datasets. The API keys do not enable tracing. Keeping these duties separate gives each integration only the access it needs.

When both API keys target the same project, configure them together:

valcore config set-logfire-key

Agent Evaluator touchpoints

The tracing token connects the evaluation runtime to the Valcore Project. Valcore configures its Logfire service as valcore and instruments Pydantic AI globally.

Agent Evaluator execution

Inspect the model request, structured response, timing, errors, and any Pydantic AI tool or capability activity emitted by a Gateway-backed Agent Evaluator.

Generation and refinement

Dataset generation, Agent Evaluator generation, and plain-language refinement use the same Pydantic AI instrumentation.

Run hierarchy

Each normal run emits a valcore.run parent span and one valcore.score_row child per Dataset row, with Agent Evaluator, version, Dataset, and concurrency context.

Dataset touchpoints

Build a Dataset from trace SQL

The read API key powers Datasets → New dataset → Logfire query. Write SQL against the Source Agent Project, choose how many top-level trace trees to sample, map columns to the Dataset contract, and optionally identify a result column to use as the label.

valcore logfire pull   --sql-file support-cases.sql   --name support-cases   --count 100

Queries default to the last 24 hours and can cover at most 14 days. Child spans can be nested into a children JSON field. Valcore saves the SQL and pull settings as Dataset provenance.

Import or publish hosted Datasets

The read key lists and fetches Datasets hosted in the Source Agent Project. The write key publishes a local Dataset to the Valcore Project.

# Read from the Source Agent Project
valcore logfire list
valcore logfire fetch support-golden-set

# Publish to the Valcore Project
valcore logfire push support-golden-set

In the interface, use the corresponding Import from Logfire and Push to Logfire actions. Pushing does not require the tracing token; importing does not require the write key.

Experiment Run touchpoints

The tracing token is the only Logfire credential required to sync an Experiment Run. In Runs → New run, select the Dataset and Agent Evaluator, choose Validation, and enable Run as a Logfire experiment.

valcore experiment <agent-evaluator> <dataset>

Valcore executes the cases through pydantic_evals.Dataset.evaluate. The Experiment Run appears in Logfire's Evals view with an experiment span, case spans, task execution, evaluator assertions, inputs, expected labels, actual judgments, durations, and traces. Valcore also keeps its local run record and statistical results.

Logfire-backed Experiment Runs cannot be cancelled or retried row by row. Start a new Experiment Run when configuration or individual cases need to change.

Verify the integration

Run valcore config get and confirm the required credential is reported as configured.
Start valcore serve, open Settings, and check the Logfire status cards. Stored secrets stay masked.
Run a small labeled Dataset with Run as a Logfire experiment enabled.
Follow the Logfire link from the completed Experiment Run and confirm the experiment and case traces appear in the Valcore Project.
For Dataset access, run valcore logfire list or preview a trace query to confirm the read key points to the intended Source Agent Project.

If Valcore cannot resolve a Logfire project link from an API key, configure a fallback SQL Workbench URL:

valcore config set-logfire-explore-url   https://logfire.pydantic.dev/<organization>/<project>/explore

For Logfire's credential UI and native concepts, see the official guides for write tokens, hosted Dataset API keys, and Pydantic Evals.