Pydantic-native observability

Logfire integration

Connect production evidence to the full Valcore loop: build Datasets from traces, observe Agents and Agent Evaluators, sync Agent prompts, and open Pydantic Experiment Runs with every case attached.

Choose a project layout

For most teams, use two Logfire projects. This keeps production access separate from evaluation output while still giving Valcore one connected workflow.

Source Agent ProjectProduction traces and source Datasets

Valcore reads from this project with the read API key.

Local Valcore workspaceAgents, Agent Evaluators, Datasets, Experiment Runs

Your authoring and results remain available locally.

Valcore ProjectEvaluation traces, experiments, published Datasets, Agent prompt variables

The tracing token and write API key point here.

Set up Logfire

Create an account and organization. Sign in at Logfire, then create the projects you chose above.
Create the tracing token. Open the Valcore Project, then Settings → Write tokens → New write token. Copy it when it is shown; Logfire only displays the full value once.
Create the read API key. In the Source Agent Project, open Settings → API Keys. Grant project:read and project:read_datasets.
Create the write API key. In the Valcore Project, create an API key with project:read_datasets and project:write_datasets.
Add the credentials to Valcore. Open Settings → Logfire and paste each value into its matching field, or use the CLI commands below.

valcore config set-logfire-token
valcore config set-logfire-read-key
valcore config set-logfire-write-key
valcore config get

Valcore stores secrets in ~/.valcore/config.toml ($HOME\.valcore\config.toml on Windows). It uses file mode 0600 on macOS and Linux and the current user's filesystem ACLs on Windows. Settings and config get report whether a secret is configured without returning its value.

Set up the Pydantic AI Gateway

The Gateway is managed from your Logfire organization, but its API key has a different job from Logfire's tracing token and Dataset API keys: it authorizes hosted model requests.

Enable the Gateway. In Logfire, open your organization and go to AI Engineering → Gateway. Activate the Gateway if it is not already enabled.
Choose how models are funded. Use Pydantic's built-in providers with a Gateway balance, or add your own upstream provider credentials under Providers. Built-in providers may require a payment method and prepaid balance.
Create a Gateway key. Open the Gateway's API Keys tab, create a project-scoped key for Valcore, and apply the spending limits appropriate for evaluation runs.
Add the key to Valcore. Open Settings → Pydantic AI Gateway and paste the key, or configure it from the terminal:

valcore config set-key
valcore config get

Next, open Settings → Model Selection, clear the local CLI default, and use a gateway/<provider>:<model> route. One Gateway key can reach models from multiple configured providers.

gateway/anthropic:claude-sonnet-5
gateway/openai:gpt-5
gateway/google:gemini-2.5-pro

Dataset generation

Generate rows, suggested labels, controlled label distributions, and Dataset contracts with a hosted model.

Agent and Agent Evaluator authoring

Draft Agents from a prompt, generate or refine Agent Evaluator prompts and contracts, then run a pinned hosted model with Pydantic AI tools and harness capabilities.

Experiment Runs

Execute the Agent Evaluator across a Dataset without depending on a locally installed and authenticated coding CLI.

See Pydantic's official AI Gateway setup guide for provider configuration, project and user keys, regions, endpoints, spending limits, and usage telemetry.

Credentials and scopes

Credential	Logfire location and scope	What it enables
Gateway API key`gateway_api_key`	Organization → AI Engineering → Gateway Project-scoped Gateway key	Hosted model calls for Dataset generation, Agent and Agent Evaluator authoring, and Experiment Runs.
Tracing token`logfire_token`	Valcore Project Project write token	Valcore service telemetry, Pydantic AI traces, and Pydantic Experiment Runs.
Read API key`logfire_read_key`	Source Agent Project `project:read` `project:read_datasets`	SQL trace queries, trace import, and listing or fetching hosted Datasets.
Write API key`logfire_write_key`	Valcore Project `project:read_datasets` `project:write_datasets` For prompt sync, also `project:read_variables` and `project:write_variables`	Publishing local Datasets to Logfire's hosted Dataset store, and syncing Agent prompts with managed variables.
Frontend application`logfire_frontend_trace_urllogfire_frontend_token`	Valcore Project Project settings → Frontend applications Restricted public token	Optional browser tracing of the workbench, and opt-in session replay.

The tracing token does not query traces or publish Datasets. The API keys do not enable tracing. Keeping these duties separate gives each integration only the access it needs.

When both API keys target the same project, configure them together:

valcore config set-logfire-key

Agent Evaluator touchpoints

The tracing token connects the evaluation runtime to the Valcore Project. Valcore configures its Logfire service as valcore and instruments Pydantic AI globally.

Agent Evaluator execution

Inspect the model request, structured response, timing, errors, and any Pydantic AI tool or capability activity emitted by a Gateway-backed Agent Evaluator.

Generation and refinement

Agent, Dataset, and Agent Evaluator generation and plain-language refinement use the same Pydantic AI instrumentation. Agent drafts appear as agent_generator.

Run hierarchy

Each normal run emits a valcore.run parent span and one valcore.score_row child per Dataset row, with run kind, version, Dataset, and concurrency context. The parent records the run's status and metrics when it closes.

Dataset touchpoints

Build a Dataset from trace SQL

The read API key powers Datasets → New dataset → Logfire query. Write SQL against the Source Agent Project, choose how many top-level trace trees to sample, map columns to the Dataset contract, and optionally identify a result column to use as the label.

valcore logfire pull   --sql-file support-cases.sql   --name support-cases   --count 100

Queries default to the last 24 hours and can cover at most 14 days. Child spans can be nested into a children JSON field. Valcore saves the SQL and pull settings as Dataset provenance.

Import or publish hosted Datasets

The read key lists and fetches Datasets hosted in the Source Agent Project. The write key publishes a local Dataset to the Valcore Project.

# Read from the Source Agent Project
valcore logfire list
valcore logfire fetch support-golden-set

# Publish to the Valcore Project
valcore logfire push support-golden-set

In the interface, use the corresponding Import from Logfire and Push to Logfire actions. Pushing does not require the tracing token; importing does not require the write key.

Experiment Run touchpoints

The tracing token is the only Logfire credential required to sync an Experiment Run. In Runs → New run, select the Dataset and Agent Evaluator, choose either Eval or Validation, and enable Run as a Logfire experiment.

Valcore executes either kind through pydantic_evals.Dataset.evaluate. Both appear in Logfire's Evals view with experiment, case, task, and evaluator spans, plus inputs, actual judgments, durations, and traces. A Validation run additionally sends confirmed labels as expected outputs and calculates agreement statistics; an Eval run does neither.

The CLI shortcut below currently creates a Validation run. Use the interface to create an Eval run with the Logfire experiment engine.

valcore experiment <agent-evaluator> <dataset>

Valcore keeps its local run record and statistical results. Logfire-backed Experiment Runs cannot be cancelled or retried row by row; start a new Experiment Run when configuration or individual cases need to change.

Agent touchpoints

The tracing token covers Agents exactly as it covers Agent Evaluators. A trial or Dataset run is a Pydantic AI call, so the model request, structured output, capability activity, and timing appear in the Valcore Project.

Derive runs

An Agent run over a Dataset emits the same valcore.run parent with kind derive and one child span per row, so Agent and Agent Evaluator runs can be queried side by side.

Prompt sync

The write API key, with project:read_variables and project:write_variables, links an Agent's instructions and input template to two managed variables in the key's project.

No runtime dependency

Agent runs always use the local version and never fetch prompts from Logfire. Syncing is an explicit edit, never a live lookup.

Sync writes a new variable version under a dedicated valcore_sync label and never promotes production. See Logfire prompt sync for linking, conflicts, and template limits.

Browser observability

Valcore can also trace the workbench itself. In Logfire, create a frontend application under Project settings → Frontend applications. Then open Valcore's Settings → Logfire Frontend Observability and paste the generated trace URL and restricted public token. Browser tracing starts on the next page load.

Record session replays is a separate early-access option and is off by default. When it is on, UI text and input values are recorded everywhere except the Settings page, console capture stays off, and recordings go to your Logfire project.

valcore config set logfire_frontend_trace_url https://logfire-us.pydantic.dev/v1/traces
valcore config set logfire_frontend_token <restricted-public-token>
valcore config set logfire_session_replay true

See Logfire's frontend observability guide for the Logfire-side setup.

Verify the integration

Run valcore config get and confirm the required credential is reported as configured.
Start valcore serve, open Settings, and check the Logfire status cards. Stored secrets stay masked.
Run a small labeled Dataset with Run as a Logfire experiment enabled.
Follow the Logfire link from the completed Experiment Run and confirm the experiment and case traces appear in the Valcore Project.
For Dataset access, run valcore logfire list or preview a trace query to confirm the read key points to the intended Source Agent Project.

If Valcore cannot resolve a Logfire project link from an API key, configure a fallback SQL Workbench URL:

valcore config set-logfire-explore-url   https://logfire.pydantic.dev/<organization>/<project>/explore

For Logfire's credential UI and native concepts, see the official guides for write tokens, hosted Dataset API keys, and Pydantic Evals.