Agents

Agents and Agent Evaluators

An Agent is the system being measured. An Agent Evaluator is the agent that scores it. Both are versioned Pydantic AI agents, but they play opposite roles: an Agent turns Dataset rows into responses, and an Agent Evaluator turns rows—including those responses—into scores.

In the app, Agents leads the Author section of the sidebar, followed by Evals (the app's short label for Agent Evaluators), Datasets, and Annotations. Overview counts Agents alongside everything else.

1. Define an Agent version. Choose a model, write instructions, and declare which Dataset columns reach the Agent.
2. Run it over a Dataset. Responses are staged as they arrive and become a numbered derivation of that Dataset once you save them.
3. Build the Agent Evaluator. Generate one from the Agent's input and response fields, or reuse one whose columns match.
4. Score the responses. Run an Eval over the Dataset plus the saved derivation, then compare derivations from different Agent versions on the exact same rows.

Agent panel tour

The Agents list shows each Agent's name, description, and version count. Opening one switches to the version editor, where every action applies to the selected version.

1. Start with the version bar. Select a version, see whether it is frozen, and reach every action: Create evaluator, Run agent, Logfire sync, New version, Import, Export, Delete, and Freeze or Copy.
2. Bind the Agent to your data. Instructions, an optional input template, input fields, and a variables mapping decide what the Agent receives from each row.
3. Edit the spec directly when you need to. Capabilities and the raw AgentSpec JSON carry everything else pydantic-ai supports, including an output schema.
4. Check the response columns. They are derived from the spec and are the columns an Agent Evaluator can read once the Agent has run.

Defining an Agent version

Select Agents → New agent and give it a name and description. An Agent with no versions opens straight into a draft named v1; complete it and select Create version.

For a structured Agent, add an output schema to the spec. This one produces reply and escalate response columns:

{
  "instructions": "You are a retail support agent. Reply in two or three sentences.",
  "output_schema": {
    "type": "object",
    "properties": {
      "reply": { "type": "string", "description": "The message sent to the customer" },
      "escalate": { "type": "boolean", "description": "Whether a human should take over" }
    },
    "required": ["reply", "escalate"]
  }
}

Valcore validates the whole version when it is saved: a bad model string, an unknown capability, a template placeholder outside the input fields, or an unmapped required dependency is rejected with the reason.

Importing and exporting specs

Export downloads the selected version as a YAML AgentSpec. Valcore's binding—model, input template, input fields, and variables mapping—rides in metadata.valcore, so the file is still a valid spec that plain Agent.from_spec can load.

instructions: You are a retail support agent. Reply in two or three sentences.
output_schema:
  type: object
  properties:
    reply: { type: string }
    escalate: { type: boolean }
metadata:
  valcore:
    model: gateway/anthropic:claude-sonnet-5
    prompt_template: "Order status: {order_status}\n\n{request}"
    required_columns: [request, order_status]
    deps_mapping: {}

Import accepts YAML or JSON pasted into the dialog and fills the editor. Nothing is stored until you save the version. From the terminal, import creates a new Agent named from the spec's name, the --name option, or the filename:

valcore agent export support-agent --version v2 --out support-agent.yaml
valcore agent import support-agent.yaml --name support-agent-copy

Versioning Agents

Agent versions follow the same lifecycle as Agent Evaluator versions. The active version is used whenever a command does not name one. Any run freezes the version it used, and you can also select Freeze yourself.

A frozen version is read-only. Select Copy to get an editable copy, or New version to start a draft from the current values. Pulling text from Logfire always creates a new version rather than editing one in place.

Trying an input

Select Run agent to open the run panel, then stay on Try an input. The panel shows one field per input field, or a single free-form Input when the Agent declares none. Run returns the rendered prompt, each response column, and the latency.

A trial result is temporary. Running again replaces it, and Valcore asks before discarding an unsaved result—including when you close the panel or leave the page. To keep a result, choose a Dataset and select Save: the typed inputs are appended to that Dataset as a new row, and the response is saved as a new one-row derivation. A result that errored cannot be saved.

Running over a Dataset

In the run panel, switch to Run a dataset, choose a Dataset, and select Run dataset. Valcore checks that the Dataset has every input field before creating anything, then starts a Derive run and opens it on the Runs page, where it has the same progress, concurrency, and cancellation as any other run.

Responses are written as each row completes, so a refresh or restart does not lose them. They stay staged—invisible to the rest of Valcore—until you accept them. When the run finishes, its detail page lists every row's inputs, responses, errors, and latency, with Save and Discard.

Derivations and derived views

A derivation is one saved run of an Agent version over a Dataset. It adds the Agent's response columns to that Dataset's rows without copying them: the inputs stay in one place, and every derivation joins to the same row IDs. That is what makes two Agent versions directly comparable.

On a Dataset's detail page, the View menu appears once a derivation exists. Choose Original for the editable rows, or an entry such as support-agent · v2 · run 0 for a read-only table of inputs, responses, errors, and latency. If a response column has the same name as an input column, the response wins in the joined view.

Evaluating Agent responses

Generating an Agent Evaluator

Select Create evaluator on an Agent version. Describe what makes a response good, choose which of the Agent's fields the Agent Evaluator should read—at least one input and one response—and add notes on how each one should factor in. Optionally prescribe a label space. Valcore generates an editable draft and opens it under Agent Evaluators to review and save. Generation needs a Gateway key or a local CLI default.

Choosing the data contract

In Runs → New run, the Data menu lists each Dataset and, beneath it, each saved derivation with the columns it adds:

support-cases — request, order_status
support-cases + support-agent v1 · run 0 — request, order_status, reply, escalate
support-cases + support-agent v2 · run 0 — request, order_status, reply, escalate

An Agent Evaluator that reads {reply} cannot run on bare support-cases. Those entries are disabled and name the missing columns, so the mismatch shows before you start. Pick a derivation and choose Eval.

Rows the derivation does not cover, or where the Agent errored, are skipped rather than scored. The run detail reports them, for example Skipped rows: response error: 3, so a score over fewer rows than the Dataset holds is always visible.

Comparing Agent versions

Run the same Agent Evaluator over a derivation from each Agent version, then open Runs → Compare. Both runs share the same Dataset, so every row lines up and changed scores are listed first. The same approach compares run 0 and run 1 of one version to measure how much its answers vary between runs.

Logfire prompt sync

Select Logfire sync to link an Agent's instructions and input template to two Logfire managed variables, so a prompt can be edited in Logfire and pulled back, or written in Valcore and pushed. Nothing else syncs: model, capabilities, mappings, and output schema stay local.

Syncing is only for editing. Agent runs always use the local version and never read prompts from Logfire.

1. Create a key. In the Logfire project to sync with, create an API key with project:read_variables and project:write_variables. Add the Dataset scopes to the same key if you also publish Datasets with it.
2. Store it as the write key. Use Settings or valcore config set-logfire-write-key. The key decides the project; there is no project ID to enter.
3. Inspect and link. Inspect shows each field as in_sync, local_changed, remote_changed, conflict, remote_missing, or unsupported. Link with local text to publish it, or with remote text to create a new local version from Logfire.
4. Pull and push explicitly. Pull creates a new active Agent version from the remote text. Push saves a new variable version under a valcore_sync label and never moves production or any other label.

The variables are named from the Agent's ID: valcore_agent_<agent_id>_instructions and valcore_agent_<agent_id>_input_template. Valcore compares against each variable's latest saved version, not a label.

CLI reference

Agents are created in the app or imported from a spec file. Everything else works from the terminal against the same workspace. Agents, versions, and Datasets resolve by name or unique ID prefix.

# Is it alive? Sends literal text, bypassing the input template (cannot be saved)
valcore run agent support-agent -p "Where is order 1042?"

# Try ad-hoc inputs, or one existing row
valcore run agent support-agent \
  --input request="Where is my order?" --input order_status=shipped
valcore run agent support-agent --dataset support-cases --row 3

# Run the whole Dataset; staged until saved
valcore run agent support-agent --dataset support-cases --concurrency 8 --watch
valcore run agent support-agent --dataset support-cases --save

# Review, accept, or discard derivations
valcore agent derivation list --dataset support-cases
valcore agent derivation save support-agent/v2/0
valcore agent derivation discard <id-prefix>

# Score the saved responses
valcore run evaluator reply-quality --dataset support-cases \
  --derivation support-agent/v2/0 --kind eval

A single-row, ad-hoc, or -p run is temporary. Add --save with --dataset to keep a row or input run as a derivation. Derivation references take an ID prefix or agent/version/ordinal; valcore list derivations shows saved ones and agent derivation list also marks staged ones.

valcore agent prompt-sync status support-agent
valcore agent prompt-sync link support-agent --initial local
valcore agent prompt-sync pull support-agent --field instructions
valcore agent prompt-sync push support-agent --yes
valcore agent prompt-sync resolve support-agent --choice remote --field input_template
valcore agent prompt-sync unlink support-agent

Pull, push, and resolve preview the affected text and ask for confirmation; --yes skips the prompt. With --json, the preview goes to stderr so stdout holds only the result.