Agent judges

Evaluators

Turn evaluation criteria into repeatable agent judges with explicit inputs, score contracts, harness capabilities, and immutable run history.

Evaluator panel tour

The Evaluators list is the catalog of reusable agent judges. Open one to work on its versions; each version is a complete, attributable runtime configuration rather than only a prompt.

Valcore interface map Evaluator version editor
EVALUATOR

Response quality

Run
v4-quality-passACTIVEFROZENNew version
JUDGMENTInstructions

Assess accuracy, relevance, and completeness…

Required columns
questionanswer
OUTPUT CONTRACTScore kind

categorical

Score field

quality

passreviewfail
⌄ Capabilities & tools☑ CodeMode☑ FileSystem☐ Shell
Version stateJudgment and inputsOutput contractCapabilities & tools
  1. Start with the version bar. It identifies the selected version, whether it is active, and whether a completed run has frozen it. Use New version to branch safely.
  2. Define the judgment and its inputs. Instructions describe the standard; the prompt template inserts row values with braces such as {answer}. Every placeholder must appear in Required columns.
  3. Make the output machine-checkable. Add structured output fields, choose categorical or numeric scoring, then select which compatible field is the score.
  4. Expand capabilities only when needed. Capabilities and tools are collapsed by default because they widen what the agent judge can do. Their configuration is saved with the version.

The action bar connects this object to the rest of the workflow: Generate dataset creates compatible cases, Run moves to the run launcher, and Export produces runnable Python or a portable JSON package.

Creating evaluators

Manual creation

Each evaluator runs as an agent judge. Name it, write the judging prompt, choose a Gateway model, and list every required dataset column. Ask the agent judge to explain its reasoning before returning a score.

Select New evaluator → From scratch to create the container, then complete the first version editor from top to bottom: Identity, Judgment, Inputs, Output contract, and—only when needed—Capabilities & tools. Valcore validates the draft continuously and the footer names the next field blocking Save.

Numeric labels

Use a bounded numeric score for graded qualities. Set the minimum and maximum explicitly and explain what the ends—and useful points between them—mean.

Categorical labels

Use a small fixed label set for clear decisions. Define every label in the prompt; three to five well-separated categories is often easier to validate than a large ambiguous set.

Generating evaluators

From prompts and data contracts

Provide the task, required columns, column notes, and score schema. Valcore generates an editable evaluator draft that conforms to those inputs.

From existing datasets

Seed generation from a dataset to inherit its columns and label schema. Review the draft prompt carefully before creating the first version.

Versioning evaluators

The active version is used whenever a run does not name another. A version becomes frozen after it is used by a run, preserving the prompt, model, output contract, tools, and capabilities behind that result.

To iterate, select the frozen version and choose New version. The editor begins with a copy of the selected configuration, so you can make one deliberate change and compare it against the original. The version diff shows the exact configuration changes rather than relying on names or timestamps.

Before a run, Valcore checks that evaluator columns are a subset of dataset columns, label kinds match, and categorical label sets match exactly.

Evaluation harness capabilities

Every evaluator runs as an agent judge inside the Pydantic AI harness. Capabilities give that agent more ways to investigate a case than reading the dataset row alone, and are opt-in per evaluator version.

Configure capabilities in the UI

  1. Open Evaluators and select the evaluator you want to edit.
  2. Open an editable version—or create a new version if the current one is frozen.
  3. Expand Capabilities & tools near the bottom of the version editor.
  4. Enable only the capabilities the agent judge needs, configure any revealed settings, and save the version.

CodeMode

Lets the agent judge solve multi-step work in a code-driven execution loop.

SubAgents

Lets the agent judge delegate bounded parts of a complex evaluation to sub-agents.

Planning

Gives the agent judge a structured planning workflow for longer evaluation tasks.

FileSystem

Lets the agent judge read from a rooted directory. After enabling it, set the root dir shown in the UI.

Shell

Lets the agent judge run commands from an explicit allow-list. Configure comma-separated allowed commands and a default timeout.

CLI reference

Capability authoring happens in the UI. Once saved, the CLI uses the same versioned configuration automatically when you run or export the evaluator.

# Find the evaluator and its active version
valcore list evaluators

# Run it with its saved capabilities
valcore run <evaluator> <dataset> --watch

# Export a specific version as runnable Python
valcore export <evaluator> --version <version> -o evaluator.py

# Move a complete evaluator package between workspaces
valcore export <evaluator> --format json -o evaluator.json
valcore import evaluator.json

Names and unique ID prefixes are accepted anywhere an evaluator, version, or dataset is requested.