Reference

CLI & automation

Use the terminal over the same local workspace as the app, move evaluator packages between environments, and turn validation into a release signal.

App and CLI

The visual workbench and the valcore command are two interfaces over the same SQLite workspace. The app is usually fastest for authoring and labeling; the CLI is better for repeatable runs, exports, automation, and agent-driven work.

The CLI opens the database directly. valcore serve does not need to be running for terminal commands to work.

# Start the visual workbench
valcore serve

# Address resources by name or a unique ID prefix
valcore run response-quality support-quality --watch

Command reference

`valcore serve`

Starts the API and workbench. Use --host, --port, or --no-browser to control how it launches.

`valcore list`

Lists evaluators, datasets, or runs. Add --json for structured output.

`valcore run`

Runs an evaluator version over a dataset. Important options are --version, --kind, --concurrency, --watch, --json, and --min-accuracy.

`valcore experiment`

Runs a validation through pydantic_evals.Dataset.evaluate and records it in Logfire experiments. It supports --version, --concurrency, and --json, but not watch or cancellation.

`valcore export` / `import`

Moves evaluators and datasets as runnable Python or portable JSON eval packages.

`valcore logfire`

pull queries traces, list shows hosted datasets, fetch imports one, and push publishes a local dataset.

`valcore config`

Sets credentials and defaults, prints the config path, or opens the config in your editor. config get masks stored secrets by default.

`valcore skills`

Installs, lists, or removes the bundled coding-agent skill. valcore version prints the installed version.

valcore list evaluators
valcore list datasets --json
valcore run my-evaluator my-dataset --kind validation --watch
valcore experiment my-evaluator my-dataset
valcore logfire fetch qa-set
valcore version

Portable eval packages

Evaluator and dataset exports support two formats. Code exports are standalone Python. JSON exports use a pydantic_ai AgentSpec and a pydantic_evals Dataset, with a small Valcore metadata block that preserves the prompt template, required columns, score field, and tool names.

# Standalone evaluator program
valcore export my-judge -o my-judge.py

# One portable evaluator + dataset bundle
valcore export my-judge --dataset my-data --format json -o package.json

# Separate agent and dataset JSON files
valcore export my-judge --dataset my-data --format json --split -o package.json

# Restore a JSON package to the local workspace
valcore import package.json

Running an exported dataset

from pydantic_evals import Dataset
from valcore_judge import ValcoreJudge

dataset = Dataset.from_file(
    "my-data.json",
    custom_evaluator_types=[ValcoreJudge],
)
report = dataset.evaluate_sync(task)

Using Valcore in CI

Use a fully labeled categorical dataset, emit JSON to stdout, and set a minimum accuracy. Progress is written to stderr, so redirecting stdout produces a clean result file.

valcore run my-evaluator my-dataset   --kind validation   --min-accuracy 0.90   --json > run.json

Exit 0

The run completed and met the threshold when one was supplied.

Exit 1

The run failed or Valcore reported a domain or configuration error.

Exit 2

Categorical accuracy was below --min-accuracy.

Agent skills

Valcore ships a skill that teaches coding agents its data model, compatibility rules, workflow, and CLI. Install it in the repository the agent works in:

valcore skills install                    # .agents/skills/
valcore skills install --claude           # .claude/skills/
valcore skills install --copilot          # .github/skills/
valcore skills install --all              # all three
valcore skills install --claude --global  # home-level install

Destination flags are additive. Existing identical copies are skipped; use --force to replace an edited copy or --symlink to follow upgrades to the packaged skill. Use valcore skills list to inspect installations and valcore skills uninstall to remove them.

Workspace and databases

By default, state lives under ~/.valcore: the configuration file, SQLite database, and server logs. The directory is created with mode 0700 and the config file with mode 0600.

~/.valcore/
  config.toml
  valcore.db
  logs/

Use VALCORE_HOME to relocate the whole workspace, or pass --db before the command group to select another SQLite database for one invocation:

VALCORE_HOME=./.valcore valcore list runs
valcore --db ./scratch.sqlite list datasets

See Configuration for credential precedence and the boundary between the source Logfire project and Valcore's own project.