Getting started

1. Install Valcore

Choose either path. If you already work with a coding agent, hand it the Valcore skill and let it install and operate the tool for you. If you prefer the terminal, use Homebrew on macOS or Linux, or uv on any supported platform including Windows.

Install with your coding agent

Copy this prompt into Claude Code, Codex, or Cursor:

Inspect the skill source on GitHub →

Or install from the terminal

macOS or Linux with Homebrew:

brew install duncankmckinnon/tap/valcore
valcore --help

macOS, Linux, or Windows PowerShell with uv:

uv tool install valcore
valcore --help

2. Launch the workbench

valcore serve

Valcore opens the interface at http://127.0.0.1:8000. The interface, CLI, and installed agent skill all use the same local SQLite workspace.

valcore config set-logfire-token

3. Choose a model route

Open Settings → Model Selection. Under Local CLI default, select the coding agent already installed and authenticated on your machine:

This route is used for both generation and Experiment Runs, with no additional model key. The Default model card on Overview confirms the effective selection.

4. Generate the Dataset

Open Datasets → New dataset → Generate. Fill the form with the following values.

Name

retail-support-demo

Description

Customer-support conversations about retail orders. Each row contains one customer request and the support response to it. The Dataset should include straightforward resolutions, unresolved cases that need more work, and clearly hostile support responses.

Generation instructions

Generate realistic customer-support exchanges for an online retailer. Cover order status, late or missing deliveries, damaged items, returns, refunds, cancellations, address changes, and incorrect products. Vary the difficulty and writing style. Some rows should belong to the same multi-turn conversation and therefore repeat the same session value.

Assign labels using these definitions:
- resolved: the response correctly answers the request or provides a clear, sufficient next action.
- unresolved: the response is incomplete, incorrect, irrelevant, or leaves the customer without a workable next step.
- hostile: the response is rude, aggressive, blaming, threatening, mocking, or otherwise antagonistic toward the customer.

Make each generated response consistent with its assigned label.

Columns

session, request, response

Categorical labels

Keep Label kind set to Categorical, then add resolved, unresolved, and hostile. Their definitions are already included in the generation instructions above:

Size and label distribution

Enable the label mix, enter those percentages, confirm the preview shows 4 / 4 / 2, then select Generate.

5. Confirm the labels

Generated labels begin as suggestions, not human ground truth. Validation becomes available only after all 10 rows are labeled. Select Annotate from the Dataset's detail page to open its Label Set's Annotation Queue.

1. Read each request and response and check the suggested label.
2. Select Accept when it is correct, or choose a different label.
3. Continue until the Dataset summary shows 10 labeled and 0 unlabeled.

6. Generate the Agent Evaluator

From the Dataset detail page, select Generate evaluator. Keep Use this dataset's label space enabled, include request and response, and deselect session. Paste this into Criteria:

Create an Agent Evaluator for a retail customer-support chatbot. Evaluate the response in the context of the customer's request.

Return exactly one judgment from the Dataset's label space:
- resolved: the response correctly answers the request or gives a clear and sufficient next action.
- unresolved: the response is incomplete, incorrect, irrelevant, or does not give the customer a workable next step.
- hostile: the response is rude, aggressive, blaming, threatening, mocking, or otherwise antagonistic.

Use hostile whenever hostility is present, even if the response also contains useful information. Otherwise distinguish resolved from unresolved based on whether the customer's need was actually addressed. Include concise reasoning before the judgment.

Select Generate evaluator. Review the generated draft, give it a clear name such as retail-support-resolution, then save the first version. The generated Agent Evaluator inherits the Dataset's categorical labels, so the two are compatible by construction.

7. Run the experiment

Open Runs → New run and configure:

Keep the active version and default concurrency, then select Start. A validation Experiment Run executes the Agent Evaluator over all 10 rows and compares every judgment with the confirmed human label.

8. Read the results

Start with Accuracy and Cohen's κ, then inspect the per-label precision, recall, F1, and confusion matrix. Open disagreements to compare the Agent Evaluator's reasoning and judgment with the Dataset label.

If you enabled Logfire, follow the experiment link to see the native Pydantic Experiment Run and the trace for each case. Back in Valcore, create a new Agent Evaluator version, change one thing, and run it against the same Dataset—the beginning of a repeatable experimentation loop.

Next: measure your own agent. This demo scored responses that were generated with the Dataset. To score what your own agent says, define it under Agents or describe it and let a model draft it, run it over a Dataset of requests, and point an Agent Evaluator at the saved responses.