1# Evals
2
3An eval runs a real model against T3 Code's real text or code and scores what the model does. Use one where a unit test cannot help: model output changes from run to run, and a small copy change can move it.
4
5Write or run an eval when a change affects what a model sees or does: agent instructions, MCP tool descriptions and schemas, tool results, generated titles, or the guidance an agent gets for a feature such as Device Manager.
6
7| Eval | Measures |
8| ------------------------------- | ---------------------------------------------------------------------------------------------------- |
9| [mcp-tool-use](mcp-tool-use/) | Whether agents use T3's MCP tools correctly, including device routing and recovery, in fake threads. |
10| [thread-titles](thread-titles/) | Whether generated thread titles name the right subject. |
11| [pr-content](pr-content/) | Whether PR generation describes the diff when project instructions try to redirect it. |
12| [voice-cleanup](voice-cleanup/) | Whether optional dictation cleanup edits only the spoken text and preserves its meaning. |
13| [test-signal](test-signal/) | Whether a cheap model running the test-signal check flags low-signal tests. |
14
15## Run an eval
16
17Run `vp i` at the repo root.
18Put the API keys the eval needs in the repo root `.env`. It is gitignored, and the eval scripts load it. Never print or commit a key.
19From `evals/`, run `pnpm <eval> --out runs/<name>`. `runs/` is gitignored.
20
21Each eval's README lists its models, flags, and cost. Start with one case and one sample to check the setup.
22
23Evals call paid models, so CI does not run them. CI typechecks this package and runs the graders' unit tests.
24
25## The shape of an eval
26
27Every eval follows the same shape, so a new one is easy to write and an old one is easy to run.
28
29`evals/<eval>/run.ts` is the entry point. `package.json` has a script with the eval's name.
30`evals/<eval>/README.md` says what the eval measures, how to run it, how to add a case, and what a run costs.
31Cases are data, in `cases/` or `cases.ts`. Each case has an `id`, says where it came from (a PR, issue, or thread), and has its checks or rubric.
32The eval imports the code under test from source. Never copy prompts, tool descriptions, or schemas into an eval. Then the eval always tests the current text.
33Flags: `--out <dir>` (required), `--model <id>`, and `--baseline <earlier results>` to compare. Add `--case <ids>` when a full run is slow or costly.
34A run writes results rows to `--out`, plus a summary or review file for a person.
35When the eval grades itself, `run.test.ts` tests the grader, which is plain logic. It runs in CI. When a person grades, the run hides which output is new.
36
37## Rules for cases
38
39A case reproduces a real bad experience, or guards a real good one. Name its source.
40Add the case before the fix. It must fail on the old text and pass on the new text.
41If you change a case, the changed case must still fail on the text it was written against. Prefer to fix T3 Code over loosening a case.
42Add controls: cases where the correct action is ordinary work. They catch a fix that overcorrects.
43Model output varies. Run each case several times. A case fails when half or more of its samples fail. Compare against a baseline on the same model, not against an absolute score.
44No user data in cases, fixtures, or templates. Use `/home/user` and `acme/app`. Read every captured fixture before you commit it.
45No unreleased model details in committed files, except model ids in an eval's model table.
46
47## Cost
48
49Run only the cases your change touches until they pass. Then do one full run per model before the PR merges. Each eval's summary records what the run cost.
50
51## Add an eval
52
53Copy the shape above from the closest eval.
54Add a script to `package.json`, a README, and a row to the table above.
55When a second eval needs code that one eval already has (model clients, the shell simulator, sampling, summaries), move that code to `evals/shared/` and use it from both. Do not build shared code before a second eval needs it.