Open History to see the last commit
1# TanStack AI context-optimization evals
2
3Measures whether adornment-driven context projection (`@tanstack/ai-optimize`)
4cuts spend without cutting accuracy, using the [Harbor](https://github.com/harbor-framework/harbor)
5runner (Apache-2.0) for comparability with the Strands numbers.
6
7**These are TypeScript numbers and say so.** No TypeScript-vs-anything claim is
8made (see the brief, Part B non-goals).
9
10## Layout
11
12```
13evals/
14 README.md this file
15 harbor/
16 tanstack_ai.py Harbor BaseInstalledAgent adapter (E1)
17 tanstack_ai_runner.mjs the Node agent Harbor runs in the task container (E2)
18 arms.mjs the four arms (baseline / global / adorned / universal)
19 tools.mjs shell + file-editor tools, implemented over child_process/fs
20 report.mjs per-trial JSONL -> per-model x arm cost table (E4)
21 run-matrix.mjs suite x arms x tasks x trials -> results/*.jsonl
22 chart.mjs results/*.jsonl -> results/charts/*.png (headless Chromium)
23 run-config.example.json maxTrials + hard USD cap (budget discipline)
24 results/
25 RESULTS.md findings writeup (tables + charts + recommendations)
26 *.jsonl raw per-trial records
27 charts/*.png rendered bar charts
28 suites/
29 middle-critical/generate.mjs held-out synthetic suite generator (E3)
30 agentic-coding/generate.mjs read a large file, transform, write it back (single write)
31 multi-write/generate.mjs read+edit+write SEVERAL files in one session (write-arg drop compounds)
32 code-mode/generate.mjs same work through execute_typescript; agent logs for debugging
33 terminal-bench/generate.mjs run noisy shell commands; answer in the tail (disposable logs, the Strands regime)
34 disposable-read/generate.mjs read a large log via read_file (no shell), answer in the tail — the disposable-log
35 test that survives a capable model (Sonnet pipes around the shell variant)
36```
37
38## Arms (selected by `ARM` env, nothing else changes)
39
40Two modes. The tool-call arms give the agent `read_file` / `write_file` (and
41`run_shell` when enabled). The code-mode arms (`cm-*`) give it one tool,
42`execute_typescript`, that runs its program in a QuickJS sandbox with the file
43tools exposed as `external_*` functions.
44
45| Arm | Mode | Middleware |
46| ------------- | ---------- | -------------------------------------------------------------------------------------------------------------------------- |
47| `baseline` | tool-calls | none |
48| `global` | tool-calls | `optimize()` `defaults.mode: 'flatten'`, 1500/750 head_tail (Strands replica) |
49| `adorned` | tool-calls | leaves mode + per-tool rules: drop completed `write_file.content`, clip `run_shell.stdout`, keep `read_file.content` whole |
50| `universal` | tool-calls | leaves-mode defaults only |
51| `cm-baseline` | code-mode | none |
52| `cm-adorned` | code-mode | drop completed `execute_typescript.typescriptCode`, clip `logs` |
53| `cm-global` | code-mode | flatten the whole `execute_typescript` result (over-truncation control) |
54
55System prompt and turn limits are held constant across arms.
56
57## What each suite is for
58
59- **middle-critical / agentic-coding** — a single large read/write. Proves the
60 plumbing but the write-arg saving only lands on one call, so at small `n` the
61 agent's step-count variance dominates.
62- **multi-write** — the write-arg drop (D3) compounds: each completed
63 `write_file` content lingers for the rest of the session until `adorned` drops
64 it. Also the clearest place to see that full-file rewrites grow context
65 quadratically.
66- **code-mode** — code mode already keeps file contents in the sandbox, so the
67 model context is tiny. This suite isolates what `optimize()` adds _on top of_
68 code mode: clipping the debug `logs` an agent prints. `cm-baseline` carries
69 every dump; `cm-adorned` clips them and keeps the returned result.
70- **terminal-bench** — the disposable-log regime Strands' headline comes from:
71 noisy shell commands whose answer is a summary line in the tail. Size-based
72 truncation is nearly free here (~96% input cut, no retrievals) on a model naive
73 enough to dump output. A capable model (Sonnet) pipes around it (`| tail`), so
74 the big logs never enter context — use `disposable-read` there instead.
75- **disposable-read** — the same disposable-log idea forced through `read_file`
76 with shell disabled, so a capable model can't pipe around it. On Sonnet,
77 truncating the read cut input ~94% and cost ~14x, 4/4 solved. See
78 `results/RESULTS.md`.
79
80## Model ladder (never Fable-anchored)
81
82Report per model, never pooled. A frontier model is the best case for surviving
83lossy context, so safety measured there is an upper bound.
84
85- frontier: Fable 5 (`--fable` flag required, it is the most expensive cell)
86- production-default: a Sonnet-class model and a GPT equivalent
87- low-cost / open tier
88
89## Budget discipline
90
91Every run carries `maxTrials` and a hard USD cap (`run-config.json`). The runner
92aborts when the cap is hit and reports partial results **as partial**. Dev
93iterations use small task subsets and the cheaper tiers. Fable runs need
94`--fable`.
95
96## Running locally (smoke test, no Harbor)
97
98The runner is self-contained and can run one task directly for a smoke test.
99Keys come from `.env.local` at the repo root (`ANTHROPIC_API_KEY`,
100`OPENAI_API_KEY`, ...).
101
102```bash
103# one task, cheapest tier, adorned arm
104ARM=adorned MODEL=claude-sonnet-4-6 \
105 node evals/harbor/tanstack_ai_runner.mjs --smoke evals/harbor/suites/middle-critical/out/task-000.json
106```
107
108Generate a suite first, then run a matrix across arms and trials:
109
110```bash
111# generate tasks
112node evals/harbor/suites/multi-write/generate.mjs --count 3 --files 3 --out evals/harbor/suites/multi-write/out
113node evals/harbor/suites/code-mode/generate.mjs --count 3 --files 3 --out evals/harbor/suites/code-mode/out
114
115# run suite x arms x trials -> one JSONL (sequential; bounded by TURN_LIMIT)
116cd evals/harbor
117TURN_LIMIT=18 node run-matrix.mjs \
118 --tasks "suites/multi-write/out/task-*.json" \
119 --arms baseline,adorned,universal,global,cm-baseline,cm-adorned \
120 --model gemini-3.8-flash --trials 3 --out results/multi-write.jsonl \
121 --max-usd 5
122```
123
124`--max-usd <n>` stops before the next request once the estimated spend (from the
125`report.mjs` rate card) passes the cap, keeping the partial results. A Fable
126model requires `--fable` to confirm the spend.
127
128Summarize per-trial JSONL into the headline table:
129
130```bash
131node evals/harbor/report.mjs evals/harbor/results/*.jsonl
132```
133
134## Status
135
136E1 (adapter) is modeled on Harbor's `langgraph.py` + `langgraphjs_runner.mjs`
137precedent. The Harbor container-install contract details are marked `TODO(harbor)`
138where they need to be checked against the Harbor version you pin. The runner
139(E2), arms, suite generator (E3), and report (E4) are runnable today.