Skip to content
Source
Sign inSign up
TanStack/ai
Open History to see the last commit

README.md

139 lines7.5 KB

TanStack AI context-optimization evals

Measures whether adornment-driven context projection (@tanstack/ai-optimize) cuts spend without cutting accuracy, using the Harbor runner (Apache-2.0) for comparability with the Strands numbers.

These are TypeScript numbers and say so. No TypeScript-vs-anything claim is made (see the brief, Part B non-goals).

Layout

evals/
  README.md                         this file
  harbor/
    tanstack_ai.py                  Harbor BaseInstalledAgent adapter (E1)
    tanstack_ai_runner.mjs          the Node agent Harbor runs in the task container (E2)
    arms.mjs                        the four arms (baseline / global / adorned / universal)
    tools.mjs                       shell + file-editor tools, implemented over child_process/fs
    report.mjs                      per-trial JSONL -> per-model x arm cost table (E4)
    run-matrix.mjs                  suite x arms x tasks x trials -> results/*.jsonl
    chart.mjs                       results/*.jsonl -> results/charts/*.png (headless Chromium)
    run-config.example.json         maxTrials + hard USD cap (budget discipline)
    results/
      RESULTS.md                    findings writeup (tables + charts + recommendations)
      *.jsonl                       raw per-trial records
      charts/*.png                  rendered bar charts
    suites/
      middle-critical/generate.mjs  held-out synthetic suite generator (E3)
      agentic-coding/generate.mjs   read a large file, transform, write it back (single write)
      multi-write/generate.mjs      read+edit+write SEVERAL files in one session (write-arg drop compounds)
      code-mode/generate.mjs        same work through execute_typescript; agent logs for debugging
      terminal-bench/generate.mjs   run noisy shell commands; answer in the tail (disposable logs, the Strands regime)
      disposable-read/generate.mjs  read a large log via read_file (no shell), answer in the tail — the disposable-log
                                    test that survives a capable model (Sonnet pipes around the shell variant)

Arms (selected by ARM env, nothing else changes)

Two modes. The tool-call arms give the agent read_file / write_file (and run_shell when enabled). The code-mode arms (cm-*) give it one tool, execute_typescript, that runs its program in a QuickJS sandbox with the file tools exposed as external_* functions.

ArmModeMiddleware
baselinetool-callsnone
globaltool-callsoptimize() defaults.mode: 'flatten', 1500/750 head_tail (Strands replica)
adornedtool-callsleaves mode + per-tool rules: drop completed write_file.content, clip run_shell.stdout, keep read_file.content whole
universaltool-callsleaves-mode defaults only
cm-baselinecode-modenone
cm-adornedcode-modedrop completed execute_typescript.typescriptCode, clip logs
cm-globalcode-modeflatten the whole execute_typescript result (over-truncation control)

System prompt and turn limits are held constant across arms.

What each suite is for

  • middle-critical / agentic-coding — a single large read/write. Proves the plumbing but the write-arg saving only lands on one call, so at small n the agent's step-count variance dominates.
  • multi-write — the write-arg drop (D3) compounds: each completed write_file content lingers for the rest of the session until adorned drops it. Also the clearest place to see that full-file rewrites grow context quadratically.
  • code-mode — code mode already keeps file contents in the sandbox, so the model context is tiny. This suite isolates what optimize() adds on top of code mode: clipping the debug logs an agent prints. cm-baseline carries every dump; cm-adorned clips them and keeps the returned result.
  • terminal-bench — the disposable-log regime Strands' headline comes from: noisy shell commands whose answer is a summary line in the tail. Size-based truncation is nearly free here (~96% input cut, no retrievals) on a model naive enough to dump output. A capable model (Sonnet) pipes around it (| tail), so the big logs never enter context — use disposable-read there instead.
  • disposable-read — the same disposable-log idea forced through read_file with shell disabled, so a capable model can't pipe around it. On Sonnet, truncating the read cut input ~94% and cost ~14x, 4/4 solved. See results/RESULTS.md.

Model ladder (never Fable-anchored)

Report per model, never pooled. A frontier model is the best case for surviving lossy context, so safety measured there is an upper bound.

  • frontier: Fable 5 (--fable flag required, it is the most expensive cell)
  • production-default: a Sonnet-class model and a GPT equivalent
  • low-cost / open tier

Budget discipline

Every run carries maxTrials and a hard USD cap (run-config.json). The runner aborts when the cap is hit and reports partial results as partial. Dev iterations use small task subsets and the cheaper tiers. Fable runs need --fable.

Running locally (smoke test, no Harbor)

The runner is self-contained and can run one task directly for a smoke test. Keys come from .env.local at the repo root (ANTHROPIC_API_KEY, OPENAI_API_KEY, ...).

# one task, cheapest tier, adorned arm
ARM=adorned MODEL=claude-sonnet-4-6 \
  node evals/harbor/tanstack_ai_runner.mjs --smoke evals/harbor/suites/middle-critical/out/task-000.json

Generate a suite first, then run a matrix across arms and trials:

# generate tasks
node evals/harbor/suites/multi-write/generate.mjs --count 3 --files 3 --out evals/harbor/suites/multi-write/out
node evals/harbor/suites/code-mode/generate.mjs  --count 3 --files 3 --out evals/harbor/suites/code-mode/out

# run suite x arms x trials -> one JSONL (sequential; bounded by TURN_LIMIT)
cd evals/harbor
TURN_LIMIT=18 node run-matrix.mjs \
  --tasks "suites/multi-write/out/task-*.json" \
  --arms baseline,adorned,universal,global,cm-baseline,cm-adorned \
  --model gemini-3.8-flash --trials 3 --out results/multi-write.jsonl \
  --max-usd 5

--max-usd <n> stops before the next request once the estimated spend (from the report.mjs rate card) passes the cap, keeping the partial results. A Fable model requires --fable to confirm the spend.

Summarize per-trial JSONL into the headline table:

node evals/harbor/report.mjs evals/harbor/results/*.jsonl

Status

E1 (adapter) is modeled on Harbor's langgraph.py + langgraphjs_runner.mjs precedent. The Harbor container-install contract details are marked TODO(harbor) where they need to be checked against the Harbor version you pin. The runner (E2), arms, suite generator (E3), and report (E4) are runnable today.