feat(ai, ai-optimize): adornment-driven tool context optimization #2148
tanstack-source added labelsfeature
- docs/advanced/context-optimization.md line 178Outdated@@ -0,0 +158,21 @@158+159+Drop an argument only when nothing later in the run needs to see it. A **terminal** write (the last thing the agent does with that file) is free to drop. Dropping writes an agent keeps working against mid-session can cost more than it saves, because the model re-checks work it can no longer see in its own history.160+161+## Code mode162+163+[Code mode](../code-mode/code-mode) already keeps most data out of the model context: the agent writes one `execute_typescript` program that reads files, transforms them, and writes them back, all inside the sandbox. The model never sees the file contents. But two things still ride in history and grow fast:164+165+The **program source** (`typescriptCode`). It ran once; after that the model needs the *result*, not the code it already submitted. This is the `write_file` story again.166+The **logs**. Agents `console.log` to debug ("let me print what I'm working with"), and one program can dump whole files into `logs`. Pure noise once the step is done.167+168+Adorn `execute_typescript` the same way you adorn any tool: drop the completed program, clip the logs, keep the returned result whole.169+170+```ts ignore171+import { chat } from '@tanstack/ai'172+import { createCodeMode } from '@tanstack/ai-code-mode'173+import { createQuickJSIsolateDriver } from '@tanstack/ai-isolate-quickjs'174+import { optimize } from '@tanstack/ai-optimize'175+176+const { tools, systemPrompt } = createCodeMode({177+ driver: createQuickJSIsolateDriver(),178+ tools: [readFile, writeFile],evals/harbor/chart.mjs line 23Outdated@@ -0,0 +3,21 @@3+// and fonts (ImageMagick's SVG path mangles both here).4+//5+// node chart.mjs # writes results/charts/*.png from results/*.jsonl6+//7+// Charts:8+// A terminal-bench input tokens per arm (the Strands-regime replication)9+// B optimize() effect vs baseline across three workloads (the honest story)10+// C code mode vs tool-calls input (code mode is the big lever)11+12+import { readFileSync, mkdirSync } from 'node:fs'13+import path from 'node:path'14+import { fileURLToPath } from 'node:url'15+16+const HERE = path.dirname(fileURLToPath(import.meta.url))17+const PW = path.join(18+ HERE,19+ '..',20+ '..',21+ 'node_modules',22+ '.pnpm',23+ 'playwright-core@1.57.0',evals/harbor/run-matrix.mjs lines 76 to 80Outdated@@ -0,0 +60,21 @@60+ if (taskFiles.length === 0) {61+ console.error(`no task files matched ${taskGlob}`)62+ process.exit(1)63+ }64+ mkdirSync(path.dirname(outFile), { recursive: true })65+ writeFileSync(outFile, '') // fresh file for this run66+67+ const total = taskFiles.length * arms.length * trials68+ let done = 069+ console.error(70+ `matrix: ${taskFiles.length} tasks x ${arms.length} arms x ${trials} trials = ${total} runs on ${model}`,71+ )72+73+ for (const tf of taskFiles) {74+ const task = JSON.parse(readFileSync(tf, 'utf8'))75+ for (const arm of arms) {76+ for (let trial = 0; trial < trials; trial++) {77+ done += 178+ const tag = `[${done}/${total}] ${task.id} ${arm} t${trial}`79+ try {80+ const rec = await runTask(task, { arm, model })evals/harbor/run-matrix.mjs line 96Outdated@@ -0,0 +76,21 @@76+ for (let trial = 0; trial < trials; trial++) {77+ done += 178+ const tag = `[${done}/${total}] ${task.id} ${arm} t${trial}`79+ try {80+ const rec = await runTask(task, { arm, model })81+ rec.trial = trial82+ appendFileSync(outFile, JSON.stringify(rec) + '\n')83+ console.error(84+ `${tag} -> ${rec.outcome} in=${rec.tokens.input} proj=${rec.truncation.projections} retr=${rec.truncation.retrievals}`,85+ )86+ } catch (err) {87+ const rec = {88+ taskId: task.id,89+ suite: task.suite,90+ arm,91+ model,92+ trial,93+ outcome: 'error',94+ failureClass: 'runner-error',95+ error: String(err?.message ?? err),96+ tokens: { input: 0, output: 0 },evals/harbor/suites/middle-critical/generate.mjs lines 69 to 70Outdated@@ -0,0 +50,21 @@50+ const lines = []51+ for (let i = 0; i < totalLines; i++) {52+ lines.push(53+ i === needleAt54+ ? `2026-11-12T10:30:00Z INFO billing: provisioning token issued value=${token}`55+ : noiseLine(rng, i),56+ )57+ }58+ const logPath = path.join(outDir, `fixture-${String(n).padStart(3, '0')}.log`)59+ writeFileSync(logPath, lines.join('\n') + '\n')60+61+ const prompt =62+ `Read the log file at ${path.resolve(logPath)} and find the single provisioning ` +63+ `token. Report ONLY the token value (the string after "value=").`64+65+ return {66+ id,67+ suite: 'middle-critical',68+ prompt,69+ cwd: path.resolve(outDir),70+ answer: token,evals/harbor/tanstack_ai_runner.mjs line 162Outdated@@ -0,0 +142,21 @@142+ )143+ const adapter = await resolveAdapter(model)144+145+ let finalText = ''146+ const stream = chat({147+ adapter,148+ model,149+ systemPrompts,150+ messages: [{ role: 'user', content: task.prompt }],151+ threadId: `eval-${task.id}`,152+ tools,153+ middleware: [makeUsageMiddleware(usage), ...middleware],154+ agentLoopStrategy: maxIterations(TURN_LIMIT),155+ })156+157+ for await (const chunk of stream) {158+ if (159+ chunk.type === 'TEXT_MESSAGE_CONTENT' &&160+ typeof chunk.delta === 'string'161+ )162+ finalText += chunk.deltaevals/harbor/tanstack_ai.py lines 36 to 41Outdated@@ -0,0 +21,21 @@21+import os22+from dataclasses import dataclass23+from pathlib import Path24+25+try:26+ # TODO(harbor): import path for the installed-agent base class in your pin.27+ from harbor.agents import BaseInstalledAgent # type: ignore28+except Exception: # pragma: no cover - lets this file import without Harbor present29+ class BaseInstalledAgent: # minimal stand-in for type-checking / docs30+ pass31+32+33+RUNNER = "tanstack_ai_runner.mjs"34+# Pin these to the exact versions whose numbers you publish (brief §7: "every35+# number names its implementation — language, package, version").36+PINNED_PACKAGES = [37+ "@tanstack/ai@<pin>",38+ "@tanstack/ai-optimize@<pin>",39+ "@tanstack/ai-anthropic@<pin>",40+ "@tanstack/ai-openai@<pin>",41+]evals/harbor/tanstack_ai.py lines 37 to 40Outdated@@ -0,0 +20,21 @@20+import json21+import os22+from dataclasses import dataclass23+from pathlib import Path24+25+try:26+ # TODO(harbor): import path for the installed-agent base class in your pin.27+ from harbor.agents import BaseInstalledAgent # type: ignore28+except Exception: # pragma: no cover - lets this file import without Harbor present29+ class BaseInstalledAgent: # minimal stand-in for type-checking / docs30+ pass31+32+33+RUNNER = "tanstack_ai_runner.mjs"34+# Pin these to the exact versions whose numbers you publish (brief §7: "every35+# number names its implementation — language, package, version").36+PINNED_PACKAGES = [37+ "@tanstack/ai@<pin>",38+ "@tanstack/ai-optimize@<pin>",39+ "@tanstack/ai-anthropic@<pin>",40+ "@tanstack/ai-openai@<pin>",evals/harbor/tools.mjs line 67Outdated@@ -0,0 +47,21 @@47+ exitCode: { type: 'number' },48+ },49+ },50+ }).server(async ({ command }) => run('bash', ['-lc', command], cwd))51+52+ const readFileTool = toolDefinition({53+ name: 'read_file',54+ description: 'Read a UTF-8 file and return its full content.',55+ inputSchema: {56+ type: 'object',57+ properties: { path: { type: 'string' } },58+ required: ['path'],59+ additionalProperties: false,60+ },61+ outputSchema: {62+ type: 'object',63+ properties: { path: { type: 'string' }, content: { type: 'string' } },64+ },65+ }).server(async ({ path }) => ({66+ path,67+ content: await readFile(path, 'utf8'),packages/ai-optimize/src/policies.ts line 129Outdated@@ -0,0 +109,21 @@109+ * surface — keep the history honest). */110+ args(tool: PolicyTool): ResolvedArgsPolicy | undefined {111+ if (tool.name === RETRIEVE_TOOL_NAME) return undefined112+ if (this.exclude.has(tool.name)) return undefined113+ if (tool.needsApproval) return undefined // forbidden in v1114+115+ const override = this.config.tools?.[tool.name]?.args116+ if (!override?.fields || Object.keys(override.fields).length === 0)117+ return undefined118+119+ const fields: Record<string, ResolvedFieldProjection> = {}120+ for (const [path, rule] of Object.entries(override.fields)) {121+ fields[path] = resolveFieldRule(rule)122+ }123+ return { fields }124+ }125+126+ /** True when any applicable (non-excluded) rule across the tool set is127+ * retrievable — i.e. `retrieve_context` should be injected. */128+ anyRetrievable(tools: Array<PolicyTool>): boolean {129+ for (const tool of tools) {packages/ai-optimize/src/project.ts line 51Outdated@@ -0,0 +31,21 @@31+ * Non-string values are projected against their `JSON.stringify` form and the32+ * leaf is replaced with the resulting preview *string*; the type change is33+ * acceptable in the projection only.34+ */35+export function truncateValue(input: {36+ raw: unknown37+ projection: ResolvedFieldProjection38+ fieldPath: string39+ toolName: string40+ toolCallId: string41+ stash: Stash42+}): { value: string; event: OptimizationEvent } | null {43+ const { raw, projection, fieldPath, toolName, toolCallId, stash } = input44+ const isText = typeof raw === 'string'45+ const s = isText ? (raw as string) : safeStringify(raw)46+ if (alreadyProjected(s)) return null47+48+ const origTokens = estimateChars(s.length, isText)49+ if (origTokens <= projection.aboveTokens) return null50+51+ const ref = makeRef(toolCallId, fieldPath)packages/ai-optimize/src/project.ts line 80Outdated@@ -0,0 +60,21 @@60+ if (projection.truncate === 'drop_with_ref') {61+ const value = stashed62+ ? `${DROPPED_PREFIX} ${fieldPath}, ~${origTokens} tokens. Full content: retrieve_context(ref="${ref}")]`63+ : `${DROPPED_PREFIX} ${fieldPath}, ~${origTokens} tokens.]`64+ // If the projected form would be >= the original, keep the original.65+ if (value.length >= s.length) return fail()66+ return {67+ value,68+ event: event(69+ toolName,70+ toolCallId,71+ fieldPath,72+ 'drop_with_ref',73+ origTokens,74+ 0,75+ stashed ? ref : undefined,76+ ),77+ }78+ }79+80+ const budget = projection.maxTokens * 4 // charspackages/ai-optimize/src/project.ts line 307Outdated@@ -0,0 +287,21 @@287+ const res = truncateValue({288+ raw: joined,289+ projection: rule,290+ fieldPath: '$',291+ toolName,292+ toolCallId,293+ stash,294+ })295+ if (!res) return { content, events: [] }296+ return {297+ content: [{ type: 'text', content: res.value }, ...nonText],298+ events: [res.event],299+ }300+ }301+302+ const explicitPaths = Object.keys(policy.fields).filter((p) => p !== '$')303+ const flatten = policy.default?.mode === 'flatten'304+305+ // Structured output (plain object): field surgery, unless flattening.306+ if (307+ !flatten &&packages/ai-optimize/src/runtime.ts lines 37 to 43Outdated@@ -0,0 +23,21 @@23+ middleware: ChatMiddleware24+ retrieveTool: ReturnType<typeof createRetrieveTool>25+ stash: Stash26+}27+28+/**29+ * Build a projection runtime over a shared stash. `optimize()` builds one of30+ * these internally; call `createContextRuntime()` directly for the31+ * author-adorned path (no `optimize()` config needed).32+ */33+export function createContextRuntime(34+ config: OptimizeConfig = {},35+): ContextRuntime {36+ const resolver = new PolicyResolver(config)37+ const stash = createStash()38+39+ // Run-scoped observability. Projection re-runs identically each iteration40+ // (cache honesty), so dedupe projected events by field; retrievals are41+ // distinct actions and always pass through. Accumulate for the log record.42+ const collected: Array<OptimizationEvent> = []43+ const emittedProjections = new Set<string>()packages/ai-optimize/src/stash.ts line 33Outdated@@ -0,0 +13,21 @@13+export interface StashEntry {14+ toolName: string15+ toolCallId: string16+ fieldPath: string17+ /** The original field value, verbatim — never a serialization. */18+ value: unknown19+}20+21+export type Stash = Map<string, StashEntry>22+23+export function createStash(): Stash {24+ return new Map<string, StashEntry>()25+}26+27+/** Deterministic ref: `ctx_<toolCallId>_<slug>` where slug is the field path28+ * with non-alphanumerics collapsed to `_` (`$` → `root`). */29+export function makeRef(toolCallId: string, fieldPath: string): string {30+ const slug =31+ fieldPath === '$'32+ ? 'root'33+ : fieldPath.replace(/[^a-zA-Z0-9]+/g, '_').replace(/^_+|_+$/g, '')
- docs/advanced/context-optimization.md line 148Outdated@@ -0,0 +128,21 @@128+ exclude: ['get_account_balance'],129+ onEvent: (e) => console.log(e),130+})131+```132+133+Then pass `opt.tools` and `opt.middleware` to `chat()`:134+135+```ts ignore136+chat({ adapter, messages, tools: opt.tools, middleware: [opt.middleware] })137+```138+139+`defaults.mode` picks how un-ruled content shrinks:140+141+`leaves` (default): each oversized string leaf in the result is truncated on its own. Short leaves like IDs and statuses fall under the threshold and pass through whole.142+`flatten`: the whole result is serialized and truncated as one string.143+144+### Trim the arguments of a completed call145+146+Trimming is not only for results. The **arguments** a tool was called with sit in history too, and some are large: the full file body passed to `write_file`, the whole program passed to a code interpreter. Once the call has run, the model rarely needs those arguments again, the result is what matters.147+148+`optimize()` can drop them. `dropWithRef()` on an argument field replaces it in history with a one-line marker:docs/advanced/context-optimization.md line 153Outdated@@ -0,0 +133,21 @@133+Then pass `opt.tools` and `opt.middleware` to `chat()`:134+135+```ts ignore136+chat({ adapter, messages, tools: opt.tools, middleware: [opt.middleware] })137+```138+139+`defaults.mode` picks how un-ruled content shrinks:140+141+`leaves` (default): each oversized string leaf in the result is truncated on its own. Short leaves like IDs and statuses fall under the threshold and pass through whole.142+`flatten`: the whole result is serialized and truncated as one string.143+144+### Trim the arguments of a completed call145+146+Trimming is not only for results. The **arguments** a tool was called with sit in history too, and some are large: the full file body passed to `write_file`, the whole program passed to a code interpreter. Once the call has run, the model rarely needs those arguments again, the result is what matters.147+148+`optimize()` can drop them. `dropWithRef()` on an argument field replaces it in history with a one-line marker:149+150+```ts ignore151+write_file: {152+ // The file is on disk now; the content argument no longer earns its place in history.153+ args: { fields: { content: dropWithRef() } },docs/advanced/context-optimization.md line 256Outdated@@ -0,0 +236,21 @@236+```237+[Truncated for context: content, ~2000 tokens, kept ~750. Full content: retrieve_context(ref="ctx_call_17_content")]238+239+<first 1800 chars>240+241+[... 5000 chars elided ...]242+243+<last 1200 chars>244+```245+246+The model calls `retrieve_context(ref="ctx_call_17_content")` and gets the full original field back. That result is itself exempt from projection, so it never gets re-truncated.247+248+Retrieval is also a tuning signal. If a field is retrieved on most turns, it was the wrong field to truncate. Watch for it through `onEvent` or the `tanstack.optimize` log record, which carries one entry per projection and per retrieval.249+250+The stash that backs retrieval is scoped to a single run. Within the run that truncated a field, retrieval is guaranteed. The canonical saved messages still hold every original, so projected history rebuilds identically on the next run.251+252+## When it helps, and when it hurts253+254+Trimming is not a free win. Whether it saves tokens or costs them depends on one thing: **is the field disposable, or load-bearing?** A build log the agent reads once is disposable. A file the agent is still editing is load-bearing. Truncate the first and you save almost all of it for free. Truncate the second and the model calls `retrieve_context` to pull it back, or works around the gap, and you spend *more* than you saved.255+256+In our evals the same `truncateHeadTail()` rule cut input ~96% on disposable shell logs and *raised* it ~99% on load-bearing file edits. Pick by the field, not by size.
- docs/advanced/context-optimization.md line 166Outdated@@ -0,0 +146,21 @@146+Trimming is not only for results. The **arguments** a tool was called with sit in history too, and some are large: the full file body passed to `write_file`, the whole program passed to a code interpreter. Once the call has run, the model rarely needs those arguments again, the result is what matters.147+148+`optimize()` can drop them. `dropWithRef()` on an argument field replaces it in history with a one-line marker:149+150+```ts ignore151+write_file: {152+ // The file is on disk now; the content argument no longer earns its place in history.153+ args: { fields: { content: dropWithRef() } },154+},155+```156+157+This changes the **completed call** only. The approval prompt, the live stream, and execution all saw the full argument; this trims the model-facing copy of a call that already finished. Two rules keep it safe:158+159+It is `optimize()`-only. A tool author cannot declare argument trimming, it is a consumer decision about history.160+It never applies to `needsApproval` tools. Their arguments are the exact thing a human approved, so they stay whole.161+162+`dropWithRef()` still honors the size gate: it uses the default 1500-token threshold (wrap it in `aboveTokens(n, ...)` to change that), so a small argument stays whole. A large one stays whole too if the marker would not be shorter than it.163+164+Two cases to watch:165+166+**Drop only when nothing later in the run reads it.** A terminal write (the last thing the agent does with that file) is free to drop. Dropping writes an agent keeps working against mid-session can cost more than it saves, because the model re-checks work it can no longer see in its own history.evals/harbor/run-matrix.mjs line 34Outdated@@ -0,0 +14,21 @@14+ writeFileSync,15+ mkdirSync,16+ appendFileSync,17+ globSync,18+} from 'node:fs'19+import path from 'node:path'20+import { runTask } from './tanstack_ai_runner.mjs'21+import { DEFAULT_RATES } from './report.mjs'22+23+function arg(flag, dflt) {24+ const i = process.argv.indexOf(flag)25+ return i === -1 ? dflt : process.argv[i + 1]26+}27+function hasFlag(flag) {28+ return process.argv.includes(flag)29+}30+31+/** Est. USD for one trial from the rate card ($/MTok). null if the rate is unknown. */32+function trialCost(rec) {33+ const r = DEFAULT_RATES[rec.model]34+ if (!r) return nullevals/harbor/tanstack_ai_runner.mjs line 161Outdated@@ -0,0 +141,21 @@141+ task.disableShell !== true,142+ )143+ const adapter = await resolveAdapter(model)144+145+ let finalText = ''146+ const stream = chat({147+ adapter,148+ model,149+ systemPrompts,150+ messages: [{ role: 'user', content: task.prompt }],151+ threadId: `eval-${task.id}`,152+ tools,153+ middleware: [makeUsageMiddleware(usage), ...middleware],154+ agentLoopStrategy: maxIterations(TURN_LIMIT),155+ })156+157+ // Catch a mid-stream failure so the record still carries the tokens the run158+ // consumed before it threw (the usage middleware has already counted them).159+ let runError160+ try {161+ for await (const chunk of stream) {
github-actions[bot](GitHub profile, opens in a new tab) assignedtombeckenham
github-actions[bot](GitHub profile, opens in a new tab) added labelswaiting-on: maintainer
tanstack-source added labelsfactory:needs-maintainer
tanstack-source reviewed
packages/ai-optimize/src/runtime.ts line 32Outdated@@ -0,0 +12,21 @@12+13+function safeParse(content: string): unknown {14+ try {15+ return JSON.parse(content)16+ } catch {17+ return content18+ }19+}20+21+/** A projection runtime: one stash, the middleware, and the retrieval tool. */22+export interface ContextRuntime {23+ middleware: ChatMiddleware24+ retrieveTool: ReturnType<typeof createRetrieveTool>25+ stash: Stash26+}27+28+/**29+ * Build a projection runtime over a shared stash. `optimize()` builds one of30+ * these internally; call `createContextRuntime()` directly for the31+ * author-optimized path (no `optimize()` config needed).32+ */evals/harbor/tanstack_ai_runner.mjs line 161Outdated@@ -0,0 +141,21 @@141+ task.disableShell !== true,142+ )143+ const adapter = await resolveAdapter(model)144+145+ let finalText = ''146+ const stream = chat({147+ adapter,148+ model,149+ systemPrompts,150+ messages: [{ role: 'user', content: task.prompt }],151+ threadId: `eval-${task.id}`,152+ tools,153+ middleware: [makeUsageMiddleware(usage), ...middleware],154+ agentLoopStrategy: maxIterations(TURN_LIMIT),155+ })156+157+ // Catch a mid-stream failure so the record still carries the tokens the run158+ // consumed before it threw (the usage middleware has already counted them).159+ let runError160+ try {161+ for await (const chunk of stream) {- evals/harbor/tanstack_ai_runner.mjs lines 165 to 166Outdated@@ -0,0 +146,21 @@146+ // Wall-clock: total time, and time-to-first-token (round-trip latency floor).147+ const startedAt = performance.now()148+ let firstTokenMs149+ const stream = chat({150+ adapter,151+ model,152+ systemPrompts,153+ messages: [{ role: 'user', content: task.prompt }],154+ threadId: `eval-${task.id}`,155+ tools,156+ middleware: [makeUsageMiddleware(usage), ...middleware],157+ agentLoopStrategy: maxIterations(TURN_LIMIT),158+ })159+160+ // Catch a mid-stream failure so the record still carries the tokens the run161+ // consumed before it threw (the usage middleware has already counted them).162+ let runError163+ try {164+ for await (const chunk of stream) {165+ if (firstTokenMs === undefined)166+ firstTokenMs = performance.now() - startedAt
github-actions[bot](GitHub profile, opens in a new tab) added labelsmerge-conflicts
github-actions[bot](GitHub profile, opens in a new tab) added labelswaiting-on: author
github-actions[bot](GitHub profile, opens in a new tab) removed labelswaiting-on: maintainer
tanstack-source added labelsfactory:needs-maintainer
Pushed0c9b4b4
Pushed39e1fd1
github-actions[bot](GitHub profile, opens in a new tab) added labelswaiting-on: maintainer
github-actions[bot](GitHub profile, opens in a new tab) removed labelswaiting-on: author
github-actions[bot](GitHub profile, opens in a new tab) removed labelsmerge-conflicts
Sign in to join this conversation.
🦋 Changeset detected#
Latest commit: 39e1fd1e9a6a007de7c963b9b09e43c957910316
The changes in this PR will be included in the next version bump.
This PR includes changesets to release 3 packages
Not sure what this means? Click here to learn what changesets are.
Click here if you're a maintainer who wants to add another changeset to this PR