TanStack AI E2E Tests#
End-to-end tests for TanStack AI using Playwright and aimock for deterministic LLM mocking.
Architecture: Playwright drives a TanStack Start app (testing/e2e/) which routes requests through provider adapters pointing at aimock. Fixtures define mock responses. No real API keys needed. All scenarios (including tool execution flows) use aimock fixtures. Tests run in parallel with per-test X-Test-Id isolation.
Providers tested: openai, anthropic, gemini, ollama, groq, grok, openrouter, openrouter-responses, bedrock, bedrock-responses, openai-compatible, mistral, byteplus, elevenlabs
Claude Code (
@tanstack/ai-claude-code) is excluded from the standard matrix. It's a harness adapter that spawns the Claude Code runtime as a subprocess, so aimock's per-testX-Test-Idheader isolation can't be injected into its requests. It's covered by unit tests in the package plus a gated live smoke test intests/claude-code.spec.ts— run it withCLAUDE_CODE_E2E=1and anANTHROPIC_API_KEY(or a localclaude login).
What's tested#
Provider-coverage tests#
Each test iterates over supported providers using providersFor('feature'):
| Feature | Providers | Spec file |
|---|---|---|
| chat | 7 | tests/chat.spec.ts |
| one-shot-text | 7 | tests/one-shot-text.spec.ts |
| multi-turn | 7 | tests/multi-turn.spec.ts |
| structured-output | 7 | tests/structured-output.spec.ts |
| structured-output-stream | 4 | tests/structured-output-stream.spec.ts |
| tool-calling | 7 | tests/tool-calling.spec.ts |
| parallel-tool-calls | 6 | tests/parallel-tool-calls.spec.ts |
| tool-approval | 6 | tests/tool-approval.spec.ts |
| text-tool-text | 6 | tests/text-tool-text.spec.ts |
| agentic-structured | 7 | tests/agentic-structured.spec.ts |
| reasoning | 3 | tests/reasoning.spec.ts |
| multimodal-image | 5 | tests/multimodal-image.spec.ts |
| multimodal-structured | 5 | tests/multimodal-structured.spec.ts |
| summarize | 6 | tests/summarize.spec.ts |
| summarize-stream | 6 | tests/summarize-stream.spec.ts |
| image-gen | 7 | tests/image-gen.spec.ts |
| image-to-image | 1 | tests/image-to-image.spec.ts |
| image-to-video | 1 | tests/image-to-video.spec.ts |
| tts | 3 | tests/tts.spec.ts |
| transcription | 3 | tests/transcription.spec.ts |
| audio-gen | 1 | tests/audio-gen.spec.ts |
Tools-test page#
Deterministic scenarios covering tool execution flows:
| Spec file | Tests | What it covers |
|---|---|---|
tests/tools-test/chat-flow.spec.ts | 5 | Text-only, server tool, client tool, tool call structure |
tests/tools-test/approval-flow.spec.ts | 6 | Approve, deny, sequential, parallel, mixed flows |
tests/tools-test/client-tool.spec.ts | 5 | Single, sequential, parallel, triple, server+client |
tests/tools-test/race-conditions.spec.ts | 8 | No blocking, no deadlocks, timing, mixed flows |
tests/tools-test/server-client-sequence.spec.ts | 5 | Server→client, parallel server, ordering |
Interrupt playground#
Deterministic coverage of every AG-UI interrupt shape, ported from the
ts-react-chat wildlife example (route /interrupts-test, scenarios in
src/lib/interrupt-scenario-tools.ts). Each scenario isolates one interrupt
behavior — server/client × boolean, shared payload, branch payload, edited
args, plus a generic (non-tool) interrupt and batch sets — and is exercised
across allow / deny / cancel via both per-item (interrupt.resolveInterrupt /
interrupt.cancel) and root batch (resolveInterrupts / cancelInterrupts)
resolution.
| Spec file | Tests | What it covers |
|---|---|---|
tests/interrupts-test/per-item.spec.ts | 24 | Approve / deny / cancel each single scenario via the bound interrupt |
tests/interrupts-test/generic.spec.ts | 2 | Generic interrupt resolve-with-payload and cancel |
tests/interrupts-test/batch.spec.ts | 5 | Root approve-all / deny-all / cancel-all, per-item, and mixed resolver |
Notes:
- Server tool results are not surfaced to the client as message parts, so
server execution is asserted via the
approval-respondedstate; server edited-args are verified through aemitCustomEventecho. - A shared
approvalSchemarequires a payload on the deny decision too, so those scenarios carry adenyPayload.
Advanced feature tests#
| Spec file | What it covers |
|---|---|
tests/abort.spec.ts | Stop button cancels in-flight generation |
tests/lazy-tools.spec.ts | __lazy__tool__discovery__ discovers and uses lazy tools |
tests/custom-events.spec.ts | Server tool emitCustomEvent received by client |
tests/middleware.spec.ts | onChunk transform, onBeforeToolCall skip |
tests/error-handling.spec.ts | Server RUN_ERROR, aimock error fixture |
tests/tool-error.spec.ts | Tool throws error, agentic loop continues |
Durable / detachable run tests#
| Spec file | What it covers |
|---|---|
tests/delivery-durability.spec.ts | Transport layer: offset-tagged log, Last-Event-ID reconnect, second-tab join, SSE + NDJSON |
tests/persistence-durability.spec.ts | Client layer: a browser refresh restores the conversation and any pending interrupt |
tests/sandbox-durability.spec.ts | Sandbox instances: a second run resumes the persisted sandbox |
tests/durable-takeover.spec.ts | Takeover with log alignment, detach-on-disconnect, out-of-band cancel in both bands, cancel-vs-disconnect divergence, and the superseded-driver epoch fence |
durable-takeover.spec.ts drives /api/durable-takeover, a provider-free harness
whose fake agent journal advances only on an explicit ?action=tick, so every
disconnect happens at a known point in the stream. Two things to know before
extending it:
- The disconnect is injected server-side (
?action=drop, a plain abort with no reason) — a genuine precondition, not a workaround. A clientfetchabort does not propagate to the server through this app's dev server, so a test cannot otherwise produce a mid-stream socket close the run observes; the spec documents this in theSseStreamcomments (around lines 141-146). ?action=seedis also used, and is still legitimate: it lets a test reach the cross-host takeover case — attaching from a process that never held the original connection — which a single test process cannot otherwise produce. A detached run's log now stays open for takeover:RunDetachedCapabilityandpackages/ai/src/delivery-detach.tscarry the run's detach verdict to the sink, which skips both the syntheticRUN_ERRORanddurability.close()for an unrequested disconnect on a detachable run. That path is pinned by'a real disconnect, then an attach, continues the stream'— a normal passing test, not atest.fail()case.
What to add for your change#
E2E coverage is mandatory for every feature, bug fix, or behavior change (see the root CLAUDE.md). Mirror of that table, kept in sync here:
| Change type | What E2E test to add |
|---|---|
| New provider adapter | Add provider to feature-support.ts + test-matrix.ts. Existing feature tests auto-run. |
| New feature (e.g., new generation type) | Add feature to types, feature config, support matrix. Create fixture + spec file. |
| Bug fix in chat/streaming | Add a test case to chat.spec.ts or tools-test/ that reproduces the bug. |
| Tool system change | Add scenario to tools-test-scenarios.ts + test in tools-test/ specs. |
| Middleware change | Add test to middleware.spec.ts with appropriate scenario. |
| Client-side change (useChat, etc.) | Add test covering the observable behavior change. |
| Durable/detachable run + takeover | Add a spec that disconnects mid-stream, reconnects with the same runId, and asserts the resumed stream continues from the aligned offset instead of restarting the agent. |
| Out-of-band cancel vs. plain disconnect | Add a case distinguishing requestRunCancel/wasCancelRequested (durable cancel — record ends 'aborted') from an unrequested disconnect on a detachable run (record stays 'running', resumable). |
| New run-store backend | Run runPersistenceConformance from packages/ai-persistence/src/testkit/ against it; it now pins undefined vs. explicit false on cancelRequested and absent vs. explicit undefined on update. |
| New sandbox provider | Run the journal conformance suite from @tanstack/ai-sandbox/testkit against it, using makeFakeShellSpawn (also from that testkit) to exercise the journal/claim/driver seam without a real sandbox. |
1. Quick Start#
# Install dependencies
pnpm install
# Run all E2E tests
pnpm --filter @tanstack/ai-e2e test:e2e
# Run with Playwright UI (useful for debugging)
pnpm --filter @tanstack/ai-e2e test:e2e:ui
# Run a specific spec
pnpm --filter @tanstack/ai-e2e test:e2e -- --grep "openai -- chat"
# Run only the tools-test specs
pnpm --filter @tanstack/ai-e2e test:e2e -- tests/tools-test/
2. Recording a New Fixture#
# 1. Set your API keys
export OPENAI_API_KEY=sk-...
export ANTHROPIC_API_KEY=sk-ant-...
# 2. Start the app in record mode
pnpm --filter @tanstack/ai-e2e record
# 3. Navigate to the feature: http://localhost:3010/openai/tool-calling
# 4. Chat — aimock proxies to the real API and saves the fixture
# 5. Find it in testing/e2e/fixtures/recorded/
# 6. Stop the dev server (Ctrl+C)
3. Organizing the Recorded Fixture#
Move from recorded/ to the appropriate feature directory:
mv fixtures/recorded/openai-*.json fixtures/tool-calling/my-new-scenario.json
Clean up the fixture:
- Simplify the
matchfield — use a unique[prefix]userMessage - Verify the
response— check content, toolCalls, or reasoning fields - Remove provider-specific artifacts — fixtures should be provider-agnostic
Important: aimock uses substring matching for userMessage. Always use a unique [prefix] to prevent collisions:
{
"fixtures": [
{
"match": { "userMessage": "[myfeature] describe the guitar" },
"response": {
"content": "The Fender Stratocaster is a versatile electric guitar..."
}
}
]
}
Existing prefixes: [chat], [oneshot], [reasoning], [multiturn-1], [multiturn-2], [toolcall], [parallel], [approval], [approval-deny], [text-tool-text], [structured], [structured-stream], [structured-stream-abort], [agentic], [mmimage], [mmstruct], [summarize], [imagegen], [tts], [transcription], [abort-test], [error-test].
4. Writing a Test#
Tests import from ./fixtures to get the testId and aimockPort for parallel isolation:
import { test, expect } from './fixtures'
import {
sendMessage,
waitForResponse,
getLastAssistantMessage,
featureUrl,
} from './helpers'
import { providersFor } from './test-matrix'
for (const provider of providersFor('my-feature')) {
test.describe(`${provider} — my-feature`, () => {
test('does the thing', async ({ page, testId, aimockPort }) => {
await page.goto(featureUrl(provider, 'my-feature', testId, aimockPort))
await sendMessage(page, '[myfeature] describe the guitar')
await waitForResponse(page)
const response = await getLastAssistantMessage(page)
expect(response).toContain('Stratocaster')
})
})
}
For tool tests that use sequenceIndex, use waitForAssistantText — the agentic loop produces multiple responses:
await sendMessage(page, '[toolcall] what guitars do you have in stock')
await waitForResponse(page)
const toolCalls = await getToolCalls(page)
expect(toolCalls[0].name).toBe('getGuitars')
// Wait for text response AFTER tool execution
await waitForAssistantText(page, 'Fender Stratocaster')
5. Adding a New Feature#
- Add to
src/lib/types.ts—Featureunion +ALL_FEATURESarray - Add to
src/lib/features.ts— define tools, modelOptions - Add to
src/lib/feature-support.ts— mark which providers support it - Add to
tests/test-matrix.ts— mirror the support matrix - Create fixture —
fixtures/my-new-feature/basic.jsonwith[prefix]userMessage - Create test spec —
tests/my-new-feature.spec.tsusingprovidersFor()
6. Adding a New Provider#
- Add adapter factory to
src/lib/providers.ts— withdefaultHeadersforX-Test-Id - Add to
src/lib/feature-support.ts— mark which features it supports - Add to
tests/test-matrix.ts— mirror the support matrix - No fixture changes needed — aimock translates to correct wire format
Bedrock Converse coverage gap#
The bedrock and bedrock-responses providers in this matrix use createBedrockText with a baseURL pointing at aimock — they speak Bedrock's OpenAI-compatible endpoint, which aimock's OpenAI replay handles fine.
The default bedrock-converse adapter (introduced later) uses @aws-sdk/client-bedrock-runtime and speaks AWS's binary event-stream (vnd.amazon.eventstream) Converse protocol, which @copilotkit/aimock does not currently mock. Adding bedrock-converse to the live matrix would fail without a Converse-capable aimock provider.
Coverage today: the Converse translation layer (message converter, tool converter, stream processor, structured output, adapter) is covered by unit tests in packages/ai-bedrock/tests/converse/ (64 tests). The OpenAI-compatible bedrock and bedrock-responses entries remain in the E2E matrix as-is.
Follow-up: a Bedrock/Converse provider will be added to aimock to close this gap and enable full E2E coverage of the Converse path.
BytePlus (Ark) path handling and record-mode gap#
BytePlus splits across two products, and the E2E wiring reflects that split:
- Ark (chat, Seedream image, Seedance video) serves everything under
/api/v3, so the chat, image and video adapters getbaseURL: <mock>/api/v3. - Seed Speech (TTS, ASR) is a separate host with a separate key, and its adapters append
/api/v3/...themselves — so they get the bare<mock>base.
Chat and image need no mock changes. aimock's compat-path normalizer rewrites any non-/v1/, non-/v2/ path ending in a known OpenAI suffix to /v1/<suffix>, so /api/v3/chat/completions and /api/v3/images/generations land on the native handlers and the provider-agnostic fixtures apply unchanged. Seedream's request body differs from OpenAI's (size as a 1K/2K token, no n, watermark), but aimock's image handler only reads model and prompt and answers with the { created, data: [...] } envelope Seedream also returns.
Three endpoints have no aimock equivalent and are mounted in global-setup.ts, all on the /api/v3 prefix — each returns false for paths it doesn't own so chat and image still fall through:
byteplusSeedanceMount()—POST/GET /contents/generations/tasks[/{id}]byteplusTTSMount()—POST /tts/createbyteplusASRMount()—POST /auc/bigmodel/recognize/flash
Record-mode gap: aimock's RecordProviderKey union has no byteplus entry, so pnpm record can't proxy ark.ap-southeast.bytepluses.com or the Seed Speech host to capture real fixtures — the same situation as the Bedrock Converse gap above. Chat features reuse the existing provider-agnostic fixtures (aimock matches on message content, not provider); the media endpoints are served by the hand-written mounts listed above. Update those mounts by hand if the wire shapes change, and cross-check against the adapter unit tests in packages/ai-byteplus/tests/.
SDK baseURL notes:
- OpenAI, Grok:
LLMOCK_OPENAI(with/v1) +defaultHeaders - Groq:
LLMOCK_BASE(SDK appends/openai/v1/internally) +defaultHeaders - Anthropic:
LLMOCK_BASE+defaultHeaders - Gemini:
httpOptions: { baseUrl: LLMOCK_BASE, headers } - Ollama:
{ host: LLMOCK_BASE, headers }(config object) - OpenRouter:
serverURLwith?testId=query param (SDK doesn't support headers) - BytePlus:
LLMOCK_BASE + /api/v3+defaultHeadersfor Ark (chat, image, video); bareLLMOCK_BASE+defaultHeadersfor Seed Speech (TTS, ASR)
7. Adding a Tool Test Scenario#
- Create fixture —
fixtures/tools-test/my-scenario.jsonwith[my-scenario] run testuserMessage - Add tool definitions to
src/lib/tools-test-tools.tsif new tools needed - Update
getToolsForScenario— map scenario to tools - Add to
SCENARIO_LIST— for the UI dropdown - Create test in
tests/tools-test/— use helpers fromtests/tools-test/helpers.ts
8. Adding a Middleware Test#
- Create fixture —
fixtures/middleware-test/my-scenario.json - Add middleware to
src/routes/api.middleware-test.ts— new mode - Add test to
tests/middleware.spec.ts
9. Fixture Matching Tips#
userMessage— substring matching, always use unique[prefix]stringssequenceIndex— tracks request count per match pattern perX-Test-Id. Use for multi-step tool flows.tool— matches when the model calls a specific toolmodel— matches a specific model name (avoid — breaks provider-agnosticism)- Fixtures are matched in order — first match wins
10. Troubleshooting#
- Test times out: Check
userMessagein fixture exactly matchessendMessage()(including[prefix]) - Wrong fixture matched: Overlapping
userMessagesubstrings. Make prefixes more specific. - Tool test wrong response:
sequenceIndexcounter issue. Each test gets a uniqueX-Test-Idfor isolation — make sure you importtestfrom./fixtures, not@playwright/test - Fixture works for OpenAI but not Anthropic: Remove
modelfrom match field - Port 4010 already in use: Kill stale aimock process. aimock starts in
globalSetupand lives for the entire test run.
Architecture#
testing/e2e/
├── src/
│ ├── routes/
│ │ ├── $provider/$feature.tsx # Dynamic chat UI per provider+feature
│ │ ├── tools-test.tsx # Tools test page with event tracking
│ │ ├── middleware-test.tsx # Middleware test page
│ │ ├── api.chat.ts # Chat endpoint → aimock
│ │ ├── api.tools-test.ts # Tools endpoint → aimock
│ │ └── api.middleware-test.ts # Middleware endpoint → aimock + middleware
│ ├── lib/
│ │ ├── providers.ts # Provider → adapter factory (X-Test-Id headers)
│ │ ├── features.ts # Feature → config (tools, modelOptions)
│ │ ├── feature-support.ts # Provider × feature support matrix
│ │ └── tools-test-tools.ts # Server + client tool definitions
│ └── components/
│ └── ChatUI.tsx # Chat interface with stop button + image rendering
├── fixtures/ # aimock fixture JSON files (organized by feature)
├── test-assets/ # Images for multimodal tests
├── global-setup.ts # Starts aimock on port 4010
├── global-teardown.ts # Stops aimock
├── tests/
│ ├── fixtures.ts # Playwright fixture (testId + aimockPort)
│ ├── helpers.ts # sendMessage, waitForResponse, featureUrl, etc.
│ ├── test-matrix.ts # providersFor() — which providers support which features
│ ├── *.spec.ts # Provider-coverage + advanced feature tests
│ └── tools-test/ # Tool execution flow tests
│ ├── helpers.ts # selectScenario, runTest, getMetadata, etc.
│ └── *.spec.ts # Tool scenario tests
└── playwright.config.ts # fullyParallel: true, retries: 2, globalSetup/Teardown