1# Scenarios
2
3A scenario is an [e2e](https://e2e.com/docs) test that shows one behavior in the real app: a real
4server on its own T3 home, real HTTP and WebSocket, and a real browser on the web client or a
5pooled iPhone simulator running the mobile app. The
6dev replay agent stands in for the model, so a run is free and repeatable. Use one as the proof
7for a bug fix: it fails on the base commit and passes on the fix.
8
9Scenarios live in [`scripts/scenarios`](../../scripts/scenarios) as `*.e2e.ts`. e2e is the
10runner; its docs ship in `scripts/node_modules/e2e/docs` and `e2e guide` prints them.
11
12## Write one
13
15import { expect } from "e2e";
16
17import { test, WEB } from "./t3.ts";
18
19test(
20 "Tab moves from the add-project Git URL field to its submit button",
21 WEB,
22 async ({ t3, app, screen, browser }) => {
23 await t3.project("demo");
24 await t3.open();
25
26 await screen.getByRole("button", "Add project").tap();
27 await screen.getByText("Git URL").tap();
28 await screen.getByRole("combobox").fill("https://github.com/pingdotgg/t3code.git");
29 const submit = screen.getByRole("button", "Continue (Enter)");
30 await expect(submit).toBeEnabled();
31
32 await browser.keyboard.press("Tab");
33 await app.screenshot("after-tab");
34 await expect(submit).toBeFocused();
35 },
36);
38
39`screen`, `browser`, `app`, and `expect` are e2e's own. [`t3.ts`](../../scripts/scenarios/t3.ts)
40adds one fixture for what e2e cannot know about T3 Code:
41
42`t3.project(title, files)` creates a git repository and adds it as a project.
43`t3.thread({ project, title, messages })` starts a thread and plays one replayed turn per
44 message before returning, so the thread has real history. `model` picks the recording.
45`t3.open(path)` opens the app, already signed in.
46
47State is created through the app's own CLI, so it is real events and works on a base commit
48whose schema differs. New threads default to the replay agent's `showcase` recording: type a
49message and send it. To play something else, add a recording to the
50[replay agent](development.md#replay-provider).
51
52Write one test per file: a proof runs the file and needs exactly one test in it.
53
54Take an `app.screenshot("label")` where the bug shows. The proof pairs base and head captures by
55label. e2e adds a capture at the point of failure itself.
56
57Write only `expect` assertions for the thing the bug breaks. A proof counts a failed assertion as
58the bug. A control that cannot be found, or an app that does not start, counts as a scenario that
59could not run.
60
61Keep `agent.act` and `agent.assert` out of scenarios: they call a model, and no model is
62configured.
63
64## Run
65
67vp run scenarios # all of them
68cd scripts && pnpm exec e2e run scenarios/replay-turn-streams.e2e.ts --headed
70
71e2e starts the dev stack through [`start-app.ts`](../../scripts/scenarios/start-app.ts) on a free
72port, runs the tests, and stops it. Results land in `scripts/.e2e`: `report.json`, `summary.md`,
73and per test a `trace.md`, its screenshots, and a video. `vp run scenarios --target web` runs one
74target; flags go straight after the script name, without `--`.
75
76The first run downloads e2e's browser (about 120 MB). To fetch it ahead of time, such as in a
77machine image, run this from the repository root after `vp i`:
78
80node scripts/node_modules/playwright-core/cli.js install --only-shell chromium ffmpeg
82
83e2e opens the web app at `127.0.0.1`, and the start script tells the web dev server to bind that
84address. It does not use `localhost`, which some machines resolve to `::1` first.
85
86`e2e run --help` lists the rest: `--grep`, `--watch`, `--last-failed`, `--changed-since`.
87
88Run one at a time per checkout: a run recreates the checkout's scenario T3 home and its `.e2e`
89output. Use a second worktree for a second run.
90
91## Prove a fix
92
94vp run scenarios:prove scripts/scenarios/add-project-tab-reaches-submit.e2e.ts \
95 --base e509db3a2f^ --head e509db3a2f --upload-command "ghdrop upload"
97
98[`prove-scenario.ts`](../../scripts/prove-scenario.ts) runs `e2e run` for that scenario on both
99commits and compares the two reports. It prints, and writes to `scripts/.e2e/prove/proof.json`,
100`failsOnBase`, `passesOnHead`, `baseSha`, `headSha`, `verdict`, each side's `steps`, `captures`,
101`videos`, and `markdown`: a table with base and head captures side by side, each labelled with
102its commit, ready for a pull request description. `--upload-command` names a command that takes
103the capture paths and prints one URL per line. `--head` defaults to `HEAD`.
104
105It exits 0 only when the scenario fails on base and passes on head, 1 when it ran and did not
106prove the fix, and 2 when it could not run. A scenario that passes on base does not reproduce the
107bug, and the verdict says so.
108
109The scenario and e2e always come from your checkout. Only the app comes from the commit under
110test: a temporary worktree beside your checkout, installed with `vp i` and removed afterwards. A
111commit that is your clean `HEAD` runs in place. So a new scenario can prove a fix against a base
112that predates it.
113
114## iOS
115
116An iOS scenario imports from [`t3-mobile.ts`](../../scripts/scenarios/t3-mobile.ts) and passes
117`IOS` where a web one passes `WEB`:
118
120import { expect } from "e2e";
121
122import { IOS, test } from "./t3-mobile.ts";
123
124test(
125 "The iOS app lists a seeded thread and opens its history",
126 IOS,
127 async ({ t3, app, screen }) => {
128 const project = await t3.project("demo");
129 await t3.thread({ project, title: "Earlier work", messages: ["first", "second"] });
130 await t3.open();
131
132 const thread = screen.getByText("Earlier work", { exact: false }).first();
133 await expect(thread).toBeVisible({ timeout: 120_000 });
134 await thread.tap();
135 await expect(screen.getByText("second fixture turn complete", { exact: false })).toBeVisible();
136 },
137);
139
140`t3.project` and `t3.thread` are the same. `t3.open()` installs the dev client for the checkout's
141native fingerprint, loads it from Metro, and pairs it with the scenario's server. `device` is
142e2e's [device fixture](https://e2e.com/docs/mobile#the-device-fixture) for system actions.
143
144The run leases an iPhone 18 Pro from the Simlock pool, so it needs `SIMLOCK_AGENT_DEVICE_URL`
145and `AGENT_DEVICE_DAEMON_AUTH_TOKEN`, as `mobile:e2e` does, and a gateway on Simlock 0.7.0 or
146later. A web-only run leases nothing. The device reaches this machine over the tailnet on ports
14713990 (server) and 8190 (Metro); set `T3_SCENARIO_SERVER_PORT` and `T3_SCENARIO_METRO_PORT` when
148another run on the machine holds them. An iOS run takes two to three minutes, most of it the
149device and the first bundle. `e2e run --target web` or `--target ios` runs one side.
150
151`scenarios:prove` works the same for an iOS scenario. Each commit gets the dev client built for
152its own native code.
153
154## Other tiers
155
156**Android.** Add a target with `mobile({ platform: "android", device: simlock("Pixel 9") })`
157 and open the dev client with the Android form of the Expo link. Not built.
158**Live provider.** Point the default model in `start-app.ts` at a real provider's cheapest
159 model. Assertions on exact reply text would not hold.
160
161## In CI
162
163No job runs scenarios, and none should run all of them on every pull request. e2e can select the
164ones a branch touches with `e2e run --changed-since origin/main` once
165[test filters](https://e2e.com/docs/filters) are configured, and `@e2e-dev/github` posts the
166result as a pull request comment.
167
168## What a scenario cannot cover
169
170Desktop shell behavior (menus, windows, IPC), and Android.
171Provider-specific behavior. The replay agent speaks generic ACP.
172State no command can create, such as a run stopped by a usage limit.
173Timing. Replay streaming is simulated, so a scenario cannot show a performance regression.