1# Devices
2
3The environment server owns simulators and emulators the way it owns
4terminals: discovery, streaming, and agent access all run there, and every
5client reaches them through the environment connection. This is what makes the
6Device panel work over Tailscale and T3 Connect, including when an SSH host runs the devices.
7
8## Two external tools, one seam
9
10[expo-device-hub](../../apps/server/src/device/LocalDeviceHost.ts) streams and
11[agent-device](../../apps/server/src/device/AgentDeviceShim.ts) drives. Each is
12npm-installed at a pinned version into the T3 home after its matching Device
13panel consent step. Manual setup installs and starts only expo-device-hub;
14agent-device remains absent and stopped until agent access is granted. Both run
15with the server's Node; `npx` would make the first `device_open` after a reboot
16depend on the registry. The hub is a supervised child rather than an imported
17middleware because serve-sim loads private CoreSimulator frameworks through a
18native addon, and a crash there must not take the server down.
19
20Everything platform-specific sits behind
21[`DeviceHost`](../../apps/server/src/device/DeviceHost.ts). The service lists,
22boots, and shuts down devices through the host's device operations, and the
23proxy and the MCP tools only see a hub origin and an agent-device endpoint.
24Hosts that run expo-device-hub share one hub-backed implementation of those
25operations, so a host whose devices live elsewhere only replaces them.
26SSH hosts forward both endpoints to server loopback. Every proxied request
27also carries the host id; device ids alone are not unique across hosts.
28
29## EAS Simulator hosts are billed sessions
30
31An [EAS host](../../apps/server/src/device/EasDeviceHost.ts) has no machine:
32each opened device is an EAS Simulator session that bills the Expo account
33until it stops. Every session is created with the host's run and idle limits,
34the host refuses to start past its session limit, and closing the host stops
35its sessions. Sessions it starts are recorded under the state directory and
36stopped by the next server if this one dies; sessions other tools started on
37the same account are never touched.
38
39Each session brings its own preview host (serve-sim, or serve-emu for Android)
40and agent-device daemon, with their own tokens. The proxy asks the host to
41route each request (`DeviceHostReady.upstream`) and adds the session token server-side, so
42clients never hold it. Agent config files are written per device
43(`DeviceOperations.agentEndpoint`), which is why `device_open` opens the
44device before targeting it. Device settings and tools run `simctl` or `adb` on
45the device's machine, which EAS does not expose, so they are unavailable there.
46
47## Simlock hosts lease through the agent-device front
48
49A [Simlock host](../../apps/server/src/device/SimlockDeviceHost.ts) (alpha) opens a
50device by running T3's own agent-device CLI against the gateway's agent-device
51front (`connect proxy`, then `open`), not with Simlock's lease HTTP API:
52agent-device cannot adopt a lease it did not allocate. The lease is bound to
53that one agent-device session and to the directory `open` ran in, so
54`agentEndpoint` hands every thread the same session, a `stateDir` holding its
55connection record, and no `--udid`/`--serial` (the gateway refuses selectors).
56Installs work only from that directory; `device_open` names it.
57
58Simlock serves a lease's screen and input to its holder in serve-sim's formats
59on both platforms: AVCC envelopes at `/v1/leases/<id>/stream` and serve-sim's
60input messages at `/v1/leases/<id>/input`, both with the agent token. Clients
61therefore treat a Simlock Android lease like iOS (`serveSimOnly` in the stream
62client), and `upstream` maps serve-sim's `stream.avcc` and `helper/ws` paths to
63those routes. Simlock has no MJPEG, screenshot, or host command routes, so the
64panel offers only input there.
65Leases are renewed while tracked and released at the host's age limit; one a
66crashed server left behind ends on its own one TTL later, so there is no ledger.
67
68## The hub is never exposed
69
70serve-sim has a shell-exec route whose token is readable from its own
71unauthenticated `/api`, and serve-emu's action routes have no auth at all. The
72hub binds loopback and the only way in is the
73[proxy](../../apps/server/src/device/DeviceHubProxy.ts), which allowlists the
74stream, config, and screenshot routes and authenticates every request as an
75environment session. `<img>` and `WebSocket` cannot carry headers, so the proxy
76authenticates like the `/ws` upgrade: cookie, or a short-lived `wsTicket` that
77bearer and DPoP clients mint over authenticated HTTP. The ticket is stripped
78before the request reaches the hub.
79
80Stream responses carry `Cache-Control: no-transform`; the compression
81middleware would otherwise buffer an MJPEG body that never ends. Browsers paint an
82MJPEG part only once the next part begins and the hub writes one only when the
83screen changes, so the proxy
84[re-sends the last part](../../apps/server/src/device/mjpegStillFrame.ts) after a
85short quiet period. In browser dev, the Vite proxy must forward WebSocket
86upgrades for `/api`, not only `/ws`.
87
88## Device settings never go through the hub
89
90serve-sim's preview drives its Tools panel by sending shell commands over that
91same exec channel. Proxying it, even allowlisted, would hand any environment
92session arbitrary command execution on the host, so T3 does not. The
93[`device.action`](../../apps/server/src/device/DeviceActions.ts) RPC runs the
94underlying `simctl`, `adb`, and serve-sim helper binaries itself through
95`DeviceHostReady.run`, one typed action per control, and returns the settings
96it reads back. The proxy allowlist grows only with read routes (accessibility
97tree, foreground app, event log) and refuses non-GET methods everywhere except
98screenshot capture and stream tuning.
99
100## Agents drive through the CLI
101
102The `device_*` toolkit is deliberately four tools: list, open, screenshot, and
103close. Driving happens through the `agent-device` CLI, which has the semantic
104snapshot model agents need and stays current with its own releases. T3 prepends
105a shim directory to the provider's PATH. The CLI installs on the environment
106server even when that server cannot run simulators. Hosts start on demand.
107
108That environment is fixed when the provider subprocess spawns, so
109[`prepareMcpSession`](../../apps/server/src/orchestration-v2/ProviderSessionManager.ts)
110starts agent-device only when device support and agent access have both been
111enabled, the session has the `device` capability, and the machine can run at
112least one platform. Starting it later from `device_open` would leave the
113already-running agent without the CLI.
114
115How to drive a device is returned from `device_open`, not kept in an
116always-loaded prompt or skill: it costs nothing in threads that never open a
117device and cannot drift from the pinned CLI version. `device_open` returns a
118launcher bound to its agent-device session, so agents need no target flags.
119The launcher leaves out the flags a command refuses (`cdp` talks to Metro, not
120to the device), refuses the commands that would replace T3's connection, and
121checks that connection first: agent-device reports a sandboxed shell and a
122dropped tunnel alike as an unreachable daemon, so the launcher names which one it
123is. Ending a session (`device_close`, the Device panel, a shutdown) also closes
124its agent-device session, which quits the app that session opened and releases the CLI's claim on the device.
125
126Agents choose between these tools and ssh before they open anything, so
127sessions that hold `device` get a short block from
128[`orchestrationInstructions.ts`](../../packages/provider-core/src/server/orchestrationInstructions.ts)
129through their adapter's instruction channel. It points simulator and emulator UI
130work on local and SSH hosts at the tools; `simctl`, `adb` and ssh stay
131available for what the tools do not cover. Pi and Muse do not send it yet.
132Claude's native subagents never see the session's system prompt, so a
133`SubagentStart` hook passes them the block, where the runtime runs hooks and the
134subagent's context is not isolated.
135
136## The viewer decodes both vendored protocols
137
138The hub vendors two streaming servers with different wire formats. iOS video is
139an HTTP body of AVCC envelopes decoded with WebCodecs, with input on a separate
140binary WebSocket; Android multiplexes SEMU-framed H.264 and JSON gestures over
141one WebSocket. [`DeviceStreamView.tsx`](../../apps/web/src/components/device/DeviceStreamView.tsx)
142speaks both so one panel covers both platforms.
143
144WebCodecs is secure-context only, so the viewer probes `isConfigSupported` and falls back to
145MJPEG on iOS when decoding is unavailable. Android's SEMU socket has no MJPEG;
146there the panel reports that it cannot decode.
147
148serve-sim can also be locked to WebRTC, as on EAS Simulator, and then refuses
149AVCC with a 409 naming the transport. The viewer still asks for AVCC first, so
150hubs that serve it pay nothing, and switches on that 409, or when its MJPEG
151image fails in a browser that has `RTCPeerConnection`. Offers are not trickled:
152serve-sim takes one offer with every candidate. The video element only feeds
153the same canvas sink, so the panel and the mobile WebView keep one surface.
154Android asks serve-emu's `/api` before opening its video socket and streams
155WebRTC when it is advertised (EAS Simulator), keeping a separate `video=0`
156socket for input so input can reconnect without restarting video.