1# T3 Connect
2
3T3 Connect uses Clerk for cloud identity. The relay manages environment links,
4credentials for reaching environments, and managed tunnel allocations. After
5bootstrap, clients send application traffic through the environment's tunnel
6hostname; the relay Worker does not proxy their HTTP or WebSocket sessions.
7The one exception is automation webhooks: the relay forwards
8`/v1/hooks/:endpointKey/:hookId/:token` to the environment's tunnel so
9senders get a stable URL. `endpointKey` is the 16-hex hash suffix of the
10managed tunnel's name, not the environment id. It keeps bodies and tokens out
11of its traces and leaves token and signature verification to the environment
12([forwarder](../../infra/relay/src/hooks/HookForwarder.ts)).
13
14By default the forwarder stores nothing. An environment can opt in to having
15the relay hold requests while it is offline
16(`hold_webhooks_while_offline` on its links). Only then does the relay store the
17raw request, including the hook token in the path, in a Durable Object with
18SQLite storage. There is one object per managed link, keyed by its endpoint
19key: each account that links the environment gets its own held requests,
20order, and caps. The object pushes held requests back through the tunnel from
21its alarm, oldest first within each hook, and backs off while the environment
22stays away. A hook that answers busy is skipped for the rest of that run, so
23other hooks on the link can overtake it. When
24the tunnel reconnects, the environment asks the relay to deliver right away to
25each of its links. Requests are deleted once the environment answers, after 24
26hours, when their link is unlinked, or when the environment turns holding off,
27which clears every link it proved. Both clears are best effort: unlinking logs a
28failed clear and continues, and turning holding off stops at the first link it
29cannot clear, so those requests wait for the 24-hour expiry. The relay still
30never checks the token;
31delivery goes through the same environment route.
32Every forward carries `x-t3-relay-delivery-id`, so a request that reached the
33environment before a timeout and is delivered again later runs once
34([inbox object](../../infra/relay/src/hooks/HookInboxObject.ts)).
35
36A Durable Object, not Postgres or Queues, because held requests are write-once,
37read-once bodies of up to 1 MiB that need per-link order, caps, and retry
38timing. Queues cap messages at 128 KB and cannot hold one link's requests back
39while its environment is away.
40
41Clerk, deployment, and native authentication setup live in the
42[Connect setup runbook](../operations/connect-setup.md).
43
44## The relay is a trusted broker
45
46An authenticated cloud user still needs an active environment link. The relay
47asks that environment to mint a one-time bootstrap credential bound to the
48client's DPoP key. The client exchanges it directly with the environment for an
49[environment session](./environment-auth.md). The relay never receives that
50session token, and possessing the bootstrap credential alone does not permit
51redeeming it without the client's private key.
52
53Both sides authenticate this exchange. The environment accepts only bounded,
54replay-guarded relay proofs for its own identity, linked user, and requested
55operation. Signed environment responses bind the result to the request nonce;
56mint responses also bind the credential to the client proof key. The relay
57verifies those bindings before returning a credential. This prevents a different
58process behind the tunnel from impersonating the linked environment. The checks
59meet in the
60[environment link service](../../apps/server/src/cloud/CloudLink.ts) and
61[relay connector](../../infra/relay/src/environments/EnvironmentConnector.ts).
62
63The relay holds the signing authority for mint requests. DPoP protects an honest
64exchange from credential reuse; it does not make a compromised relay signing
65key harmless. Keep that trust assumption explicit when changing the protocol.
66
67Managed tunnels expose only a validated loopback HTTP origin. Link proof checks
68reject forwarded authority headers, and the relay resolves endpoints from its
69own managed allocations rather than a caller-supplied URL. Health and mint
70requests must not follow redirects. These restrictions keep endpoint discovery
71from turning into arbitrary relay egress or exposing another service on the
72environment host.
73
74## A link outlives a connector process
75
76CLI authorization, desired exposure, and a running connector have different
77lifetimes. Linking can record intent while the server is stopped. Startup
78reconciles that intent. CLI logout removes the stored cloud credential and
79disables exposure without uninstalling the environment's background service.
80
81Managed allocations belong to a user/environment pair. Provisioning checkpoints
82external tunnel and DNS resources so retries can reconcile partial work. A
83normal shutdown of a CLI-managed link releases its tunnel to avoid paying for
84an idle resource, retaining the hostname reservation for the next startup.
85It also retains the allocation record so the environment remains "offline"
86rather than becoming "not authorized".
87
88Two cases must retain the tunnel across shutdown. A link installed through a
89client has no startup provisioning path and depends on its stored connector
90token. An update handoff immediately starts a replacement server, and replacing
91the tunnel would add routing propagation delay to every update. These exceptions
92belong to [shutdown handling](../../apps/server/src/cloud/CloudLink.ts).
93
94Release and unlink claim the allocation generation before deleting external
95resources. A delayed cleanup must not delete a tunnel reused by a concurrent
96restart or relink. Unlink commits authorization revocation before external
97teardown, because a database failure must leave the active link usable. Failed
98teardown retains enough state to retry. See the
99[managed endpoint lifecycle](../../infra/relay/src/environments/ManagedEndpointProvider.ts).
100
101## Idle tunnels are reclaimed and recovered
102
103Cloudflare bills a tunnel whether or not a connector is attached, so a laptop
104that sleeps with a linked environment leaves a paid tunnel behind. The relay's
105five-minute maintenance job can reclaim those tunnels. `RELAY_TUNNEL_CLEANUP_MODE`
106selects `off`, `dry-run`, or `enabled`, with `off` as the default. The mode is
107read at deploy time, so changing it means a relay deploy, not a variable flip.
108A candidate is a same-stage tunnel that Cloudflare reports down for at least
109five minutes, or one that never connected and is at least an hour old. The
110longer grace for never-connected tunnels covers a pairing still in progress.
111
112Cleanup deletes only tunnels whose host has registered recovery. Allocations
113without recovery registration belong to hosts that cannot replace a deleted
114tunnel and are left alone. Allocations with no recorded tunnel ID, or a
115different tunnel ID, are skipped because a provision may own them. A tunnel with
116no allocation row at all is counted as `skippedOrphan` and never deleted: there
117is no row to lock, so a relink that adopts it by name could race the delete.
118Clear those by hand. Each sweep is bounded: at most ten list requests, 100 deletions, a
119two-minute deadline, and an early stop on a Cloudflare rate limit. Each sweep
120starts one budget further along the candidate list, so a block of deletes that
121keep failing cannot starve the tunnels listed after them. See the
122[reaper](../../infra/relay/src/environments/ManagedEndpointReaper.ts).
123
124A host registers recovery at startup by sending its tunnel ID and loopback
125origin with a short-lived signature from the environment key. Registration
126touches Cloudflare only when the local host or port changed, and once per
127existing allocation on the first registration after the upgrade because the
128stored origin is empty. First registrations are jittered so an auto-update wave
129does not hit the relay at once. The host stores a confirmed-origin marker with
130the connector config, and a later boot starts the connector before registration
131only when that marker matches the current config and port. If registration
132cannot reach the relay for ten minutes, the host starts its stored config anyway
133and keeps registering in the background until it can reconcile the origin.
134If the connector exits, or `cloudflared` reports repeated tunnel
135rejections, the host asks the relay for a replacement, at most once every two
136minutes. The relay
137provisions under the same allocation, so the hostname and DNS record survive
138and clients keep their bindings. Every mutation on an allocation bumps its
139`generation`, and deletion locks the row at the generation it claimed, so a
140host that reconnects mid-sweep wins.
141
142## A paid T3 Chat plan raises the tunnel cap
143
144T3 Chat uses WorkOS, not Clerk, so the two accounts are linked by consent
145rather than by a shared identity. The relay and T3 Chat share one secret
146(`T3CHAT_LINK_SECRET` here, `T3CODE_LINK_SECRET` there). The relay signs a
147short-lived token naming the Clerk user and email; the user confirms it while
148signed in to T3 Chat, and T3 Chat stores the link. T3 Chat owns the link and the
149plan, so the relay keeps no copy. It asks T3 Chat only when an account reaches
150its normal cap for a new environment, and any failure keeps the normal cap.
151An environment that already holds a counted slot restarts without asking, so a
152downgrade or a T3 Chat outage only blocks new environments. A manual override
153row replaces the default for new environments, and a T3 Chat plan does not
154raise it
155([limits](../../infra/relay/src/environments/ManagedTunnelLimits.ts)).
156
157## The relay client follows the server's pin
158
159The host runs `cloudflared` pinned by `CLOUDFLARED_VERSION` in
160[`relayClient.ts`](../../packages/shared/src/relayClient.ts), and bumping it there
161(version, URLs, and checksums) is the whole release step. A linked host that
162starts a server with a new pin keeps its connector up on the newest older managed
163release, installs the pinned one in the background, then starts a second
164connector on it for the same tunnel. The old connector stops only after the new
165one registers, so the swap never takes the host off the relay. If the new one
166fails to spawn, the old one keeps serving and the install retries later. A host
167with no relay client installs one first. A `cloudflared` on `PATH` or an explicit `T3CODE_CLOUDFLARED_PATH` is the
168user's choice and is never replaced, since linking asks before downloading.
169
170Once the pinned release registers a connection, managed releases older than the
171pin are deleted except the newest of them. That one, and any newer release, can
172belong to another server sharing the same T3 home, such as a rollback after a
173failed update, which could not run again without it.
174
175The version comes from running `cloudflared version`, not from the folder name.
176Connectors from before `--no-autoupdate` replaced themselves in place, so a
177managed folder can hold another release. Such a binary still runs as a fallback
178while the pin installs. Every binary needs `CLOUDFLARED_MIN_VERSION`, the oldest
179release that accepts every flag the connector is started with. Raise it whenever
180a new flag is added.
181
182## OAuth traps
183
184Interactive clients and the headless CLI use the same Clerk application but
185different credentials. The relay accepts both session-template JWTs and CLI
186OAuth tokens; requiring a JWT template for the CLI would reject valid logins.
187The CLI is a public OAuth client using PKCE and stores no client secret.
188
189Loopback CLI authorization starts on the hosted `/connect` page so sign-in
190completes before entering Clerk's authorize endpoint. Sending a signed-out
191browser straight to that endpoint loses the authorize parameters during the
192sign-in redirect. The [shared flow](../../packages/shared/src/connectAuth.ts)
193preserves PKCE and state for the loopback callback.
194
195SSH and headless sessions use Clerk's OAuth device authorization grant because
196the browser cannot ordinarily reach a listener on the remote machine. The CLI
197polls Clerk's token endpoint directly while the user approves a short code on
198Clerk's hosted device page; the hosted app plays no part and there is no
199redirect URI or PKCE. The grant must be enabled on the CLI OAuth application
200or the device endpoint returns an error before any prompt is shown.