1# Relay observability
2
3 For maintainers. Using T3 Code? See [docs/user](../user/).
4
5The relay Alchemy stack owns a shared Axiom trace setup:
6
7`t3-code-relay-traces-prod`, the OpenTelemetry trace dataset shared by the Worker, mobile app, and
8 first-party relay clients
9`t3-code-relay-otel-ingest-prod`, the dataset-scoped Worker ingest token
10`t3-code-mobile-otel-ingest-prod`, the dataset-scoped mobile ingest token
11`t3-code-relay-client-otel-ingest-prod`, the dataset-scoped first-party relay-client ingest token
12`t3-code-relay-recent-spans-prod`, a view of recent request and endpoint spans
13`t3-code-mobile-logs-prod` and `t3-code-mobile-metrics-prod`, the mobile app's crash, error, and
14 startup data from `expo-observe` (see [Mobile crashes and errors](#mobile-crashes-and-errors))
15`t3-code-mobile-telemetry-ingest-prod`, the ingest token the Worker forwards that data with
16`t3-code-mobile-sourcemaps-prod`, the R2 bucket of mobile JavaScript source maps, and
17 `t3-code-mobile-sourcemaps-upload-prod`, a Cloudflare token that can only write objects to it
18
19Alchemy stages append their sanitized stage name to isolate resources, for example
20`t3-code-relay-traces-dev-julius` for a personal stage.
21
22Deploy from `infra/relay` with the normal Alchemy workflow:
23
25vp run deploy
27
28Alchemy resolves account-level Axiom deployment credentials through its provider. At runtime, the
29Worker receives only its scoped ingest token. Mobile and relay clients use their own separately
30provisioned scoped ingest tokens.
31
32The Worker emits Effect's built-in HTTP server spans plus endpoint and database child spans.
33Effect's OpenTelemetry exporter stores semantic HTTP attributes below the `attributes.` prefix.
34For example:
35
37['t3-code-relay-traces-prod']
38| where kind == 'server'
39| extend endpoint = column_ifexists('attributes.http.route', ''),
40 customAttributes = column_ifexists('attributes.custom', dynamic({}))
41| project _time, name, trace_id, duration,
42 ['attributes.http.request.method'],
43 ['attributes.url.path'],
44 ['attributes.http.response.status_code'],
45 endpoint,
46 relayOperation = customAttributes['relay']['operation']
47| order by _time desc
48| limit 200
50
51The provisioned view also reads the endpoint from `attributes.http.route`. Relay-specific span
52annotations are stored under `attributes.custom`; `relay.operation` is one of the emitted custom
53attributes.
54
55Agents should prefer the provisioned view or APL queries for completed incidents instead of
56tailing the Cloudflare Worker. The stack does not provision a separate query token. Responders who
57need scripted query access use the authorized account-level `AXIOM_TOKEN` together with
58`AXIOM_ORG_ID`; scoped ingest tokens remain write-only credentials for their producers.
59
60DPoP proof failures include the stable `relay.dpop.failure_code` span attribute. A `time_window`
61failure means that a signed proof was too old or too far in the future for the relay's allowed
62window. It can point to a date or time problem on either device, but it can also result from a
63delayed request. The client uses this category, and the absence of a category from an older relay,
64to decide whether clock skew is confirmed or only one possible cause.
65
66## Webhooks
67
68A public webhook request is one `relay.hooks.forward` span. Its `relay.hook.outcome` says what
69happened: `forwarded`, `held`, `rate_limited`, `inbox_full`, `not_found`, `payload_too_large`,
70`environment_unavailable`, or `environment_timeout`. `relay.hook.endpoint_key` identifies the
71managed endpoint, and with it the environment. On a forward, `relay.hook.upstream_status` or
72`relay.hook.upstream_error` records the environment's answer, and
73`relay.hook.upstream_outcome` what it did with the request (`accepted`, `duplicate`,
74`prompt_too_long`, `queue_full`, `rejected_signature`, `expired`, `disabled`, ...), from its
75`x-t3-hook-outcome` response header. A held request's delivery records the same on its
76`relay.inbox.deliver` span. The relay sends its own `traceparent` with each forward and drops any a
77sender supplied, and signs each forward (`x-t3-relay-delivery`) so the environment trusts the
78relay's delivery id, receive time and trace context only from the relay, so on environments that export to T3 Connect, the environment's
79`ScheduledTaskService.triggerWebhook` span lands in the same trace. `relay.hook.rate_limit` says which
80budget ran out: `endpoint` or `hook`. `relay.hook.rate_limiter_failed_open` is set when the
81Cloudflare rate limiter was unavailable and the request went through unlimited.
82
83Held requests are handled in the endpoint's `HookInboxObject`, and each call into it is its own
84trace, not a child of the forward span: `relay.inbox.hold`, `relay.inbox.wake`, and
85`relay.inbox.deliver` for each alarm run. Each carries `relay.hook.endpoint_key`, so they join to
86forwards. The details are on the child spans: `HookInboxStore.hold` carries
87`relay.inbox.refused` when the inbox refused a request (`max_per_hook`, `max_requests`,
88`max_bytes`, or `already_held`). `HookInboxStore.deliverDue` carries `relay.inbox.run_result`
89(`drained`, `more_pending`, `busy`, or `unreachable`), how many requests it sent and delivered,
90the consecutive failures behind the retry delay, the longest time a delivered request waited, and
91the backlog left. A run that errored outright has `relay.inbox.run_result = failed` on the root
92`relay.inbox.deliver` span instead.
93
94Questions these answer:
95
97// Webhook outcomes per hour
98['t3-code-relay-traces-prod']
99| where name == 'relay.hooks.forward'
100| summarize count() by bin(_time, 1h), outcome = tostring(['attributes.custom']['relay.hook.outcome'])
101
102// Inboxes stuck behind an unreachable environment, by endpoint
103['t3-code-relay-traces-prod']
104| where name == 'HookInboxStore.deliverDue'
105| extend c = ['attributes.custom']
106| where tostring(c['relay.inbox.run_result']) == 'unreachable'
107| project trace_id, failures = toint(c['relay.inbox.consecutive_failures']),
108 backlog = toint(c['relay.inbox.held_count'])
109| join kind=inner (
110 ['t3-code-relay-traces-prod']
111 | where name == 'relay.inbox.deliver'
112 | project trace_id, endpoint = tostring(['attributes.custom']['relay.hook.endpoint_key'])
113 ) on trace_id
114| summarize runs = count(), failures = max(failures), backlog = max(backlog) by endpoint
116
117## Alerts
118
119The stack provisions threshold monitors and a `t3-code-relay-health` dashboard from
120[`infra/relay/src/monitoring.ts`](../../infra/relay/src/monitoring.ts) when the deployment sets
121`RELAY_ALERT_NOTIFIER_ID`. Paging monitors go to that notifier and need two failing runs in a row.
122Warnings go to `RELAY_WARNING_NOTIFIER_ID` when it is set and are otherwise only visible in Axiom.
123Each monitor's description gives its healthy baseline.
124
125Monitors count HTTP status codes, not span errors. A request the relay cuts off at its 9 s deadline
126ends as an interrupted span without an error flag, but still answers 504. `/connect` 502s are
127normal: offline hosts produce most of them, retried by their clients every few seconds.
128
129The dashboard's "Connecting" charts show connections as users see them. Web, desktop, and mobile
130export a root `relay.connection.attempt` span per attempt, which lasts until initial sync finishes, in
131the same trace as the relay's and the host's spans. Most reconnects reuse a cached relay token and go
132straight to the host, so relay latency only affects cold connects. A slow total with fast phases
133usually means the client waited out an unreachable direct (LAN) route before falling back to the relay.
134
135**Links failing / tunnel recovery failing.** Open the dashboard's "Link failures by reason" table.
136`provisioning: ensure-tunnel` or `configure-tunnel` with Cloudflare 429s on the "Cloudflare API calls
137by status" chart is the #16810 failure: the account's Cloudflare API rate limit. Ask Cloudflare to
138raise it, and check whether something is calling the API in a loop (tunnel recovery from hosts is the
139largest caller). Slow Postgres instead points at the database or Hyperdrive.
140
141**Environments unreachable.** Clients reach fewer hosts than usual. Check Cloudflare Tunnel status and
142recent relay deploys before suspecting hosts: a host-side problem does not move this many
143environments at once.
144
145**Push delivery failing.** Look at the `relay.apns_deliveries.process_signed_job` or `relay.fcm.process`
146error messages. An expired APNs key or FCM service account fails every job.
147
148**Cloudflare API rate limited / near its limit.** The warning fires on sustained 429s; the page fires
149when they are frequent enough to stall provisioning. Compare the call volume on the dashboard with the
150account limit, which Cloudflare raised to 20,000 calls per 5 minutes in October 2026 (peak use then
151was about 2,100).
152
153**Traffic dropped.** The Worker is down, a deploy broke it, or spans stopped reaching Axiom. Check
154`https://relay.t3.codes/health` first.
155
156**Tunnel cleanup stopped / tunnel deletions failing.** Warnings for the five-minute reaper that deletes
157idle Cloudflare tunnels. "Stopped" counts sweeps that finished, so it also fires when every sweep
158fails; a sweep fails when listing tunnels does, usually behind Cloudflare 429s. Idle tunnels are
159billed until the reaper catches up, which the "Idle tunnels in Cloudflare" chart shows. A failed
160sweep records no counts, so while sweeps fail the chart stops at the last good one; check the tunnel
161list in the Cloudflare dashboard for current numbers. For failed
162deletions, read the relay's "Failed to delete an inactive managed tunnel" warnings. To pause cleanup,
163see [Managed tunnel cleanup rollout](release.md#managed-tunnel-cleanup-rollout).
164
165**Links degraded / database slow.** Warnings for slow degradation. Link failures above 20% over six
166hours have so far meant provisioning running into the 9 s deadline, with Postgres latency as the
167usual cause.
168
169## Mobile crashes and errors
170
171The mobile app reports through [`expo-observe`](https://docs.expo.dev/eas/observe/). Its
172`endpointUrl` points at the relay (`/v1/mobile/observe`, set in `apps/mobile/app.config.ts` from
173`T3CODE_RELAY_URL`), because Observe cannot send auth headers and prefixes every path with the EAS
174project id. The relay forwards each request to Axiom with its own token: logs (JavaScript errors,
175native crashes, log events) to `t3-code-mobile-logs-*`, metrics to `t3-code-mobile-metrics-*`, and
176spans to the shared traces dataset. Expo's Observe dashboard receives nothing. The endpoint URL is
177compiled into the native app, so pointing it elsewhere takes a store build.
178
179Hermes stacks from release builds are minified bytecode offsets. On the way through, the relay
180rewrites each JavaScript exception's `exception.stacktrace` with the bundle's source map from R2
181and records the outcome on the log record:
182
183`relay.symbolication.status`: `symbolicated`, or why not. `no_bundle_id` is a development or
184 local build. `source_map_missing` means no map was uploaded for the bundle;
185 `source_map_too_large`, `source_map_invalid` and `source_map_unavailable` (R2 unreadable, retried
186 shortly) mean one was. `source_map_limit` is a request naming more bundles than the relay decodes
187 at once, and `no_bundle_frames` a stack with no frames from the bundle.
188`relay.symbolication.source_map`: the R2 key used
189`relay.symbolication.raw_stacktrace`: the original stack, when it was rewritten
190
191iOS frames lose their install path (`/private/var/containers/Bundle/Application/<uuid>/...`), which
192differs per install, so `exception.stacktrace` groups across users.
193
194A fatal JavaScript error ends in a native abort, which Observe reports as a second crash with a
195generic React Native stack. The relay marks that one `relay.crash.js_fatal` when both arrive in one
196request, as they do on the next launch. Leave it out to count real native crashes:
197
199['t3-code-mobile-logs-prod']
200| where ['attributes.expo.error.source'] == 'nativeCrash'
201| where isnull(['attributes.relay.crash.js_fatal'])
202| summarize count() by ['attributes.exception.type'], ['attributes.exception.message']
204
205Native crash stacks (`expo.error.source == 'nativeCrash'`) are not symbolicated. On iOS, app frames
206read like `main (T3Code + 19476)`: the offset into the app binary. The build hook uploads that
207build's dSYMs to `ios/dsym/<easBuildId>.zip`; download it with an account R2 token, then:
208
210unzip <easBuildId>.zip
211atos -arch arm64 -o T3Code.app.dSYM/Contents/Resources/DWARF/T3Code -l 0x100000000 \
212 $(printf '0x%x' $((0x100000000 + 19476)))
214
215Maps are keyed by the bundle that crashed: `<platform>/ota/<updateId>.map` for OTA updates and
216`<platform>/build/<easBuildId>.map` for the bundle embedded in a store build. The production
217mobile workflow uploads OTA maps after `eas update`, and the `eas-build-on-success` hook in
218`apps/mobile/package.json` uploads a build's map, both with `apps/mobile/scripts/upload-sourcemaps.mts`.
219They need the `MOBILE_SOURCEMAPS_R2` secret, a JSON value the stack outputs as
220`mobileSourceMapsUploadCredentials` from a token that can only write to the bucket. Creating that
221token needs **Account API Tokens: Edit** on the deploy token. Deploy output shows it redacted; read
222it from state, where it is under `__redacted__`:
223
225cd infra/relay && vp exec alchemy state read T3CodeRelay/prod/output
227
228Set it as a GitHub Actions secret, for OTA updates, and as an EAS environment variable with
229`secret` visibility in the `production` environment only, for builds. Keep it out of `preview`:
230PR preview builds run the PR's code in that environment. In CI only the upload step gets it, not
231`eas update` and the dependencies it runs. If the deploy replaces the token, update both.
232
233The `t3-code-mobile-health` dashboard is the place to look every few days: crash and handled
234error counts, top crashes with the installs and app versions they hit, crashes by version, whether
235stacks are symbolicated, and whether reports reach the relay at all. A crash counts once: the
236native abort after a JS crash (`relay.crash.js_fatal`) is left out. Startup times are metrics, so
237they are not on it; query `t3-code-mobile-metrics-*` with MPL. Prod and PR previews get the
238dashboard, and deployments with `RELAY_ALERT_NOTIFIER_ID` get these monitors
239([`infra/relay/src/mobileTelemetry/monitoring.ts`](../../infra/relay/src/mobileTelemetry/monitoring.ts)):
240
241**crash spike** (pages): crashes per 15 min well above the same time over the last 7 days. Until
242 a week of data exists it has no baseline. A spike right after a release usually means a bad OTA
243 update; roll it back in EAS.
244**one crash hitting many installs** (pages): a single crash on 25 installs within an hour.
245**reports stopped** (warning): no startup metrics forwarded for an hour, so crashes are not
246 arriving either.
247**crash stacks not symbolicated** (warning): over half of release JS stacks stayed minified for
248 6 h. `relay.symbolication.status` names the cause.
249
250Thresholds are starting points, not measured baselines; revisit them after two weeks of
251production data.
252
253Without it, uploads are skipped with a warning and stacks stay minified. Each ingest request is one
254`relay.mobile_telemetry.ingest` span; `relay.mobile_telemetry.outcome` is `forwarded`,
255`rate_limited`, `body_too_large`, `upstream_unavailable`, or `upstream_timeout`.