1# Observability
2
3 For maintainers. Using T3 Code? See [docs/user](../user/).
4
5T3 Code has one server-side observability model:
6
7pretty logs go to stdout for humans
8completed spans go to a local NDJSON trace file
9traces, metrics, and logs can also be exported over OTLP to a real backend like Grafana LGTM
10
11The local trace file is the persisted source of truth for normal local launches. Those launches do not
12write a separate server log file, but SSH-managed launches also persist the remote process's
13stdout/stderr at `~/.t3/ssh-launch/<state>/server.log`.
14
15## Where To Find Things
16
17### Logs
18
19Logs are human-facing:
20
21destination: stdout
22format: `Logger.consolePretty()`
23normal local persistence: none
24SSH-managed launch persistence: `~/.t3/ssh-launch/<state>/server.log`
25remote export: OTLP only, when configured
26
27If you want a log message to show up in the trace file, emit it inside an active span with `Effect.log...`. `Logger.tracerLogger` will attach it as a span event.
28
29Configuring a logs endpoint takes over that job. The server then exports log records, which cover
30every message instead of only the ones inside an active span and carry the trace and span ids so
31they still line up with the trace. `Logger.tracerLogger` is dropped in that mode, so the same
32message is not exported twice and the trace file stops carrying log messages. stdout output and
33SSH-managed launch persistence stay unchanged either way.
34
35### Traces
36
37Completed spans are written as NDJSON records to `serverTracePath`. The default depends on how the
38server starts: production and explicitly configured homes use
39`<home>/userdata/logs/server.trace.ndjson` (so `~/.t3/userdata/...` by default, or
40`/custom/path/userdata/...` with `--home-dir /custom/path`), a linked worktree dev run uses
41`<worktree>/.t3/userdata/logs/server.trace.ndjson`, and an implicit dev run outside a linked
42worktree uses `~/.t3/dev/logs/server.trace.ndjson`.
43
44Important fields common to both record types:
45
46`type`: `effect-span` or `otlp-span`
47`name`: span name
48`traceId`, `spanId`, `parentSpanId`: correlation
49`durationMs`: elapsed time
50`attributes`: structured context
51`events`: embedded logs and custom events
52
53`effect-span` records also contain `exit` with `Success`, `Failure`, or `Interrupted`. `otlp-span`
54records instead carry OTLP resource, scope, and optional status fields.
55
56The `TraceRecord`, `EffectTraceRecord`, and `OtlpTraceRecord` schemas live in
57`packages/shared/src/observability.ts`.
58
59SQL statement spans (named `sqlite`) and `sql.transaction` spans carry `db.connection` and
60`db.lock_wait_ms`. `db.connection` is `writer` for the read-write connection or `reader` for a
61read-only one. For a database file both run in worker threads; an in-memory database has only a
62writer, on the event loop. `db.lock_wait_ms` is the part of the
63span spent waiting for that connection, behind other SQL. That wait does not block the event loop,
64so subtract it before you blame a slow span on the query. Statements inside a transaction use its
65connection and do not carry these attributes.
66
67An MCP request over HTTP has a `McpHttpServer.authenticateRequest` span that covers only admitting
68it: the credential check, the mode limit, and the Linked environments gate. The request runs after
69it, under the `McpServer.…/<method>` span, and each tool call has its own `mcp.tool.<name>` span
70that times it. That span's attributes name the tool, never its arguments.
71
72DPoP proof failures include the safe `environment.dpop.failure_code` span
73attribute. A `time_window` failure means that a signed proof was too old or too
74far in the future for the environment server's allowed window. It can point to
75a date or time problem on either device, but it can also result from a delayed
76request.
77
78#### Summarize the trace file
79
80`t3 trace summary` reads the trace file and its rotated backups directly, so it works while the
81server is stalled or stopped. It prints counts, rates, and latency percentiles per span name. Use
82it to measure background work or to compare two builds.
83
85t3 trace summary --since 30m --limit 40
87
88It reads `T3CODE_TRACE_FILE` if set, else `<home>/userdata/logs/server.trace.ndjson` for
89`--base-dir` or `T3CODE_HOME`, plus the `T3CODE_TRACE_MAX_FILES` rotated backups. For a dev run or
90a copied file, set `T3CODE_TRACE_FILE`. `--since 30m` keeps spans that ended in the last 30
91minutes. The rate is per minute between the first and last span end.
92
93### Metrics
94
95Metrics are not written to a local file.
96
97local persistence: none
98remote export: OTLP only, when configured
99current definitions: `apps/server/src/observability/Metrics.ts`
100
101If OTLP is not configured, metrics still exist in-process, but you will not have a local artifact to inspect.
102
103### Event Loop Stalls
104
105`apps/server/src/observability/EventLoopMonitor.ts` samples the server's event loop every 30 s. When
106the loop stalled for more than 2 s since the previous sample, it records a root
107`server.eventLoop.stall` span with a warning. The span has trace level `Warn`, so it stays when
108`T3CODE_TRACE_MIN_LEVEL` is `Warn`. The warning shows in Settings > Diagnostics unless OTLP logs are
109on. The span time is when the sample ran, not when the stall happened.
110
111The trace file rotates in well under an hour on a busy server, so each stall is also appended as one
112NDJSON line to `event-loop-stalls.ndjson`, next to the trace file. A line has `time` and the span
113attributes. The file keeps the last 1000 or more stalls, with one rotated backup
114(`event-loop-stalls.ndjson.1`). Use it to see how often a server stalls over days.
115
116Some delay is not recorded:
117
118`delayMaxMs` is the longest stall, and can undercount it by up to 1 s. The 2 s threshold applies to
119 this value, so a stall over 3 s is normally recorded, and a shorter one can be missed. A stall
120 that ends just as a sample runs can be missed too.
121Time the computer spends asleep reads as delay on macOS and Windows. So a sample only counts when
122 the loop was busy, not waiting for events, for at least `delayMaxMs`. Busy time covers the whole
123 window, so a short sleep in an otherwise busy window can still record a false stall. The span then
124 shows CPU time far below `delayMaxMs`.
125The first sample after launch is skipped. Startup work such as migrations and projection bootstrap
126 can block the loop for seconds on a large database.
127
128CPU times and page faults cover the whole process over the whole window since the previous sample.
129The window is nominally 30 s, but a long stall delays the sample and makes the window longer. Other
130work in the window can hide a wait, so only CPU time far below `delayMaxMs` proves the thread was
131waiting. Read CPU together with page faults:
132
133High `cpuSystemMs` with many page faults means memory pressure. Major faults are reads from disk or swap.
134 On macOS, reads from compressed memory are minor faults plus system CPU.
135High `cpuUserMs` with few page faults means JavaScript work or garbage collection.
136Low CPU with few major page faults points at synchronous disk I/O, such as SQLite reads or trace
137 file writes.
138Many `involuntaryContextSwitches` mean other processes were competing for the CPU.
139
140### Related Artifacts
141
142Provider event NDJSON files still exist for provider runtime streams. Those are separate from the main server trace file.
143
144## Run The Server In Instrumented Mode
145
146There are two useful modes:
147
148local-only: stdout + local `server.trace.ndjson`
149full local observability: stdout + local trace file + OTLP export to Grafana/Tempo/Prometheus
150
151The local trace file is always on. OTLP export is opt-in.
152
153### Option 1: Local Traces Only
154
155You do not need any extra env vars. Just run the app normally and inspect `server.trace.ndjson`.
156
157Examples:
158
160t3
162
164node --run dev
166
168node --run dev:desktop
170
171### Option 2: Run With A Local LGTM Stack
172
173#### 1. Start Grafana LGTM
174
176docker run --name lgtm \
177 -p 3000:3000 \
178 -p 4317:4317 \
179 -p 4318:4318 \
180 --rm -ti \
181 grafana/otel-lgtm
183
184Then open `http://localhost:3000`.
185
186Default Grafana login:
187
188username: `admin`
189password: `admin`
190
191#### 2. Export OTLP env vars
192
193For endpoint settings and startup precedence, see the
194[user telemetry guide](../user/telemetry.md#export-diagnostics-to-your-own-receiver).
195In a worktree, use `vp run dev` instead of `t3` to keep state isolated.
196
198export T3CODE_OTLP_TRACES_URL=http://localhost:4318/v1/traces
199export T3CODE_OTLP_METRICS_URL=http://localhost:4318/v1/metrics
200export T3CODE_OTLP_LOGS_URL=http://localhost:4318/v1/logs
201export OTEL_RESOURCE_ATTRIBUTES=deployment.environment.name=development
203
204Optional:
205
207export T3CODE_TRACE_MIN_LEVEL=Info
208export T3CODE_TRACE_TIMING_ENABLED=true
210
211#### 3. Launch the app from that same shell
212
213CLI:
214
216t3
218
219Monorepo web/server dev:
220
222node --run dev
224
225Monorepo desktop dev:
226
228node --run dev:desktop
230
231Packaged desktop app:
232
233Launch the actual app executable from the same shell so the desktop app and embedded backend inherit `T3CODE_OTLP_*`.
234
235macOS app bundle example:
236
238T3CODE_OTLP_TRACES_URL=http://localhost:4318/v1/traces \
239T3CODE_OTLP_METRICS_URL=http://localhost:4318/v1/metrics \
240T3CODE_OTLP_LOGS_URL=http://localhost:4318/v1/logs \
241"/Applications/T3 Code.app/Contents/MacOS/T3 Code"
243
244Direct binary example:
245
247T3CODE_OTLP_TRACES_URL=http://localhost:4318/v1/traces \
248T3CODE_OTLP_METRICS_URL=http://localhost:4318/v1/metrics \
249T3CODE_OTLP_LOGS_URL=http://localhost:4318/v1/logs \
250./path/to/your/desktop-app-binary
252
253Do not rely on launching from Finder, Spotlight, the dock, or the Start menu after setting shell env vars. Those launches usually will not pick them up.
254
255#### 4. Fully restart after changing env
256
257The backend reads observability config at process start. If you change OTLP env vars, stop the app completely and start it again.
258
259## How To Use Traces And Metrics To Debug The Server
260
261### Start With The Local Trace File
262
263The trace file is the fastest way to inspect raw span data.
264
265Resolve the path for the launch mode once. Production and explicitly configured homes store runtime
266state under the base directory's `userdata` folder:
267
269TRACE_FILE="${T3CODE_HOME:-$HOME/.t3}/userdata/logs/server.trace.ndjson"
271
272A dev server started from a linked worktree defaults to that worktree's local home:
273
275TRACE_FILE="$WORKTREE/.t3/userdata/logs/server.trace.ndjson"
277
278Only an implicit dev run outside a linked worktree uses the shared dev directory:
279
281TRACE_FILE="$HOME/.t3/dev/logs/server.trace.ndjson"
283
284Tail the selected file:
285
287tail -f "$TRACE_FILE"
289
290Show failed spans:
291
293jq -c 'select(.type == "effect-span" and .exit._tag != "Success") | {
294 name,
295 durationMs,
296 exit,
297 attributes
298}' "$TRACE_FILE"
300
301Show slow spans:
302
304jq -c 'select(.durationMs > 1000) | {
305 name,
306 durationMs,
307 traceId,
308 spanId
309}' "$TRACE_FILE"
311
312Inspect embedded log events:
313
315jq -c 'select(any(.events[]?; .attributes["effect.logLevel"] != null)) | {
316 name,
317 durationMs,
318 events: [
319 .events[]
320 | select(.attributes["effect.logLevel"] != null)
321 | {
322 message: .name,
323 level: .attributes["effect.logLevel"]
324 }
325 ]
326}' "$TRACE_FILE"
328
329Follow one trace:
330
332jq -r 'select(.traceId == "TRACE_ID_HERE") | [
333 .name,
334 .spanId,
335 (.parentSpanId // "-"),
336 .durationMs
337] | @tsv' "$TRACE_FILE"
339
340Filter orchestration commands:
341
343jq -c 'select(.attributes["orchestration_v2.command_type"] != null) | {
344 name,
345 durationMs,
346 commandType: .attributes["orchestration_v2.command_type"],
347 threadId: .attributes["orchestration_v2.thread_id"]
348}' "$TRACE_FILE"
350
351Filter git activity:
352
354jq -c 'select(.attributes["git.operation"] != null) | {
355 name,
356 durationMs,
357 operation: .attributes["git.operation"],
358 cwd: .attributes["git.cwd"],
359 hookEvents: [
360 .events[]
361 | select(.name == "git.hook.started" or .name == "git.hook.finished")
362 ]
363}' "$TRACE_FILE"
365
366### Use Tempo When You Need A Real Trace Viewer
367
368Tempo is better than raw NDJSON when you want to:
369
370search across many traces
371inspect parent/child relationships visually
372compare many slow traces
373drill into one failing request without hand-joining by `traceId`
374
375Recommended flow in Grafana:
376
377Open `Explore`.
378Pick the `Tempo` data source.
379Set the time range to something recent like `Last 15 minutes`.
380Start broad. Do not begin with a very narrow query.
381Look for spans from the `t3code-server` or `t3code-desktop` service, then narrow by span name or
382 attributes.
383
384Good first searches:
385
386service name `t3code-server` or `t3code-desktop`, plus a resource attribute such as
387 `deployment.environment.name`
388span names like `sendTurn` or a Git operation such as `GitVcsDriver.statusDetails.status`
389Git spans whose `git.operation` attribute identifies the operation
390orchestration spans with attributes like `orchestration_v2.command_type`
391
392Once you know traces are arriving, narrower TraceQL queries for names such as `sendTurn` or Git
393operation names become useful.
394
395### Use Metrics To See Systemic Problems
396
397Traces are best for one request. Metrics are best for trends.
398
399Good metric families to watch:
400
401`t3_rpc_request_duration`
402`t3_orchestration_command_duration`
403`t3_provider_turn_duration` (how long the provider adapter takes to start a turn, not the turn's run time)
404`t3_git_command_duration`
405
406Counters tell you volume and failure rate:
407
408`t3_rpc_requests_total`
409`t3_orchestration_commands_total`
410`t3_provider_turns_total`
411`t3_git_commands_total`
412
413Webhooks have their own families:
414
415`t3_webhook_deliveries_total` by `outcome` and `source` (`relay` or `direct`). Beyond what the
416 sender sees, `queue_full` means a task already had its limit of deliveries waiting,
417 `prompt_too_long` means the filled-in prompt passed the provider limit, and `duplicate` means
418 the relay delivered a request this environment had already run.
419`t3_webhook_runs_total` by `outcome` (`started`, `skipped`, `failed`) for the runs those
420 deliveries start, which happen after the sender has its answer.
421`t3_webhook_held_delay` for how long requests the relay held waited before arriving.
422
423`t3_secret_requests_total` by `status` (`saved`, `declined`, `cancelled`, `timed_out`) for secrets
424 agents asked users for, and `t3_secret_refs_consumed_total` by `result` (`used`, `rejected`) for
425 tools redeeming them. Neither ever carries a value.
426
427`ScheduledTaskService.triggerWebhook` spans carry the same outcome per request, and each run
428started from a delivery is its own `ScheduledTaskService.runWebhookDelivery` trace. For a request
429the relay forwarded, the span also goes to the T3 Connect trace export as a child of the relay's
430span; requests that reach the environment directly never join a sender's trace.
431
432Use metrics when the question is:
433
434"is this always slow?"
435"did this get worse after a change?"
436"which command type is failing most often?"
437
438Use traces when the question is:
439
440"what happened in this specific request?"
441"which child span caused this one slow interaction?"
442"what logs were emitted inside the failing flow?"
443
444## Common Workflows
445
446### "Why did this request fail?"
447
448Start with the local NDJSON file.
449Find `effect-span` records where `exit._tag != "Success"`.
450Group by `traceId`.
451Inspect sibling spans and span events.
452If needed, move to Tempo for the full trace tree.
453
454### "Why is the UI feeling slow?"
455
456Search for slow top-level spans in the trace file or Tempo.
457Check child spans for sqlite, git, provider, or terminal work.
458Look at the matching duration metrics to see whether the slowness is systemic.
459
460### "Are git hooks causing latency?"
461
462Filter `git.operation` spans.
463Inspect `git.hook.started` and `git.hook.finished` events.
464Compare hook timing to the enclosing git span duration.
465
466### "Why do I have spans locally but nothing in Grafana?"
467
468Usually one of these is true:
469
470`T3CODE_OTLP_TRACES_URL` was not set
471the app was launched from a different environment than the one where you exported the vars
472the app was not fully restarted after changing env
473Grafana is looking at the wrong time range or service name
474
475If the local NDJSON file is updating, local tracing is working. The problem is almost always OTLP export configuration or process startup.
476
477## How To Think About Adding Tracing To Future Code
478
479### Prefer Boundaries Over Tiny Helpers
480
481Good span boundaries:
482
483RPC methods
484orchestration command handling
485provider adapter calls
486external process calls
487persistence writes
488queue handoffs
489
490Avoid tracing every tiny helper. Most helpers should inherit the active span rather than create a new one.
491
492### Reuse `Effect.fn(...)` Where It Already Exists
493
494The codebase already uses `Effect.fn("name")` heavily. That should usually be your first tracing boundary.
495
496For ad hoc work:
497
499import { Effect } from "effect";
500
501const runThing = Effect.gen(function* () {
502 yield* Effect.annotateCurrentSpan({
503 "thing.id": "abc123",
504 "thing.kind": "example",
505 });
506
507 yield* Effect.logInfo("starting thing");
508 return yield* doWork();
509}).pipe(Effect.withSpan("thing.run"));
511
512### Put High-Cardinality Detail On Spans
513
514Use span annotations for IDs, paths, and other detailed context:
515
517yield *
518 Effect.annotateCurrentSpan({
519 "provider.thread_id": input.threadId,
520 "provider.request_id": input.requestId,
521 "git.cwd": input.cwd,
522 });
524
525### Keep Metric Labels Low Cardinality
526
527Good metric labels:
528
529operation kind
530method name
531provider kind
532aggregate kind
533outcome
534
535Bad metric labels:
536
537raw thread IDs
538command IDs
539file paths
540cwd
541full prompts
542full model strings when a normalized family label would do
543
544Detailed context belongs on spans, not metrics.
545
546### Use Logs As Span Events
547
548Logs inside a span become part of the trace story:
549
551yield * Effect.logInfo("starting provider turn");
552yield * Effect.logDebug("waiting for approval response");
554
555Those messages show up as span events because `Logger.tracerLogger` is installed.
556
557### Use The Pipeable Metrics API
558
559`withMetrics(...)` is the default way to attach a counter and timer to an effect:
560
562import { someCounter, someDuration, withMetrics } from "../observability/Metrics.ts";
563
564const program = doWork().pipe(
565 withMetrics({
566 counter: someCounter,
567 timer: someDuration,
568 attributes: {
569 operation: "work",
570 },
571 }),
572);
574
575## Detailed API Reference
576
577### Runtime Wiring
578
579The server observability layer is assembled in `apps/server/src/observability/Observability.ts`.
580
581It provides:
582
583pretty stdout logger
584`Logger.tracerLogger`
585local NDJSON tracer
586optional OTLP trace exporter
587optional OTLP metrics exporter
588optional OTLP log exporter
589Effect trace-level and timing refs
590
591The desktop main process is a second producer, assembled in
592`apps/desktop/src/app/DesktopObservability.ts`. It reads the same `T3CODE_OTLP_*` names and the same
593Settings entries as the backend it supervises, and covers work the backend cannot see: app startup,
594window and menu handling, backend supervision, and updates. It reports as service
595`t3code-desktop`, so a collector shows it alongside the backend rather than mixed into it. It
596exports traces and logs only; the main process records no metrics, so the metrics endpoint applies
597to the backend alone.
598
599### Env Vars
600
601Local trace file:
602
603`T3CODE_TRACE_FILE`: override trace file path
604`T3CODE_TRACE_MAX_BYTES`: per-file rotation size, default `10485760`
605`T3CODE_TRACE_MAX_FILES`: rotated file count, default `10`
606`T3CODE_TRACE_BATCH_WINDOW_MS`: flush window, default `1000`
607`T3CODE_TRACE_MIN_LEVEL`: minimum trace level, default `Info`
608`T3CODE_TRACE_TIMING_ENABLED`: enable timing metadata, default `true`
609
610OTLP export:
611
612`T3CODE_OTLP_TRACES_URL`: OTLP trace endpoint
613`T3CODE_OTLP_METRICS_URL`: OTLP metric endpoint
614`T3CODE_OTLP_LOGS_URL`: OTLP log endpoint
615`T3CODE_OTLP_EXPORT_INTERVAL_MS`: export interval, default `10000`
616`T3CODE_OTLP_HEADERS`: extra headers for all three exporters, same format as
617 `OTEL_EXPORTER_OTLP_HEADERS`: comma-separated `key=value` pairs with percent-encoded values.
618`T3CODE_OTLP_PROTOCOL`: `http/json` (default) or `http/protobuf`
619
620The server and the desktop app also read the standard
621`OTEL_EXPORTER_OTLP_{TRACES,METRICS,LOGS}_ENDPOINT` and generic `OTEL_EXPORTER_OTLP_ENDPOINT` (with
622`/v1/traces`, `/v1/metrics`, or `/v1/logs` appended), for a collector expecting those instead. A
623non-blank `T3CODE_OTLP_*_URL` wins over either, and a per-signal endpoint wins over the generic one
624for its signal. A blank value counts as unset. A signal with an OTEL endpoint takes its headers from
625`OTEL_EXPORTER_OTLP_HEADERS` and its protocol from `OTEL_EXPORTER_OTLP_PROTOCOL` (default
626`http/protobuf`, read case-insensitively), and a per-signal
627`OTEL_EXPORTER_OTLP_{TRACES,METRICS,LOGS}_HEADERS` or `_PROTOCOL` wins over the generic one for its
628signal. `T3CODE_OTLP_HEADERS` and `T3CODE_OTLP_PROTOCOL` never apply to it. An endpoint that is not
629an `http` or `https` URL, a protocol other than `http/protobuf` or `http/json` such as `grpc`, or
630headers that are not `key=value` pairs with percent-encoded values turn that signal's export off
631with a startup warning, rather than sending it to the Settings endpoint.
632
633Service names are fixed: `t3code-server` for the backend and `t3code-desktop` for the desktop main
634process, both in `service.namespace` `t3code`. `OTEL_SERVICE_NAME` and a `service.name` or
635`service.namespace` in `OTEL_RESOURCE_ATTRIBUTES` are ignored. Tell installations apart with other
636resource attributes, such as `OTEL_RESOURCE_ATTRIBUTES=deployment.environment.name=staging`.
637
638Each process reports its own build, not the one it talks to: the server, the desktop main process,
639the UI, and the mobile app each carry `t3code.release.channel` (`dev`, `stable`, `nightly`, or
640`preview`, from their own version; desktop PR builds count as `preview`, and a server not built by a
641release workflow reports `dev`) and `deployment.environment.name` (`development`, `production`, or
642`staging` for `preview`). On the server and the desktop main process either key in
643`OTEL_RESOURCE_ATTRIBUTES` wins. The UI and mobile app never read that variable: Vercel PR previews
644set `VITE_RELEASE_CHANNEL=preview` because their `APP_VERSION` must stay comparable with the
645server's, and the mobile app maps its build variant (`development`, `preview`, `production`), or
646reports `dev` from a development bundle.
647
648Attributes specific to T3 Code use the `t3code.` prefix. The UI runs as `t3code-web` in both a
649browser and the desktop window, since it is the same code, and `t3code.client.surface` is `desktop`
650or `web`. The server's `t3code.server.managed_by` is `desktop` when the desktop app launched it and
651`standalone` otherwise. The desktop main process reports `t3code.desktop.mode` as `development` or
652`packaged` (formerly `service.mode`; dashboards filtering on that key must switch). Node processes
653also report the semantic `process.runtime.name`, `process.runtime.version`, and
654`process.runtime.description`, and UI traces report `browser.language`, `user_agent.original`, and,
655in Chromium, `browser.platform`, `browser.mobile`, and `browser.brands`.
656
657Every resource also carries a `service.instance.id`: random per process for the server and desktop,
658per page load for the UI, and the Expo session on mobile. A `service.instance.id` in
659`OTEL_RESOURCE_ATTRIBUTES` replaces the generated one on the server and desktop.
660
661If the OTLP URLs are unset, local tracing still works, metrics stay in-process only, and logs stay
662on stdout only.
663
664### The Kill Switch
665
666`T3CODE_OTEL_SDK_DISABLED` and `OTEL_SDK_DISABLED` turn off every OTLP export in both the server and
667the desktop main process, overriding any endpoint from the environment or Settings. Local trace
668files and stdout logs are unaffected.
669
670`T3CODE_OTEL_SDK_DISABLED` wins when set, so `T3CODE_OTEL_SDK_DISABLED=false` re-enables export on a
671machine that sets `OTEL_SDK_DISABLED` for everything else. It accepts the usual boolean spellings
672(`true`/`false`, `yes`/`no`, `on`/`off`, `1`/`0`, `y`/`n`). `OTEL_SDK_DISABLED` follows the
673OpenTelemetry specification and only `true` disables export, so `OTEL_SDK_DISABLED=1` does not.
674Values are case-insensitive and trimmed. An unrecognized value is ignored with a startup warning.
675
676`OTEL_TRACES_EXPORTER`, `OTEL_METRICS_EXPORTER`, or `OTEL_LOGS_EXPORTER` set to `none` turns off
677just that signal, overriding an OTEL endpoint and the Settings endpoint. A `T3CODE_OTLP_*_URL` still
678wins for its signal. `otlp` is the default, and any other exporter name, such as `console` or
679`prometheus`, is ignored with a startup warning.
680
681### What Is Instrumented Today
682
683Current high-value span and metric boundaries include:
684
685WebSocket RPC request spans (`ws.rpc.<method>`) and metrics in
686 `apps/server/src/observability/RpcInstrumentation.ts`
687startup phases
688orchestration command processing
689provider session and turn operations
690git command execution and git hook events
691terminal session lifecycle
692sqlite query execution, except the statements inside `orchestrationV2.EventSink.write`, which
693 its `sql.transaction` span times as one
694event loop stalls (`server.eventLoop.stall`)
695
696### Current Constraints
697
698logs outside spans are not persisted in the trace file; SSH-managed launch stdout/stderr is still
699 captured in its launcher log
700metrics are not snapshotted locally
701
702## Heap Snapshots
703
704To see what a long-running server holds in memory, send it `SIGUSR2`. The server writes a V8 heap
705snapshot to its logs dir and logs the path. This works for desktop, `npx t3`, and service installs
706on macOS and Linux. Windows has no `SIGUSR2`.
707
708Send the signal to the server pid in `server-runtime.json`, which sits in the server's state dir
709next to the `logs` dir. For a dev server or a `--home-dir` launch, use that server's state dir from
710[Traces](#traces). Do not send it to the desktop app or the service launcher: a process without the
711handler exits on `SIGUSR2`. After a crash the file can keep a stale pid that now belongs to a
712different process, so check the pid first.
713
715pid="$(jq .pid "${T3CODE_HOME:-$HOME/.t3}/userdata/server-runtime.json")"
716ps -p "$pid" -o command=
718
719If `ps` shows the T3 Code server, send the signal:
720
722kill -USR2 "$pid"
724
725The file is `<logsDir>/server-<pid>-<timestamp>.heapsnapshot`, next to `server.trace.ndjson`. To
726open it, use the Memory tab in Chrome DevTools and select Load.
727
728Before you take one:
729
730The server stops while it writes the file. For a large heap this can take a minute or more.
731 Connected clients can reconnect during the pause, and an event loop monitor, if the server has
732 one, records the pause as a stall. Send the signal once. A second signal sent during a write
733 takes another snapshot after the first one finishes.
734The write needs about as much free memory as the heap uses. On a machine that is already
735 swapping, it can make the problem worse or crash the server.
736The file contains everything in server memory, including tokens, secrets, and thread content. Do
737 not share it publicly. Delete it when you are done, because storage cleanup does not remove it.