Skip to content
Source
Sign inSign up

product-analytics.md

154 lines9.1 KB

1# Product analytics
2​
3The server owns analytics delivery, opt-out, and identity for every connected client.
4Clients do not load the PostHog browser SDK. Client-use events require an
5authenticated connection, so visiting the hosted app without connecting does not
6count as product use.
7​
8## Installs and owners
9​
10`distinct_id` keeps the rule every build has used: the hashed Codex workspace,
11else the Claude machine id, else the install's anonymous id. It can change when
12someone logs in, and a Codex Team shares one, but keeping it means PostHog charts
13and history never break. Do not change it; add a field instead.
14​
15Every event also carries `installId` when the server can save its state: the hash
16of a random id in the state directory, which never changes for the install. If
17that file cannot be written, events still go out under `distinct_id`, as before. Count installs
18with it. Who an install belongs to is a separate fact, sent as
19`install.owner_observed`. Owners are resolved at query time in our own store, so
20a rule can change and apply to all history.
21​
22[Owner sources](../../apps/server/src/telemetry/Identify.ts), strongest first:
23T3 Connect, then the OpenAI and Claude person ids, then the client device. The
24server reports what it can read itself: the linked Connect account and provider
25logins. A connecting client reports its signed-in Connect account and its device,
26which links every environment opened from one phone, browser, or desktop. Use the
27OpenAI `user_id` claim, not the Codex `account_id` (the ChatGPT workspace, which a
28whole Team shares) or the token `sub` (one per login method). Two Connect accounts are never the same person, whatever
29weaker ids they share.
30​
31Every id is SHA-256 hashed with [one helper](../../packages/shared/src/telemetryId.ts)
32on both sides, so a Connect account hashes the same on the client and the server.
33The server accepts only hashes from clients.
34​
35## Attribution boundaries
36​
37Client dimensions belong to the event's WebSocket connection. A server-global
38"current client" would misattribute simultaneous web, desktop, and mobile use.
39Provider execution has its own events because a turn can outlive the requesting
40connection.
41​
42Keep client and server dimensions separate. A desktop host can serve a phone or a
43remote browser, and a direct connection can cross a network. Older clients omit
44metadata. Missing client values must stay unknown rather than being backfilled
45from server properties. The legacy `clientType` property describes how the server
46runs; use `surface` for the connected client.
47​
48## Interpreting events
49​
50Use `client.turn.requested` for active-use reports. There is no connection event,
51because T3 Connect sessions expire hourly and each reconnect would count again.
52A command whose id was already accepted is a retry, so it records nothing again.
53One identity can appear in several client groups during a period; adding those
54groups double-counts users.
55​
56Provider send and completion counts need not match. `provider.turn.completed`
57is one provider turn, and `segment` says what started it: the run's first
58attempt (`initial`), a `steering_restart`, a provider `continuation` such as a
59Codex goal turn, or a `native_child` subagent. `createdBy`, `creationSource`,
60`scheduled`, and `subagent` describe the run's starting message, and `childKind`
61separates native subagents from T3-managed child threads. Use this event for
62per-turn measurements, not for counting requests.
63​
64`provider.request.completed` is one run, recorded when the run settles in the
65server process that ran it. It adds each root turn once, so continuations,
66steering, and replayed provider events do not add rows. `requestKind: "user"`
67counts what people asked for. Native subagent turns are summed apart from the
68root (`childUsageStatus`, `childInputTokens`); T3-managed child threads report
69their own requests as `t3_child`. A child still running at settlement is never
70added later, and the summary sets `childrenRunning`. A server restart loses the
71summary of a run in progress. A run that continues after a restart reports its
72own usage with `restartContinuation` and `lifecycleObserved: false`; it cannot
73recover the original request's count or kind. A run whose provider session never
74opens has no summary, and a start the provider rejects reports
75`providerTurns: 0`. A Stop that lands while a steering restart is starting ends
76the run without a summary. Collection is best effort, with no scan or backfill of
77provider history.
78​
79Counts cover the main agent. `otherInputTokens` and `otherOutputTokens` are what
80a provider's own ledger recorded beyond it during the turn: Claude's per-process
81`modelUsage` growth, which covers native subagents, compaction, and helper
82calls. They need a baseline from the same process, so they stay absent when it
83is unknown or the ledger goes backwards, and background work that ends between
84turns lands in the next one. Claude subagents have no turns of their own, so
85this never overlaps child turns.
86​
87`model`, `effort`, and `serviceTier` are what the run selected. `serviceTier`
88is `standard`, `flex`, `fast`, or `ultrafast`, from Codex's tier option or
89Claude's fast mode, and is absent when the selection does not set one. `observedModel` and
90`mixedModels` appear only when the provider reported the models for the counted
91work (Codex thread settings and reroutes, Claude root messages, OpenCode steps).
92`mixedModels: false` without `reportVersion` was never checked. Model labels
93come from the bundled model manifest and the Cursor and Grok catalogs, also for
94routed ids such as `anthropic/claude-opus-5-5`, and efforts come from a closed
95list, so user-configured names stay out.
96​
97Status and counters answer different questions. `usageReported: false` means
98the adapter attached no usage; a harness where every row says so is unsupported,
99not idle. `partial` keeps valid counts that cannot establish a whole-turn total,
100such as an OpenCode step that ended without a report, an OpenCode stream
101reconnect, or an interrupted segment.
102Explicit zeros are sent and unknown counts stay absent. `localCommand` marks a
103provider-local command such as Claude's `/compact`, whose result counters are not
104model work. Reasoning tokens are a subset of output.
105​
106Charts built on these events should say "output tokens per measured provider
107completion", compare matching provider, observed model, effort, interaction mode,
108and terminal status, and show eligible and measured counts by harness, origin,
109and status so unknown models and unsupported adapters stay in the denominator.
110Divide summed totals for output/input ratios; averaging per-turn ratios lets
111small-input turns dominate. Rows without `reportVersion` keep their missing
112fields unknown; a missing origin is never a person.
113​
114`feature.used` counts WebSocket requests that start a feature. RPCs are observed in
115the RPC authorization middleware after the request succeeds; dispatched commands
116are recorded by the dispatch handler, which can tell a retry apart.
117[Feature mapping](../../apps/server/src/telemetry/FeatureUsage.ts) allowlists
118the methods and commands, and takes variants only from closed contract enums,
119so paths, ids, and text never reach analytics. Guests of a shared thread have no
120WebSocket connection, so the guest share service records their joins and messages
121itself, with no `surface`.
122​
123Agent tool events report driver kinds, never instance ids: users name
124instances, so an instance id can identify an account or employer.
125[Provider dimensions](../../apps/server/src/telemetry/ProviderDimensions.ts)
126report a model only when it is a non-custom entry in a vendor catalog. OpenCode,
127Pi, and ACP agents list models from user configuration, such as local Ollama
128tags, so their models stay out. Handoff settings come from closed enums and
129booleans in tool arguments; prompts, titles, and ids are never read. Thread ids
130are used only to look up the threads on each side and never leave the server,
131which is also why tool events carry no per-thread sequence. A handoff to several
132threads, such as `create_threads`, records one `mcp.tool.invoked` per target, each
133repeating the call's outcome and duration, so event counts are not call counts.
134​
135## Delivery
136​
137The server sends batches to the ingest endpoint at `ingest.t3.codes`, which archives
138every event and forwards some of them to PostHog. A send can fail after the endpoint
139has stored the batch, so every retry is a copy.
140[Delivery](../../apps/server/src/telemetry/AnalyticsService.ts) gives each event a
141uuid when it is recorded (the endpoint rejects events without one), backs off after
142a failed send, and drops a batch after a few tries. Without these limits, one stuck
143batch was sent every second for days.
144​
145## Collection boundary
146​
147Keep analytics payloads to product metadata and normalized measurements. Do not
148send prompts, authentication material, raw provider payloads, user-assigned device
149names, or conversation identifiers. Client metadata is best effort; invalid values
150must not reject a connection. PostHog person profiles remain disabled.
151​
152Dev-runner servers and the CI archive smoke test turn telemetry off, which also
153stops relay tracing, because each would count as a new install. Set
154`T3CODE_TELEMETRY_ENABLED=true` to test analytics or relay tracing from a dev server.