Startup still blocks for ~7 minutes on 0.1.10 migrations after the #35421 fix, and clients only get an empty 503 #35995

What happened#

After the service auto-updated, T3 Code "would not come back up": systemd showed active (running), but the server was unusable for ~7 minutes and nothing said a database upgrade was running, so the user assumed it hung.

Diagnosis#

#35421 (startup migrations 82/83 taking 17 min on a 32 GB statev2.sqlite) was closed by #35486. That PR stopped requests from hanging (immediate 503), committed each migration on its own and logged "N of M" progress. Its own description says it does not make readiness faster and that the 503 has an empty body. On a build with the fix, that gap is the whole user-visible problem:

1. Readiness is still blocked by every pending migration. On a 15 GB database the server answered nothing useful for 6 min 48 s. Index builds over the turn_items / nodes projection tables still finish before the server serves anything.
2. The 503 carries no information. rejectRequestsUntilServing (apps/server/src/httpResponseErrorGuard.ts) answers every HTTP request and WebSocket upgrade with 503, Retry-After: 1 and an empty body. The progress lines exist only in boot-service.log. Users can't tell an upgrade from a hang, and restarting throws away the migration in progress.
3. Each release adds another expensive startup migration. On this build migration 86 (86_QueuedTurnInputIndex, a partial index on json_extract(payload_json, '$.inputIntent')) took as long as 82. #35421 only covered 82/83, and more full-table index builds keep shipping.
4. The root cause keeps growing. statev2.sqlite grows without bound (#26591, #17845). A long-lived install hits this on every update that adds an index.

Not a duplicate of #35421: the hang is fixed. What's left is the server staying unusable for the whole migration with no user-visible progress, which #35486 left out of scope.

Steps to reproduce#

1. Standalone systemd service with a ~15 GB statev2.sqlite, on 0.1.8-nightly.20261108.4353.
2. Update to 0.1.10-nightly.20261113.4616 (migrations 81–87 pending); service restarts.
3. Open the app or curl -i http://127.0.0.1:3773/.well-known/t3/environment.
4. Empty-body 503 for ~7 min; progress only in boot-service.log.

Version#

Bug seen on 0.1.10-nightly.20261113.4616 (updated from 0.1.8-nightly.20261108.4353). Triage CLI: 0.1.10-nightly.20261113.4616.

Environment#

Linux x64 on WSL2 (kernel 6.18.40.1-microsoft-standard-WSL2), Node v26.12.1. Standalone server as a systemd user service bound to 0.0.0.0:3773. statev2.sqlite 15.1 GB.

Evidence#

`boot-service.log` (the per-migration progress added by #35486):


[05:53:28.988] Running database migrations
[05:53:28.995] 81_ProjectionMessageSearchText   1 of 7
[05:53:45.644] 82_TurnItemConversationIndex     2 of 7
[05:56:16.942] 83_CommandReadIndexes            3 of 7
[05:57:34.330] 84_LastExecutedRunIndex          4 of 7
[05:57:35.645] 85_TurnItemCounts                5 of 7
[05:57:41.971] 86_QueuedTurnInputIndex          6 of 7
[06:00:11.420] 87_ProjectionProjectsCustomTitle 7 of 7
[06:00:11.443] Migrations ran successfully
[06:00:12.811] Listening on http://0.0.0.0:3773



`GET /.well-known/t3/environment` returned `503` at 05:56 and 05:58 and `200` at 06:00:18 (`databaseMigration: 87`). Service start was 05:53:24.

Related issues#

• #35421 / #35486: same symptom; fix removed the hang, not the blocking or missing progress.
• #35313: desktop loading screen during migrations; web/standalone has no equivalent.
• #23372: desktop windowless 8+ min during startup recovery.
• #26591, #17845: unbounded statev2.sqlite growth.

Suggestions: build indexes after ready or in batches (note ProjectionStore.ts pins reads with INDEXED BY on the migration-82 index, so deferring needs fallback queries); put "upgrading database, N of M" in the startup 503 body so clients can show it.

Fix applied or workaround#

None. Waited it out; nothing changed on the install or database.

Filed by#

Claude Code (Claude Opus 5.5), via t3 triage.