fix(server): migration 64 skips thread payloads that are not valid JSON #35884
main fromrefs/pull/35884/headmerged by juliusmarminge View on GitHubMigration 64 filtered orchestration_v2_projection_threads and V2 thread orchestration_events with json_type(payload_json, ...). SQLite raises 'malformed JSON' when json_type meets an empty or invalid payload, so one damaged row aborted the migration and with it server startup. Both SELECTs now add AND json_valid(payload_json) before the json_type filters, the guard migration 77 uses on the same table. Rows with unreadable payloads are left untouched and every valid row is still canonicalized. No new regex or parser.
Fixes pingdotgg/t3code#35864
Reproduction on main
On origin/main 95cb068c10 (apps/server, Node, in-memory SQLite via @t3tools/shared/nodeSqliteClient): ran runMigrations({toMigrationInclusive: 63}), inserted one valid thread projection whose payload has a legacy linkedPullRequest, then for case 'projection' also inserted an orchestration_v2_projection_threads row with payload_json = '' and for case 'event' an orchestration_events row (application_event_version 2, aggregate_kind 'thread', event_type 'thread.created') with payload_json = '', then ran runMigrations({toMigrationInclusive: 64}). Control: no error, ledger 64. Projection case and event case: 'Migration "64_CanonicalizeThreadPullRequestLinks" failed' caused by SQLite 'malformed JSON', ledger stays at 63. Also extended the existing test in 064_CanonicalizeThreadPullRequestLinks.test.ts with both damaged rows; it fails on main with 'malformed JSON'. I did not launch a full server against such a database; that startup is blocked follows from the migration runner being part of persistence setup.
Evidence
Run the real server against fresh synthetic migration-63 databases containing valid legacy-linked projection/event payloads. Compare a clean control with an empty thread projection and, separately, an empty V2 thread.created event. Capture startup output, HTTP readiness, migration ledger, and resulting payloads.
Environment on both sides: Linux x86_64, Node 26.12.1, native SQLite, loopback HTTP; live-data seeding disabled.
| Before | After |
|---|---|
| 95cb068c10 | acfc208e4c |
| Captured server output and persisted state | Captured server output and persisted state |
Both damaged cases exit 1: migration 64 fails with malformed JSON; ledger remains 63. | Both damaged cases reach server readiness and HTTP 200; ledger reaches 87. Empty payloads remain unchanged; valid projection and event links are canonicalized. |
| Clean control succeeds. | Clean control succeeds. |
Driver · Synthetic seed script · Observations
Scope: migration and server startup are proven. The damaged-projection case still produces background reader/sweep JSON warnings after migrations finish; these remain visible in the after capture.
Not shown:
- Full client usability with a damaged projection was not verified; background projection readers still report JSON errors on head despite successful migration and server readiness.
- Nonempty malformed JSON variants were not separately exercised; both damaged cases used the reported empty-string payload.
Tests
vp test run src/persistence/Migrations/064_CanonicalizeThreadPullRequestLinks.test.ts (apps/server): fails without the fix with 'malformed JSON', passes with it (1/1). vp run check:changed: format, lint, knip, typecheck (3 packages) and tests (2 files) all pass.
Who did the work
- Written by Claude Opus 5.5, in Claude Code
- Reviewed by GPT-6.1 Sol, in Codex, and by Claude Opus 5.5, each on its own
- Proof captured by GPT-6.1 Sol and checked by Claude Opus 5.5
Run by the t3code factory in T3 Code.
tanstack-source added labelsbug
github-actions[bot](GitHub profile, opens in a new tab) added labelsvouch:trusted
github-actions[bot](GitHub profile, opens in a new tab) added labelssize:XS
Approvability
Verdict: Approved at
acfc208Macroscope's review found this PR approvable — Migration 64 now skips malformed JSON in thread projections and events rather than letting those payloads abort migration. Invalid rows remain untouched, and valid-row transformations are unchanged.
You can add or adjust custom eligibility rules. [Learn more](https://docs.macroscope.com/approvability#example-custom-eligibility-rules).Thread transfer impact
✅ Thread transfer remains within every enforced ceiling.
ℹ️ The exact PR base did not have a successful artifact. Baseline uses the latest successful
mainmeasurement shown below.Provider Metric Main baseline This PR Impact PR ceiling Codex Total thread wire 5.0 KiB 5.0 KiB 0 B (0.0%) 6.8 KiB ✅ Codex Thread snapshot wire 3.8 KiB 3.8 KiB 0 B (0.0%) 4.9 KiB ✅ Codex Live turn WebSocket wire 1.2 KiB 1.2 KiB 0 B (0.0%) 2.0 KiB ✅ Codex Live turn WebSocket decoded 21.5 KiB 21.5 KiB 0 B (0.0%) 24.4 KiB ✅ Codex Live turn messages 1 1 0 (0.0%) 8 ✅ Claude Total thread wire 5.0 KiB 5.0 KiB 0 B (0.0%) 6.8 KiB ✅ Claude Thread snapshot wire 3.8 KiB 3.8 KiB 0 B (0.0%) 4.9 KiB ✅ Claude Live turn WebSocket wire 1.2 KiB 1.2 KiB 0 B (0.0%) 2.0 KiB ✅ Claude Live turn WebSocket decoded 21.9 KiB 21.9 KiB 0 B (0.0%) 24.4 KiB ✅ Claude Live turn messages 1 1 0 (0.0%) 8 ✅ Other scenarios
All 64 scenario measurements match
main.All 64 scenario measurements
Provider Scenario Metric Main baseline This PR Impact PR ceiling Codex short turn resumeWebSocket decoded10.5 KiB 10.5 KiB 0 B (0.0%) 12.7 KiB ✅ Codex short turn resumeWebSocket messages2 2 0 (0.0%) 2 ✅ Codex long turn running, cold openHTTP decoded281.7 KiB 281.7 KiB 0 B (0.0%) 302.7 KiB ✅ Codex long turn running, cold openrows75 75 0 (0.0%) 75 ✅ Codex long turn running, history pageHTTP decoded92.4 KiB 92.4 KiB 0 B (0.0%) 107.4 KiB ✅ Codex long turn running, history pagerows75 75 0 (0.0%) 75 ✅ Codex long turn finished, cold openHTTP decoded381.5 KiB 381.5 KiB 0 B (0.0%) 410.2 KiB ✅ Codex long turn finished, cold openrows151 151 0 (0.0%) 151 ✅ Codex long turn finished, socket cold openWebSocket decoded381.6 KiB 381.6 KiB 0 B (0.0%) 410.2 KiB ✅ Codex long turn finished, socket cold openWebSocket messages2 2 0 (0.0%) 2 ✅ Codex long turn finished, socket cold openrows151 151 0 (0.0%) 151 ✅ Codex long turn finished, work pageHTTP decoded186.2 KiB 186.2 KiB 0 B (0.0%) 200.2 KiB ✅ Codex long turn finished, work pagerows150 150 0 (0.0%) 150 ✅ Codex idle thread resumeWebSocket decoded65 B 65 B 0 B (0.0%) 200 B ✅ Codex idle thread resumeWebSocket messages1 1 0 (0.0%) 1 ✅ Codex fleet shell snapshotHTTP decoded42.4 KiB 42.4 KiB 0 B (0.0%) 53.7 KiB ✅ Codex fleet shell snapshotrows27 27 0 (0.0%) 27 ✅ Codex fleet shell subscribeWebSocket decoded65 B 65 B 0 B (0.0%) 200 B ✅ Codex fleet shell subscribeWebSocket messages1 1 0 (0.0%) 1 ✅ Codex fleet shell resume checkHTTP decoded18 B 18 B 0 B (0.0%) 100 B ✅ Codex fleet shell resume checkrows0 0 0 0 ✅ Codex stale shell reloadHTTP decoded42.4 KiB 42.4 KiB 0 B (0.0%) 53.7 KiB ✅ Codex stale shell reloadrows27 27 0 (0.0%) 27 ✅ Codex stale shell resumeWebSocket decoded140 B 140 B 0 B (0.0%) 400 B ✅ Codex stale shell resumeWebSocket messages2 2 0 (0.0%) 2 ✅ Codex stale todo list cold openHTTP decoded72.4 KiB 72.4 KiB 0 B (0.0%) 83.0 KiB ✅ Codex stale todo list cold openrows75 75 0 (0.0%) 75 ✅ Codex stale todo list cold opennodes2 2 0 (0.0%) 4 ✅ Codex stale todo list cold opensubagents2 2 0 (0.0%) 4 ✅ Codex stale todo list cold openplans1 1 0 (0.0%) 2 ✅ Codex stale todo list cold openproviderTurns1 1 0 (0.0%) 2 ✅ Codex stale todo list cold opencheckpoints1 1 0 (0.0%) 2 ✅ Claude short turn resumeWebSocket decoded10.6 KiB 10.6 KiB 0 B (0.0%) 12.7 KiB ✅ Claude short turn resumeWebSocket messages2 2 0 (0.0%) 2 ✅ Claude long turn running, cold openHTTP decoded283.3 KiB 283.3 KiB 0 B (0.0%) 302.7 KiB ✅ Claude long turn running, cold openrows75 75 0 (0.0%) 75 ✅ Claude long turn running, history pageHTTP decoded93.4 KiB 93.4 KiB 0 B (0.0%) 107.4 KiB ✅ Claude long turn running, history pagerows75 75 0 (0.0%) 75 ✅ Claude long turn finished, cold openHTTP decoded384.2 KiB 384.2 KiB 0 B (0.0%) 410.2 KiB ✅ Claude long turn finished, cold openrows151 151 0 (0.0%) 151 ✅ Claude long turn finished, socket cold openWebSocket decoded384.3 KiB 384.3 KiB 0 B (0.0%) 410.2 KiB ✅ Claude long turn finished, socket cold openWebSocket messages2 2 0 (0.0%) 2 ✅ Claude long turn finished, socket cold openrows151 151 0 (0.0%) 151 ✅ Claude long turn finished, work pageHTTP decoded188.1 KiB 188.1 KiB 0 B (0.0%) 200.2 KiB ✅ Claude long turn finished, work pagerows150 150 0 (0.0%) 150 ✅ Claude idle thread resumeWebSocket decoded66 B 66 B 0 B (0.0%) 200 B ✅ Claude idle thread resumeWebSocket messages1 1 0 (0.0%) 1 ✅ Claude fleet shell snapshotHTTP decoded43.0 KiB 43.0 KiB 0 B (0.0%) 53.7 KiB ✅ Claude fleet shell snapshotrows27 27 0 (0.0%) 27 ✅ Claude fleet shell subscribeWebSocket decoded66 B 66 B 0 B (0.0%) 200 B ✅ Claude fleet shell subscribeWebSocket messages1 1 0 (0.0%) 1 ✅ Claude fleet shell resume checkHTTP decoded18 B 18 B 0 B (0.0%) 100 B ✅ Claude fleet shell resume checkrows0 0 0 0 ✅ Claude stale shell reloadHTTP decoded43.0 KiB 43.0 KiB 0 B (0.0%) 53.7 KiB ✅ Claude stale shell reloadrows27 27 0 (0.0%) 27 ✅ Claude stale shell resumeWebSocket decoded140 B 140 B 0 B (0.0%) 400 B ✅ Claude stale shell resumeWebSocket messages2 2 0 (0.0%) 2 ✅ Claude stale todo list cold openHTTP decoded73.4 KiB 73.4 KiB 0 B (0.0%) 83.0 KiB ✅ Claude stale todo list cold openrows75 75 0 (0.0%) 75 ✅ Claude stale todo list cold opennodes2 2 0 (0.0%) 4 ✅ Claude stale todo list cold opensubagents2 2 0 (0.0%) 4 ✅ Claude stale todo list cold openplans1 1 0 (0.0%) 2 ✅ Claude stale todo list cold openproviderTurns1 1 0 (0.0%) 2 ✅ Claude stale todo list cold opencheckpoints1 1 0 (0.0%) 2 ✅ Baseline:
36371de· PR result:acfc208· Source CI: successScenario and decoded snapshot size
10 historical turns, 5 command tools per turn, 878.9 KiB retained MCP result per historical turn, and a 1.05 MiB retained result in the measured turn.
- Codex decoded thread snapshot: 108.5 KiB
- Claude decoded thread snapshot: 108.8 KiB
Updated in place by a trusted workflow. PR artifacts are strictly validated and never executed.
juliusmarminge merged this
496856djuliusmarminge closed this
Sign in to join this conversation.
macroscopeapp[bot](GitHub profile, opens in a new tab) approved these changes