Load-subset replay oracle: failed marker load after marker truncate leaves the live query in a terminal error #2239
Send to my agentSummary#
The random campaign of the load-subset replay refinement oracle fails on main (427a7c31e). A live query ends in the error status after a failed marker load. The oracle's model says no fatal load failed in this history, so it expects the window to recover.
- Test:
packages/db/tests/query/load-subset-replay-refinement-oracle.test.ts› "Ordered windows across truncate replays › keeps the ordered window under load and release faults (random campaign)" - Assertion:
convergence: terminal error: expected false to be true(line 1875:expect(fatalLoadFailed(), ...).toBe(true))
Reproduce#
cd packages/db
TANSTACK_DB_ORACLE_SEED=1930131799 npx vitest run \
tests/query/load-subset-replay-refinement-oracle.test.ts \
-t "load and release faults \(random"
fast-check output:
{ seed: 1930131799, path: "33:3:1:4:0:2:3:2:2", endOnFailure: true }
Counterexample: [[{"type":"hold"},{"type":"truncateMarkers"},{"type":"settle","ok":false}],{"direction":"asc","offset":0,"limit":1,"ties":false,"peer":false,"joined":"none","markerMode":"on-demand","includes":true}]
Shrunk 8 time(s)
The shrunk history:
- hold the loads;
- truncate the marker source;
- settle the held loads with a failure.
The scenario is an ascending window (offset 0, limit 1) with on-demand markers and includes, no ties, no peer and no join.
What the oracle checks#
After the faults, the recovery step truncates every source, settles, and moves the window again explicitly. A failed replay keeps the last complete publication until a later authoritative replay, and a failed window operation stays private until a later explicit move revalidates it (see ARCHITECTURE.md). After that, the live query must either show the expected window, or be in error only if fatalLoadFailed() (demandFailed || failedWhileLoading) holds. In this history the query is in error, but fatalLoadFailed() is false.
Undecided#
Two explanations fit this result, and I haven't decided between them:
- Production defect: a marker load that fails while held across a marker truncate becomes a terminal query error, though the documented recovery should clear it.
- Model gap: this failure really is fatal, and
fatalLoadFailed()misses the case where a marker load fails after a marker truncate.
The fixed campaign passes. I found this while working on #2220 (branch issue-2220-sync-metadata-held-reads), but it reproduces on a clean main checkout, and that branch doesn't touch this oracle or its production path.
KyleAMathews(GitHub profile, opens in a new tab) closed this
Sign in to join this conversation.
Duplicate of #2224, which is fixed by #2229. The shrunk trace here (hold,
truncateMarkers, then a failedsettleon an ascendinglimit: 1window with on-demand markers and includes) passes onmainat1640de834. Details are in #2224.