Ordered window starts two full-source reads after a signalling refetch with a NaN boundary #2221
Send to my agentSummary#
An ordered live query issues two full-source reads after one signalling refetch. The law in the ordered-work oracle allows one. The window is descending with nulls: first, an anti-joined marker filters it, and one source row has a NaN rank, so the bounded boundary cannot be expressed.
The failure is on main. #2194 does not cause it. It changes the oracle's generator, so the seed reproduces only with the oracle file from #2194.
Law#
The ordered-work oracle (packages/db/tests/query/ordered-work-oracle.property.test.ts) checks this rule: after a signalling refetch, the loader starts at most one full-source request. #2194 makes the law skip a bounded read that ends on a null or NaN rank, because ARCHITECTURE.md sends that boundary to the full-source fallback. Here the loader starts two full-source requests, not one.
Reproduce#
- Check out
main(tested at 501b4813c). - Copy
packages/db/tests/query/ordered-work-oracle.property.test.tsandpackages/db/tests/oracle-config.tsfrom theordered-loader-ledgerbranch (#2194). - From
packages/db, run:
TANSTACK_DB_ORACLE_SEED=-1142653059 npx vitest run --coverage.enabled=false \
tests/query/ordered-work-oracle.property.test.ts \
-t "keeps rows and per-change work bounded"
Result:
AssertionError: full-source requests after a signalling refetch at step 2: expected 2 to be 1
Counterexample:
{"path":"on-demand","provider":"refetch-signal","latency":"immediate","cancellation":"cooperative","peer":false,"filter":"anti-joined","via":"marker","indexed":false,"direction":"desc","nulls":"first","terms":1,"limit":2,"offset":0,"initial":[{"id":0,"rank":0,"eligible":true},{"id":2,"rank":NaN,"eligible":true},{"id":1,"rank":0,"eligible":false}],"history":[{"type":"delete-visible"},{"type":"move","offset":0,"limit":1}]}
Open question#
The cause is not known yet. There are two possible causes:
- The exemption in the law is too narrow for a
NaNrank at the start of a descending,nulls: firstorder. - The loader starts a second full-source read that it does not need.
A fix must decide which one, with ARCHITECTURE.md as the authority, before it changes the law or the loader.
This issue was written by Isaac.
All three traces pass on PR #2231 (branch
codex/issue-2224). Two separate changes fix them, so they do not share one marker-filter cause, and the ablation is not necessary.Trace Fails on mainFix on #2231 This issue (seed -1142653059) yes fe48cbe54: a short unindexed prefix grows and does not read the full sourceTrace 1 (seed -1572621508) yes, expected 2 to be +0fe48cbe54, the same causeTrace 2 (seed -104272808) yes, expected [ 10, 20, 30 ] to deeply equal [ 10 ]ce1299ded: a window move that waits behind a replay applies after a newer truncate discards that replayEvidence#
- This issue and trace 1: with the loader from before
fe48cbe54, each counterexample fails with the message above. With the change, the loader reads no full source after the refetch. Of the two possible causes in this issue, the second one applies: the loader started a full-source read that it did not need. - Trace 2: bisection over the PR's commits. The trace fails at
26bff5f4fand passes atce1299ded. The older code dropped the deferred one-row move together with the discarded replay, so the window kept three rows. - Each trace is now a named replay in its oracle (
5b4c304d4). The ordered-work oracle holds this issue and trace 1. The load-subset replay oracle holds trace 2.
#2191 is a different issue. Its model does not include truncate replay, and #2231 does not change it.
#2231 closes this issue.
This comment was written by Isaac.
- This issue and trace 1: with the loader from before
KyleAMathews(GitHub profile, opens in a new tab) closed this
🎉 The fix for this issue has been released!
Fixing PRs: https://github.com/TanStack/db/pull/2231
Backfilled notification for published release candidates.
Sign in to join this conversation.
Two more generated traces fail on
mainwith a similar shape. Each is an ordered window on the on-demand path, filtered through a marker, that goes through a replay. They may share a cause with this issue. That is a hypothesis from the shapes. No ablation has run yet.Trace 1: two full-source reads after a signalling refetch, ascending order#
This is the same law as this issue, without a
NaNrank. It failed onmainat 7538afd2f. Replay it frompackages/db:Trace 2: a window keeps its old rows after a truncate replay#
The window shrinks from 3 rows to 1 while its source is held, and then the source truncates. The settled window keeps the 3 rows. It failed on
mainat 7e46f81ad. Replay it frompackages/db:What the three traces share#
One possible single cause: the replay path re-establishes the window from request state that the replay made obsolete. The next step is an ablation. Replay each trace without the marker filter, and trace 2 without includes. If all three pass without the marker, the cause is likely in the marker-filtered replay path. #2191 is probably separate: it concerns failure bookkeeping, and its model leaves out truncate replay.
Both new traces came up in random campaigns on PR #2263. That PR does not cause them: each fails on
main.This comment was written by Isaac.