Electric on-demand skips updates to unknown rows that enter a served ordered range #2138
Send to my agentProblem#
An on-demand Electric collection with the default replica mode can show a wrong ordered window. A row that the Collection did not load can move into a served ordered range, and the Collection never receives it.
Electric skips an update for a row that the Collection does not know, unless the shape uses replica: 'full' (packages/electric-db-collection/src/electric.ts, near line 2382). A default-replica update contains only the changed columns, so the client cannot make the row from it. The documentation states this behavior: "unknown partial rows are ignored" (docs/collections/electric-collection.md:414).
This conflicts with the LoadSubset rule from #2136: a provider keeps each served ordered range whole. After a load serves an ordered, limited page, every row that later sorts inside that range must arrive. Electric's move-in and tag messages apply to rows that Electric already knows. They do not cover a row that newly sorts into a limited ordered range.
Reproduction#
On-demand mode, default replica, live query orderBy(rank desc).limit(2).
- Rows with rank 10, 4, 1, 0. The window is
[10, 4]. - The shape stream sends an update for the row with rank 0, which the Collection did not load. Its rank changes to 7.
- Expected window:
[10, 7]. Actual window:[10, 4].
With replica: 'full', the row arrives and the window is [10, 7].
A witness test uses the package's mocked ShapeStream. It will go into the fix PR. It does not prove which messages a real Electric server sends for this row. The fix PR must confirm that against the server or its documentation.
Options#
- When an update arrives for an unknown row, fetch the full row, or reload the served subsets that the row can enter.
- Invalidate the limited ordered subsets when such an update arrives, so that the DB loader requests them again.
- Accept the gap as a limit, and document that ordered on-demand windows need
replica: 'full'.
Related: #2136 (the same rule for query-db).
This issue was written by Isaac.
Sign in to join this conversation.
A possible route, from #2154 (the #2136 fix). It is not merged yet.
#2154 adds a sync-adapter call,
invalidateLoadedSubsets(). A sync adapter that cannot keep a served ordered range whole calls it. Core then makes each ordered live query and Effect on the Collection re-establish its window with the existing bounded repair: arefetch: trueprefix from the start of the order throughoffset + limit, then a tie request and any refill. It does not read the full source. TheLoadSubsetFnrule becomes: a sync adapter that refetches or receives changes for loaded subsets on its own either keeps served prefixes whole, or callsinvalidateLoadedSubsets()once the rows apply.For Electric's default replica mode, this means:
updatefor a row the Collection does not know, it callsinvalidateLoadedSubsets()instead of only ignoring it.loadSubsetpath.Core coalesces the call to one event per turn, so a burst of skipped updates starts one repair per ordered window. Plans that already hold the full source ignore it.
Points to check before choosing this route: