Fix unbounded expression growth in DeduplicatedLoadSubset #1348
main fromrefs/pull/1348/head·merged by KyleAMathews(GitHub profile, opens in a new tab) Fix unbounded WHERE expression growth in DeduplicatedLoadSubset when loading all data after accumulating specific predicates — prevents production 500 errors from oversized query bodies.
How the bug happens#
When multiple components query the same collection with specific filters (e.g., eq(task_id, uuid) as a user navigates between tasks), DeduplicatedLoadSubset accumulates those predicates into an IN(task_id, [uuid-1, ..., uuid-N]) expression — this is correct and working as designed.
The bug triggers when a page then loads the entire collection with no WHERE clause (e.g., a list/index page). The deduplication layer correctly computes a difference query — "give me everything I don't already have" → NOT(IN(task_id, [...])) — and sends it to the backend. But it incorrectly records that optimized expression as what was "loaded" instead of recording that all data was requested. So the system never learns it has everything, and each subsequent unfiltered load compounds the expression further, eventually producing a 193KB deeply nested WHERE clause that crashes downstream parsers.
Root Cause#
When loadSubset({}) (load all data) was called after accumulating many eq(task_id, uuid) predicates, the code computed an optimized difference query (NOT(IN(task_id, [...]))) and sent it to the backend — correct. But it then passed that same optimized expression to updateTracking(), which meant:
hasLoadedAllDatawas never set totrue(becausewherewasn'tundefined)unlimitedWherebecameIN([...]) OR NOT(IN([...]))— logically tautological but not simplified- Each subsequent "load all" compounded the expression further
- After enough cycles: 193KB deeply nested expression that crashes downstream parsers
Approach#
Separate cloneOptions into two copies at the point where the difference is computed:
trackingOptions: Preserves the original predicate (e.g.,where: undefined) forupdateTracking()— so "load all" correctly setshasLoadedAllData = trueloadOptions: May be narrowed with the difference expression for the actual backend request
Both the sync return path (return true) and async path (.then()) use trackingOptions for tracking and loadOptions for the backend call and in-flight matching.
Key Invariants#
trackingOptions.wherealways equals the caller's originalwherevalueloadOptions.wheremay differ (narrowed byminusWherePredicates)- In-flight entries store
loadOptions(matching must reflect what data the backend will actually return) updateTrackingreceivestrackingOptions(tracking must reflect what was requested)
Non-goals#
- Fixing the downstream parser crash (separate concern — Electric should return 400 not 500 for unparseable expressions)
- Adding a max-size threshold for accumulated predicates (possible future improvement)
- Optimizing
unionWherePredicatesto simplify tautological expressions
Verification#
pnpm vitest run packages/db/tests/query/subset-dedupe.test.ts
27 tests pass, including 4 new regression tests:
- Async: eq accumulation → load all → verify deduplication
- Sync: same scenario with synchronous
_loadSubsetreturn - Repeated "load all" without expression growth
- Extended existing test to verify post-load-all deduplication
Files Changed#
packages/db/src/query/subset-dedupe.ts— Split singlecloneOptionsintotrackingOptions+loadOptions; use appropriate copy for tracking vs backend callspackages/db/tests/query/subset-dedupe.test.ts— 4 new tests covering the production bug scenario (async + sync paths, repeated loads, expression structure verification); removed debug artifacts
- 1
More templates
@tanstack/angular-db
npm i https://pkg.pr.new/@tanstack/angular-db@1348@tanstack/db
npm i https://pkg.pr.new/@tanstack/db@1348@tanstack/db-ivm
npm i https://pkg.pr.new/@tanstack/db-ivm@1348@tanstack/electric-db-collection
npm i https://pkg.pr.new/@tanstack/electric-db-collection@1348@tanstack/offline-transactions
npm i https://pkg.pr.new/@tanstack/offline-transactions@1348@tanstack/powersync-db-collection
npm i https://pkg.pr.new/@tanstack/powersync-db-collection@1348@tanstack/query-db-collection
npm i https://pkg.pr.new/@tanstack/query-db-collection@1348@tanstack/react-db
npm i https://pkg.pr.new/@tanstack/react-db@1348@tanstack/rxdb-db-collection
npm i https://pkg.pr.new/@tanstack/rxdb-db-collection@1348@tanstack/solid-db
npm i https://pkg.pr.new/@tanstack/solid-db@1348@tanstack/svelte-db
npm i https://pkg.pr.new/@tanstack/svelte-db@1348@tanstack/trailbase-db-collection
npm i https://pkg.pr.new/@tanstack/trailbase-db-collection@1348@tanstack/vue-db
npm i https://pkg.pr.new/@tanstack/vue-db@1348commit:
d643e0dSize Change: +6 B (+0.01%)
Total Size: 93.2 kB
Filename Size Change ./packages/db/dist/esm/query/subset-dedupe.js927 B +6 B (+0.65%) compressed-size-action::db-package-sizeℹ️ View Unchanged
Filename Size ./packages/db/dist/esm/collection/change-events.js1.39 kB ./packages/db/dist/esm/collection/changes.js1.22 kB ./packages/db/dist/esm/collection/events.js388 B ./packages/db/dist/esm/collection/index.js3.32 kB ./packages/db/dist/esm/collection/indexes.js1.1 kB ./packages/db/dist/esm/collection/lifecycle.js1.75 kB ./packages/db/dist/esm/collection/mutations.js2.34 kB ./packages/db/dist/esm/collection/state.js3.49 kB ./packages/db/dist/esm/collection/subscription.js3.71 kB ./packages/db/dist/esm/collection/sync.js2.41 kB ./packages/db/dist/esm/deferred.js207 B ./packages/db/dist/esm/errors.js4.83 kB ./packages/db/dist/esm/event-emitter.js748 B ./packages/db/dist/esm/index.js2.72 kB ./packages/db/dist/esm/indexes/auto-index.js742 B ./packages/db/dist/esm/indexes/base-index.js766 B ./packages/db/dist/esm/indexes/btree-index.js2.17 kB ./packages/db/dist/esm/indexes/lazy-index.js1.1 kB ./packages/db/dist/esm/indexes/reverse-index.js538 B ./packages/db/dist/esm/local-only.js808 B ./packages/db/dist/esm/local-storage.js2.1 kB ./packages/db/dist/esm/optimistic-action.js359 B ./packages/db/dist/esm/paced-mutations.js496 B ./packages/db/dist/esm/proxy.js3.75 kB ./packages/db/dist/esm/query/builder/functions.js733 B ./packages/db/dist/esm/query/builder/index.js4.1 kB ./packages/db/dist/esm/query/builder/ref-proxy.js1.05 kB ./packages/db/dist/esm/query/compiler/evaluators.js1.43 kB ./packages/db/dist/esm/query/compiler/expressions.js430 B ./packages/db/dist/esm/query/compiler/group-by.js2.23 kB ./packages/db/dist/esm/query/compiler/index.js2.05 kB ./packages/db/dist/esm/query/compiler/joins.js2.11 kB ./packages/db/dist/esm/query/compiler/order-by.js1.45 kB ./packages/db/dist/esm/query/compiler/select.js1.09 kB ./packages/db/dist/esm/query/expression-helpers.js1.43 kB ./packages/db/dist/esm/query/ir.js673 B ./packages/db/dist/esm/query/live-query-collection.js360 B ./packages/db/dist/esm/query/live/collection-config-builder.js5.55 kB ./packages/db/dist/esm/query/live/collection-registry.js264 B ./packages/db/dist/esm/query/live/collection-subscriber.js2.42 kB ./packages/db/dist/esm/query/live/internal.js145 B ./packages/db/dist/esm/query/optimizer.js2.62 kB ./packages/db/dist/esm/query/predicate-utils.js2.97 kB ./packages/db/dist/esm/query/query-once.js359 B ./packages/db/dist/esm/scheduler.js1.3 kB ./packages/db/dist/esm/SortedMap.js1.3 kB ./packages/db/dist/esm/strategies/debounceStrategy.js247 B ./packages/db/dist/esm/strategies/queueStrategy.js428 B ./packages/db/dist/esm/strategies/throttleStrategy.js246 B ./packages/db/dist/esm/transactions.js2.9 kB ./packages/db/dist/esm/utils.js924 B ./packages/db/dist/esm/utils/browser-polyfills.js304 B ./packages/db/dist/esm/utils/btree.js5.61 kB ./packages/db/dist/esm/utils/comparison.js952 B ./packages/db/dist/esm/utils/cursor.js457 B ./packages/db/dist/esm/utils/index-optimization.js1.51 kB ./packages/db/dist/esm/utils/type-guards.js157 B Size Change: 0 B
Total Size: 3.85 kB
compressed-size-action::react-db-package-sizeℹ️ View Unchanged
Filename Size ./packages/react-db/dist/esm/index.js225 B ./packages/react-db/dist/esm/useLiveInfiniteQuery.js1.32 kB ./packages/react-db/dist/esm/useLiveQuery.js1.34 kB ./packages/react-db/dist/esm/useLiveSuspenseQuery.js559 B ./packages/react-db/dist/esm/usePacedMutations.js401 B KyleAMathews(GitHub profile, opens in a new tab) closed this
Sign in to join this conversation.
🦋 Changeset detected#
Latest commit: 3a1731954bd31038499f2fc48ec383a4cc271391
The changes in this PR will be included in the next version bump.
This PR includes changesets to release 12 packages
Not sure what this means? Click here to learn what changesets are.
Click here if you're a maintainer who wants to add another changeset to this PR