Skip to content
Source
Sign inSign up
TanStack/db

Fix unbounded expression growth in DeduplicatedLoadSubset #1348

MergedKyleAMathews(GitHub profile, opens in a new tab)merged 4 commits intomain fromrefs/pull/1348/headmerged by KyleAMathews(GitHub profile, opens in a new tab)

Fix unbounded WHERE expression growth in DeduplicatedLoadSubset when loading all data after accumulating specific predicates — prevents production 500 errors from oversized query bodies.

How the bug happens

When multiple components query the same collection with specific filters (e.g., eq(task_id, uuid) as a user navigates between tasks), DeduplicatedLoadSubset accumulates those predicates into an IN(task_id, [uuid-1, ..., uuid-N]) expression — this is correct and working as designed.

The bug triggers when a page then loads the entire collection with no WHERE clause (e.g., a list/index page). The deduplication layer correctly computes a difference query — "give me everything I don't already have" → NOT(IN(task_id, [...])) — and sends it to the backend. But it incorrectly records that optimized expression as what was "loaded" instead of recording that all data was requested. So the system never learns it has everything, and each subsequent unfiltered load compounds the expression further, eventually producing a 193KB deeply nested WHERE clause that crashes downstream parsers.

Root Cause

When loadSubset({}) (load all data) was called after accumulating many eq(task_id, uuid) predicates, the code computed an optimized difference query (NOT(IN(task_id, [...]))) and sent it to the backend — correct. But it then passed that same optimized expression to updateTracking(), which meant:

  1. hasLoadedAllData was never set to true (because where wasn't undefined)
  2. unlimitedWhere became IN([...]) OR NOT(IN([...])) — logically tautological but not simplified
  3. Each subsequent "load all" compounded the expression further
  4. After enough cycles: 193KB deeply nested expression that crashes downstream parsers

Approach

Separate cloneOptions into two copies at the point where the difference is computed:

  • trackingOptions: Preserves the original predicate (e.g., where: undefined) for updateTracking() — so "load all" correctly sets hasLoadedAllData = true
  • loadOptions: May be narrowed with the difference expression for the actual backend request

Both the sync return path (return true) and async path (.then()) use trackingOptions for tracking and loadOptions for the backend call and in-flight matching.

Key Invariants

  • trackingOptions.where always equals the caller's original where value
  • loadOptions.where may differ (narrowed by minusWherePredicates)
  • In-flight entries store loadOptions (matching must reflect what data the backend will actually return)
  • updateTracking receives trackingOptions (tracking must reflect what was requested)

Non-goals

  • Fixing the downstream parser crash (separate concern — Electric should return 400 not 500 for unparseable expressions)
  • Adding a max-size threshold for accumulated predicates (possible future improvement)
  • Optimizing unionWherePredicates to simplify tautological expressions

Verification

pnpm vitest run packages/db/tests/query/subset-dedupe.test.ts

27 tests pass, including 4 new regression tests:

  • Async: eq accumulation → load all → verify deduplication
  • Sync: same scenario with synchronous _loadSubset return
  • Repeated "load all" without expression growth
  • Extended existing test to verify post-load-all deduplication

Files Changed

  • packages/db/src/query/subset-dedupe.ts — Split single cloneOptions into trackingOptions + loadOptions; use appropriate copy for tracking vs backend calls
  • packages/db/tests/query/subset-dedupe.test.ts — 4 new tests covering the production bug scenario (async + sync paths, repeated loads, expression structure verification); removed debug artifacts

https://claude.ai/code/session_01Vp75RhjVR4tV5FdjKJbBWE

    1. 🦋 Changeset detected

      Latest commit: 3a1731954bd31038499f2fc48ec383a4cc271391

      The changes in this PR will be included in the next version bump.

      This PR includes changesets to release 12 packages
      NameType
      @tanstack/dbPatch
      @tanstack/angular-dbPatch
      @tanstack/electric-db-collectionPatch
      @tanstack/offline-transactionsPatch
      @tanstack/powersync-db-collectionPatch
      @tanstack/query-db-collectionPatch
      @tanstack/react-dbPatch
      @tanstack/rxdb-db-collectionPatch
      @tanstack/solid-dbPatch
      @tanstack/svelte-dbPatch
      @tanstack/trailbase-db-collectionPatch
      @tanstack/vue-dbPatch

      Not sure what this means? Click here to learn what changesets are.

      Click here if you're a maintainer who wants to add another changeset to this PR

    2. More templates

      @tanstack/angular-db

      npm i https://pkg.pr.new/@tanstack/angular-db@1348

      @tanstack/db

      npm i https://pkg.pr.new/@tanstack/db@1348

      @tanstack/db-ivm

      npm i https://pkg.pr.new/@tanstack/db-ivm@1348

      @tanstack/electric-db-collection

      npm i https://pkg.pr.new/@tanstack/electric-db-collection@1348

      @tanstack/offline-transactions

      npm i https://pkg.pr.new/@tanstack/offline-transactions@1348

      @tanstack/powersync-db-collection

      npm i https://pkg.pr.new/@tanstack/powersync-db-collection@1348

      @tanstack/query-db-collection

      npm i https://pkg.pr.new/@tanstack/query-db-collection@1348

      @tanstack/react-db

      npm i https://pkg.pr.new/@tanstack/react-db@1348

      @tanstack/rxdb-db-collection

      npm i https://pkg.pr.new/@tanstack/rxdb-db-collection@1348

      @tanstack/solid-db

      npm i https://pkg.pr.new/@tanstack/solid-db@1348

      @tanstack/svelte-db

      npm i https://pkg.pr.new/@tanstack/svelte-db@1348

      @tanstack/trailbase-db-collection

      npm i https://pkg.pr.new/@tanstack/trailbase-db-collection@1348

      @tanstack/vue-db

      npm i https://pkg.pr.new/@tanstack/vue-db@1348

      commit: d643e0d

    3. Size Change: +6 B (+0.01%)

      Total Size: 93.2 kB

      FilenameSizeChange
      ./packages/db/dist/esm/query/subset-dedupe.js927 B+6 B (+0.65%)
      ℹ️ View Unchanged
      FilenameSize
      ./packages/db/dist/esm/collection/change-events.js1.39 kB
      ./packages/db/dist/esm/collection/changes.js1.22 kB
      ./packages/db/dist/esm/collection/events.js388 B
      ./packages/db/dist/esm/collection/index.js3.32 kB
      ./packages/db/dist/esm/collection/indexes.js1.1 kB
      ./packages/db/dist/esm/collection/lifecycle.js1.75 kB
      ./packages/db/dist/esm/collection/mutations.js2.34 kB
      ./packages/db/dist/esm/collection/state.js3.49 kB
      ./packages/db/dist/esm/collection/subscription.js3.71 kB
      ./packages/db/dist/esm/collection/sync.js2.41 kB
      ./packages/db/dist/esm/deferred.js207 B
      ./packages/db/dist/esm/errors.js4.83 kB
      ./packages/db/dist/esm/event-emitter.js748 B
      ./packages/db/dist/esm/index.js2.72 kB
      ./packages/db/dist/esm/indexes/auto-index.js742 B
      ./packages/db/dist/esm/indexes/base-index.js766 B
      ./packages/db/dist/esm/indexes/btree-index.js2.17 kB
      ./packages/db/dist/esm/indexes/lazy-index.js1.1 kB
      ./packages/db/dist/esm/indexes/reverse-index.js538 B
      ./packages/db/dist/esm/local-only.js808 B
      ./packages/db/dist/esm/local-storage.js2.1 kB
      ./packages/db/dist/esm/optimistic-action.js359 B
      ./packages/db/dist/esm/paced-mutations.js496 B
      ./packages/db/dist/esm/proxy.js3.75 kB
      ./packages/db/dist/esm/query/builder/functions.js733 B
      ./packages/db/dist/esm/query/builder/index.js4.1 kB
      ./packages/db/dist/esm/query/builder/ref-proxy.js1.05 kB
      ./packages/db/dist/esm/query/compiler/evaluators.js1.43 kB
      ./packages/db/dist/esm/query/compiler/expressions.js430 B
      ./packages/db/dist/esm/query/compiler/group-by.js2.23 kB
      ./packages/db/dist/esm/query/compiler/index.js2.05 kB
      ./packages/db/dist/esm/query/compiler/joins.js2.11 kB
      ./packages/db/dist/esm/query/compiler/order-by.js1.45 kB
      ./packages/db/dist/esm/query/compiler/select.js1.09 kB
      ./packages/db/dist/esm/query/expression-helpers.js1.43 kB
      ./packages/db/dist/esm/query/ir.js673 B
      ./packages/db/dist/esm/query/live-query-collection.js360 B
      ./packages/db/dist/esm/query/live/collection-config-builder.js5.55 kB
      ./packages/db/dist/esm/query/live/collection-registry.js264 B
      ./packages/db/dist/esm/query/live/collection-subscriber.js2.42 kB
      ./packages/db/dist/esm/query/live/internal.js145 B
      ./packages/db/dist/esm/query/optimizer.js2.62 kB
      ./packages/db/dist/esm/query/predicate-utils.js2.97 kB
      ./packages/db/dist/esm/query/query-once.js359 B
      ./packages/db/dist/esm/scheduler.js1.3 kB
      ./packages/db/dist/esm/SortedMap.js1.3 kB
      ./packages/db/dist/esm/strategies/debounceStrategy.js247 B
      ./packages/db/dist/esm/strategies/queueStrategy.js428 B
      ./packages/db/dist/esm/strategies/throttleStrategy.js246 B
      ./packages/db/dist/esm/transactions.js2.9 kB
      ./packages/db/dist/esm/utils.js924 B
      ./packages/db/dist/esm/utils/browser-polyfills.js304 B
      ./packages/db/dist/esm/utils/btree.js5.61 kB
      ./packages/db/dist/esm/utils/comparison.js952 B
      ./packages/db/dist/esm/utils/cursor.js457 B
      ./packages/db/dist/esm/utils/index-optimization.js1.51 kB
      ./packages/db/dist/esm/utils/type-guards.js157 B
      compressed-size-action::db-package-size
    4. Size Change: 0 B

      Total Size: 3.85 kB

      ℹ️ View Unchanged
      FilenameSize
      ./packages/react-db/dist/esm/index.js225 B
      ./packages/react-db/dist/esm/useLiveInfiniteQuery.js1.32 kB
      ./packages/react-db/dist/esm/useLiveQuery.js1.34 kB
      ./packages/react-db/dist/esm/useLiveSuspenseQuery.js559 B
      ./packages/react-db/dist/esm/usePacedMutations.js401 B
      compressed-size-action::react-db-package-size
    5. 🎉 This PR has been released!

      Thank you for your contribution!

Sign in to join this conversation.