Skip to content
Source
Sign inSign up
TanStack/db

fix(db): propagate changes through nested toArray includes at depth 3+ #1457

MergedKyleAMathews(GitHub profile, opens in a new tab)merged 5 commits intomain fromrefs/pull/1457/headmerged by KyleAMathews(GitHub profile, opens in a new tab)

Summary

Nested toArray() includes (depth 3+) silently drop changes. A query like toArray(runs) → toArray(texts) → concat(toArray(textDeltas)) never propagates textDelta inserts — the concatenated text stays empty forever. Also, wrapping toArray() in expressions like coalesce(toArray(...)) silently produces incorrect results instead of erroring.

Root Cause

flushIncludesState has five phases. Phase 3 (drainNestedBuffers) only drains one level of shared nested buffers — it processes the immediate children's buffer but not grandchildren's. Phase 4 then recursively flushes only entries that Phase 2 (direct child changes) or Phase 3 (buffer-drained changes) marked as dirty.

When only the deepest level changes (e.g., a textDelta insert), intermediate levels have no changes, so neither Phase 2 nor Phase 3 produce dirty entries. Phase 4 does nothing, and the deep buffer changes are permanently stranded.

Approach

Added a third pass at the end of Phase 4 that scans all child registry entries for deep nested buffer changes via hasPendingIncludesChanges(). This function already recursively checks buffers at all depths. Entries with stranded buffers are recursively flushed, and their correlation keys are added to the inline re-emit set so parent rows are re-materialized.

// Phase 4c: entries with deep nested buffer changes
const deepBufferDirty = new Set<unknown>()
if (state.nestedSetups) {
  for (const [correlationKey, entry] of state.childRegistry) {
    if (entriesWithChildChanges.has(correlationKey)) continue
    if (dirtyFromBuffers.has(correlationKey)) continue
    if (entry.includesStates && hasPendingIncludesChanges(entry.includesStates)) {
      flushIncludesState(entry.includesStates, ...)
      deepBufferDirty.add(correlationKey)
    }
  }
}

The recursive flushIncludesState call applies the same logic at each level, so this handles arbitrary nesting depth.

For the coalesce(toArray()) issue, added brand-based runtime checks in toExpression() that throw a clear error at query construction time.

Key Invariants

  • drainNestedBuffers only routes entries matching the per-entry routing index — unroutable entries stay in the shared buffer for other entries to claim
  • The deep buffer pass runs after Phase 3 drains, so it only catches genuinely stranded changes
  • Each recursive flushIncludesState call applies the same three-pass logic, enabling arbitrary depth

Non-goals

  • Making drainNestedBuffers itself recursive (would change the existing single-level routing model)
  • Adding try-catch per entry in flush loops (pre-existing pattern, out of scope)

Verification

pnpm vitest run packages/db/tests/query/repro-bug2.test.ts --reporter=verbose
pnpm vitest run packages/db/tests/ --reporter=verbose
pnpm vitest run packages/db-ivm/ --reporter=verbose

All 2382 tests pass (2060 db + 322 IVM), including 10 new regression tests.

Files changed

FileChange
packages/db/src/query/live/collection-config-builder.tsAdd deep buffer pass in Phase 4 of flushIncludesState()
packages/db/src/query/builder/ref-proxy.tsAdd brand-based runtime checks for ToArrayWrapper/ConcatToArrayWrapper in toExpression()
packages/db/src/query/builder/functions.tsAdd __brand property to wrapper classes
packages/db/tests/query/repro-bug2.test.ts10 regression tests: error throwing, flat propagation, chained collections, nested includes, spurious update guard
packages/db/tests/query/repro-many-siblings.test.tsRegression test matching the darix 5-sibling entity-timeline pattern

🤖 Generated with Claude Code

    1. More templates

      @tanstack/angular-db

      npm i https://pkg.pr.new/@tanstack/angular-db@1457

      @tanstack/browser-db-sqlite-persistence

      npm i https://pkg.pr.new/@tanstack/browser-db-sqlite-persistence@1457

      @tanstack/capacitor-db-sqlite-persistence

      npm i https://pkg.pr.new/@tanstack/capacitor-db-sqlite-persistence@1457

      @tanstack/cloudflare-durable-objects-db-sqlite-persistence

      npm i https://pkg.pr.new/@tanstack/cloudflare-durable-objects-db-sqlite-persistence@1457

      @tanstack/db

      npm i https://pkg.pr.new/@tanstack/db@1457

      @tanstack/db-ivm

      npm i https://pkg.pr.new/@tanstack/db-ivm@1457

      @tanstack/db-sqlite-persistence-core

      npm i https://pkg.pr.new/@tanstack/db-sqlite-persistence-core@1457

      @tanstack/electric-db-collection

      npm i https://pkg.pr.new/@tanstack/electric-db-collection@1457

      @tanstack/electron-db-sqlite-persistence

      npm i https://pkg.pr.new/@tanstack/electron-db-sqlite-persistence@1457

      @tanstack/expo-db-sqlite-persistence

      npm i https://pkg.pr.new/@tanstack/expo-db-sqlite-persistence@1457

      @tanstack/node-db-sqlite-persistence

      npm i https://pkg.pr.new/@tanstack/node-db-sqlite-persistence@1457

      @tanstack/offline-transactions

      npm i https://pkg.pr.new/@tanstack/offline-transactions@1457

      @tanstack/powersync-db-collection

      npm i https://pkg.pr.new/@tanstack/powersync-db-collection@1457

      @tanstack/query-db-collection

      npm i https://pkg.pr.new/@tanstack/query-db-collection@1457

      @tanstack/react-db

      npm i https://pkg.pr.new/@tanstack/react-db@1457

      @tanstack/react-native-db-sqlite-persistence

      npm i https://pkg.pr.new/@tanstack/react-native-db-sqlite-persistence@1457

      @tanstack/rxdb-db-collection

      npm i https://pkg.pr.new/@tanstack/rxdb-db-collection@1457

      @tanstack/solid-db

      npm i https://pkg.pr.new/@tanstack/solid-db@1457

      @tanstack/svelte-db

      npm i https://pkg.pr.new/@tanstack/svelte-db@1457

      @tanstack/tauri-db-sqlite-persistence

      npm i https://pkg.pr.new/@tanstack/tauri-db-sqlite-persistence@1457

      @tanstack/trailbase-db-collection

      npm i https://pkg.pr.new/@tanstack/trailbase-db-collection@1457

      @tanstack/vue-db

      npm i https://pkg.pr.new/@tanstack/vue-db@1457

      commit: 96d3b6f

    2. Size Change: +251 B (+0.22%)

      Total Size: 113 kB

      📦 View Changed
      FilenameSizeChange
      packages/db/dist/esm/query/builder/functions.js919 B+14 B (+1.55%)
      packages/db/dist/esm/query/builder/ref-proxy.js1.2 kB+151 B (+14.39%)⚠️
      packages/db/dist/esm/query/live/collection-config-builder.js7.88 kB+86 B (+1.1%)
      ℹ️ View Unchanged
      FilenameSize
      packages/db/dist/esm/collection/change-events.js1.39 kB
      packages/db/dist/esm/collection/changes.js1.38 kB
      packages/db/dist/esm/collection/cleanup-queue.js810 B
      packages/db/dist/esm/collection/events.js434 B
      packages/db/dist/esm/collection/index.js3.61 kB
      packages/db/dist/esm/collection/indexes.js1.99 kB
      packages/db/dist/esm/collection/lifecycle.js1.69 kB
      packages/db/dist/esm/collection/mutations.js2.47 kB
      packages/db/dist/esm/collection/state.js5.26 kB
      packages/db/dist/esm/collection/subscription.js3.74 kB
      packages/db/dist/esm/collection/sync.js2.88 kB
      packages/db/dist/esm/collection/transaction-metadata.js144 B
      packages/db/dist/esm/deferred.js207 B
      packages/db/dist/esm/errors.js4.92 kB
      packages/db/dist/esm/event-emitter.js748 B
      packages/db/dist/esm/index.js3 kB
      packages/db/dist/esm/indexes/auto-index.js830 B
      packages/db/dist/esm/indexes/base-index.js729 B
      packages/db/dist/esm/indexes/basic-index.js2.05 kB
      packages/db/dist/esm/indexes/btree-index.js2.17 kB
      packages/db/dist/esm/indexes/index-registry.js820 B
      packages/db/dist/esm/indexes/reverse-index.js538 B
      packages/db/dist/esm/local-only.js890 B
      packages/db/dist/esm/local-storage.js2.1 kB
      packages/db/dist/esm/optimistic-action.js359 B
      packages/db/dist/esm/paced-mutations.js496 B
      packages/db/dist/esm/proxy.js3.75 kB
      packages/db/dist/esm/query/builder/index.js5.25 kB
      packages/db/dist/esm/query/compiler/evaluators.js1.62 kB
      packages/db/dist/esm/query/compiler/expressions.js430 B
      packages/db/dist/esm/query/compiler/group-by.js2.69 kB
      packages/db/dist/esm/query/compiler/index.js3.63 kB
      packages/db/dist/esm/query/compiler/joins.js2.15 kB
      packages/db/dist/esm/query/compiler/order-by.js1.51 kB
      packages/db/dist/esm/query/compiler/select.js1.11 kB
      packages/db/dist/esm/query/effect.js4.78 kB
      packages/db/dist/esm/query/expression-helpers.js1.43 kB
      packages/db/dist/esm/query/ir.js829 B
      packages/db/dist/esm/query/live-query-collection.js360 B
      packages/db/dist/esm/query/live/collection-registry.js264 B
      packages/db/dist/esm/query/live/collection-subscriber.js1.94 kB
      packages/db/dist/esm/query/live/internal.js145 B
      packages/db/dist/esm/query/live/utils.js1.64 kB
      packages/db/dist/esm/query/optimizer.js2.62 kB
      packages/db/dist/esm/query/predicate-utils.js2.97 kB
      packages/db/dist/esm/query/query-once.js359 B
      packages/db/dist/esm/query/subset-dedupe.js960 B
      packages/db/dist/esm/scheduler.js1.3 kB
      packages/db/dist/esm/SortedMap.js1.3 kB
      packages/db/dist/esm/strategies/debounceStrategy.js247 B
      packages/db/dist/esm/strategies/queueStrategy.js428 B
      packages/db/dist/esm/strategies/throttleStrategy.js246 B
      packages/db/dist/esm/transactions.js2.9 kB
      packages/db/dist/esm/utils.js927 B
      packages/db/dist/esm/utils/array-utils.js273 B
      packages/db/dist/esm/utils/browser-polyfills.js304 B
      packages/db/dist/esm/utils/btree.js5.61 kB
      packages/db/dist/esm/utils/comparison.js1.05 kB
      packages/db/dist/esm/utils/cursor.js457 B
      packages/db/dist/esm/utils/index-optimization.js1.54 kB
      packages/db/dist/esm/utils/type-guards.js157 B
      packages/db/dist/esm/virtual-props.js360 B
      compressed-size-action::db-package-size
    3. Size Change: 0 B

      Total Size: 4.24 kB

      ℹ️ View Unchanged
      FilenameSize
      packages/react-db/dist/esm/index.js249 B
      packages/react-db/dist/esm/useLiveInfiniteQuery.js1.32 kB
      packages/react-db/dist/esm/useLiveQuery.js1.34 kB
      packages/react-db/dist/esm/useLiveQueryEffect.js355 B
      packages/react-db/dist/esm/useLiveSuspenseQuery.js567 B
      packages/react-db/dist/esm/usePacedMutations.js401 B
      compressed-size-action::react-db-package-size
    4. GPT5.2 review:


      Looks good to me.

      What I reviewed

      • Nested include propagation changes in collection-config-builder
      • New runtime guard for toArray() / concat(toArray()) inside expressions
      • Regression coverage added in includes.test.ts

      What I checked

      • The depth-3+ nested toArray() propagation fix appears consistent with the reported failure mode
      • The new toExpression guard correctly prevents toArray() / concat(toArray()) from being used inside expression helpers, with a clear error message
      • The added tests cover the main regression scenarios, including nested/chained includes and sibling-update isolation

      Validation

      • Ran the @tanstack/db test suite locally from the PR branch
      • Result: 94 test files passed, 2271 tests passed, 5 skipped, no type errors

      Notes

      • I didn’t find any concrete bugs or regressions in the PR
      • Only minor residual risk I see is that some of the new async tests use fixed setTimeout(...) delays, which could be a source of occasional CI flakiness, but not a correctness issue with the implementation itself

      LGTM

    5. Opus 4.6 Review:


      All 112 tests pass. Here's my review of PR #1457:


      PR Review: fix(db): propagate changes through nested toArray includes at depth 3+

      Overall assessment: This is a well-targeted, minimal fix for a real bug with excellent test coverage. A few suggestions below.

      What the PR does

      Two distinct fixes:

      1. Core fix — Adds a "Phase 4c" pass in flushIncludesState that catches entries with pending deep nested buffer changes (depth 3+) that were stranded because drainNestedBuffers only drains one level.

      2. Guard rail — Adds runtime brand-based detection in toExpression() so that toArray() / concat(toArray()) inside expressions like coalesce() throws a clear error instead of silently producing wrong results.

      Correctness analysis

      The core fix is correct. The key insight is sound: when only the deepest level changes (e.g., a textDelta insert at depth 3), intermediate levels have no direct or buffer-drained changes, so they're never marked dirty and never flushed by Phase 4. The new pass uses the existing recursive hasPendingIncludesChanges() to detect stranded changes and recursively flushes them.

      I traced through the shared buffer model carefully. In createPerEntryIncludesStates (line 1309), per-entry states share the same nestedSetups references as the parent:

            state.nestedSetups = setup.nestedSetups

      This means hasPendingIncludesChanges will return true for all entries sharing that setup when any deep buffer has changes — not just the entry that "owns" the change. This could cause unnecessary recursive flushes for unaffected entries. However, this is not a correctness issue: drainNestedBuffers inside the recursive flush uses the per-entry routing index, so only properly-routed changes are consumed, and unroutable entries remain in the buffer for the correct entry to claim.

      Suggestions

      1. Consider instanceof instead of __brand for the expression guard

      The wrapper classes are proper classes in the same package, so instanceof would work and be more robust than duck-typing:

      import { ToArrayWrapper, ConcatToArrayWrapper } from './functions.js'
      
      if (value instanceof ToArrayWrapper || value instanceof ConcatToArrayWrapper) {
        const name = value instanceof ToArrayWrapper ? `toArray()` : `concat(toArray())`
        throw new Error(...)
      }

      This avoids adding a runtime __brand property and is immune to false positives from arbitrary objects that happen to have __brand: 'ToArrayWrapper'. It would also mean no changes needed to functions.ts at all.

      2. Minor performance: early exit for the deep buffer pass

      The deep buffer pass iterates the entire childRegistry and calls hasPendingIncludesChanges on each entry. A quick pre-check on the shared buffers could skip the loop entirely when there are no deep changes:

      if (state.nestedSetups && hasNestedBufferChanges(state.nestedSetups)) {
        for (const [correlationKey, entry] of state.childRegistry) {
          // ... existing logic
        }
      }

      This replaces the current if (state.nestedSetups) guard. Since hasNestedBufferChanges is cheap (just checks .size > 0 on buffers), this would avoid the registry scan in the common case where no deep changes are pending.

      However, this might not be fully correct because per-entry pendingChildChanges are separate from the shared nested buffers. If changes ended up in per-entry pendingChildChanges through some other path, this early exit would miss them. Worth double-checking whether that path is possible for the stranded-changes scenario.

      3. PR description file table is inaccurate

      The description lists repro-bug2.test.ts and repro-many-siblings.test.ts as changed files, but all tests were actually added to includes.test.ts. Minor, but worth correcting for reviewers.

      Tests

      The test coverage is excellent — 10 new tests covering:

      • Error throwing for misuse of toArray()/concat(toArray()) in expressions (2 tests)
      • Sequential insert propagation through various patterns (5 tests)
      • Depth 3+ nested propagation (2 tests including a control)
      • Spurious update guard ensuring sibling parents aren't unnecessarily re-emitted (1 test)

      The spurious update test at line 4902 is particularly good — it verifies referential identity (toBe) to ensure unaffected sibling arrays aren't re-created:

            expect(data().runs[0].texts[0].text).toBe(``)
      
            expect(data().runs[0].texts).toBe(run1TextsBefore)

      The tests use setTimeout-based synchronization (50-100ms), which is consistent with existing patterns in this file but inherently timing-dependent. Not a blocker for this PR.

      Summary

      Approve with minor suggestions. The fix is minimal, correct, well-tested, and solves a real user-facing bug. The instanceof vs __brand suggestion and the early-exit optimization are quality improvements worth considering but not blockers.

    6. samwillis(GitHub profile, opens in a new tab) approved these changes

      I think this looks good, I had a good read through too while the agents were looking. I think Opus is being overly pernickety.

      :shipit:

Sign in to join this conversation.