fix(db): propagate changes through nested toArray includes at depth 3+ #1457
main fromrefs/pull/1457/head·merged by KyleAMathews(GitHub profile, opens in a new tab) Summary#
Nested toArray() includes (depth 3+) silently drop changes. A query like toArray(runs) → toArray(texts) → concat(toArray(textDeltas)) never propagates textDelta inserts — the concatenated text stays empty forever. Also, wrapping toArray() in expressions like coalesce(toArray(...)) silently produces incorrect results instead of erroring.
Root Cause#
flushIncludesState has five phases. Phase 3 (drainNestedBuffers) only drains one level of shared nested buffers — it processes the immediate children's buffer but not grandchildren's. Phase 4 then recursively flushes only entries that Phase 2 (direct child changes) or Phase 3 (buffer-drained changes) marked as dirty.
When only the deepest level changes (e.g., a textDelta insert), intermediate levels have no changes, so neither Phase 2 nor Phase 3 produce dirty entries. Phase 4 does nothing, and the deep buffer changes are permanently stranded.
Approach#
Added a third pass at the end of Phase 4 that scans all child registry entries for deep nested buffer changes via hasPendingIncludesChanges(). This function already recursively checks buffers at all depths. Entries with stranded buffers are recursively flushed, and their correlation keys are added to the inline re-emit set so parent rows are re-materialized.
// Phase 4c: entries with deep nested buffer changes
const deepBufferDirty = new Set<unknown>()
if (state.nestedSetups) {
for (const [correlationKey, entry] of state.childRegistry) {
if (entriesWithChildChanges.has(correlationKey)) continue
if (dirtyFromBuffers.has(correlationKey)) continue
if (entry.includesStates && hasPendingIncludesChanges(entry.includesStates)) {
flushIncludesState(entry.includesStates, ...)
deepBufferDirty.add(correlationKey)
}
}
}
The recursive flushIncludesState call applies the same logic at each level, so this handles arbitrary nesting depth.
For the coalesce(toArray()) issue, added brand-based runtime checks in toExpression() that throw a clear error at query construction time.
Key Invariants#
drainNestedBuffersonly routes entries matching the per-entry routing index — unroutable entries stay in the shared buffer for other entries to claim- The deep buffer pass runs after Phase 3 drains, so it only catches genuinely stranded changes
- Each recursive
flushIncludesStatecall applies the same three-pass logic, enabling arbitrary depth
Non-goals#
- Making
drainNestedBuffersitself recursive (would change the existing single-level routing model) - Adding try-catch per entry in flush loops (pre-existing pattern, out of scope)
Verification#
pnpm vitest run packages/db/tests/query/repro-bug2.test.ts --reporter=verbose
pnpm vitest run packages/db/tests/ --reporter=verbose
pnpm vitest run packages/db-ivm/ --reporter=verbose
All 2382 tests pass (2060 db + 322 IVM), including 10 new regression tests.
Files changed#
| File | Change |
|---|---|
packages/db/src/query/live/collection-config-builder.ts | Add deep buffer pass in Phase 4 of flushIncludesState() |
packages/db/src/query/builder/ref-proxy.ts | Add brand-based runtime checks for ToArrayWrapper/ConcatToArrayWrapper in toExpression() |
packages/db/src/query/builder/functions.ts | Add __brand property to wrapper classes |
packages/db/tests/query/repro-bug2.test.ts | 10 regression tests: error throwing, flat propagation, chained collections, nested includes, spurious update guard |
packages/db/tests/query/repro-many-siblings.test.ts | Regression test matching the darix 5-sibling entity-timeline pattern |
🤖 Generated with Claude Code
- 1
Size Change: +251 B (+0.22%)
Total Size: 113 kB
📦 View Changed
Filename Size Change packages/db/dist/esm/query/builder/functions.js919 B +14 B (+1.55%) packages/db/dist/esm/query/builder/ref-proxy.js1.2 kB +151 B (+14.39%) ⚠️ packages/db/dist/esm/query/live/collection-config-builder.js7.88 kB +86 B (+1.1%) compressed-size-action::db-package-sizeℹ️ View Unchanged
Filename Size packages/db/dist/esm/collection/change-events.js1.39 kB packages/db/dist/esm/collection/changes.js1.38 kB packages/db/dist/esm/collection/cleanup-queue.js810 B packages/db/dist/esm/collection/events.js434 B packages/db/dist/esm/collection/index.js3.61 kB packages/db/dist/esm/collection/indexes.js1.99 kB packages/db/dist/esm/collection/lifecycle.js1.69 kB packages/db/dist/esm/collection/mutations.js2.47 kB packages/db/dist/esm/collection/state.js5.26 kB packages/db/dist/esm/collection/subscription.js3.74 kB packages/db/dist/esm/collection/sync.js2.88 kB packages/db/dist/esm/collection/transaction-metadata.js144 B packages/db/dist/esm/deferred.js207 B packages/db/dist/esm/errors.js4.92 kB packages/db/dist/esm/event-emitter.js748 B packages/db/dist/esm/index.js3 kB packages/db/dist/esm/indexes/auto-index.js830 B packages/db/dist/esm/indexes/base-index.js729 B packages/db/dist/esm/indexes/basic-index.js2.05 kB packages/db/dist/esm/indexes/btree-index.js2.17 kB packages/db/dist/esm/indexes/index-registry.js820 B packages/db/dist/esm/indexes/reverse-index.js538 B packages/db/dist/esm/local-only.js890 B packages/db/dist/esm/local-storage.js2.1 kB packages/db/dist/esm/optimistic-action.js359 B packages/db/dist/esm/paced-mutations.js496 B packages/db/dist/esm/proxy.js3.75 kB packages/db/dist/esm/query/builder/index.js5.25 kB packages/db/dist/esm/query/compiler/evaluators.js1.62 kB packages/db/dist/esm/query/compiler/expressions.js430 B packages/db/dist/esm/query/compiler/group-by.js2.69 kB packages/db/dist/esm/query/compiler/index.js3.63 kB packages/db/dist/esm/query/compiler/joins.js2.15 kB packages/db/dist/esm/query/compiler/order-by.js1.51 kB packages/db/dist/esm/query/compiler/select.js1.11 kB packages/db/dist/esm/query/effect.js4.78 kB packages/db/dist/esm/query/expression-helpers.js1.43 kB packages/db/dist/esm/query/ir.js829 B packages/db/dist/esm/query/live-query-collection.js360 B packages/db/dist/esm/query/live/collection-registry.js264 B packages/db/dist/esm/query/live/collection-subscriber.js1.94 kB packages/db/dist/esm/query/live/internal.js145 B packages/db/dist/esm/query/live/utils.js1.64 kB packages/db/dist/esm/query/optimizer.js2.62 kB packages/db/dist/esm/query/predicate-utils.js2.97 kB packages/db/dist/esm/query/query-once.js359 B packages/db/dist/esm/query/subset-dedupe.js960 B packages/db/dist/esm/scheduler.js1.3 kB packages/db/dist/esm/SortedMap.js1.3 kB packages/db/dist/esm/strategies/debounceStrategy.js247 B packages/db/dist/esm/strategies/queueStrategy.js428 B packages/db/dist/esm/strategies/throttleStrategy.js246 B packages/db/dist/esm/transactions.js2.9 kB packages/db/dist/esm/utils.js927 B packages/db/dist/esm/utils/array-utils.js273 B packages/db/dist/esm/utils/browser-polyfills.js304 B packages/db/dist/esm/utils/btree.js5.61 kB packages/db/dist/esm/utils/comparison.js1.05 kB packages/db/dist/esm/utils/cursor.js457 B packages/db/dist/esm/utils/index-optimization.js1.54 kB packages/db/dist/esm/utils/type-guards.js157 B packages/db/dist/esm/virtual-props.js360 B Size Change: 0 B
Total Size: 4.24 kB
compressed-size-action::react-db-package-sizeℹ️ View Unchanged
Filename Size packages/react-db/dist/esm/index.js249 B packages/react-db/dist/esm/useLiveInfiniteQuery.js1.32 kB packages/react-db/dist/esm/useLiveQuery.js1.34 kB packages/react-db/dist/esm/useLiveQueryEffect.js355 B packages/react-db/dist/esm/useLiveSuspenseQuery.js567 B packages/react-db/dist/esm/usePacedMutations.js401 B GPT5.2 review:
Looks good to me.
What I reviewed#
- Nested include propagation changes in
collection-config-builder - New runtime guard for
toArray()/concat(toArray())inside expressions - Regression coverage added in
includes.test.ts
What I checked#
- The depth-3+ nested
toArray()propagation fix appears consistent with the reported failure mode - The new
toExpressionguard correctly preventstoArray()/concat(toArray())from being used inside expression helpers, with a clear error message - The added tests cover the main regression scenarios, including nested/chained includes and sibling-update isolation
Validation#
- Ran the
@tanstack/dbtest suite locally from the PR branch - Result: 94 test files passed, 2271 tests passed, 5 skipped, no type errors
Notes#
- I didn’t find any concrete bugs or regressions in the PR
- Only minor residual risk I see is that some of the new async tests use fixed
setTimeout(...)delays, which could be a source of occasional CI flakiness, but not a correctness issue with the implementation itself
LGTM
- Nested include propagation changes in
Opus 4.6 Review:
All 112 tests pass. Here's my review of PR #1457:
PR Review: fix(db): propagate changes through nested toArray includes at depth 3+#
Overall assessment: This is a well-targeted, minimal fix for a real bug with excellent test coverage. A few suggestions below.
What the PR does#
Two distinct fixes:
Core fix — Adds a "Phase 4c" pass in
flushIncludesStatethat catches entries with pending deep nested buffer changes (depth 3+) that were stranded becausedrainNestedBuffersonly drains one level.Guard rail — Adds runtime brand-based detection in
toExpression()so thattoArray()/concat(toArray())inside expressions likecoalesce()throws a clear error instead of silently producing wrong results.
Correctness analysis#
The core fix is correct. The key insight is sound: when only the deepest level changes (e.g., a textDelta insert at depth 3), intermediate levels have no direct or buffer-drained changes, so they're never marked dirty and never flushed by Phase 4. The new pass uses the existing recursive
hasPendingIncludesChanges()to detect stranded changes and recursively flushes them.I traced through the shared buffer model carefully. In
createPerEntryIncludesStates(line 1309), per-entry states share the samenestedSetupsreferences as the parent:state.nestedSetups = setup.nestedSetupsThis means
hasPendingIncludesChangeswill returntruefor all entries sharing that setup when any deep buffer has changes — not just the entry that "owns" the change. This could cause unnecessary recursive flushes for unaffected entries. However, this is not a correctness issue:drainNestedBuffersinside the recursive flush uses the per-entry routing index, so only properly-routed changes are consumed, and unroutable entries remain in the buffer for the correct entry to claim.Suggestions#
1. Consider
instanceofinstead of__brandfor the expression guardThe wrapper classes are proper classes in the same package, so
instanceofwould work and be more robust than duck-typing:import { ToArrayWrapper, ConcatToArrayWrapper } from './functions.js' if (value instanceof ToArrayWrapper || value instanceof ConcatToArrayWrapper) { const name = value instanceof ToArrayWrapper ? `toArray()` : `concat(toArray())` throw new Error(...) }This avoids adding a runtime
__brandproperty and is immune to false positives from arbitrary objects that happen to have__brand: 'ToArrayWrapper'. It would also mean no changes needed tofunctions.tsat all.2. Minor performance: early exit for the deep buffer pass
The deep buffer pass iterates the entire
childRegistryand callshasPendingIncludesChangeson each entry. A quick pre-check on the shared buffers could skip the loop entirely when there are no deep changes:if (state.nestedSetups && hasNestedBufferChanges(state.nestedSetups)) { for (const [correlationKey, entry] of state.childRegistry) { // ... existing logic } }This replaces the current
if (state.nestedSetups)guard. SincehasNestedBufferChangesis cheap (just checks.size > 0on buffers), this would avoid the registry scan in the common case where no deep changes are pending.However, this might not be fully correct because per-entry
pendingChildChangesare separate from the shared nested buffers. If changes ended up in per-entrypendingChildChangesthrough some other path, this early exit would miss them. Worth double-checking whether that path is possible for the stranded-changes scenario.3. PR description file table is inaccurate
The description lists
repro-bug2.test.tsandrepro-many-siblings.test.tsas changed files, but all tests were actually added toincludes.test.ts. Minor, but worth correcting for reviewers.Tests#
The test coverage is excellent — 10 new tests covering:
- Error throwing for misuse of
toArray()/concat(toArray())in expressions (2 tests) - Sequential insert propagation through various patterns (5 tests)
- Depth 3+ nested propagation (2 tests including a control)
- Spurious update guard ensuring sibling parents aren't unnecessarily re-emitted (1 test)
The spurious update test at line 4902 is particularly good — it verifies referential identity (
toBe) to ensure unaffected sibling arrays aren't re-created:expect(data().runs[0].texts[0].text).toBe(``) expect(data().runs[0].texts).toBe(run1TextsBefore)The tests use
setTimeout-based synchronization (50-100ms), which is consistent with existing patterns in this file but inherently timing-dependent. Not a blocker for this PR.Summary#
Approve with minor suggestions. The fix is minimal, correct, well-tested, and solves a real user-facing bug. The
instanceofvs__brandsuggestion and the early-exit optimization are quality improvements worth considering but not blockers.samwillis(GitHub profile, opens in a new tab) approved these changes
I think this looks good, I had a good read through too while the agents were looking. I think Opus is being overly pernickety.
:shipit:
KyleAMathews(GitHub profile, opens in a new tab) closed this
Sign in to join this conversation.
More templates
@tanstack/angular-db
@tanstack/browser-db-sqlite-persistence
@tanstack/capacitor-db-sqlite-persistence
@tanstack/cloudflare-durable-objects-db-sqlite-persistence
@tanstack/db
@tanstack/db-ivm
@tanstack/db-sqlite-persistence-core
@tanstack/electric-db-collection
@tanstack/electron-db-sqlite-persistence
@tanstack/expo-db-sqlite-persistence
@tanstack/node-db-sqlite-persistence
@tanstack/offline-transactions
@tanstack/powersync-db-collection
@tanstack/query-db-collection
@tanstack/react-db
@tanstack/react-native-db-sqlite-persistence
@tanstack/rxdb-db-collection
@tanstack/solid-db
@tanstack/svelte-db
@tanstack/tauri-db-sqlite-persistence
@tanstack/trailbase-db-collection
@tanstack/vue-db
commit:
96d3b6f