fix(db): nested toArray includes drop children when sibling groups share a correlation key (#1501) #1607
main fromrefs/pull/1607/head·merged by kevin-dp(GitHub profile, opens in a new tab) Fixes #1501.
The bug#
With three (or more) levels of nested toArray includes (e.g. products → priceRanges → region), when two children in different parent groups share the same deepest correlation key, only one of them receives the nested rows — the other comes back as an empty array:
T-Shirt → priceRange 1 (regionId 1) → region: [] ❌ expected [Europe]
T-Shirt → priceRange 2 (regionId 2) → region: [North America] ✓
Hoodie → priceRange 3 (regionId 1) → region: [Europe] ✓
Root cause#
The nested pipeline writes results into a buffer shared across per-parent-group states, and routing assumes one parent per nested correlation key. Two compounding issues:
- 1:1 routing + destructive drain.
nestedRoutingIndexmapped each nested correlation key to a single parent group (last-writer-wins), anddrainNestedBuffersdeleted the buffer entry after routing to the first match — so a sibling group sharing the key was dropped. - No re-emit for late arrivals. The nested pipeline does not re-emit already-materialized rows, so a parent group that starts referencing an existing correlation key after the rows were drained (e.g. a sibling inserted after the initial load) saw nothing.
Fix#
nestedRoutingIndexnow maps a nested correlation key to a Set of parent groups;drainNestedBuffersfans buffered grandchild changes out to every ready parent group before dropping the buffer entry. Routing-index inserts/deletes and parent-delete cleanup maintain the per-key parent sets.- A per-level cumulative snapshot of net-present grandchild rows seeds late-arriving parent groups with the rows their siblings already received.
Tests#
Three tests under nested toArray includes (depth 3+) cover initial load, a sibling inserted after load, and deleting one of two siblings that share a correlation key. Full tests/query/ suite (1596 tests) passes.
🤖 Generated with Claude Code
Summary by CodeRabbit#
- Bug Fixes
- Fixed nested
toArrayand live nested includes behavior when sibling parent groups share the same deepest correlation key, preventing dropped or mis-associated child results. - Improved fan-out routing so nested updates are applied to all relevant parent groups, including correct seeding for newly-associated parents without breaking sibling isolation during inserts and deletes.
- Fixed nested
- Tests
- Added regression coverage for multi-level nested includes (
regionunderpriceRangesunderproducts) for both live and materialized collection modes, including preload, post-load insert, updates, and multiple delete scenarios.
- Added regression coverage for multi-level nested includes (
- 1
More templates
@tanstack/angular-db
npm i https://pkg.pr.new/@tanstack/angular-db@1607@tanstack/browser-db-sqlite-persistence
npm i https://pkg.pr.new/@tanstack/browser-db-sqlite-persistence@1607@tanstack/capacitor-db-sqlite-persistence
npm i https://pkg.pr.new/@tanstack/capacitor-db-sqlite-persistence@1607@tanstack/cloudflare-durable-objects-db-sqlite-persistence
npm i https://pkg.pr.new/@tanstack/cloudflare-durable-objects-db-sqlite-persistence@1607@tanstack/db
npm i https://pkg.pr.new/@tanstack/db@1607@tanstack/db-ivm
npm i https://pkg.pr.new/@tanstack/db-ivm@1607@tanstack/db-sqlite-persistence-core
npm i https://pkg.pr.new/@tanstack/db-sqlite-persistence-core@1607@tanstack/electric-db-collection
npm i https://pkg.pr.new/@tanstack/electric-db-collection@1607@tanstack/electron-db-sqlite-persistence
npm i https://pkg.pr.new/@tanstack/electron-db-sqlite-persistence@1607@tanstack/expo-db-sqlite-persistence
npm i https://pkg.pr.new/@tanstack/expo-db-sqlite-persistence@1607@tanstack/node-db-sqlite-persistence
npm i https://pkg.pr.new/@tanstack/node-db-sqlite-persistence@1607@tanstack/offline-transactions
npm i https://pkg.pr.new/@tanstack/offline-transactions@1607@tanstack/powersync-db-collection
npm i https://pkg.pr.new/@tanstack/powersync-db-collection@1607@tanstack/query-db-collection
npm i https://pkg.pr.new/@tanstack/query-db-collection@1607@tanstack/react-db
npm i https://pkg.pr.new/@tanstack/react-db@1607@tanstack/react-native-db-sqlite-persistence
npm i https://pkg.pr.new/@tanstack/react-native-db-sqlite-persistence@1607@tanstack/rxdb-db-collection
npm i https://pkg.pr.new/@tanstack/rxdb-db-collection@1607@tanstack/solid-db
npm i https://pkg.pr.new/@tanstack/solid-db@1607@tanstack/svelte-db
npm i https://pkg.pr.new/@tanstack/svelte-db@1607@tanstack/tauri-db-sqlite-persistence
npm i https://pkg.pr.new/@tanstack/tauri-db-sqlite-persistence@1607@tanstack/trailbase-db-collection
npm i https://pkg.pr.new/@tanstack/trailbase-db-collection@1607@tanstack/vue-db
npm i https://pkg.pr.new/@tanstack/vue-db@1607commit:
e6bb8f3kevin-dp(GitHub profile, opens in a new tab) changed the title
Actionable comments posted: 1
🧹 Nitpick comments (2)
packages/db/tests/query/includes.test.ts (1)
5078-5079: 🩺 Stability & Availability | 🔵 TrivialReplace the fixed 50ms sleeps with a deterministic wait helper.
flushPromises()is already available in the test utilities, and the same change should be applied to the matching delete case at line 5160.🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the rest with a brief reason, keep changes minimal, and validate. In `@packages/db/tests/query/includes.test.ts` around lines 5078 - 5079, The tests in `includes.test.ts` use fixed 50ms sleeps after `priceRanges.insert(...)`, which should be replaced with the deterministic `flushPromises()` helper already available in the test utilities. Update the insert case and the matching delete case to await `flushPromises()` instead of `setTimeout`, keeping the same behavior while removing timing flakiness.Source: Coding guidelines
packages/db/src/query/live/collection-config-builder.ts (1)
1153-1158: 📐 Maintainability & Code Quality | 🔵 Trivial | ⚡ Quick winAvoid adding new
anyto snapshot state.SnapshotRow.valueand the new helper signature can useunknownhere; the code only stores/replays the value and does not need unchecked access. As per coding guidelines,**/*.{ts,tsx}: Avoid usinganytypes; useunknowninstead when the type is truly unknown.♻️ Proposed type tightening
type SnapshotRow = { - value: any + value: unknown orderByIndex: string | undefined /** Net multiplicity (inserts − deletes) currently materialized for this row */ count: number } @@ function accumulateSnapshot( setup: NestedIncludesSetup, nestedCorrelationKey: unknown, - childChanges: Map<unknown, Changes<any>>, + childChanges: Map<unknown, Changes<unknown>>, ): void {Also applies to: 1373-1377
🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the rest with a brief reason, keep changes minimal, and validate. In `@packages/db/src/query/live/collection-config-builder.ts` around lines 1153 - 1158, The snapshot state is introducing an unnecessary any type in SnapshotRow.value and the related helper signature, even though the value is only stored and replayed. Tighten the types in collection-config-builder by changing SnapshotRow and the affected helper used around the snapshot logic to use unknown instead of any, and keep the existing handling unchanged except for type-safe narrowing where needed.Source: Coding guidelines
🤖 Prompt for all review comments with AI agents
Verify each finding against current code. Fix only still-valid issues, skip the rest with a brief reason, keep changes minimal, and validate. Inline comments: In `@packages/db/src/query/live/collection-config-builder.ts`: - Around line 1387-1399: Clone snapshot values before assigning them into row state in collection-config-builder’s snapshot/replay logic, because row.value is currently sharing the same object as changes.value and later mutation of changes.value can strip routing metadata. Update the row initialization and replay paths in the affected update/merge flow (including the shared logic around the count/orderByIndex handling) to store a cloned copy instead of the original object, so later cleanup cannot affect previously buffered rows and nested routing can be rebuilt correctly. --- Nitpick comments: In `@packages/db/src/query/live/collection-config-builder.ts`: - Around line 1153-1158: The snapshot state is introducing an unnecessary any type in SnapshotRow.value and the related helper signature, even though the value is only stored and replayed. Tighten the types in collection-config-builder by changing SnapshotRow and the affected helper used around the snapshot logic to use unknown instead of any, and keep the existing handling unchanged except for type-safe narrowing where needed. In `@packages/db/tests/query/includes.test.ts`: - Around line 5078-5079: The tests in `includes.test.ts` use fixed 50ms sleeps after `priceRanges.insert(...)`, which should be replaced with the deterministic `flushPromises()` helper already available in the test utilities. Update the insert case and the matching delete case to await `flushPromises()` instead of `setTimeout`, keeping the same behavior while removing timing flakiness.🪄 Autofix (Beta)
Fix all unresolved CodeRabbit comments on this PR:
- Push a commit to this branch (recommended)
- Create a new PR with the fixes
ℹ️ Review info
⚙️ Run configuration
Configuration used: defaults
Review profile: CHILL
Plan: Pro
Run ID:
ace73dfd-754c-45cb-8106-836a50331bd6📥 Commits
Reviewing files that changed from the base of the PR and between b7229fc056df438537ed78c7382289ade4eee4ae and 5d5259ec5c50c6bf7c19155aac0d01347a5e9650.
📒 Files selected for processing (3)
.changeset/nested-toarray-shared-buffer-overlap.mdpackages/db/src/query/live/collection-config-builder.tspackages/db/tests/query/includes.test.ts
✅ Files skipped from review due to trivial changes (1)
- .changeset/nested-toarray-shared-buffer-overlap.md
@@ -1344,0 +1379,21 @@1379+ if (!snap) {1380+ snap = new Map()1381+ setup.snapshot.set(nestedCorrelationKey, snap)1382+ }1383+1384+ for (const [childKey, changes] of childChanges) {1385+ let row = snap.get(childKey)1386+ if (!row) {1387+ row = {1388+ value: changes.value,1389+ orderByIndex: changes.orderByIndex,1390+ count: 0,1391+ }1392+ snap.set(childKey, row)1393+ }1394+ row.count += changes.inserts - changes.deletes1395+ if (changes.inserts > 0) {1396+ row.value = changes.value1397+ if (changes.orderByIndex !== undefined) {1398+ row.orderByIndex = changes.orderByIndex1399+ }🎯 Functional Correctness | 🟠 Major | ⚡ Quick win
Clone snapshot values before storing or replaying them.
row.valuecurrently aliaseschanges.value; later cleanup deletesINCLUDES_ROUTINGfromchanges.valueat Line 1984. A late replay can then seed a row without routing metadata, preventing deeper nested routing from being rebuilt via Lines 1544-1547.🐛 Proposed fix
+function cloneSnapshotValue<T>(value: T): T { + if (value == null || typeof value !== `object`) return value + if (Array.isArray(value)) return [...value] as T + return { ...(value as Record<PropertyKey, unknown>) } as T +} + function accumulateSnapshot( setup: NestedIncludesSetup, nestedCorrelationKey: unknown, childChanges: Map<unknown, Changes<any>>, @@ row = { - value: changes.value, + value: cloneSnapshotValue(changes.value), orderByIndex: changes.orderByIndex, count: 0, } @@ if (changes.inserts > 0) { - row.value = changes.value + row.value = cloneSnapshotValue(changes.value) if (changes.orderByIndex !== undefined) { row.orderByIndex = changes.orderByIndex } @@ byChild.set(childKey, { deletes: 0, inserts: row.count, - value: row.value, + value: cloneSnapshotValue(row.value), orderByIndex: row.orderByIndex, })Also applies to: 1436-1443
🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the rest with a brief reason, keep changes minimal, and validate. In `@packages/db/src/query/live/collection-config-builder.ts` around lines 1387 - 1399, Clone snapshot values before assigning them into row state in collection-config-builder’s snapshot/replay logic, because row.value is currently sharing the same object as changes.value and later mutation of changes.value can strip routing metadata. Update the row initialization and replay paths in the affected update/merge flow (including the shared logic around the count/orderByIndex handling) to store a cloned copy instead of the original object, so later cleanup cannot affect previously buffered rows and nested routing can be rebuilt correctly.✅ Addressed in commits 33bfcaa to d614455
Here's a complete explanation of the bugs and the fix. This PR implements the snapshot accumulation fix but it means we store each distinct nested row in that snapshot. The explanation below also documents a potential alternative.
The example#
products → priceRanges → regionFor each product you want its price ranges, and for each price range its region. Three levels deep. Our problem data:
- product 1 (T-Shirt) has priceRange 1 → regionId 1 (Europe)
- product 2 (Hoodie) has priceRange 3 → regionId 1 (Europe) ← same region!
How includes actually compute (the important part)#
You might imagine the engine runs the "region" query fresh for each price range. It doesn't. That would be slow and wouldn't update reactively. Instead:
There is ONE shared "region" pipeline. It runs once over all regions and emits region rows tagged with a correlation key — the value that says "who do I belong to". For region, that key is
regionId. So the region pipeline emits one thing:regionId 1 → { Europe }Just one entry. Even though two price ranges want region 1, the pipeline produces it once, keyed by
regionId 1.Results land in a "buffer" — a temporary mailbox:
regionId → the region rows. Think of it as "here are freshly-computed region rows, waiting to be delivered."Then a "drain" step delivers buffered rows to the right parents. This is the routing step. To deliver region 1 to the correct price range, the engine keeps a routing index:
routingIndex: regionId 1 → "which price-range group does this belong to?"Each price-range group has its own private copy of region results (its own little child collection), so the final tree can show each one its own
regionarray. The drain reads the buffer, looks up the parent in the routing index, copies the rows into that parent's private copy, and then deletes the buffer entry (it's been delivered — don't deliver twice).Why it broke#
The whole design quietly assumed one nested key belongs to one parent. Two things follow from that assumption, and both are wrong when two siblings share
regionId 1:Bug 1 — the routing index only remembered one parent.
routingIndex.set(regionId 1, priceRange1) // T-Shirt's routingIndex.set(regionId 1, priceRange3) // Hoodie's — OVERWRITESIt's a plain key→value map, so the second write clobbers the first. Now
regionId 1points to only one group. Then drain delivers region 1 to that one group and deletes the buffer entry. The other group's lookup finds nothing → itsregioncomes back[]. That's the empty array you saw.Bug 2 — late arrivals get nothing.
Say T-Shirt's priceRange exists at load, but Hoodie's priceRange 3 is inserted later. By then the region pipeline already emitted region 1 (at load), it was drained, and the buffer entry was deleted. When priceRange 3 shows up, the pipeline does not re-emit region 1 — from its point of view nothing about regions changed; it already produced that row. So the buffer is empty and there's nothing to deliver to Hoodie. Empty array again.
Both of these were confirmed with debug logging before writing the fix.
The fix — two matching pieces#
Fix 1: let one nested key map to many parents#
The routing index value changes from a single parent to a Set of parents:
routingIndex: regionId 1 → { priceRange1, priceRange3 }- Adding a parent now adds to the set instead of overwriting (the
updateRoutingIndexinsert change). - Removing a parent removes just that one from the set, leaving the others (the delete change and
cleanRoutingIndexOnDelete). This is what keeps the deletion case working — deleting Hoodie mustn't wipe regionId 1 while T-Shirt still uses it. - Drain now loops over every parent in the set and copies the rows into each one's private copy, and only then deletes the buffer entry. That's the
for (const parentCorrelationKey of parentCorrelationKeys)loop. This fixes the initial-load case.
Fix 2: remember delivered rows so late arrivals can be seeded#
Because the pipeline won't re-emit region 1 for a price range added later, the engine needs its own memory of "what region rows currently exist for regionId 1". That's the new snapshot:
snapshot: regionId 1 → { Europe } (kept around, not deleted on drain)- Every time something is drained, it's also folded into the snapshot (
accumulateSnapshot). The snapshot tracks a running count so that if a region is later deleted, it drops out of the snapshot too — it stays an accurate picture of "what currently exists", not a pile of stale rows. - When a new parent group first references an existing key (
isNewParentinupdateRoutingIndex), we seed it straight from the snapshot (seedParentFromSnapshot) — hand it the rows its siblings already have, without waiting on the pipeline. This fixes the insert-after-load case.
One-line summary#
The nested-includes system assumed each nested key had exactly one parent and treated delivered rows as throwaway. The fix makes a nested key able to feed multiple parents (Set instead of single value + fan-out on drain), and keeps a snapshot of currently-existing rows so parents that show up late can be handed the same data their siblings already received.
Is the snapshot a memory problem?#
Partly a fair concern, but it's bounded, not unbounded:
- It stores one entry per distinct (nestedKey, childKey) — i.e. one copy of each distinct nested row, not one per parent. Siblings sharing a key share the single snapshot entry. So its size tracks the distinct nested result set, not parent count.
- It holds references to the same row
valueobjects already materialized elsewhere, not deep copies. The extra cost is Map overhead + one slot per distinct row. - It shrinks:
accumulateSnapshottracks a net count and deletes rows (and empty keys) when the count hits 0 on delete. So it's not an ever-growing log — it stays proportional to currently-live data.
So it's not a leak and not multiplied by parents. But it is a real added cost: roughly O(distinct nested rows) extra — essentially one more index over the deepest level, for the lifetime of the live query. For a query with a very large nested collection, that's non-trivial memory.
The alternative: copy from an existing sibling instead of keeping a snapshot#
The key realization#
When does a late-arriving parent need seeding at all? Only when the region rows were already delivered and the buffer entry deleted in an earlier flush, and the pipeline won't re-emit them. But "already delivered to whom?" — to a sibling. So whenever seeding is needed, at least one sibling already has the rows materialized in its own private copy.
That's the insight the alternative exploits: you don't need a separate stash of the data, because a sibling is already holding a perfectly good copy of it. Go ask the sibling.
How "copy from sibling" works#
Walk through Hoodie's priceRange 3 arriving late, referencing
regionId 1:Look up the routing index (which is now a Set thanks to Fix 1):
regionId 1 → { priceRange1, priceRange3 }priceRange3 is the newcomer; priceRange1 is an existing sibling.
Find a sibling that already has the data. priceRange1's private region state has a
childRegistrykeyed byregionId, andchildRegistry[regionId 1]is a little collection already holding{ Europe }.Copy those rows into priceRange3's pending changes, then let the normal materialize step run. priceRange3 now shows
[Europe].
So instead of consulting a snapshot, step 2 reaches into a sibling's already-materialized collection.
What it costs you in code#
The snapshot stored exactly what the delivery machinery needs: for each row, its
childKey, itsvalue, and itsorderByIndex. A live child collection only stores the rows themselves. To rebuild "changes" from it you have to look those extra bits back up:- the sibling state's
resultKeysmap (row → childKey), - its
orderByIndicesmap (row → orderByIndex),
then iterate the sibling's collection and reconstruct insert-changes from each row. It's the same idea as
seedParentFromSnapshot, just reading from a collection + two side-maps instead of from one tidy snapshot Map.You also need a small guard: "pick a sibling that is actually populated right now." The routing-index Set gives you candidates, but you'd skip the newcomer itself and any sibling that happens to be empty, and copy from the first good one. (As argued above, a good one is essentially guaranteed to exist whenever seeding is needed — if none did, the buffer path would still be handling it.)
The two approaches compared#
Snapshot (current fix) Copy-from-sibling Standing memory one extra index over distinct nested rows, kept for the query's life none — reuses data already in siblings Work at steady state a little on every drain (fold into snapshot) none Work on a late insert cheap: read tidy snapshot a bit more: find a ready sibling, rebuild changes from its collection + side-maps Code complexity self-contained, one clean structure more fiddly: sibling selection + reconstructing changes Correctness risk must keep the snapshot's net-count in sync with deletes/updates sibling is always the live truth, so no parallel structure to drift — but "is a sibling ready?" is a subtler question How to choose#
The snapshot trades memory for simplicity and predictability: one clean structure, consulted in one place, no "go find a healthy donor" logic. Copy-from-sibling trades that memory away but pays in code complexity and a reconstruction step on each late insert.
For typical includes (modest nested result sets) the snapshot's overhead is negligible and the simpler code wins. If you expect very large nested collections where doubling an index matters, copy-from-sibling is the better fit — and it's a localized swap: only
accumulateSnapshot+seedParentFromSnapshotchange; the Set-based routing fan-out (Fix 1) stays exactly as-is either way.Actionable comments posted: 1
🤖 Prompt for all review comments with AI agents
Verify each finding against current code. Fix only still-valid issues, skip the rest with a brief reason, keep changes minimal, and validate. Inline comments: In `@packages/db/tests/query/includes.test.ts`: - Around line 5171-5261: The current live/materialize regression tests only verify preload behavior; extend the existing nested-collection coverage in the relevant `it(...)` cases to assert post-load mutations as well. Use the same `createLiveQueryCollection`, `toTree`, and nested `q.from(...)` setup to add insert/delete assertions for the live and materialized variants, and include async stale-order/race scenarios that exercise the same shared-correlation-key path after initial load. Keep the new assertions anchored to the existing collection setup so the bug is reproduced across both materialization modes.🪄 Autofix (Beta)
Fix all unresolved CodeRabbit comments on this PR:
- Push a commit to this branch (recommended)
- Create a new PR with the fixes
ℹ️ Review info
⚙️ Run configuration
Configuration used: defaults
Review profile: CHILL
Plan: Pro
Run ID:
e827313e-ce5c-41ab-a147-be0bec3dfe49📥 Commits
Reviewing files that changed from the base of the PR and between 5d5259ec5c50c6bf7c19155aac0d01347a5e9650 and 7bd39a89deeb451304b5df371a16fba3798a5ec5.
📒 Files selected for processing (1)
packages/db/tests/query/includes.test.ts
@@ -4904,0 +5241,21 @@5241+ id: 1,5242+ title: `T-Shirt`,5243+ priceRanges: [5244+ { id: 1, regionId: 1, region: [{ id: 1, name: `Europe` }] },5245+ {5246+ id: 2,5247+ regionId: 2,5248+ region: [{ id: 2, name: `North America` }],5249+ },5250+ ],5251+ },5252+ {5253+ id: 2,5254+ title: `Hoodie`,5255+ priceRanges: [5256+ { id: 3, regionId: 1, region: [{ id: 1, name: `Europe` }] },5257+ ],5258+ },5259+ ])5260+ })5261+📐 Maintainability & Code Quality | 🟡 Minor | ⚡ Quick win
Expand live/materialize variants with mutation-path assertions.
These two tests validate preload only. Please add post-load insert/delete coverage (and async stale-order/race coverage) for these same materialization modes to protect the full regression surface.
As per coding guidelines,
**/*.test.{ts,tsx,js}: “Always add unit tests that reproduce a bug before fixing it…” and “Test corner cases including … async race conditions …”.Also applies to: 5262-5353
🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the rest with a brief reason, keep changes minimal, and validate. In `@packages/db/tests/query/includes.test.ts` around lines 5171 - 5261, The current live/materialize regression tests only verify preload behavior; extend the existing nested-collection coverage in the relevant `it(...)` cases to assert post-load mutations as well. Use the same `createLiveQueryCollection`, `toTree`, and nested `q.from(...)` setup to add insert/delete assertions for the live and materialized variants, and include async stale-order/race scenarios that exercise the same shared-correlation-key path after initial load. Keep the new assertions anchored to the existing collection setup so the bug is reproduced across both materialization modes.Source: Coding guidelines
✅ Addressed in commit d914565
Had a quick discussion with @samwillis on the 2 alternatives. We decided to go for correctness and simplicity so we pick the snapshot approach.
samwillis(GitHub profile, opens in a new tab) requested changes
I found one remaining routing hole in the new
Set<parentCorrelationKey>approach.A single parent correlation key can be backed by multiple child rows that share the same nested correlation key. The new route set dedupes those rows down to one parent entry, so when one of the child rows is deleted the delete path removes the whole parent route even though another child row in that same parent group still needs it. After that, later grandchild updates/deletes for the shared nested key no longer reach the surviving child.
The problematic path is around
packages/db/src/query/live/collection-config-builder.ts:1591, where the delete path doesparents.delete(correlationKey)without knowing whether another child row in that parent group still references the same nested key.I pushed a minimal failing repro here:
- Branch: https://github.com/samwillis/db/tree/codex/pr-1607-same-parent-routing-repro
- Commit: https://github.com/samwillis/db/commit/21ac924c671233ad1dc42d8debdb7509285c6239
The repro adds one product with two price ranges sharing
regionId: 1, deletes one price range, then updates region1. The surviving price range still showsEuropeinstead ofRenamed Europe.Targeted command used:
PATH=/Users/samwillis/.cache/codex-runtimes/codex-primary-runtime/dependencies/node/bin:$PATH pnpm exec vitest --run tests/query/includes.test.ts -t 'keeps routing when one of multiple same-parent siblings sharing a nested key is deleted'This likely needs per-parent route multiplicity/refcounting, or tracking routes at the child-key level, so deleting one sibling does not remove routing for the surviving sibling.
Great catch @samwillis — confirmed and fixed in 33bfcaa5.
I reproduced your repro against the branch and it failed exactly as you described: deleting one of two same-parent siblings that share a nested correlation key dropped the route for the survivor, so later grandchild updates never reached it. Root cause was exactly what you identified at
collection-config-builder.ts:1591— theSet<parentCorrelationKey>route had no notion of how many child rows in a parent group referenced a nested key, so the first sibling delete emptied the entry and dropped the whole route.Fix: the routing index now tracks the referencing child keys per
(nestedKey, parentGroup)—Map<nestedKey, Map<parentCorrelationKey, Set<childKey>>>— and a parent group is only dropped from a route once its last referencing child row is removed. Extracted aremoveChildKeyFromRoutehelper so the delete path and parent-delete cleanup share one consistent cascade.While verifying, I found a second routing hole your report exposed: an update that changes a child row's nested correlation key (e.g. a price range's
regionIdchanges) only carries the new key, so the row's stale reference under the old key was never released. With the new refcounting that stale child key kept the old route alive, and a later sibling delete then mis-routed grandchild changes (the updated row lost its grandchildren). Fixed with a per-setupchildKey → nestedKeymap that releases the previous reference on update — scoped per nested setup so a change to one nested include can't disturb another include on the same child row.New regression tests cover all three: your same-parent-sibling-delete case, the update-then-sibling-delete case, and cross-include isolation. Full
tests/query/suite passes (1601), no type errors, lint clean.Separately — and not introduced by this PR — I noticed that an atomic mutual swap (two same-parent siblings exchanging nested keys in a single tick) produces wrong results; it reproduces on
maintoo, so it predates this work. I'll open a separate issue with a repro rather than expand this PR's scope. Let me know if you'd prefer it folded in here.Actionable comments posted: 1
🤖 Prompt for all review comments with AI agents
Verify each finding against current code. Fix only still-valid issues, skip the rest with a brief reason, keep changes minimal, and validate. Inline comments: In `@packages/db/tests/query/includes.test.ts`: - Around line 5179-5184: The test currently uses fixed `setTimeout(50)` sleeps, which makes the async behavior timing-sensitive and flaky. In the affected `includes.test.ts` cases, replace those waits with state-based synchronization by polling or awaiting the expected collection state after each `currencies.update(...)` mutation, using the existing test helpers or a repo-local wait helper instead of arbitrary delays. Keep the assertions anchored around the relevant `currencies` updates so the test verifies ordering and completion directly.🪄 Autofix (Beta)
Fix all unresolved CodeRabbit comments on this PR:
- Push a commit to this branch (recommended)
- Create a new PR with the fixes
ℹ️ Review info
⚙️ Run configuration
Configuration used: defaults
Review profile: CHILL
Plan: Pro
Run ID:
7e8a55a3-f66e-4f39-aed2-6091dd08e82f📥 Commits
Reviewing files that changed from the base of the PR and between d91456575a974a0d0fd0e5fa19e0fdcd66f4c104 and ea83de90929d80c32d321ddf41821e3293a04ef7.
📒 Files selected for processing (2)
packages/db/src/query/live/collection-config-builder.tspackages/db/tests/query/includes.test.ts
🚧 Files skipped from review as they are similar to previous changes (1)
- packages/db/src/query/live/collection-config-builder.ts
@@ -4904,0 +5164,21 @@5164+ .from({ c: currencies })5165+ .where(({ c }) => eq(c.id, pr.currencyId))5166+ .select(({ c }) => ({ id: c.id, code: c.code })),5167+ ),5168+ })),5169+ ),5170+ })),5171+ })5172+ await collection.preload()5173+5174+ // Change ONLY regionId; currency must still resolve, and a later currency5175+ // rename must still reach this price range.5176+ priceRanges.update(1, (draft) => {5177+ draft.regionId = 25178+ })5179+ await new Promise((r) => setTimeout(r, 50))5180+5181+ currencies.update(9, (draft) => {5182+ draft.code = `USD`5183+ })5184+ await new Promise((r) => setTimeout(r, 50))🩺 Stability & Availability | 🟡 Minor | ⚡ Quick win
Replace fixed sleeps with state-based synchronization.
These
setTimeout(50)waits make the regressions timing-sensitive: slow CI can still flake, and fast runs can hide ordering problems instead of asserting them. Please wait on the expected collection state (or a repo-local polling helper) after each mutation rather than sleeping for an arbitrary duration. As per coding guidelines,**/*.test.{ts,tsx,js}: “Test corner cases including … async race conditions …”.Also applies to: 5269-5279, 5431-5436
🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the rest with a brief reason, keep changes minimal, and validate. In `@packages/db/tests/query/includes.test.ts` around lines 5179 - 5184, The test currently uses fixed `setTimeout(50)` sleeps, which makes the async behavior timing-sensitive and flaky. In the affected `includes.test.ts` cases, replace those waits with state-based synchronization by polling or awaiting the expected collection state after each `currencies.update(...)` mutation, using the existing test helpers or a repo-local wait helper instead of arbitrary delays. Keep the assertions anchored around the relevant `currencies` updates so the test verifies ordering and completion directly.Source: Coding guidelines
✅ Addressed in commits 33bfcaa to 62a6dd1
samwillis(GitHub profile, opens in a new tab) requested changes
Thanks for the update. The previous same-parent repro is fixed now: the added test
keeps routing when one of multiple same-parent siblings sharing a nested key is deletedpasses onea83de909.I found one remaining routing hole in the new child-key refcounting approach, though.
nestedRoutingIndexis still shared across all sibling nested includes at a level (nestedCorrelationKey -> parent -> childKeys), whilenestedRoutingChildToNestedis per nested setup. When two sibling nested includes use the same nested correlation value for the same child row, cleanup for one include can delete the shared route that the other include still needs.Concrete failing case:
- A price range has
regionId: 1andcurrencyId: 1. - The child select defines nested includes in this order:
currencyfirst,regionsecond. - Updating only
regionIdfrom1to2makes thecurrencysetup observe key1unchanged first, then theregionsetup removes child key1from the shared route for nested key1. - Because the route is shared and only keyed by child key, the later removal deletes the route needed by
currency. - A subsequent currency update no longer reaches the surviving
currencyinclude, so the row still showsEURinstead ofUSD.
The problematic shape is around
packages/db/src/query/live/collection-config-builder.ts:1216and the deletion atpackages/db/src/query/live/collection-config-builder.ts:1556: the route refcount is not scoped by nested setup/include field, so equal correlation values from different nested includes collide.Temporary repro I ran locally:
it(`keeps sibling include routing when an earlier include still references the old nested key`, async () => { // one price range: regionId: 1, currencyId: 1 // select order: currency first, region second priceRanges.update(1, (draft) => { draft.regionId = 2 }) await new Promise((r) => setTimeout(r, 50)) currencies.update(1, (draft) => { draft.code = `USD` }) await new Promise((r) => setTimeout(r, 50)) // Fails: currency remains EUR })Targeted command:
PATH=/Users/samwillis/.cache/codex-runtimes/codex-primary-runtime/dependencies/node/bin:$PATH pnpm exec vitest --run tests/query/includes.test.ts -t 'keeps sibling include routing when an earlier include still references the old nested key'This likely needs the route reference tracking to be scoped per nested setup/include field as well as by parent and child key, or otherwise include setup identity in the routing key/refcount so sibling includes with equal correlation values cannot remove each other's routes.
- A price range has
Confirmed and fixed — thanks again @samwillis, this was exactly right.
Reproduced your case (added as
resolves two nested includes on the same child independently when they share a correlation valuein 1553c4ef) and traced it: withregionId === currencyId === 1, both thecurrencyandregionsetups resolve the samenestedRoutingKeyand collapse into a singleSet<childKey>entry in the shared route store. Re-pointingregionId 1→2removes child key1from that shared set, it empties, and the whole route for key1is dropped — even thoughcurrencystill needs it. A later currency change then finds no route ([DRAIN] setup=0 nestedKey=1 routes=NONE) and the row keepsEUR.Root cause as you pinpointed: the child-key refcount tracked the removal decision per nested setup (
nestedRoutingChildToNested), but the route store itself (nestedRoutingIndex/nestedRoutingReverseIndex) was shared across all sibling includes at a level, and the routing key doesn't encode the include field — so equal correlation values from different includes can remove each other's routes.Fix (492f0852): scope the route store per nested setup —
nestedRoutingIndexandnestedRoutingReverseIndexare now arrays indexed by setup, mirroringnestedRoutingChildToNested. Each include owns its route store, so a removal in one include can't touch another's.removeChildKeyFromRoutenow takes the per-setup maps and parent-delete cleanup iterates every setup. Drain targeting was already per-setup (it routes intoentry.includesStates[i]), so only the store needed to move to the same granularity.Full
tests/query/suite passes (1602), no type errors, lint clean. The earlier same-parent/sibling-parent guarantees are unchanged — those all live within a single setup, which is now justindex[0].samwillis(GitHub profile, opens in a new tab) approved these changes
Re-reviewed latest head
29b8fb16589d48a87342001a6779f14357b2d751after the follow-up fixes.The route stores are now scoped per nested setup, so the sibling nested includes that share the same correlation value no longer collide. I also checked the same-parent sibling route cleanup and the recursive deeper-buffer path added for depth 3+ fanout.
Verification:
pnpm exec vitest --run tests/query/includes.test.ts(134 passed, no type errors)
I don't see any remaining blocking issues in this update.
kevin-dp(GitHub profile, opens in a new tab) closed this
Sign in to join this conversation.
Note
Reviews paused#
It looks like this branch is under active development. To avoid overwhelming you with review comments due to an influx of new commits, CodeRabbit has automatically paused this review. You can configure this behavior by changing the
reviews.auto_review.auto_pause_after_reviewed_commitssetting.Use the following commands to manage reviews:
@coderabbitai resumeto resume automatic reviews.@coderabbitai reviewto trigger a single review.Use the checkboxes below for quick actions:
📝 Walkthrough
Walkthrough#
Nested include routing now tracks multiple parents per deepest correlation key, replays drained nested rows from snapshots, and adds regression coverage for shared-key three-level includes across preload, insert, delete, live collection, and materialized variants.
Changes#
Nested toArray fan-out and snapshotting
packages/db/src/query/live/collection-config-builder.tspackages/db/src/query/live/collection-config-builder.ts.changeset/nested-toarray-shared-buffer-overlap.md,packages/db/tests/query/includes.test.tsregionId.Estimated code review effort#
🎯 4 (Complex) | ⏱️ ~45 minutes
Suggested reviewers#
Poem#
🚥 Pre-merge checks | ✅ 4 | ❌ 1
❌ Failed checks (1 warning)#
✅ Passed checks (4 passed)
✨ Finishing Touches
🧪 Generate unit tests (beta)
Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.
❤️ Share