fix: reconcile duplicate live query child inserts #1600
main fromrefs/pull/1600/head·merged by kevin-dp(GitHub profile, opens in a new tab) Fix live-query includes reconciliation so .update() flows that re-emit an existing joined child row update the internal child collection instead of attempting a duplicate insert.
User-visible impact: updating a queryCollectionOptions source used in a live query with joined/included child rows no longer crashes with either the secondary LIVE_QUERY_INTERNAL TypeError or the underlying duplicate child insert error.
Root Cause#
The user action is .update(), but that update triggers live-query graph reconciliation. During reconciliation, flushIncludesState writes changes into internal child collections such as:
__child-collection:detailed-surplus-agreement-line-items-product-30587
From that internal collection's perspective, a derived/joined row can be emitted as an insert even though the user operation was an update. When the child collection already contained that key, sync correctly rejected the duplicate insert:
Cannot insert document with key "302935" from sync because it already exists in the collection "__child-collection:..."
There was also a secondary issue in the duplicate-key error path: generic/internal collection configs do not necessarily have live-query internals in config.utils, but sync.ts assumed they did while constructing DuplicateKeySyncError context.
Approach#
The fix has two parts:
In
flushIncludesState, compute whether the internal child collection already has the child key before writing an insert-only child change. If the child already exists, write an internalupdateinstead of aninsert.const key = entry.syncMethods.collection.getKeyFromItem(change.value) const childAlreadyExists = entry.syncMethods.collection.has(key) if (change.inserts > 0 && change.deletes === 0) { entry.syncMethods.write({ value: change.value, type: childAlreadyExists ? `update` : `insert`, }) }In the duplicate-key sync error path, guard optional
config.utilsbefore readingLIVE_QUERY_INTERNAL, preserving enhanced context when present and defaulting the flags when absent.
Key Invariants#
- Existing internal child rows must not receive duplicate sync inserts during live-query includes reconciliation.
- New internal child rows should still be inserted normally.
- Existing live-query-specific duplicate-key diagnostics should be preserved when
LIVE_QUERY_INTERNALis present. - Collection configs without
utilsmust not throw a secondaryTypeErrorwhile constructing duplicate-key errors.
Non-goals#
- This does not change public
.update()semantics. - This does not change query collection refetch behavior.
- This does not redesign includes materialization or child collection identity.
Trade-offs#
The targeted fix keeps reconciliation local to the internal child collection write decision. A broader alternative would be changing upstream D2/includes change classification, but the child collection is the place with direct knowledge of whether a given child key is already materialized.
Verification#
cd packages/db
pnpm vitest run tests/collection.test.ts -t "config has no utils" --coverage.enabled=false
pnpm vitest run tests/query/includes.test.ts --coverage.enabled=false --pool-options.threads.maxThreads=2
cd packages/query-db-collection
pnpm vitest run tests/query.test.ts -t "duplicate insert into includes child" --coverage.enabled=false
CI is green for PR #1600, including:
- Test
- Run E2E Tests
- Build Example Site
- Preview
Files changed#
packages/db/src/collection/sync.ts- Guard access to optional
config.utilsbefore readingLIVE_QUERY_INTERNAL.
- Guard access to optional
packages/db/src/query/live/collection-config-builder.ts- Convert insert-only internal child changes to updates when the child key already exists.
packages/db/tests/collection.test.ts- Add regression coverage for duplicate-key sync errors when collection config has no
utils.
- Add regression coverage for duplicate-key sync errors when collection config has no
packages/db/tests/query/includes.test.ts- Add coverage for joined includes child reconciliation.
packages/query-db-collection/tests/query.test.ts- Add coverage for query collection update/refetch feeding a live-query includes child collection.
.changeset/fix-sync-duplicate-key-utils.md- Add patch changeset for
@tanstack/db.
- Add patch changeset for
Fixes #1599
Summary by CodeRabbit#
Bug Fixes#
- Improved duplicate-key sync failure handling in live queries, preventing unexpected TypeErrors and ensuring the correct duplicate-key error is raised.
- Refined live query includes propagation for joined child collections to avoid duplicate child-row insertions during update flows.
- Improved child write behavior to correctly choose between inserts and updates based on whether the child already exists.
Tests#
- Added regression tests covering duplicate-key sync behavior and joined include update scenarios.
- Extended existing test assertions and imports to validate the new error and propagation behavior.
- 1
More templates
@tanstack/angular-db
npm i https://pkg.pr.new/@tanstack/angular-db@1600@tanstack/browser-db-sqlite-persistence
npm i https://pkg.pr.new/@tanstack/browser-db-sqlite-persistence@1600@tanstack/capacitor-db-sqlite-persistence
npm i https://pkg.pr.new/@tanstack/capacitor-db-sqlite-persistence@1600@tanstack/cloudflare-durable-objects-db-sqlite-persistence
npm i https://pkg.pr.new/@tanstack/cloudflare-durable-objects-db-sqlite-persistence@1600@tanstack/db
npm i https://pkg.pr.new/@tanstack/db@1600@tanstack/db-ivm
npm i https://pkg.pr.new/@tanstack/db-ivm@1600@tanstack/db-sqlite-persistence-core
npm i https://pkg.pr.new/@tanstack/db-sqlite-persistence-core@1600@tanstack/electric-db-collection
npm i https://pkg.pr.new/@tanstack/electric-db-collection@1600@tanstack/electron-db-sqlite-persistence
npm i https://pkg.pr.new/@tanstack/electron-db-sqlite-persistence@1600@tanstack/expo-db-sqlite-persistence
npm i https://pkg.pr.new/@tanstack/expo-db-sqlite-persistence@1600@tanstack/node-db-sqlite-persistence
npm i https://pkg.pr.new/@tanstack/node-db-sqlite-persistence@1600@tanstack/offline-transactions
npm i https://pkg.pr.new/@tanstack/offline-transactions@1600@tanstack/powersync-db-collection
npm i https://pkg.pr.new/@tanstack/powersync-db-collection@1600@tanstack/query-db-collection
npm i https://pkg.pr.new/@tanstack/query-db-collection@1600@tanstack/react-db
npm i https://pkg.pr.new/@tanstack/react-db@1600@tanstack/react-native-db-sqlite-persistence
npm i https://pkg.pr.new/@tanstack/react-native-db-sqlite-persistence@1600@tanstack/rxdb-db-collection
npm i https://pkg.pr.new/@tanstack/rxdb-db-collection@1600@tanstack/solid-db
npm i https://pkg.pr.new/@tanstack/solid-db@1600@tanstack/svelte-db
npm i https://pkg.pr.new/@tanstack/svelte-db@1600@tanstack/tauri-db-sqlite-persistence
npm i https://pkg.pr.new/@tanstack/tauri-db-sqlite-persistence@1600@tanstack/trailbase-db-collection
npm i https://pkg.pr.new/@tanstack/trailbase-db-collection@1600@tanstack/vue-db
npm i https://pkg.pr.new/@tanstack/vue-db@1600commit:
8305b9dSize Change: +25 B (+0.02%)
Total Size: 123 kB
📦 View Changed
Filename Size Change packages/db/dist/esm/collection/sync.js2.88 kB +2 B (+0.07%) packages/db/dist/esm/query/live/collection-config-builder.js8.38 kB +23 B (+0.28%) compressed-size-action::db-package-sizeℹ️ View Unchanged
Filename Size packages/db/dist/esm/collection/change-events.js1.39 kB packages/db/dist/esm/collection/changes.js1.38 kB packages/db/dist/esm/collection/cleanup-queue.js810 B packages/db/dist/esm/collection/events.js434 B packages/db/dist/esm/collection/index.js3.62 kB packages/db/dist/esm/collection/indexes.js1.99 kB packages/db/dist/esm/collection/lifecycle.js1.69 kB packages/db/dist/esm/collection/mutations.js2.47 kB packages/db/dist/esm/collection/state.js5.33 kB packages/db/dist/esm/collection/subscription.js3.74 kB packages/db/dist/esm/collection/transaction-metadata.js144 B packages/db/dist/esm/deferred.js207 B packages/db/dist/esm/errors.js5.01 kB packages/db/dist/esm/event-emitter.js748 B packages/db/dist/esm/index.js3.09 kB packages/db/dist/esm/indexes/auto-index.js829 B packages/db/dist/esm/indexes/base-index.js729 B packages/db/dist/esm/indexes/basic-index.js2.05 kB packages/db/dist/esm/indexes/btree-index.js2.17 kB packages/db/dist/esm/indexes/index-registry.js820 B packages/db/dist/esm/indexes/reverse-index.js538 B packages/db/dist/esm/local-only.js916 B packages/db/dist/esm/local-storage.js2.12 kB packages/db/dist/esm/optimistic-action.js359 B packages/db/dist/esm/paced-mutations.js496 B packages/db/dist/esm/proxy.js3.75 kB packages/db/dist/esm/query/builder/functions.js1.47 kB packages/db/dist/esm/query/builder/index.js5.84 kB packages/db/dist/esm/query/builder/ref-proxy.js1.24 kB packages/db/dist/esm/query/compiler/evaluators.js1.83 kB packages/db/dist/esm/query/compiler/expressions.js430 B packages/db/dist/esm/query/compiler/group-by.js3.56 kB packages/db/dist/esm/query/compiler/index.js6.67 kB packages/db/dist/esm/query/compiler/joins.js2.5 kB packages/db/dist/esm/query/compiler/lazy-targets.js918 B packages/db/dist/esm/query/compiler/order-by.js1.74 kB packages/db/dist/esm/query/compiler/select.js1.4 kB packages/db/dist/esm/query/effect.js4.77 kB packages/db/dist/esm/query/expression-helpers.js1.43 kB packages/db/dist/esm/query/ir.js1.25 kB packages/db/dist/esm/query/live-query-collection.js360 B packages/db/dist/esm/query/live/collection-registry.js264 B packages/db/dist/esm/query/live/collection-subscriber.js1.93 kB packages/db/dist/esm/query/live/internal.js145 B packages/db/dist/esm/query/live/utils.js1.81 kB packages/db/dist/esm/query/optimizer.js2.92 kB packages/db/dist/esm/query/predicate-utils.js2.97 kB packages/db/dist/esm/query/query-once.js359 B packages/db/dist/esm/query/subset-dedupe.js960 B packages/db/dist/esm/scheduler.js1.3 kB packages/db/dist/esm/SortedMap.js1.3 kB packages/db/dist/esm/strategies/debounceStrategy.js247 B packages/db/dist/esm/strategies/queueStrategy.js428 B packages/db/dist/esm/strategies/throttleStrategy.js246 B packages/db/dist/esm/transactions.js3.03 kB packages/db/dist/esm/utils.js927 B packages/db/dist/esm/utils/array-utils.js273 B packages/db/dist/esm/utils/browser-polyfills.js304 B packages/db/dist/esm/utils/btree.js5.61 kB packages/db/dist/esm/utils/comparison.js1.05 kB packages/db/dist/esm/utils/cursor.js457 B packages/db/dist/esm/utils/index-optimization.js1.54 kB packages/db/dist/esm/utils/type-guards.js157 B packages/db/dist/esm/utils/uuid.js449 B packages/db/dist/esm/virtual-props.js360 B Size Change: 0 B
Total Size: 4.24 kB
compressed-size-action::react-db-package-sizeℹ️ View Unchanged
Filename Size packages/react-db/dist/esm/index.js249 B packages/react-db/dist/esm/useLiveInfiniteQuery.js1.32 kB packages/react-db/dist/esm/useLiveQuery.js1.34 kB packages/react-db/dist/esm/useLiveQueryEffect.js355 B packages/react-db/dist/esm/useLiveSuspenseQuery.js567 B packages/react-db/dist/esm/usePacedMutations.js401 B Actionable comments posted: 2
🤖 Prompt for all review comments with AI agents
Verify each finding against current code. Fix only still-valid issues, skip the rest with a brief reason, keep changes minimal, and validate. Inline comments: In `@packages/db/tests/query/includes.test.ts`: - Around line 617-621: Replace the fixed `setTimeout(resolve, 10)` sleep in this test with a condition-based wait that directly polls for the expected child items snapshot. Instead of sleeping for a fixed duration, create a polling loop that repeatedly checks if the childItems from the product matches the expected array, and only resolve the promise when the condition is satisfied. This eliminates the timing sensitivity and prevents test flakiness when async propagation is slower. In `@packages/query-db-collection/tests/query.test.ts`: - Around line 292-298: The regression assertion in the vi.waitFor() callback is validating products.get() directly, which doesn't test if the includes-child reconciliation is working properly and can miss duplicate-insert bugs in the join logic. Change the assertion to validate the included product collection from the live query row instead, so the test actually guards against regression in the child include reconciliation. Modify the expect statement inside vi.waitFor() to access the product data through the included child collection relationship rather than through direct products.get() access.🪄 Autofix (Beta)
Fix all unresolved CodeRabbit comments on this PR:
- Push a commit to this branch (recommended)
- Create a new PR with the fixes
ℹ️ Review info
⚙️ Run configuration
Configuration used: defaults
Review profile: CHILL
Plan: Pro
Run ID:
2c8e03e7-4488-41b6-bd03-dd6d99f54b83📥 Commits
Reviewing files that changed from the base of the PR and between dbb87aaae613d3096b8e9e2f76ba7ec477e46f89 and c1832723c0ea39520ed2ad12d5f495e6a4df847b.
📒 Files selected for processing (4)
packages/db/src/query/live/collection-config-builder.tspackages/db/tests/collection.test.tspackages/db/tests/query/includes.test.tspackages/query-db-collection/tests/query.test.ts
🚧 Files skipped from review as they are similar to previous changes (1)
- packages/db/tests/collection.test.ts
@@ -569,0 +601,21 @@601+ )602+ await collection.preload()603+604+ lineItems.utils.begin()605+ expect(() => {606+ lineItems.utils.write({607+ type: `delete`,608+ value: { id: 1, productId: 10, qty: 1 },609+ })610+ lineItems.utils.write({611+ type: `insert`,612+ value: { id: 1, productId: 10, qty: 2 },613+ })614+ }).not.toThrow()615+ lineItems.utils.commit()616+617+ await new Promise((resolve) => setTimeout(resolve, 10))618+619+ expect(childItems((collection.get(1) as any).product)).toEqual([620+ { id: 10, categoryId: 1, name: `Widget` },621+ ])⚠️ Potential issue | 🟡 Minor | ⚡ Quick win
Use condition-based waiting instead of fixed sleep in this async regression test.
The fixed
setTimeout(10)makes this test timing-sensitive and can flake when propagation is slower. Wait for the expected child snapshot directly.Suggested change
- await new Promise((resolve) => setTimeout(resolve, 10)) - - expect(childItems((collection.get(1) as any).product)).toEqual([ - { id: 10, categoryId: 1, name: `Widget` }, - ]) + await vi.waitFor(() => { + expect(childItems((collection.get(1) as any).product)).toEqual([ + { id: 10, categoryId: 1, name: `Widget` }, + ]) + })As per coding guidelines, "Test corner cases including: ... async race conditions ...".
📝 Committable suggestion
‼️ IMPORTANT Carefully review the code before committing. Ensure that it accurately replaces the highlighted code, contains no missing lines, and has no issues with indentation. Thoroughly test & benchmark the code to ensure it meets the requirements.
await vi.waitFor(() => { expect(childItems((collection.get(1) as any).product)).toEqual([ { id: 10, categoryId: 1, name: `Widget` }, ]) })🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the rest with a brief reason, keep changes minimal, and validate. In `@packages/db/tests/query/includes.test.ts` around lines 617 - 621, Replace the fixed `setTimeout(resolve, 10)` sleep in this test with a condition-based wait that directly polls for the expected child items snapshot. Instead of sleeping for a fixed duration, create a polling loop that repeatedly checks if the childItems from the product matches the expected array, and only resolve the promise when the condition is satisfied. This eliminates the timing sensitivity and prevents test flakiness when async propagation is slower.Source: Coding guidelines
✅ Addressed in commit 8305b9d
@@ -230,0 +278,21 @@278+ categoryId: product.categoryId,279+ name: product.name,280+ })),281+ })),282+ )283+284+ await collection.preload()285+286+ expect(() => {287+ products.update(`product-1`, (draft) => {288+ draft.categoryId = 2289+ })290+ }).not.toThrow()291+292+ await vi.waitFor(() => {293+ expect(stripVirtualProps(products.get(`product-1`) as any)).toEqual({294+ id: `product-1`,295+ categoryId: 2,296+ name: `Widget`,297+ })298+ })⚠️ Potential issue | 🟠 Major | ⚡ Quick win
This regression assertion checks the wrong target and can miss the includes-child bug.
The test currently validates
products.get(...), which can succeed even if the joined child include reconciliation is broken. Assert against the live query row’s includedproductcollection so this actually guards duplicate-insert regressions.Suggested change
- await vi.waitFor(() => { - expect(stripVirtualProps(products.get(`product-1`) as any)).toEqual({ - id: `product-1`, - categoryId: 2, - name: `Widget`, - }) - }) + await vi.waitFor(() => { + expect(childItems((collection.get(`line-1`) as any).product)).toEqual([ + { id: `product-1`, categoryId: 2, name: `Widget` }, + ]) + })As per coding guidelines, "Always add unit tests that reproduce a bug before fixing it to ensure the bug is fixed and prevent regression".
🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the rest with a brief reason, keep changes minimal, and validate. In `@packages/query-db-collection/tests/query.test.ts` around lines 292 - 298, The regression assertion in the vi.waitFor() callback is validating products.get() directly, which doesn't test if the includes-child reconciliation is working properly and can miss duplicate-insert bugs in the join logic. Change the assertion to validate the included product collection from the live query row instead, so the test actually guards against regression in the child include reconciliation. Modify the expect statement inside vi.waitFor() to access the product data through the included child collection relationship rather than through direct products.get() access.Source: Coding guidelines
✅ Addressed in commit 8305b9d
Installed : https://pkg.pr.new/@tanstack/react-db@c183272
This seems to done the trick i no longer see the error in the console and updates are working again thanks for the quick response!!
KyleAMathews(GitHub profile, opens in a new tab) changed the title
kevin-dp(GitHub profile, opens in a new tab) approved these changes
Good fix!
kevin-dp(GitHub profile, opens in a new tab) closed this
Sign in to join this conversation.
No actionable comments were generated in the recent review. 🎉
ℹ️ Recent review info
⚙️ Run configuration
Configuration used: defaults
Review profile: CHILL
Plan: Pro
Run ID:
af881539-6d7b-474f-b473-ac221f464346📥 Commits
Reviewing files that changed from the base of the PR and between 7cba2013eb4bba92872aa95530c0e14ec5246b72 and 8305b9d353f958aed4450c3de965a7b4714b32ff.
📒 Files selected for processing (2)
packages/db/tests/query/includes.test.tspackages/query-db-collection/tests/query.test.ts🚧 Files skipped from review as they are similar to previous changes (2)
📝 Walkthrough
Walkthrough#
CollectionSyncManager.write()now uses optional chaining when accessingthis.config.utilsandLIVE_QUERY_INTERNAL, preventing aTypeErrorfor collections created without live-query utils. The child include logic influshIncludesStateis refined to decide betweeninsertandupdatebased on whether the child key already exists in the collection. Regression tests and a patch changeset are added.Changes#
Sync duplicate-key fix and child include insert/update improvements
packages/db/src/collection/sync.ts,.changeset/fix-sync-duplicate-key-utils.mdsync.tsderivesinternalvia optional chaining onthis.config.utilsbefore accessingLIVE_QUERY_INTERNAL, soDuplicateKeySyncErroris constructed safely when utils is undefined. The changeset documents the patch.packages/db/tests/collection.test.tsDuplicateKeySyncErrorandSyncConfigtypes, and a new test asserts that when a collection is created without utils and a duplicate insert occurs, the error type isDuplicateKeySyncErrorrather thanTypeError.packages/db/src/query/live/collection-config-builder.ts,packages/db/tests/query/includes.test.tsflushIncludesStatenow computes the child key and checks whether it already exists in the collection, using that flag to conditionally emitinsertorupdatefor pure-insert changes. A regression test verifies that deleting and reinserting a parent row with a joined include does not cause duplicate child inserts.packages/query-db-collection/tests/query.test.tsmockSyncCollectionOptions, and a new test constructs a live query joining a sync-backed line-items collection to a query-backed products collection, updates a product, and verifies the child data reflects the update without throwing or creating duplicate inserts.Estimated code review effort#
🎯 3 (Moderate) | ⏱️ ~25 minutes
Possibly related issues#
TypeErrorwhen accessingutils[LIVE_QUERY_INTERNAL]on an undefinedutilsobject insideCollectionSyncManager.write()during sync reconciliation for collections created withqueryCollectionOptions. This PR implements the fix by guarding the access with optional chaining so undefined utils are handled safely.Suggested reviewers#
Poem#
🚥 Pre-merge checks | ✅ 5
✅ Passed checks (5 passed)
#1599: guards optional config.utils access before reading LIVE_QUERY_INTERNAL, and converts insert-only child changes to updates when the child key already exists.#1599: sync.ts guards utils access, collection-config-builder.ts prevents duplicate child inserts, and test files provide regression coverage.✨ Finishing Touches
📝 Generate docstrings
🧪 Generate unit tests (beta)
Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.
❤️ Share