Skip to content
Source
Sign inSign up
TanStack/db

fix(db): complete loadSubset refinement lifecycle #1780

ClosedKyleAMathews(GitHub profile, opens in a new tab)wants to merge 250+ commits intocodex/loadsubset-total-order fromcodex/loadsubset-refinement-oracleopened

Completes the in-scope loadSubset RFC implementation on top of the outcome, coverage, and total-order stack. Refinement, replay, publication, readiness, and adapter cleanup now use one demand/acquisition/applied-result model.

What changed

  • Keep relational state and routing in the existing D2 graph. Custom state is limited to async source work and public publication boundaries.
  • Publish reusable coverage only from an exact, authoritative result after its rows are visible.
  • Make ordered replay and graph publication atomic. A failed replacement keeps the last complete public snapshot and remains retryable.
  • Track logical leases, shared acquisitions, row-key provenance, and physical release as separate lifetimes.
  • Use one total order for indexes, top-K, cursors, and refills.
  • Apply the same restart, cancellation, late-settlement, and cleanup laws in core, Query DB, Electric, and PowerSync.

Key invariants

  • A successful subset load settles only after its establishing writes are visible.
  • Stale, canceled, failed, and released attempts establish no current coverage.
  • Failed replay or publication cannot replace the last complete snapshot.
  • Coverage compaction cannot erase lease, acquisition, row-ownership, or release facts.
  • Each physical acquisition is released once after its last logical owner leaves; failed release remains retryable.
  • Callback errors keep their exact occurrence and order without leaving partial graph state.

Compatibility and limits

  • No new query or pagination syntax.
  • The registry stores provenance and row keys, not copies of row values. Its state is bounded by active leases, acquisitions, and owned key slots, and is retired on release, reset, truncate, and cleanup.
  • Outcome-free ordered adapters remain supported without inventing coverage. Core advances the caller-local boundary until a request makes no progress; later growth refreshes from the start. Authoritative continuation, replacement, source extent, and reusable coverage still require explicit outcomes.
  • When source ordering cannot prove a safe cursor, core loads the filtered source region and lets D2 refine it.
  • Electric cannot cancel a snapshot after ShapeStream.requestSnapshot() starts because the upstream API exposes neither a request signal nor request identity.
  • The generic DBSP operator-law harness remains tracked separately in #1741.

Verification

The integrated gate passed on 303a19ca. Follow-ups through 2143f481 close the outcome-free adapter boundary found by framework CI and the exact-commit audit: local settlement never becomes reusable coverage, shrink/regrow gets a fresh request generation, Effect refill stops at a no-progress fixed point, and all reset paths retire local settlement.

  • DB: 148 files, 4,691 tests passed, 6 skipped; no type errors.
  • Registered oracle gate: 26 files, 1,518 tests; no type errors.
  • Query DB: 338 tests. Electric: 616 runtime tests / 636 checks. PowerSync: 136 tests, including the exact prior CI seed.
  • React, Vue, and Svelte shared infinite-query conformance: 33/33 each.
  • Fixed-seed and fresh-random 100× campaigns each passed all 1,518 oracle tests.
  • The four explicitly run package builds passed. ESLint had no errors; Prettier and git diff --check passed.
  • The final whole-ledger audit found no dropped defect, false-green oracle, adapter law, surviving product mutant, or architecture contradiction. A follow-up exact-commit audit found one masked flag-retirement assertion; the final test kills that mutant across reset, authoritative settlement, and live invalidation.
  • GitHub CI is green on final commit 2143f481.

Main areas changed

  • Core sync, subscription, transaction, rollback, readiness, and publication.
  • Exact subset outcomes, coverage ownership, demand identity, total ordering, and window state.
  • D2-backed live-query publication and correlated materialization boundaries.
  • Query DB, Electric, and PowerSync lifecycle conformance.
  • Architecture and oracle documentation.

Stacked on #1778. Part of #1657.

    1. Important

      Review skipped

      Auto reviews are disabled on base/target branches other than the default branch.

      Please check the settings in the CodeRabbit UI or the .coderabbit.yaml file in this repository. To trigger a single review, invoke the @coderabbitai review command.

      ⚙️ Run configuration

      Configuration used: defaults

      Review profile: CHILL

      Plan: Team

      Run ID: 8359039a-9fb6-4edb-8198-ce0af989bcd4

      You can disable this status message by setting the reviews.review_status to false in the CodeRabbit configuration file.

      Use the checkbox below for a quick retry:

      • 🔍 Trigger review

      Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

      ❤️ Share
      Comment `@coderabbitai help` to get the list of available commands.
    2. More templates

      @tanstack/angular-db

      npm i https://pkg.pr.new/@tanstack/angular-db@1780

      @tanstack/browser-db-sqlite-persistence

      npm i https://pkg.pr.new/@tanstack/browser-db-sqlite-persistence@1780

      @tanstack/capacitor-db-sqlite-persistence

      npm i https://pkg.pr.new/@tanstack/capacitor-db-sqlite-persistence@1780

      @tanstack/cloudflare-durable-objects-db-sqlite-persistence

      npm i https://pkg.pr.new/@tanstack/cloudflare-durable-objects-db-sqlite-persistence@1780

      @tanstack/db

      npm i https://pkg.pr.new/@tanstack/db@1780

      @tanstack/db-ivm

      npm i https://pkg.pr.new/@tanstack/db-ivm@1780

      @tanstack/db-sqlite-persistence-core

      npm i https://pkg.pr.new/@tanstack/db-sqlite-persistence-core@1780

      @tanstack/electric-db-collection

      npm i https://pkg.pr.new/@tanstack/electric-db-collection@1780

      @tanstack/electron-db-sqlite-persistence

      npm i https://pkg.pr.new/@tanstack/electron-db-sqlite-persistence@1780

      @tanstack/expo-db-sqlite-persistence

      npm i https://pkg.pr.new/@tanstack/expo-db-sqlite-persistence@1780

      @tanstack/node-db-sqlite-persistence

      npm i https://pkg.pr.new/@tanstack/node-db-sqlite-persistence@1780

      @tanstack/offline-transactions

      npm i https://pkg.pr.new/@tanstack/offline-transactions@1780

      @tanstack/powersync-db-collection

      npm i https://pkg.pr.new/@tanstack/powersync-db-collection@1780

      @tanstack/query-db-collection

      npm i https://pkg.pr.new/@tanstack/query-db-collection@1780

      @tanstack/react-db

      npm i https://pkg.pr.new/@tanstack/react-db@1780

      @tanstack/react-native-db-sqlite-persistence

      npm i https://pkg.pr.new/@tanstack/react-native-db-sqlite-persistence@1780

      @tanstack/react-router-with-db

      npm i https://pkg.pr.new/@tanstack/react-router-with-db@1780

      @tanstack/rxdb-db-collection

      npm i https://pkg.pr.new/@tanstack/rxdb-db-collection@1780

      @tanstack/solid-db

      npm i https://pkg.pr.new/@tanstack/solid-db@1780

      @tanstack/svelte-db

      npm i https://pkg.pr.new/@tanstack/svelte-db@1780

      @tanstack/tauri-db-sqlite-persistence

      npm i https://pkg.pr.new/@tanstack/tauri-db-sqlite-persistence@1780

      @tanstack/trailbase-db-collection

      npm i https://pkg.pr.new/@tanstack/trailbase-db-collection@1780

      @tanstack/vue-db

      npm i https://pkg.pr.new/@tanstack/vue-db@1780

      commit: 2143f48

    3. Size Change: +9.74 kB (+5.35%) 🔍

      Total Size: 192 kB

      📦 View Changed
      FilenameSizeChange
      packages/db/dist/esm/collection/change-events.js1.65 kB+175 B (+11.87%)⚠️
      packages/db/dist/esm/collection/changes.js2.48 kB+537 B (+27.57%)🚨
      packages/db/dist/esm/collection/index.js4.08 kB+91 B (+2.28%)
      packages/db/dist/esm/collection/indexes.js2.04 kB+48 B (+2.41%)
      packages/db/dist/esm/collection/lifecycle.js2 kB+135 B (+7.25%)🔍
      packages/db/dist/esm/collection/mutations.js2.62 kB+83 B (+3.27%)
      packages/db/dist/esm/collection/state.js7.47 kB+960 B (+14.75%)⚠️
      packages/db/dist/esm/collection/subscription.js13 kB+446 B (+3.56%)
      packages/db/dist/esm/collection/sync.js7.63 kB+142 B (+1.9%)
      packages/db/dist/esm/indexes/auto-index.js833 B+4 B (+0.48%)
      packages/db/dist/esm/indexes/base-index.js1 kB+218 B (+27.81%)🚨
      packages/db/dist/esm/indexes/basic-index.js2.39 kB+227 B (+10.48%)⚠️
      packages/db/dist/esm/indexes/btree-index.js2.34 kB+41 B (+1.79%)
      packages/db/dist/esm/indexes/reverse-index.js652 B+95 B (+17.06%)⚠️
      packages/db/dist/esm/query/compiler/order-by.js1.97 kB+160 B (+8.86%)🔍
      packages/db/dist/esm/query/effect.js5.62 kB+148 B (+2.71%)
      packages/db/dist/esm/query/expression-value-context.js1.49 kB+1.49 kB (new file)🆕
      packages/db/dist/esm/query/ir-stable-identity.js4.35 kB+188 B (+4.51%)🔍
      packages/db/dist/esm/query/live/bucket-facade-adapter.js3.73 kB+973 B (+35.25%)🚨
      packages/db/dist/esm/query/live/collection-config-builder.js7.69 kB+370 B (+5.05%)🔍
      packages/db/dist/esm/query/live/collection-subscriber.js3.09 kB+451 B (+17.1%)⚠️
      packages/db/dist/esm/query/live/subset-demand-controller.js1.29 kB+53 B (+4.29%)
      packages/db/dist/esm/query/live/utils.js1.48 kB+94 B (+6.78%)🔍
      packages/db/dist/esm/query/live/window-state.js2.66 kB+500 B (+23.18%)🚨
      packages/db/dist/esm/query/load-subset-options.js1.29 kB+207 B (+19.17%)⚠️
      packages/db/dist/esm/query/load-subset-outcome.js682 B+125 B (+22.44%)🚨
      packages/db/dist/esm/query/predicate-utils.js3.44 kB+55 B (+1.63%)
      packages/db/dist/esm/query/subset-dedupe.js2.34 kB+759 B (+48.07%)🚨
      packages/db/dist/esm/query/total-order.js684 B+80 B (+13.25%)⚠️
      packages/db/dist/esm/scheduler.js1.6 kB+166 B (+11.59%)⚠️
      packages/db/dist/esm/transactions.js3.52 kB+24 B (+0.69%)
      packages/db/dist/esm/utils/array-utils.js208 B-65 B (-23.81%)🎉
      packages/db/dist/esm/utils/callbacks.js174 B+174 B (new file)🆕
      packages/db/dist/esm/utils/comparison.js1.97 kB+579 B (+41.56%)🚨
      ℹ️ View Unchanged
      FilenameSize
      packages/db/dist/esm/client.js3.66 kB
      packages/db/dist/esm/collection-options.js236 B
      packages/db/dist/esm/collection/cleanup-queue.js810 B
      packages/db/dist/esm/collection/events.js434 B
      packages/db/dist/esm/collection/transaction-metadata.js144 B
      packages/db/dist/esm/deferred.js207 B
      packages/db/dist/esm/errors.js5.39 kB
      packages/db/dist/esm/event-emitter.js1.08 kB
      packages/db/dist/esm/index.js3.81 kB
      packages/db/dist/esm/indexes/index-registry.js820 B
      packages/db/dist/esm/live-query-adapter.js318 B
      packages/db/dist/esm/live-query-observer.js3.65 kB
      packages/db/dist/esm/live-query-options.js702 B
      packages/db/dist/esm/live-query-window-controller.js4.43 kB
      packages/db/dist/esm/load-subset-request-provenance.js412 B
      packages/db/dist/esm/local-only.js975 B
      packages/db/dist/esm/local-storage.js2.18 kB
      packages/db/dist/esm/optimistic-action.js359 B
      packages/db/dist/esm/paced-mutations.js496 B
      packages/db/dist/esm/proxy.js3.75 kB
      packages/db/dist/esm/query/builder/functions.js1.47 kB
      packages/db/dist/esm/query/builder/get-query-ir.js121 B
      packages/db/dist/esm/query/builder/index.js6.6 kB
      packages/db/dist/esm/query/builder/ref-proxy.js1.24 kB
      packages/db/dist/esm/query/compiler/evaluators.js1.9 kB
      packages/db/dist/esm/query/compiler/expressions.js430 B
      packages/db/dist/esm/query/compiler/group-by.js3.69 kB
      packages/db/dist/esm/query/compiler/index.js8.71 kB
      packages/db/dist/esm/query/compiler/joins.js2.95 kB
      packages/db/dist/esm/query/compiler/lazy-targets.js1.11 kB
      packages/db/dist/esm/query/compiler/parent-routes.js319 B
      packages/db/dist/esm/query/compiler/route-metadata.js419 B
      packages/db/dist/esm/query/compiler/select.js1.58 kB
      packages/db/dist/esm/query/coverage-registry.js4.75 kB
      packages/db/dist/esm/query/expression-helpers.js1.43 kB
      packages/db/dist/esm/query/ir.js1.59 kB
      packages/db/dist/esm/query/live-query-collection.js391 B
      packages/db/dist/esm/query/live/collection-registry.js264 B
      packages/db/dist/esm/query/live/internal.js145 B
      packages/db/dist/esm/query/live/materialized-pipeline.js2.47 kB
      packages/db/dist/esm/query/optimizer.js2.92 kB
      packages/db/dist/esm/query/query-once.js359 B
      packages/db/dist/esm/query/runtime-reference-identity.js463 B
      packages/db/dist/esm/SortedMap.js1.3 kB
      packages/db/dist/esm/strategies/debounceStrategy.js247 B
      packages/db/dist/esm/strategies/queueStrategy.js428 B
      packages/db/dist/esm/strategies/throttleStrategy.js246 B
      packages/db/dist/esm/utils.js927 B
      packages/db/dist/esm/utils/browser-polyfills.js304 B
      packages/db/dist/esm/utils/btree.js5.61 kB
      packages/db/dist/esm/utils/cursor.js699 B
      packages/db/dist/esm/utils/index-optimization.js2.39 kB
      packages/db/dist/esm/utils/type-guards.js157 B
      packages/db/dist/esm/utils/uuid.js449 B
      packages/db/dist/esm/virtual-props.js360 B
      compressed-size-action::db-package-size
    4. Size Change: 0 B

      Total Size: 7.25 kB

      ℹ️ View Unchanged
      FilenameSize
      packages/react-db/dist/esm/DbProvider.js317 B
      packages/react-db/dist/esm/HydrationBoundary.js263 B
      packages/react-db/dist/esm/index.js330 B
      packages/react-db/dist/esm/live-query-internals.js282 B
      packages/react-db/dist/esm/useLiveInfiniteQuery.js1.81 kB
      packages/react-db/dist/esm/useLiveQuery.js2.68 kB
      packages/react-db/dist/esm/useLiveQueryEffect.js355 B
      packages/react-db/dist/esm/useLiveSuspenseQuery.js812 B
      packages/react-db/dist/esm/usePacedMutations.js401 B
      compressed-size-action::react-db-package-size
    5. Closing as superseded by merged #1797, which replaces this refinement/lifecycle stack with the consolidated implementation. The earlier source-outcome and general-coverage completion claims do not carry over; remaining work stays tracked in RFC #1657.

Sign in to join this conversation.