Skip to content
Source
Sign inSign up
TanStack/db

fix(sqlite-persistence): preserve expression index use #1867

MergedKyleAMathews(GitHub profile, opens in a new tab)merged 12 commits intomain fromrefs/pull/1867/headmerged by KyleAMathews(GitHub profile, opens in a new tab)

Persisted SQLite queries can now use the expression indexes that their collection configuration creates. Previously, queries returned expected rows while SQLite scanned the collection table.

Index DDL stored JSON paths as SQL literals. Runtime predicates bound the same paths as parameters. SQLite did not treat these forms as one expression.

This change gives index DDL and runtime predicates one canonical expression shape. It also rebuilds stale physical indexes after compiler output changes.

Why SQLite missed the index

The index and the old predicate could use the same path and return the same rows:

-- Persisted index expression
json_extract(value, '$.profile.score')

-- Old runtime predicate
json_extract(value, ?)

SQLite matches expression indexes by SQL structure. A parameter that contains $.profile.score does not match the literal path in the index definition.

The compiler now emits the validated path as a canonical SQL literal:

json_extract(value, '$.profile.score')

Ordinary comparison values remain parameters. Scalar BigInt targets become validated SQL integers when SQLite needs that numeric expression shape.

How the change works

  • The SQLite compiler validates each JSON path before it creates a SQL literal.
  • Index DDL and runtime references use the same canonical CASE and json_extract expressions.
  • Explicit sourceAlias metadata distinguishes qualified references from legacy nested property paths.
  • Remote subset transport retains that metadata and rejects an alias that disagrees with its path.
  • Existing PropRef.path values keep their prior meaning. The compiler does not guess that a nested path contains an alias.
  • The index registry compares normalized compiler output. It rebuilds the physical index when the output changes without a signature change.
  • New persisted BigInts and query values use SQLite's signed 64-bit domain. Older oversized values remain readable from stored rows.

This keeps strings, dates, booleans, numbers, and IN list values bound. Compiler-owned paths and index constants use validated literals.

Oracle coverage

The Node oracle uses the public SQLite-core adapter with the real BetterSQLite3 driver. It makes these observations independently:

  • expected result keys from a small reference model.
  • exact SQL and bindings before adapter filtering.
  • direct SQL result keys.
  • ordered results when the query promises an order.
  • named-index use through EXPLAIN QUERY PLAN.

The same property, generators, observations, and eight-run budget run in fixed-seed and seedless-random campaigns. A seed and shrink path select one exact replay.

The generated grammar covers equality, normal IN, 901-value batched IN, ranges, conjunctions, ordering, and lower(ref). Each axis has an ablation and a plausible wrong-answer control.

Deterministic cases cover index upgrades, constant-bearing expressions, aliases, legacy nested paths, dates, and signed 64-bit BigInts. A new legacy-byte case covers oversized BigInt reads. Coordinator tests cover alias transport and reject conflicting alias metadata.

Exact replay:

TANSTACK_DB_WS5A_SEED=1659005 TANSTACK_DB_WS5A_PATH=0 \
pnpm --filter @tanstack/node-db-sqlite-persistence test:oracles

Scope boundaries

  • This change does not add arbitrary raw SQL support.
  • The oracle does not claim null or general Unicode and collation behavior.
  • The planner proof covers Node BetterSQLite3. It does not prove native-host query planning.
  • The generator uses bounded JSON paths and scalar values.
  • New BigInt writes, comparisons, and index constants outside SQLite's signed 64-bit range fail with a specific error. Legacy stored values still decode.

Implementation trailhead

  • Query IR records an explicit source alias without changing the property path.
  • SQLite persistence compiles canonical expressions, validates new BigInts, and replaces stale indexes.
  • Remote subset projection preserves qualified references across coordinator routing.
  • The Node oracle captures production SQL and compares rows and query plans separately.
  • Oracle documentation records the campaign, replay command, and proof limits.

Verification

  • @tanstack/db: 6,385 tests passed before the review follow-up. The follow-up did not change this package.
  • SQLite persistence core: 516 tests passed. Two existing tests remain marked as todo.
  • Node SQLite persistence: 135 tests passed.
  • Browser coordinator: 204 tests passed.
  • Focused query compiler and IR suites: 264 tests passed before the review follow-up.
  • Exact seed-and-path replay: one runtime target passed and 48 unrelated runtime tests skipped.
  • SQLite persistence-core and Node persistence builds and standalone typechecks passed.
  • ESLint reported zero errors and 19 existing warnings. Prettier passed, and the final diff had no whitespace errors.

✅ Checklist

  • I tested this code locally with the commands and campaigns above.

🚀 Release Impact

  • This change affects published code, and I generated a changeset.
  • This change is docs/CI/dev-only and does not need a release.

Part of #1659

    1. Review in Change Stack →

      Navigate logical layers of code changes, visualize relationships, and explore their blast radius.

      Note

      Reviews paused

      It looks like this branch is under active development. To avoid overwhelming you with review comments due to an influx of new commits, CodeRabbit has automatically paused this review. You can configure this behavior by changing the reviews.auto_review.auto_pause_after_reviewed_commits setting.

      Use the following commands to manage reviews:

      • @coderabbitai resume to resume automatic reviews.
      • @coderabbitai review to trigger a single review.

      Use the checkboxes below for quick actions:

      • ▶️ Resume reviews
      • 🔍 Trigger review

      No actionable comments were generated in the recent review. 🎉

      ℹ️ Recent review info
      ⚙️ Run configuration

      Configuration used: defaults

      Review profile: CHILL

      Plan: Advanced

      Run ID: 899a38a8-117f-4c84-860d-123e7a0c53a6

      📥 Commits

      Reviewing files that changed from the base of the PR and between 6db8fe7535c3f4a686e7172c17093f22535a4927 and 28bbddd1c9154fd62a5553664372209755469782.

      📒 Files selected for processing (1)
      • packages/node-db-sqlite-persistence/tests/expression-index-oracle.test.ts
      🚧 Files skipped from review as they are similar to previous changes (1)
      • packages/node-db-sqlite-persistence/tests/expression-index-oracle.test.ts

      Included review availability: Your plan provides up to 8 included reviews per hour; 2 remain after this review.


      📝 Walkthrough

      Walkthrough

      compileRefExpressionSql now emits JSON paths as SQL literals so runtime predicates match persisted SQLite expression indexes. A property-based oracle verifies query results and index usage through EXPLAIN QUERY PLAN.

      Changes

      SQLite expression-index planning

      Layer / File(s)Summary
      Inline reference paths
      packages/db-sqlite-persistence-core/src/sqlite-core-adapter.ts, .changeset/fix-sqlite-expression-index-planning.md
      compileRefExpressionSql emits validated JSON paths as SQL literals and removes the four path bindings. A patch changeset records the fix.
      Validate expression-index usage
      packages/node-db-sqlite-persistence/tests/expression-index-oracle.test.ts, packages/node-db-sqlite-persistence/package.json, docs/contributing/oracle-coverage.md
      The property-based oracle generates nested paths and scalar values, compares results with an independent scan, verifies named-index usage through EXPLAIN QUERY PLAN, and covers quoted identifiers, rejected DDL bindings, and prefix collisions. A package script and coverage documentation expose the oracle.

      Priority: ➖ Normal

      Estimated code review effort: 3 (Moderate) | ~25 minutes

      Change: Bug fix

      Merge Risk: ⚪ Minimal · up to 28bbd

      The adapter emits canonical SQL literals for persisted JSON reference paths while comparison values remain bound, and the reported validation passes with no actionable merge-blocking risk evidenced.

      🚥 Pre-merge checks | ✅ 4 | ❌ 1

      ❌ Failed checks (1 warning)

      Check nameStatusExplanationResolution
      Docstring Coverage⚠️ WarningDocstring coverage is 0.00% which is insufficient. The required threshold is 80.00%. Docstring coverage is scoped to functions touched by this diff. Analyzed 14 functions across 2 files.Write docstrings for the functions missing them to satisfy the coverage threshold.
      ✅ Passed checks (4 passed)
      Check nameStatusExplanation
      Linked Issues check✅ PassedCheck skipped because no linked issues were found for this pull request.
      Out of Scope Changes check✅ PassedCheck skipped because no linked issues were found for this pull request.
      Title check✅ PassedThe title clearly identifies the SQLite persistence change and its primary outcome: preserving expression index use.
      Description check✅ PassedThe description explains the change, motivation, implementation, scope, verification, testing, and release impact. It includes the required checklist and changeset information, although it uses detail…
      • Fix all pre-merge checks with AI
      ✨ Finishing Touches 💡 2
      📝 Generate docstrings 💡
      • Commit to this branch
      • Create a new PR
      🧪 Generate unit tests (beta)
      • Commit to this branch
      • Create a new PR
      🛠️ Fix failing CI checks 💡
      • Commit to this branch
      • Create a new PR

      Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

      ❤️ Share
      Comment `@coderabbitai help` to get the list of available commands.
    2. More templates

      @tanstack/angular-db

      npm i https://pkg.pr.new/@tanstack/angular-db@1867

      @tanstack/browser-db-sqlite-persistence

      npm i https://pkg.pr.new/@tanstack/browser-db-sqlite-persistence@1867

      @tanstack/capacitor-db-sqlite-persistence

      npm i https://pkg.pr.new/@tanstack/capacitor-db-sqlite-persistence@1867

      @tanstack/cloudflare-durable-objects-db-sqlite-persistence

      npm i https://pkg.pr.new/@tanstack/cloudflare-durable-objects-db-sqlite-persistence@1867

      @tanstack/db

      npm i https://pkg.pr.new/@tanstack/db@1867

      @tanstack/db-ivm

      npm i https://pkg.pr.new/@tanstack/db-ivm@1867

      @tanstack/db-sqlite-persistence-core

      npm i https://pkg.pr.new/@tanstack/db-sqlite-persistence-core@1867

      @tanstack/electric-db-collection

      npm i https://pkg.pr.new/@tanstack/electric-db-collection@1867

      @tanstack/electron-db-sqlite-persistence

      npm i https://pkg.pr.new/@tanstack/electron-db-sqlite-persistence@1867

      @tanstack/expo-db-sqlite-persistence

      npm i https://pkg.pr.new/@tanstack/expo-db-sqlite-persistence@1867

      @tanstack/node-db-sqlite-persistence

      npm i https://pkg.pr.new/@tanstack/node-db-sqlite-persistence@1867

      @tanstack/offline-transactions

      npm i https://pkg.pr.new/@tanstack/offline-transactions@1867

      @tanstack/powersync-db-collection

      npm i https://pkg.pr.new/@tanstack/powersync-db-collection@1867

      @tanstack/query-db-collection

      npm i https://pkg.pr.new/@tanstack/query-db-collection@1867

      @tanstack/react-db

      npm i https://pkg.pr.new/@tanstack/react-db@1867

      @tanstack/react-native-db-sqlite-persistence

      npm i https://pkg.pr.new/@tanstack/react-native-db-sqlite-persistence@1867

      @tanstack/react-router-with-db

      npm i https://pkg.pr.new/@tanstack/react-router-with-db@1867

      @tanstack/rxdb-db-collection

      npm i https://pkg.pr.new/@tanstack/rxdb-db-collection@1867

      @tanstack/solid-db

      npm i https://pkg.pr.new/@tanstack/solid-db@1867

      @tanstack/svelte-db

      npm i https://pkg.pr.new/@tanstack/svelte-db@1867

      @tanstack/tauri-db-sqlite-persistence

      npm i https://pkg.pr.new/@tanstack/tauri-db-sqlite-persistence@1867

      @tanstack/trailbase-db-collection

      npm i https://pkg.pr.new/@tanstack/trailbase-db-collection@1867

      @tanstack/vue-db

      npm i https://pkg.pr.new/@tanstack/vue-db@1867

      commit: bf763eb

    3. Size Change: +507 B (+0.3%)

      Total Size: 170 kB

      📦 View Changed
      FilenameSizeChange
      packages/db/dist/esm/collection/subscription.js8.8 kB+21 B (+0.24%)
      packages/db/dist/esm/query/builder/ref-proxy.js1.28 kB+42 B (+3.38%)
      packages/db/dist/esm/query/compiler/evaluators.js2.04 kB+79 B (+4.02%)
      packages/db/dist/esm/query/compiler/expressions.js603 B+43 B (+7.68%)🔍
      packages/db/dist/esm/query/compiler/group-by.js4.14 kB+14 B (+0.34%)
      packages/db/dist/esm/query/compiler/lazy-targets.js1.12 kB+13 B (+1.18%)
      packages/db/dist/esm/query/compiler/select.js1.59 kB+11 B (+0.7%)
      packages/db/dist/esm/query/ir-stable-identity.js4.18 kB+135 B (+3.34%)
      packages/db/dist/esm/query/ir.js1.74 kB+149 B (+9.37%)🔍
      ℹ️ View Unchanged
      FilenameSize
      packages/db/dist/esm/client.js3.66 kB
      packages/db/dist/esm/collection-options.js236 B
      packages/db/dist/esm/collection/change-events.js1.44 kB
      packages/db/dist/esm/collection/changes.js2.4 kB
      packages/db/dist/esm/collection/cleanup-queue.js794 B
      packages/db/dist/esm/collection/events.js481 B
      packages/db/dist/esm/collection/index.js4.36 kB
      packages/db/dist/esm/collection/indexes.js1.99 kB
      packages/db/dist/esm/collection/lifecycle.js2.15 kB
      packages/db/dist/esm/collection/mutations.js2.61 kB
      packages/db/dist/esm/collection/state.js6.94 kB
      packages/db/dist/esm/collection/sync.js4.62 kB
      packages/db/dist/esm/collection/transaction-metadata.js144 B
      packages/db/dist/esm/deferred.js207 B
      packages/db/dist/esm/errors.js5.34 kB
      packages/db/dist/esm/event-emitter.js964 B
      packages/db/dist/esm/index.js3.82 kB
      packages/db/dist/esm/indexes/auto-index.js829 B
      packages/db/dist/esm/indexes/base-index.js1.14 kB
      packages/db/dist/esm/indexes/basic-index.js2.07 kB
      packages/db/dist/esm/indexes/btree-index.js2.26 kB
      packages/db/dist/esm/indexes/index-registry.js820 B
      packages/db/dist/esm/indexes/reverse-index.js376 B
      packages/db/dist/esm/live-query-adapter.js318 B
      packages/db/dist/esm/live-query-observer.js3.69 kB
      packages/db/dist/esm/live-query-options.js702 B
      packages/db/dist/esm/live-query-window-controller.js4.36 kB
      packages/db/dist/esm/local-only.js989 B
      packages/db/dist/esm/local-storage.js2.17 kB
      packages/db/dist/esm/optimistic-action.js359 B
      packages/db/dist/esm/paced-mutations.js496 B
      packages/db/dist/esm/proxy.js3.32 kB
      packages/db/dist/esm/query/builder/clone-query.js748 B
      packages/db/dist/esm/query/builder/functions.js1.47 kB
      packages/db/dist/esm/query/builder/index.js6.72 kB
      packages/db/dist/esm/query/builder/query-ir.js116 B
      packages/db/dist/esm/query/compiler/index.js9.11 kB
      packages/db/dist/esm/query/compiler/joins.js2.99 kB
      packages/db/dist/esm/query/compiler/order-by.js1.91 kB
      packages/db/dist/esm/query/compiler/parent-routes.js319 B
      packages/db/dist/esm/query/compiler/query-equivalence.js455 B
      packages/db/dist/esm/query/compiler/route-metadata.js1.24 kB
      packages/db/dist/esm/query/effect.js5.13 kB
      packages/db/dist/esm/query/equality-value-identity.js591 B
      packages/db/dist/esm/query/expression-helpers.js1.43 kB
      packages/db/dist/esm/query/live-query-collection.js391 B
      packages/db/dist/esm/query/live/bucket-facade-adapter.js2.73 kB
      packages/db/dist/esm/query/live/collection-config-builder.js6.97 kB
      packages/db/dist/esm/query/live/collection-registry.js264 B
      packages/db/dist/esm/query/live/collection-subscriber.js2.26 kB
      packages/db/dist/esm/query/live/internal.js145 B
      packages/db/dist/esm/query/live/materialized-pipeline.js2.32 kB
      packages/db/dist/esm/query/live/ordered-source-loader.js3.81 kB
      packages/db/dist/esm/query/live/subset-demand-controller.js1.26 kB
      packages/db/dist/esm/query/live/utils.js1.14 kB
      packages/db/dist/esm/query/optimizer.js3.11 kB
      packages/db/dist/esm/query/query-once.js359 B
      packages/db/dist/esm/query/runtime-reference-identity.js572 B
      packages/db/dist/esm/query/subset-dedupe.js497 B
      packages/db/dist/esm/scheduler.js1.34 kB
      packages/db/dist/esm/SortedMap.js1.3 kB
      packages/db/dist/esm/strategies/debounceStrategy.js247 B
      packages/db/dist/esm/strategies/queueStrategy.js428 B
      packages/db/dist/esm/strategies/throttleStrategy.js246 B
      packages/db/dist/esm/sync-persistence.js530 B
      packages/db/dist/esm/transactions.js3.71 kB
      packages/db/dist/esm/utils.js1.21 kB
      packages/db/dist/esm/utils/array-utils.js270 B
      packages/db/dist/esm/utils/browser-polyfills.js304 B
      packages/db/dist/esm/utils/btree.js4.51 kB
      packages/db/dist/esm/utils/callbacks.js174 B
      packages/db/dist/esm/utils/comparison.js1.49 kB
      packages/db/dist/esm/utils/cursor.js677 B
      packages/db/dist/esm/utils/error.js167 B
      packages/db/dist/esm/utils/get-or-create.js155 B
      packages/db/dist/esm/utils/index-optimization.js2.42 kB
      packages/db/dist/esm/utils/type-guards.js230 B
      packages/db/dist/esm/utils/uuid.js449 B
      packages/db/dist/esm/virtual-props.js360 B
      compressed-size-action::db-package-size
    4. Size Change: 0 B

      Total Size: 7.34 kB

      ℹ️ View Unchanged
      FilenameSize
      packages/react-db/dist/esm/DbProvider.js317 B
      packages/react-db/dist/esm/HydrationBoundary.js263 B
      packages/react-db/dist/esm/index.js330 B
      packages/react-db/dist/esm/live-query-internals.js282 B
      packages/react-db/dist/esm/useLiveInfiniteQuery.js1.9 kB
      packages/react-db/dist/esm/useLiveQuery.js2.68 kB
      packages/react-db/dist/esm/useLiveQueryEffect.js355 B
      packages/react-db/dist/esm/useLiveSuspenseQuery.js812 B
      packages/react-db/dist/esm/usePacedMutations.js401 B
      compressed-size-action::react-db-package-size

Sign in to join this conversation.