fix(sqlite-persistence): preserve expression index use #1867
main fromrefs/pull/1867/head·merged by KyleAMathews(GitHub profile, opens in a new tab) Persisted SQLite queries can now use the expression indexes that their collection configuration creates. Previously, queries returned expected rows while SQLite scanned the collection table.
Index DDL stored JSON paths as SQL literals. Runtime predicates bound the same paths as parameters. SQLite did not treat these forms as one expression.
This change gives index DDL and runtime predicates one canonical expression shape. It also rebuilds stale physical indexes after compiler output changes.
Why SQLite missed the index#
The index and the old predicate could use the same path and return the same rows:
-- Persisted index expression
json_extract(value, '$.profile.score')
-- Old runtime predicate
json_extract(value, ?)
SQLite matches expression indexes by SQL structure. A parameter that contains $.profile.score does not match the literal path in the index definition.
The compiler now emits the validated path as a canonical SQL literal:
json_extract(value, '$.profile.score')
Ordinary comparison values remain parameters. Scalar BigInt targets become validated SQL integers when SQLite needs that numeric expression shape.
How the change works#
- The SQLite compiler validates each JSON path before it creates a SQL literal.
- Index DDL and runtime references use the same canonical
CASEandjson_extractexpressions. - Explicit
sourceAliasmetadata distinguishes qualified references from legacy nested property paths. - Remote subset transport retains that metadata and rejects an alias that disagrees with its path.
- Existing
PropRef.pathvalues keep their prior meaning. The compiler does not guess that a nested path contains an alias. - The index registry compares normalized compiler output. It rebuilds the physical index when the output changes without a signature change.
- New persisted BigInts and query values use SQLite's signed 64-bit domain. Older oversized values remain readable from stored rows.
This keeps strings, dates, booleans, numbers, and IN list values bound. Compiler-owned paths and index constants use validated literals.
Oracle coverage#
The Node oracle uses the public SQLite-core adapter with the real BetterSQLite3 driver. It makes these observations independently:
- expected result keys from a small reference model.
- exact SQL and bindings before adapter filtering.
- direct SQL result keys.
- ordered results when the query promises an order.
- named-index use through
EXPLAIN QUERY PLAN.
The same property, generators, observations, and eight-run budget run in fixed-seed and seedless-random campaigns. A seed and shrink path select one exact replay.
The generated grammar covers equality, normal IN, 901-value batched IN, ranges, conjunctions, ordering, and lower(ref). Each axis has an ablation and a plausible wrong-answer control.
Deterministic cases cover index upgrades, constant-bearing expressions, aliases, legacy nested paths, dates, and signed 64-bit BigInts. A new legacy-byte case covers oversized BigInt reads. Coordinator tests cover alias transport and reject conflicting alias metadata.
Exact replay:
TANSTACK_DB_WS5A_SEED=1659005 TANSTACK_DB_WS5A_PATH=0 \
pnpm --filter @tanstack/node-db-sqlite-persistence test:oracles
Scope boundaries#
- This change does not add arbitrary raw SQL support.
- The oracle does not claim null or general Unicode and collation behavior.
- The planner proof covers Node BetterSQLite3. It does not prove native-host query planning.
- The generator uses bounded JSON paths and scalar values.
- New BigInt writes, comparisons, and index constants outside SQLite's signed 64-bit range fail with a specific error. Legacy stored values still decode.
Implementation trailhead#
- Query IR records an explicit source alias without changing the property path.
- SQLite persistence compiles canonical expressions, validates new BigInts, and replaces stale indexes.
- Remote subset projection preserves qualified references across coordinator routing.
- The Node oracle captures production SQL and compares rows and query plans separately.
- Oracle documentation records the campaign, replay command, and proof limits.
Verification#
@tanstack/db: 6,385 tests passed before the review follow-up. The follow-up did not change this package.- SQLite persistence core: 516 tests passed. Two existing tests remain marked as todo.
- Node SQLite persistence: 135 tests passed.
- Browser coordinator: 204 tests passed.
- Focused query compiler and IR suites: 264 tests passed before the review follow-up.
- Exact seed-and-path replay: one runtime target passed and 48 unrelated runtime tests skipped.
- SQLite persistence-core and Node persistence builds and standalone typechecks passed.
- ESLint reported zero errors and 19 existing warnings. Prettier passed, and the final diff had no whitespace errors.
✅ Checklist#
- I tested this code locally with the commands and campaigns above.
🚀 Release Impact#
- This change affects published code, and I generated a changeset.
- This change is docs/CI/dev-only and does not need a release.
Part of #1659
- 1
More templates
- todos
- offline-first-electron
- @tanstack/db-example-react-next-ssr-e2e
- tanstack-start-example-basic
- @tanstack/db-example-paced-mutations-demo
- tanstack-start-db-projects-example
- @tanstack/db-example-react-start-ssr-e2e
- @tanstack/db-example-react-todo
- @tanstack/db-example-solid-todo
- offline-transactions-react-native
- shopping-list-react-native
@tanstack/angular-db
npm i https://pkg.pr.new/@tanstack/angular-db@1867@tanstack/browser-db-sqlite-persistence
npm i https://pkg.pr.new/@tanstack/browser-db-sqlite-persistence@1867@tanstack/capacitor-db-sqlite-persistence
npm i https://pkg.pr.new/@tanstack/capacitor-db-sqlite-persistence@1867@tanstack/cloudflare-durable-objects-db-sqlite-persistence
npm i https://pkg.pr.new/@tanstack/cloudflare-durable-objects-db-sqlite-persistence@1867@tanstack/db
npm i https://pkg.pr.new/@tanstack/db@1867@tanstack/db-ivm
npm i https://pkg.pr.new/@tanstack/db-ivm@1867@tanstack/db-sqlite-persistence-core
npm i https://pkg.pr.new/@tanstack/db-sqlite-persistence-core@1867@tanstack/electric-db-collection
npm i https://pkg.pr.new/@tanstack/electric-db-collection@1867@tanstack/electron-db-sqlite-persistence
npm i https://pkg.pr.new/@tanstack/electron-db-sqlite-persistence@1867@tanstack/expo-db-sqlite-persistence
npm i https://pkg.pr.new/@tanstack/expo-db-sqlite-persistence@1867@tanstack/node-db-sqlite-persistence
npm i https://pkg.pr.new/@tanstack/node-db-sqlite-persistence@1867@tanstack/offline-transactions
npm i https://pkg.pr.new/@tanstack/offline-transactions@1867@tanstack/powersync-db-collection
npm i https://pkg.pr.new/@tanstack/powersync-db-collection@1867@tanstack/query-db-collection
npm i https://pkg.pr.new/@tanstack/query-db-collection@1867@tanstack/react-db
npm i https://pkg.pr.new/@tanstack/react-db@1867@tanstack/react-native-db-sqlite-persistence
npm i https://pkg.pr.new/@tanstack/react-native-db-sqlite-persistence@1867@tanstack/react-router-with-db
npm i https://pkg.pr.new/@tanstack/react-router-with-db@1867@tanstack/rxdb-db-collection
npm i https://pkg.pr.new/@tanstack/rxdb-db-collection@1867@tanstack/solid-db
npm i https://pkg.pr.new/@tanstack/solid-db@1867@tanstack/svelte-db
npm i https://pkg.pr.new/@tanstack/svelte-db@1867@tanstack/tauri-db-sqlite-persistence
npm i https://pkg.pr.new/@tanstack/tauri-db-sqlite-persistence@1867@tanstack/trailbase-db-collection
npm i https://pkg.pr.new/@tanstack/trailbase-db-collection@1867@tanstack/vue-db
npm i https://pkg.pr.new/@tanstack/vue-db@1867commit:
bf763ebSize Change: +507 B (+0.3%)
Total Size: 170 kB
📦 View Changed
Filename Size Change packages/db/dist/esm/collection/subscription.js8.8 kB +21 B (+0.24%) packages/db/dist/esm/query/builder/ref-proxy.js1.28 kB +42 B (+3.38%) packages/db/dist/esm/query/compiler/evaluators.js2.04 kB +79 B (+4.02%) packages/db/dist/esm/query/compiler/expressions.js603 B +43 B (+7.68%) 🔍 packages/db/dist/esm/query/compiler/group-by.js4.14 kB +14 B (+0.34%) packages/db/dist/esm/query/compiler/lazy-targets.js1.12 kB +13 B (+1.18%) packages/db/dist/esm/query/compiler/select.js1.59 kB +11 B (+0.7%) packages/db/dist/esm/query/ir-stable-identity.js4.18 kB +135 B (+3.34%) packages/db/dist/esm/query/ir.js1.74 kB +149 B (+9.37%) 🔍 compressed-size-action::db-package-sizeℹ️ View Unchanged
Filename Size packages/db/dist/esm/client.js3.66 kB packages/db/dist/esm/collection-options.js236 B packages/db/dist/esm/collection/change-events.js1.44 kB packages/db/dist/esm/collection/changes.js2.4 kB packages/db/dist/esm/collection/cleanup-queue.js794 B packages/db/dist/esm/collection/events.js481 B packages/db/dist/esm/collection/index.js4.36 kB packages/db/dist/esm/collection/indexes.js1.99 kB packages/db/dist/esm/collection/lifecycle.js2.15 kB packages/db/dist/esm/collection/mutations.js2.61 kB packages/db/dist/esm/collection/state.js6.94 kB packages/db/dist/esm/collection/sync.js4.62 kB packages/db/dist/esm/collection/transaction-metadata.js144 B packages/db/dist/esm/deferred.js207 B packages/db/dist/esm/errors.js5.34 kB packages/db/dist/esm/event-emitter.js964 B packages/db/dist/esm/index.js3.82 kB packages/db/dist/esm/indexes/auto-index.js829 B packages/db/dist/esm/indexes/base-index.js1.14 kB packages/db/dist/esm/indexes/basic-index.js2.07 kB packages/db/dist/esm/indexes/btree-index.js2.26 kB packages/db/dist/esm/indexes/index-registry.js820 B packages/db/dist/esm/indexes/reverse-index.js376 B packages/db/dist/esm/live-query-adapter.js318 B packages/db/dist/esm/live-query-observer.js3.69 kB packages/db/dist/esm/live-query-options.js702 B packages/db/dist/esm/live-query-window-controller.js4.36 kB packages/db/dist/esm/local-only.js989 B packages/db/dist/esm/local-storage.js2.17 kB packages/db/dist/esm/optimistic-action.js359 B packages/db/dist/esm/paced-mutations.js496 B packages/db/dist/esm/proxy.js3.32 kB packages/db/dist/esm/query/builder/clone-query.js748 B packages/db/dist/esm/query/builder/functions.js1.47 kB packages/db/dist/esm/query/builder/index.js6.72 kB packages/db/dist/esm/query/builder/query-ir.js116 B packages/db/dist/esm/query/compiler/index.js9.11 kB packages/db/dist/esm/query/compiler/joins.js2.99 kB packages/db/dist/esm/query/compiler/order-by.js1.91 kB packages/db/dist/esm/query/compiler/parent-routes.js319 B packages/db/dist/esm/query/compiler/query-equivalence.js455 B packages/db/dist/esm/query/compiler/route-metadata.js1.24 kB packages/db/dist/esm/query/effect.js5.13 kB packages/db/dist/esm/query/equality-value-identity.js591 B packages/db/dist/esm/query/expression-helpers.js1.43 kB packages/db/dist/esm/query/live-query-collection.js391 B packages/db/dist/esm/query/live/bucket-facade-adapter.js2.73 kB packages/db/dist/esm/query/live/collection-config-builder.js6.97 kB packages/db/dist/esm/query/live/collection-registry.js264 B packages/db/dist/esm/query/live/collection-subscriber.js2.26 kB packages/db/dist/esm/query/live/internal.js145 B packages/db/dist/esm/query/live/materialized-pipeline.js2.32 kB packages/db/dist/esm/query/live/ordered-source-loader.js3.81 kB packages/db/dist/esm/query/live/subset-demand-controller.js1.26 kB packages/db/dist/esm/query/live/utils.js1.14 kB packages/db/dist/esm/query/optimizer.js3.11 kB packages/db/dist/esm/query/query-once.js359 B packages/db/dist/esm/query/runtime-reference-identity.js572 B packages/db/dist/esm/query/subset-dedupe.js497 B packages/db/dist/esm/scheduler.js1.34 kB packages/db/dist/esm/SortedMap.js1.3 kB packages/db/dist/esm/strategies/debounceStrategy.js247 B packages/db/dist/esm/strategies/queueStrategy.js428 B packages/db/dist/esm/strategies/throttleStrategy.js246 B packages/db/dist/esm/sync-persistence.js530 B packages/db/dist/esm/transactions.js3.71 kB packages/db/dist/esm/utils.js1.21 kB packages/db/dist/esm/utils/array-utils.js270 B packages/db/dist/esm/utils/browser-polyfills.js304 B packages/db/dist/esm/utils/btree.js4.51 kB packages/db/dist/esm/utils/callbacks.js174 B packages/db/dist/esm/utils/comparison.js1.49 kB packages/db/dist/esm/utils/cursor.js677 B packages/db/dist/esm/utils/error.js167 B packages/db/dist/esm/utils/get-or-create.js155 B packages/db/dist/esm/utils/index-optimization.js2.42 kB packages/db/dist/esm/utils/type-guards.js230 B packages/db/dist/esm/utils/uuid.js449 B packages/db/dist/esm/virtual-props.js360 B Size Change: 0 B
Total Size: 7.34 kB
compressed-size-action::react-db-package-sizeℹ️ View Unchanged
Filename Size packages/react-db/dist/esm/DbProvider.js317 B packages/react-db/dist/esm/HydrationBoundary.js263 B packages/react-db/dist/esm/index.js330 B packages/react-db/dist/esm/live-query-internals.js282 B packages/react-db/dist/esm/useLiveInfiniteQuery.js1.9 kB packages/react-db/dist/esm/useLiveQuery.js2.68 kB packages/react-db/dist/esm/useLiveQueryEffect.js355 B packages/react-db/dist/esm/useLiveSuspenseQuery.js812 B packages/react-db/dist/esm/usePacedMutations.js401 B KyleAMathews(GitHub profile, opens in a new tab) closed this
Sign in to join this conversation.
Navigate logical layers of code changes, visualize relationships, and explore their blast radius.
Note
Reviews paused#
It looks like this branch is under active development. To avoid overwhelming you with review comments due to an influx of new commits, CodeRabbit has automatically paused this review. You can configure this behavior by changing the
reviews.auto_review.auto_pause_after_reviewed_commitssetting.Use the following commands to manage reviews:
@coderabbitai resumeto resume automatic reviews.@coderabbitai reviewto trigger a single review.Use the checkboxes below for quick actions:
No actionable comments were generated in the recent review. 🎉
ℹ️ Recent review info
⚙️ Run configuration
Configuration used: defaults
Review profile: CHILL
Plan: Advanced
Run ID:
899a38a8-117f-4c84-860d-123e7a0c53a6📥 Commits
Reviewing files that changed from the base of the PR and between 6db8fe7535c3f4a686e7172c17093f22535a4927 and 28bbddd1c9154fd62a5553664372209755469782.
📒 Files selected for processing (1)
packages/node-db-sqlite-persistence/tests/expression-index-oracle.test.ts🚧 Files skipped from review as they are similar to previous changes (1)
Included review availability: Your plan provides up to 8 included reviews per hour; 2 remain after this review.
📝 Walkthrough
Walkthrough#
compileRefExpressionSqlnow emits JSON paths as SQL literals so runtime predicates match persisted SQLite expression indexes. A property-based oracle verifies query results and index usage throughEXPLAIN QUERY PLAN.Changes#
SQLite expression-index planning
packages/db-sqlite-persistence-core/src/sqlite-core-adapter.ts,.changeset/fix-sqlite-expression-index-planning.mdcompileRefExpressionSqlemits validated JSON paths as SQL literals and removes the four path bindings. A patch changeset records the fix.packages/node-db-sqlite-persistence/tests/expression-index-oracle.test.ts,packages/node-db-sqlite-persistence/package.json,docs/contributing/oracle-coverage.mdEXPLAIN QUERY PLAN, and covers quoted identifiers, rejected DDL bindings, and prefix collisions. A package script and coverage documentation expose the oracle.Priority: ➖ Normal
Estimated code review effort: 3 (Moderate) | ~25 minutes
Change: Bug fix
Merge Risk: ⚪ Minimal · up to
28bbdThe adapter emits canonical SQL literals for persisted JSON reference paths while comparison values remain bound, and the reported validation passes with no actionable merge-blocking risk evidenced.
🚥 Pre-merge checks | ✅ 4 | ❌ 1
❌ Failed checks (1 warning)#
✅ Passed checks (4 passed)
✨ Finishing Touches 💡 2
📝 Generate docstrings 💡
🧪 Generate unit tests (beta)
🛠️ Fix failing CI checks 💡
Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.
❤️ Share