Skip to content
Source
Sign inSign up
TanStack/db

Virtual props implementation #1213

Mergedsamwillis(GitHub profile, opens in a new tab)merged 97 commits intomain fromrefs/pull/1213/headmerged by samwillis(GitHub profile, opens in a new tab)

🎯 Changes

This PR implements the virtual properties system as described in the RFC. This introduces computed, read-only metadata fields prefixed with $ to every row, providing information about its sync status, origin, key, and source collection ID.

Key aspects of this implementation:

  • $synced: Indicates whether a row reflects confirmed state from the backend (true) or has pending optimistic mutations (false).
  • $origin: Specifies if the last confirmed change originated from the local client ('local') or was received via sync ('remote').
  • $key: The primary identifier of the row.
  • $collectionId: The ID of the collection the row originated from.
  • Pass-through semantics: Virtual properties are preserved and passed through in live queries and nested collections using an "add-if-missing" pattern.
  • Queryability: Virtual properties are available for use in query builder expressions (e.g., where(({ item }) => eq(item.$synced, true))).
  • Type Integration: Updated RefProxy and Ref types to include virtual properties, ensuring type safety and discoverability.

This feature enhances data visibility and enables more sophisticated UI logic based on the state and origin of data.

✅ Checklist

  • I have followed the steps in the Contributing guide.
  • I have tested this code locally with pnpm test:pr.

🚀 Release Impact

  • This change affects published code, and I have generated a changeset.
  • This change is docs/CI/dev-only (no release).

Open in Cursor Open in Web
    1. Cursor Agent can help with this pull request. Just @cursor in comments and I'll start working on changes in this branch. Learn more about Cursor Agents

    2. 🦋 Changeset detected

      Latest commit: 9e7dfb65a8c902f9edcec1998c58ae350efe63b7

      The changes in this PR will be included in the next version bump.

      This PR includes changesets to release 12 packages
      NameType
      @tanstack/dbPatch
      @tanstack/angular-dbPatch
      @tanstack/electric-db-collectionPatch
      @tanstack/offline-transactionsPatch
      @tanstack/powersync-db-collectionPatch
      @tanstack/query-db-collectionPatch
      @tanstack/react-dbPatch
      @tanstack/rxdb-db-collectionPatch
      @tanstack/solid-dbPatch
      @tanstack/svelte-dbPatch
      @tanstack/trailbase-db-collectionPatch
      @tanstack/vue-dbPatch

      Not sure what this means? Click here to learn what changesets are.

      Click here if you're a maintainer who wants to add another changeset to this PR

    3. More templates

      @tanstack/angular-db

      npm i https://pkg.pr.new/@tanstack/angular-db@1213

      @tanstack/db

      npm i https://pkg.pr.new/@tanstack/db@1213

      @tanstack/db-ivm

      npm i https://pkg.pr.new/@tanstack/db-ivm@1213

      @tanstack/electric-db-collection

      npm i https://pkg.pr.new/@tanstack/electric-db-collection@1213

      @tanstack/offline-transactions

      npm i https://pkg.pr.new/@tanstack/offline-transactions@1213

      @tanstack/powersync-db-collection

      npm i https://pkg.pr.new/@tanstack/powersync-db-collection@1213

      @tanstack/query-db-collection

      npm i https://pkg.pr.new/@tanstack/query-db-collection@1213

      @tanstack/react-db

      npm i https://pkg.pr.new/@tanstack/react-db@1213

      @tanstack/rxdb-db-collection

      npm i https://pkg.pr.new/@tanstack/rxdb-db-collection@1213

      @tanstack/solid-db

      npm i https://pkg.pr.new/@tanstack/solid-db@1213

      @tanstack/svelte-db

      npm i https://pkg.pr.new/@tanstack/svelte-db@1213

      @tanstack/trailbase-db-collection

      npm i https://pkg.pr.new/@tanstack/trailbase-db-collection@1213

      @tanstack/vue-db

      npm i https://pkg.pr.new/@tanstack/vue-db@1213

      commit: 9e7dfb6

    4. Size Change: +3.33 kB (+3.2%)

      Total Size: 107 kB

      FilenameSizeChange
      ./packages/db/dist/esm/collection/changes.js1.38 kB+157 B (+12.83%)⚠️
      ./packages/db/dist/esm/collection/index.js3.45 kB+126 B (+3.79%)
      ./packages/db/dist/esm/collection/mutations.js2.47 kB+135 B (+5.77%)🔍
      ./packages/db/dist/esm/collection/state.js5.2 kB+1.72 kB (+49.23%)🚨
      ./packages/db/dist/esm/collection/sync.js2.43 kB+24 B (+1%)
      ./packages/db/dist/esm/collection/transaction-metadata.js144 B+144 B (new file)🆕
      ./packages/db/dist/esm/index.js2.78 kB+29 B (+1.05%)
      ./packages/db/dist/esm/indexes/auto-index.js777 B+35 B (+4.72%)🔍
      ./packages/db/dist/esm/local-only.js890 B+82 B (+10.15%)⚠️
      ./packages/db/dist/esm/query/compiler/group-by.js2.69 kB+339 B (+14.41%)⚠️
      ./packages/db/dist/esm/query/compiler/index.js3.62 kB+122 B (+3.49%)
      ./packages/db/dist/esm/query/live/collection-config-builder.js7.63 kB+28 B (+0.37%)
      ./packages/db/dist/esm/utils/index-optimization.js1.54 kB+36 B (+2.39%)
      ./packages/db/dist/esm/virtual-props.js360 B+360 B (new file)🆕
      ℹ️ View Unchanged
      FilenameSize
      ./packages/db/dist/esm/collection/change-events.js1.39 kB
      ./packages/db/dist/esm/collection/events.js388 B
      ./packages/db/dist/esm/collection/indexes.js1.1 kB
      ./packages/db/dist/esm/collection/lifecycle.js1.75 kB
      ./packages/db/dist/esm/collection/subscription.js3.71 kB
      ./packages/db/dist/esm/deferred.js207 B
      ./packages/db/dist/esm/errors.js4.83 kB
      ./packages/db/dist/esm/event-emitter.js748 B
      ./packages/db/dist/esm/indexes/base-index.js766 B
      ./packages/db/dist/esm/indexes/btree-index.js2.17 kB
      ./packages/db/dist/esm/indexes/lazy-index.js1.1 kB
      ./packages/db/dist/esm/indexes/reverse-index.js538 B
      ./packages/db/dist/esm/local-storage.js2.1 kB
      ./packages/db/dist/esm/optimistic-action.js359 B
      ./packages/db/dist/esm/paced-mutations.js496 B
      ./packages/db/dist/esm/proxy.js3.75 kB
      ./packages/db/dist/esm/query/builder/functions.js792 B
      ./packages/db/dist/esm/query/builder/index.js5.15 kB
      ./packages/db/dist/esm/query/builder/ref-proxy.js1.05 kB
      ./packages/db/dist/esm/query/compiler/evaluators.js1.43 kB
      ./packages/db/dist/esm/query/compiler/expressions.js430 B
      ./packages/db/dist/esm/query/compiler/joins.js2.11 kB
      ./packages/db/dist/esm/query/compiler/order-by.js1.5 kB
      ./packages/db/dist/esm/query/compiler/select.js1.11 kB
      ./packages/db/dist/esm/query/effect.js4.78 kB
      ./packages/db/dist/esm/query/expression-helpers.js1.43 kB
      ./packages/db/dist/esm/query/ir.js784 B
      ./packages/db/dist/esm/query/live-query-collection.js360 B
      ./packages/db/dist/esm/query/live/collection-registry.js264 B
      ./packages/db/dist/esm/query/live/collection-subscriber.js1.94 kB
      ./packages/db/dist/esm/query/live/internal.js145 B
      ./packages/db/dist/esm/query/live/utils.js1.57 kB
      ./packages/db/dist/esm/query/optimizer.js2.62 kB
      ./packages/db/dist/esm/query/predicate-utils.js2.97 kB
      ./packages/db/dist/esm/query/query-once.js359 B
      ./packages/db/dist/esm/query/subset-dedupe.js960 B
      ./packages/db/dist/esm/scheduler.js1.3 kB
      ./packages/db/dist/esm/SortedMap.js1.3 kB
      ./packages/db/dist/esm/strategies/debounceStrategy.js247 B
      ./packages/db/dist/esm/strategies/queueStrategy.js428 B
      ./packages/db/dist/esm/strategies/throttleStrategy.js246 B
      ./packages/db/dist/esm/transactions.js2.9 kB
      ./packages/db/dist/esm/utils.js927 B
      ./packages/db/dist/esm/utils/browser-polyfills.js304 B
      ./packages/db/dist/esm/utils/btree.js5.61 kB
      ./packages/db/dist/esm/utils/comparison.js1.05 kB
      ./packages/db/dist/esm/utils/cursor.js457 B
      ./packages/db/dist/esm/utils/type-guards.js157 B
      compressed-size-action::db-package-size
    5. Size Change: 0 B

      Total Size: 4.23 kB

      ℹ️ View Unchanged
      FilenameSize
      ./packages/react-db/dist/esm/index.js249 B
      ./packages/react-db/dist/esm/useLiveInfiniteQuery.js1.32 kB
      ./packages/react-db/dist/esm/useLiveQuery.js1.34 kB
      ./packages/react-db/dist/esm/useLiveQueryEffect.js355 B
      ./packages/react-db/dist/esm/useLiveSuspenseQuery.js559 B
      ./packages/react-db/dist/esm/usePacedMutations.js401 B
      compressed-size-action::react-db-package-size
    6. samwillis(GitHub profile, opens in a new tab) changed the title

    7. kevin-dp(GitHub profile, opens in a new tab) requested changes

      We're adding a bunch of meta properties $key, $origin, etc. Each of these could conflict with user properties. I'd like to minimize potential conflicts by having a single $meta property and having all these props nested under it, so $meta.synced, $meta.origin, etc.


      I reviewed this big PR with Claude. Here's our feedback:

      Semantic concerns with virtual props in joins and aggregations

      Nice work on the virtual props system — the core idea of $synced, $origin, $key, and $collectionId is really useful for single-collection queries. However, I think the "pass-through / add-if-missing" semantics break down in two important cases.

      Joins: $collectionId becomes ambiguous

      After a join, each result row is a composite of two (or more) collections:

      q.from({ todo: todoCollection })
        .join({ list: listCollection }, ({ todo, list }) => eq(list.id, todo.list_id), "inner")
        .select(({ todo, list }) => ({
          id: todo.id,
          text: todo.text,
          listName: list.name,
        }))

      A single $collectionId string can't represent this — it will just carry forward whichever value was set first (likely the from source), silently losing provenance from the joined side. If consumers use $collectionId to route mutations back to the right collection, this is actively misleading rather than just incomplete.

      Options:

      • Make $collectionId an array or a map keyed by alias after a join (e.g. { todo: "todo-collection-id", list: "list-collection-id" })
      • Strip/nullify $collectionId on join results since it's now ambiguous
      • Keep it but rename it to make clear it only refers to the primary source

      Aggregations: $synced, $origin, and $key lose their meaning

      After a groupBy + aggregate, each result row represents a group of source rows, not a single entity:

      q.from({ user: usersCollection })
        .groupBy(({ user }) => user.departmentId)
        .select(({ user }) => ({
          departmentId: user.departmentId,
          userCount: count(user.id),
          avgAge: avg(user.age),
        }))

      For this output:

      • $key: The aggregate row doesn't correspond to any single source row. Whatever key is passed through is arbitrary and meaningless.
      • $synced: A group may contain a mix of synced and unsynced rows. Is the aggregate true only if all rows in the group are synced? Or is it just pass-through from an arbitrary row? The difference matters hugely for UI that conditionally shows "saving..." indicators.
      • $origin: Same problem — the group may mix 'local' and 'remote' rows.

      The relax join and group-by type assertions commit and the +20% size bump in group-by.js suggest the implementation is patching around these issues by loosening types and carrying props through regardless, rather than addressing the semantic mismatch.

      Suggestion

      The pass-through pattern works well for operations that don't change row identity (where, orderBy, limit, select without aggregation). The problem is specifically joins (which merge identities) and groupBy (which destroys individual identity).

      I'd suggest one of:

      1. Strip or nullify virtual props after operations that break their semantics. Aggregation strips $key / $synced / $origin; joins make $collectionId an array or strip it. Reflect this in the types.

      2. Make virtual props conditional at the type level — a from().where() result gets the full { $synced, $origin, $key, $collectionId }, but a groupBy() result gets a narrower type that omits $key / $synced / $origin. More work, but more honest.

      3. At minimum, derive aggregate-level $synced meaningfully — e.g. $synced = false if any row in the group is unsynced. This would actually be useful ("some data in this group is still pending"). But $key and $origin still don't have good aggregate semantics and should probably be omitted.

      OutdatedResolvedby samwillis(GitHub profile, opens in a new tab)
      OutdatedResolvedby samwillis(GitHub profile, opens in a new tab)
    8. @kevin-dp

      The aggregation concern is already addressed in the current implementation. We derive aggregate-level virtual props intentionally:

      • $synced is false if any row in the group is unsynced
      • $origin is local if any contributing row is local
      • $key is a derived group key rather than a source-row key

      On joins, I think the ambiguity is narrower than described. After a join, the virtual props remain attached per alias: todo.$collectionId, list.$collectionId. The place where ambiguity does exist is flattened or spread select output, where top-level merges can overwrite one source's virtual props with another's, but I would see that as a users concern when projecting in the select.

      Removed the object copes you flagged.

    9. kevin-dp(GitHub profile, opens in a new tab) approved these changes

    10. Code Review: Virtual Props Implementation

      Nice architecture overall — the "add-if-missing" enrichment pattern, WeakMap caching, and aggregate semantics (all-synced / any-local) are well-designed. A few issues worth addressing before merge:


      Bug: Truncate doesn't clear origin tracking maps

      packages/db/src/collection/state.ts ~line 839

      The truncate phase in commitPendingTransactions clears syncedData, syncedMetadata, and syncedKeys, but not rowOrigins, pendingLocalChanges, or pendingLocalOrigins. After truncate, if the server re-inserts the same keys, getRowOrigin(key) returns the stale value from rowOrigins instead of defaulting to 'remote'.

      The full cleanup() method at line 1257 does clear these maps, but truncate-during-sync doesn't.

      // After line 841 in the truncate block:
      this.rowOrigins.clear()
      this.pendingLocalChanges.clear()
      this.pendingLocalOrigins.clear()

      Bug: pendingLocalOrigins not cleaned up on transaction failure

      packages/db/src/collection/mutations.ts + state.ts

      markPendingLocalOrigins() is called before commit(), but if the transaction fails, the .catch(() => undefined) on the commit promise doesn't clean up pendingLocalOrigins. The recomputeOptimisticState failed branch clears pendingOptimistic* maps but doesn't touch pendingLocalOrigins for non-optimistic mutations. Result: $origin stays 'local' for affected keys indefinitely after a failed mutation.

      Similarly, scheduleTransactionCleanup's empty catch block (line 1198) keeps the transaction for reference but never cleans up origin tracking state.


      Minor: previousValue virtual props in optimistic path use post-change state

      packages/db/src/collection/state.ts — enrichChangeMessage (line 215)

      When enriching previousValue, it calls enrichWithVirtualProps(change.previousValue, change.key) which reads current isRowSynced/getRowOrigin — the post-mutation state. The sync commit path handles this correctly with getPreviousVirtualProps, but the optimistic recompute path at line 533 (collectOptimisticChanges → emitEvents → enrichChangeMessage) doesn't capture pre-change virtual props.

      Impact is limited to previousValue.$synced / previousValue.$origin in change events during optimistic transitions.


      Cleanup: Dead VirtualRefProps type

      packages/db/src/virtual-props.ts:105 — VirtualRefProps is exported but never imported anywhere. The actual ref typing is handled by VirtualPropsRef in types.ts and VirtualPropsRefProxy in ref-proxy.ts. Should be removed.


      Suggestion: Derive types from single source

      WithoutVirtualProps hardcodes the four property names as string literals rather than using Omit<T, keyof VirtualRowProps>. Similarly, VirtualPropsRef and VirtualPropsRefProxy duplicate the same four properties independently. If a fifth virtual prop is added, three files need manual updates. Consider deriving from VirtualRowProps:

      // WithoutVirtualProps
      export type WithoutVirtualProps<T> = Omit<T, keyof VirtualRowProps>
      
      // Ref types
      type VirtualRefShape<TKey extends string | number = string | number> = {
        readonly [K in keyof VirtualRowProps<TKey>]: RefLeaf<VirtualRowProps<TKey>[K]>
      }

      Suggestion: hasVirtualProps should check all 4 properties

      packages/db/src/virtual-props.ts:161 — Currently only checks for $synced and $origin, but the type guard asserts VirtualRowProps which also requires $key and $collectionId. Code relying on this guard could access $key on an object that doesn't have it.

Sign in to join this conversation.