Skip to content
Source
Sign inSign up
TanStack/db

Add optional compareOptions to the Collection Config which act as col… #762

Mergedkevin-dp(GitHub profile, opens in a new tab)merged 9 commits intoquery-driven-sync fromrefs/pull/762/headmerged by kevin-dp(GitHub profile, opens in a new tab)

This PR adds support for collection-wide string ordering/locale configuration. It is possible to override these in a specific query by providing specific compareOptions in the query's orderBy clause. The priority is as follows (high to low):

  • query provided compareOptions
  • collection's compareOptions if none specified in query
  • default compare options (machine's locale) if the user didn't pass compareOptions when creating the collection

The idea behind providing collection-wide locale configuration is that most applications should use the same locale configuration as their backend. This is needed because tanstack DB does a mix of loading ordered data from the backend and loading ordered data that is available locally. Hence, the local ordering must be the same as the one on the backend otherwise we get inconsistent behavior.

Implementation

High-level

  • We store the collection-wide StringSortOpts in the collection itself. If none are provided, we store the default locale configuration.
  • Modified the builder such that it no longer hydrates the orderBy clause with defaults for the locale, if no locale is specified, we keep them undefined in the orderBy clause
  • At runtime, when processing an ordered query, we build the locale configuration from the orderBy clause's config and the collection's config. If the orderBy clause specifies locale configuration, we use that one. Otherwise, we use the locale configuration from the collection.

Details

  • Indexes are sensitive to locale options (i.e. different locale options require different indexes).
  • We had to change all index creations and index lookups such that they keep into account the locale configuration.
    • When optimising ordered queries, we need to build a locale configuration (take the locale config from the orderBy clause and if not provided then use the locale config from the collection) and pass that config when we create and/or lookup indexes
    • Other places in the code were passing the default locale configuration when creating/looking up indexes. We had to change this to pass the collection's locale configuration instead
    1. 🦋 Changeset detected

      Latest commit: 464c74c60e5849c3e1dca86c45fd539b01605de7

      The changes in this PR will be included in the next version bump.

      This PR includes changesets to release 16 packages
      NameType
      @tanstack/dbPatch
      @tanstack/angular-dbPatch
      @tanstack/electric-db-collectionPatch
      @tanstack/offline-transactionsPatch
      @tanstack/powersync-db-collectionPatch
      @tanstack/query-db-collectionPatch
      @tanstack/react-dbPatch
      @tanstack/rxdb-db-collectionPatch
      @tanstack/solid-dbPatch
      @tanstack/svelte-dbPatch
      @tanstack/trailbase-db-collectionPatch
      @tanstack/vue-dbPatch
      todosPatch
      @tanstack/db-example-paced-mutations-demoPatch
      @tanstack/db-example-react-todoPatch
      @tanstack/db-example-solid-todoPatch

      Not sure what this means? Click here to learn what changesets are.

      Click here if you're a maintainer who wants to add another changeset to this PR

    2. More templates

      @tanstack/angular-db

      npm i https://pkg.pr.new/@tanstack/angular-db@762

      @tanstack/db

      npm i https://pkg.pr.new/@tanstack/db@762

      @tanstack/db-ivm

      npm i https://pkg.pr.new/@tanstack/db-ivm@762

      @tanstack/electric-db-collection

      npm i https://pkg.pr.new/@tanstack/electric-db-collection@762

      @tanstack/offline-transactions

      npm i https://pkg.pr.new/@tanstack/offline-transactions@762

      @tanstack/powersync-db-collection

      npm i https://pkg.pr.new/@tanstack/powersync-db-collection@762

      @tanstack/query-db-collection

      npm i https://pkg.pr.new/@tanstack/query-db-collection@762

      @tanstack/react-db

      npm i https://pkg.pr.new/@tanstack/react-db@762

      @tanstack/rxdb-db-collection

      npm i https://pkg.pr.new/@tanstack/rxdb-db-collection@762

      @tanstack/solid-db

      npm i https://pkg.pr.new/@tanstack/solid-db@762

      @tanstack/svelte-db

      npm i https://pkg.pr.new/@tanstack/svelte-db@762

      @tanstack/trailbase-db-collection

      npm i https://pkg.pr.new/@tanstack/trailbase-db-collection@762

      @tanstack/vue-db

      npm i https://pkg.pr.new/@tanstack/vue-db@762

      commit: 464c74c

    3. Size Change: +346 B (+0.42%)

      Total Size: 83.3 kB

      FilenameSizeChange
      ./packages/db/dist/esm/collection/change-events.js1.36 kB+22 B (+1.64%)
      ./packages/db/dist/esm/collection/index.js3.25 kB+121 B (+3.87%)
      ./packages/db/dist/esm/indexes/auto-index.js742 B+11 B (+1.5%)
      ./packages/db/dist/esm/query/builder/index.js3.84 kB-15 B (-0.39%)
      ./packages/db/dist/esm/query/compiler/order-by.js1.25 kB+79 B (+6.75%)🔍
      ./packages/db/dist/esm/query/live/collection-config-builder.js5.11 kB+108 B (+2.16%)
      ./packages/db/dist/esm/utils/index-optimization.js1.51 kB+20 B (+1.34%)
      ℹ️ View Unchanged
      FilenameSize
      ./packages/db/dist/esm/collection/changes.js977 B
      ./packages/db/dist/esm/collection/events.js388 B
      ./packages/db/dist/esm/collection/indexes.js1.1 kB
      ./packages/db/dist/esm/collection/lifecycle.js1.67 kB
      ./packages/db/dist/esm/collection/mutations.js2.26 kB
      ./packages/db/dist/esm/collection/state.js3.43 kB
      ./packages/db/dist/esm/collection/subscription.js2.42 kB
      ./packages/db/dist/esm/collection/sync.js2.02 kB
      ./packages/db/dist/esm/deferred.js207 B
      ./packages/db/dist/esm/errors.js3.9 kB
      ./packages/db/dist/esm/event-emitter.js748 B
      ./packages/db/dist/esm/index.js2.49 kB
      ./packages/db/dist/esm/indexes/base-index.js766 B
      ./packages/db/dist/esm/indexes/btree-index.js1.87 kB
      ./packages/db/dist/esm/indexes/lazy-index.js1.1 kB
      ./packages/db/dist/esm/indexes/reverse-index.js513 B
      ./packages/db/dist/esm/local-only.js837 B
      ./packages/db/dist/esm/local-storage.js2.04 kB
      ./packages/db/dist/esm/optimistic-action.js359 B
      ./packages/db/dist/esm/paced-mutations.js496 B
      ./packages/db/dist/esm/proxy.js3.22 kB
      ./packages/db/dist/esm/query/builder/functions.js606 B
      ./packages/db/dist/esm/query/builder/ref-proxy.js917 B
      ./packages/db/dist/esm/query/compiler/evaluators.js1.26 kB
      ./packages/db/dist/esm/query/compiler/expressions.js674 B
      ./packages/db/dist/esm/query/compiler/group-by.js1.78 kB
      ./packages/db/dist/esm/query/compiler/index.js1.95 kB
      ./packages/db/dist/esm/query/compiler/joins.js2 kB
      ./packages/db/dist/esm/query/compiler/select.js1.07 kB
      ./packages/db/dist/esm/query/ir.js673 B
      ./packages/db/dist/esm/query/live-query-collection.js360 B
      ./packages/db/dist/esm/query/live/collection-registry.js214 B
      ./packages/db/dist/esm/query/live/collection-subscriber.js1.77 kB
      ./packages/db/dist/esm/query/optimizer.js2.6 kB
      ./packages/db/dist/esm/query/predicate-utils.js2.88 kB
      ./packages/db/dist/esm/query/subset-dedupe.js921 B
      ./packages/db/dist/esm/scheduler.js1.21 kB
      ./packages/db/dist/esm/SortedMap.js1.18 kB
      ./packages/db/dist/esm/strategies/debounceStrategy.js237 B
      ./packages/db/dist/esm/strategies/queueStrategy.js418 B
      ./packages/db/dist/esm/strategies/throttleStrategy.js236 B
      ./packages/db/dist/esm/transactions.js2.9 kB
      ./packages/db/dist/esm/utils.js881 B
      ./packages/db/dist/esm/utils/browser-polyfills.js304 B
      ./packages/db/dist/esm/utils/btree.js5.61 kB
      ./packages/db/dist/esm/utils/comparison.js660 B
      ./packages/db/dist/esm/utils/type-guards.js157 B
      compressed-size-action::db-package-size
    4. Size Change: 0 B

      Total Size: 3.34 kB

      ℹ️ View Unchanged
      FilenameSize
      ./packages/react-db/dist/esm/index.js225 B
      ./packages/react-db/dist/esm/useLiveInfiniteQuery.js1.17 kB
      ./packages/react-db/dist/esm/useLiveQuery.js1.11 kB
      ./packages/react-db/dist/esm/useLiveSuspenseQuery.js431 B
      ./packages/react-db/dist/esm/usePacedMutations.js401 B
      compressed-size-action::react-db-package-size
    5. packages/db/src/types.ts lines 616 to 622Outdated
      @@ -582,3 +613,10 @@
      582613 */
      583614 onDelete?: DeleteMutationFn<T, TKey, TUtils, TReturn>
      584615
      616+ /**
      617+ * Specifies how to compare data in the collection.
      618+ * This should be configured to match data ordering on the backend.
      619+ * E.g., when using the Electric DB collection these options
      620+ * should match the database's collation settings.
      621+ */
      622+ compareOptions?: StringSortOpts

      @KyleAMathews this is a new collection config option, you may have some to say on naming.

      It's a bit ambiguous at the collection level what is compared — defaultCollation or defaultStringCollation is more clear.

      Ok, went for defaultStringCollation and renamed StringSortOpts type to StringCollationConfig (cc @KyleAMathews @samwillis)

    6. @kevin-dp as discussed we need to traverse the from in live queries to determine the compare options for the live query. Also remember to pull as I rebased!

    7. KyleAMathews(GitHub profile, opens in a new tab)replied on packages/db/src/types.ts lines 616 to 622

      It's a bit ambiguous at the collection level what is compared — defaultCollation or defaultStringCollation is more clear.

    8. kevin-dp(GitHub profile, opens in a new tab)replied on packages/db/src/types.ts lines 616 to 622

      Ok, went for defaultStringCollation and renamed StringSortOpts type to StringCollationConfig (cc @KyleAMathews @samwillis)

Sign in to join this conversation.