Skip to content
Source
Sign inSign up
TanStack/db

Handle pushed down predicates in Electric collection #618

Mergedkevin-dp(GitHub profile, opens in a new tab)merged 5 commits intoquery-driven-sync fromrefs/pull/618/headmerged by samwillis(GitHub profile, opens in a new tab)

stacked on #669


This PR is a follow up on https://github.com/TanStack/db/pull/617 and modifies the Electric collection to handle predicates that are being pushed down to the Electric collection.

    1. 🦋 Changeset detected

      Latest commit: 80284c2b983261e11439b311f00d558f6c7f9ec1

      The changes in this PR will be included in the next version bump.

      This PR includes changesets to release 2 packages
      NameType
      @tanstack/electric-db-collectionPatch
      @tanstack/db-example-react-todoPatch

      Not sure what this means? Click here to learn what changesets are.

      Click here if you're a maintainer who wants to add another changeset to this PR

    2. More templates

      @tanstack/angular-db

      npm i https://pkg.pr.new/@tanstack/angular-db@618

      @tanstack/db

      npm i https://pkg.pr.new/@tanstack/db@618

      @tanstack/db-ivm

      npm i https://pkg.pr.new/@tanstack/db-ivm@618

      @tanstack/electric-db-collection

      npm i https://pkg.pr.new/@tanstack/electric-db-collection@618

      @tanstack/offline-transactions

      npm i https://pkg.pr.new/@tanstack/offline-transactions@618

      @tanstack/powersync-db-collection

      npm i https://pkg.pr.new/@tanstack/powersync-db-collection@618

      @tanstack/query-db-collection

      npm i https://pkg.pr.new/@tanstack/query-db-collection@618

      @tanstack/react-db

      npm i https://pkg.pr.new/@tanstack/react-db@618

      @tanstack/rxdb-db-collection

      npm i https://pkg.pr.new/@tanstack/rxdb-db-collection@618

      @tanstack/solid-db

      npm i https://pkg.pr.new/@tanstack/solid-db@618

      @tanstack/svelte-db

      npm i https://pkg.pr.new/@tanstack/svelte-db@618

      @tanstack/trailbase-db-collection

      npm i https://pkg.pr.new/@tanstack/trailbase-db-collection@618

      @tanstack/vue-db

      npm i https://pkg.pr.new/@tanstack/vue-db@618

      commit: 80284c2

    3. Size Change: 0 B

      Total Size: 83 kB

      ℹ️ View Unchanged
      FilenameSize
      ./packages/db/dist/esm/collection/change-events.js1.34 kB
      ./packages/db/dist/esm/collection/changes.js977 B
      ./packages/db/dist/esm/collection/events.js388 B
      ./packages/db/dist/esm/collection/index.js3.12 kB
      ./packages/db/dist/esm/collection/indexes.js1.1 kB
      ./packages/db/dist/esm/collection/lifecycle.js1.67 kB
      ./packages/db/dist/esm/collection/mutations.js2.26 kB
      ./packages/db/dist/esm/collection/state.js3.43 kB
      ./packages/db/dist/esm/collection/subscription.js2.42 kB
      ./packages/db/dist/esm/collection/sync.js2.02 kB
      ./packages/db/dist/esm/deferred.js207 B
      ./packages/db/dist/esm/errors.js3.9 kB
      ./packages/db/dist/esm/event-emitter.js748 B
      ./packages/db/dist/esm/index.js2.49 kB
      ./packages/db/dist/esm/indexes/auto-index.js731 B
      ./packages/db/dist/esm/indexes/base-index.js766 B
      ./packages/db/dist/esm/indexes/btree-index.js1.87 kB
      ./packages/db/dist/esm/indexes/lazy-index.js1.1 kB
      ./packages/db/dist/esm/indexes/reverse-index.js513 B
      ./packages/db/dist/esm/local-only.js837 B
      ./packages/db/dist/esm/local-storage.js2.08 kB
      ./packages/db/dist/esm/optimistic-action.js359 B
      ./packages/db/dist/esm/paced-mutations.js496 B
      ./packages/db/dist/esm/proxy.js3.22 kB
      ./packages/db/dist/esm/query/builder/functions.js606 B
      ./packages/db/dist/esm/query/builder/index.js3.85 kB
      ./packages/db/dist/esm/query/builder/ref-proxy.js917 B
      ./packages/db/dist/esm/query/compiler/evaluators.js1.26 kB
      ./packages/db/dist/esm/query/compiler/expressions.js674 B
      ./packages/db/dist/esm/query/compiler/group-by.js1.78 kB
      ./packages/db/dist/esm/query/compiler/index.js1.95 kB
      ./packages/db/dist/esm/query/compiler/joins.js2 kB
      ./packages/db/dist/esm/query/compiler/order-by.js1.17 kB
      ./packages/db/dist/esm/query/compiler/select.js1.07 kB
      ./packages/db/dist/esm/query/ir.js673 B
      ./packages/db/dist/esm/query/live-query-collection.js360 B
      ./packages/db/dist/esm/query/live/collection-config-builder.js5 kB
      ./packages/db/dist/esm/query/live/collection-registry.js214 B
      ./packages/db/dist/esm/query/live/collection-subscriber.js1.77 kB
      ./packages/db/dist/esm/query/optimizer.js2.6 kB
      ./packages/db/dist/esm/query/predicate-utils.js2.88 kB
      ./packages/db/dist/esm/query/subset-dedupe.js921 B
      ./packages/db/dist/esm/scheduler.js1.21 kB
      ./packages/db/dist/esm/SortedMap.js1.18 kB
      ./packages/db/dist/esm/strategies/debounceStrategy.js237 B
      ./packages/db/dist/esm/strategies/queueStrategy.js418 B
      ./packages/db/dist/esm/strategies/throttleStrategy.js236 B
      ./packages/db/dist/esm/transactions.js2.9 kB
      ./packages/db/dist/esm/utils.js881 B
      ./packages/db/dist/esm/utils/browser-polyfills.js304 B
      ./packages/db/dist/esm/utils/btree.js5.61 kB
      ./packages/db/dist/esm/utils/comparison.js660 B
      ./packages/db/dist/esm/utils/index-optimization.js1.49 kB
      ./packages/db/dist/esm/utils/type-guards.js157 B
      compressed-size-action::db-package-size
    4. Size Change: 0 B

      Total Size: 3.34 kB

      ℹ️ View Unchanged
      FilenameSize
      ./packages/react-db/dist/esm/index.js225 B
      ./packages/react-db/dist/esm/useLiveInfiniteQuery.js1.17 kB
      ./packages/react-db/dist/esm/useLiveQuery.js1.11 kB
      ./packages/react-db/dist/esm/useLiveSuspenseQuery.js431 B
      ./packages/react-db/dist/esm/usePacedMutations.js401 B
      compressed-size-action::react-db-package-size
    5. OutdatedResolvedby kevin-dp(GitHub profile, opens in a new tab)
      OutdatedResolvedby kevin-dp(GitHub profile, opens in a new tab)
    6. kevin-dp(GitHub profile, opens in a new tab)replied on packages/electric-db-collection/src/sql-compiler.ts line 85

      Even if we default to lexical sort, what do we do if the user explicitly chooses a locale compare? Translating that into SQL is non-trivial.

    7. kevin-dp(GitHub profile, opens in a new tab)replied on packages/electric-db-collection/tests/electric-live-query.test.ts lines 840 to 846

      You simulated an initial sync with just 1 user (Alice)

      simulateInitialSync([sampleUsers[0]!])

      Then you make this live query collection:

      createLiveQueryCollection({
        id: `expanding-live-query`,
        startSync: true,
        query: (q) =>
          q
            .from({ user: electricCollection })
            .orderBy(({ user }) => user.age, `asc`)
            .limit(3),
      })

      The live query asks for 3 results, so we request a limited snapshot (limit = 3). requestLimitedSnapshot immediately tries to load this data from the collection, so it loads Alice. Then it also asks the collection to sync more but it doesn't await this (L218 in packages/db/src/collection/subscription.ts). So this is the original snapshot request with limit 3:

      callArgs(0) is {
        params: {},
        orderBy: 'age NULLS FIRST',
        limit: 3
      }

      At the end of the live query run, we notice that the top 3 is not filled since it contains only 1 user. So we request a snapshot for 2 more users:

      callArgs(1) is {
        params: { '1': '25' },
        where: 'age > $1',
        orderBy: 'age NULLS FIRST',
        limit: 2
      }

      We will only receive Bob and Charlie from this snapshot request (whose result is mocked).

      However, notice that we simulated an initial snapshot containing only Alice (she has age 25):

      const sampleUsers: Array<User> = [
        {
          id: 1,
          name: `Alice`,
          age: 25,
          email: `alice@example.com`,
          active: true,
        },
        {
          id: 2,
          name: `Bob`,
          age: 19,
          email: `bob@example.com`,
          active: true,
        },
        {
          id: 3,
          name: `Charlie`,
          age: 30,
          email: `charlie@example.com`,
          active: false,
        },
        {
          id: 4,
          name: `Dave`,
          age: 22,
          email: `dave@example.com`,
          active: true,
        },
      ]

      So, since we did a full initial sync, we believe we saw everything up to Alice (age 25) and so the 2nd snapshot request includes where age > 25.

      So the logic of this test is a bit flawed. In practice, we should either have synced the user with the lowest age during the initial sync and then the snapshot request should return the next 2 users. Or, we can still sync only Alice during initial sync but then we need to mimick live updates coming in for any users with age smaller than Alice's age.

    8. samwillis(GitHub profile, opens in a new tab)replied on packages/electric-db-collection/src/sql-compiler.ts line 85

      Options:

      1. not push down orderBy when it's using not postgres compatible ordering
      2. try and convert to something postgres compatible where possible
      3. throw and error or warning that its using the wrong ordering

      or some combination of above

    9. "syncs data synced" --> need to remove "syncs" or "synced"

      This is the only place in this docstring where we mention shapes and it is a bit confusing that we mention shape here and not in the other modes. I would remove it and say "syncs all data in the background"

      I'm not sure what is meant by "initial sync" here. The previous bullet says:

      uses incremental snapshots during the initial sync to provide a fast path to the data required for queries

      The part in bolt makes me think that my queries will use the data immediately? Since on-demand marks as ready after the first snapshot is synced, i'm expecting this to mark it as ready immediately (even before first snapshit sync)? But then this sentence now says:

      collection will be marked as ready once the initial sync is complete

      What does this mean? What is the initial sync in this mode? Is it the first snapshot sync, all snapshot syncs, ...?

      nit: missing some words between "layer" and "can". Perhaps:

      this indicates to the sync layer that it can load more data on demand ...

      OutdatedResolvedby samwillis(GitHub profile, opens in a new tab)
      Resolvedby samwillis(GitHub profile, opens in a new tab)
      Resolvedby samwillis(GitHub profile, opens in a new tab)
      OutdatedResolvedby samwillis(GitHub profile, opens in a new tab)
    10. samwillis(GitHub profile, opens in a new tab)replied on packages/electric-db-collection/tests/electric-live-query.test.ts lines 840 to 846

      @kevin-dp and I chatted about this, it is doing what we would expect at this stage as there is no deduplication of snapshot requests yet. Once we implement that there will not be this additional request.

    11. Will change to "full sync"

      As of #658 live queries run eagerly while data is loading, so we still don't mark the collection as "ready" till its fully synced, but it can use snapshots during that time to fulfil queries. Users will have to handle the fast the isReady=false themselves on the queries. We may yet want to iterate on the status of live queries in this case.

    12. samwillis(GitHub profile, opens in a new tab)replied on packages/electric-db-collection/tests/electric-live-query.test.ts line 501

      This spread enables us to add the up-to-date at the end of the message batch.

    13. OutdatedResolvedby samwillis(GitHub profile, opens in a new tab)
      OutdatedResolvedby samwillis(GitHub profile, opens in a new tab)
    14. kevin-dp(GitHub profile, opens in a new tab)replied on packages/electric-db-collection/src/sql-compiler.ts line 85

      TODO: leave a fixme, we will need to do this soon-ish -> locale per collection

Sign in to join this conversation.