Skip to content
Source
Sign inSign up
TanStack/db

refactor the Collection class into multiple manager classes #560

Mergedsamwillis(GitHub profile, opens in a new tab)merged 16 commits intomain fromrefs/pull/560/headmerged by samwillis(GitHub profile, opens in a new tab)

Collection had grown too large and complex - this moves everything into manager classes.

It also moves a bunch of the internal, albeit publicly accessible, state into _state and other underscore prefixed class instances. Of note is the syncedData map, this was public at the top level so that it was accessible in tests and to the Transaction system. It is now under _state.syncedData making it a little more private. If anyone depended on this its moved, but I think thats good as its an internal detail that they should not be depending on.

    1. 🦋 Changeset detected

      Latest commit: 77e8303ae11357c6864fa7552dc5bb9ae2e094aa

      The changes in this PR will be included in the next version bump.

      This PR includes changesets to release 12 packages
      NameType
      @tanstack/electric-db-collectionPatch
      @tanstack/query-db-collectionPatch
      @tanstack/rxdb-db-collectionPatch
      @tanstack/dbPatch
      @tanstack/db-example-react-todoPatch
      @tanstack/angular-dbPatch
      @tanstack/react-dbPatch
      @tanstack/solid-dbPatch
      @tanstack/svelte-dbPatch
      @tanstack/trailbase-db-collectionPatch
      @tanstack/vue-dbPatch
      todosPatch

      Not sure what this means? Click here to learn what changesets are.

      Click here if you're a maintainer who wants to add another changeset to this PR

    2. More templates

      @tanstack/angular-db

      npm i https://pkg.pr.new/@tanstack/angular-db@560

      @tanstack/db

      npm i https://pkg.pr.new/@tanstack/db@560

      @tanstack/db-ivm

      npm i https://pkg.pr.new/@tanstack/db-ivm@560

      @tanstack/electric-db-collection

      npm i https://pkg.pr.new/@tanstack/electric-db-collection@560

      @tanstack/query-db-collection

      npm i https://pkg.pr.new/@tanstack/query-db-collection@560

      @tanstack/react-db

      npm i https://pkg.pr.new/@tanstack/react-db@560

      @tanstack/rxdb-db-collection

      npm i https://pkg.pr.new/@tanstack/rxdb-db-collection@560

      @tanstack/solid-db

      npm i https://pkg.pr.new/@tanstack/solid-db@560

      @tanstack/svelte-db

      npm i https://pkg.pr.new/@tanstack/svelte-db@560

      @tanstack/trailbase-db-collection

      npm i https://pkg.pr.new/@tanstack/trailbase-db-collection@560

      @tanstack/vue-db

      npm i https://pkg.pr.new/@tanstack/vue-db@560

      commit: 77e8303

    3. Size Change: +3.88 kB (+5.61%) 🔍

      Total Size: 73 kB

      FilenameSizeChange
      ./packages/db/dist/esm/change-events.js0 B-957 B (removed)🏆
      ./packages/db/dist/esm/collection-events.js0 B-672 B (removed)🏆
      ./packages/db/dist/esm/collection-subscription.js0 B-1.69 kB (removed)🏆
      ./packages/db/dist/esm/collection.js0 B-10.5 kB (removed)🏆
      ./packages/db/dist/esm/index.js1.56 kB+6 B (+0.39%)
      ./packages/db/dist/esm/query/builder/index.js3.93 kB+4 B (+0.1%)
      ./packages/db/dist/esm/query/live-query-collection.js340 B+7 B (+2.1%)
      ./packages/db/dist/esm/transactions.js3.03 kB+4 B (+0.13%)
      ./packages/db/dist/esm/collection/change-events.js958 B+958 B (new file)🆕
      ./packages/db/dist/esm/collection/changes.js1.01 kB+1.01 kB (new file)🆕
      ./packages/db/dist/esm/collection/events.js684 B+684 B (new file)🆕
      ./packages/db/dist/esm/collection/index.js3.14 kB+3.14 kB (new file)🆕
      ./packages/db/dist/esm/collection/indexes.js1.16 kB+1.16 kB (new file)🆕
      ./packages/db/dist/esm/collection/lifecycle.js1.37 kB+1.37 kB (new file)🆕
      ./packages/db/dist/esm/collection/mutations.js2.59 kB+2.59 kB (new file)🆕
      ./packages/db/dist/esm/collection/state.js3.78 kB+3.78 kB (new file)🆕
      ./packages/db/dist/esm/collection/subscription.js1.69 kB+1.69 kB (new file)🆕
      ./packages/db/dist/esm/collection/sync.js1.32 kB+1.32 kB (new file)🆕
      ℹ️ View Unchanged
      FilenameSize
      ./packages/db/dist/esm/deferred.js230 B
      ./packages/db/dist/esm/errors.js3.1 kB
      ./packages/db/dist/esm/indexes/auto-index.js745 B
      ./packages/db/dist/esm/indexes/base-index.js605 B
      ./packages/db/dist/esm/indexes/btree-index.js1.76 kB
      ./packages/db/dist/esm/indexes/lazy-index.js1.25 kB
      ./packages/db/dist/esm/local-only.js827 B
      ./packages/db/dist/esm/local-storage.js2.02 kB
      ./packages/db/dist/esm/optimistic-action.js294 B
      ./packages/db/dist/esm/proxy.js3.87 kB
      ./packages/db/dist/esm/query/builder/functions.js615 B
      ./packages/db/dist/esm/query/builder/ref-proxy.js938 B
      ./packages/db/dist/esm/query/compiler/evaluators.js1.52 kB
      ./packages/db/dist/esm/query/compiler/expressions.js631 B
      ./packages/db/dist/esm/query/compiler/group-by.js2.08 kB
      ./packages/db/dist/esm/query/compiler/index.js2.29 kB
      ./packages/db/dist/esm/query/compiler/joins.js2.5 kB
      ./packages/db/dist/esm/query/compiler/order-by.js1.23 kB
      ./packages/db/dist/esm/query/compiler/select.js1.28 kB
      ./packages/db/dist/esm/query/ir.js508 B
      ./packages/db/dist/esm/query/live/collection-config-builder.js2.68 kB
      ./packages/db/dist/esm/query/live/collection-subscriber.js1.91 kB
      ./packages/db/dist/esm/query/optimizer.js3.05 kB
      ./packages/db/dist/esm/SortedMap.js1.24 kB
      ./packages/db/dist/esm/utils.js943 B
      ./packages/db/dist/esm/utils/btree.js6.02 kB
      ./packages/db/dist/esm/utils/comparison.js718 B
      ./packages/db/dist/esm/utils/index-optimization.js1.62 kB
      compressed-size-action::db-package-size
    4. Size Change: 0 B

      Total Size: 1.44 kB

      ℹ️ View Unchanged
      FilenameSize
      ./packages/react-db/dist/esm/index.js152 B
      ./packages/react-db/dist/esm/useLiveQuery.js1.29 kB
      compressed-size-action::react-db-package-size
    5. Pull Request Overview

      This PR refactors the large Collection class into multiple focused manager classes to improve maintainability and reduce complexity. The monolithic Collection class was broken down into specialized managers for different responsibilities while maintaining the same public API.

      Key Changes:

      • Split Collection class into multiple manager classes (state, changes, lifecycle, sync, indexes, mutations)
      • Moved internal state properties from public access to _state namespace
      • Updated all test files to use new import paths and access patterns

      Reviewed Changes

      Copilot reviewed 78 out of 79 changed files in this pull request and generated 7 comments.

      Show a summary per file
      FileDescription
      packages/db/src/collection/index.tsNew main collection implementation using manager pattern
      packages/db/src/collection/state.tsState management and optimistic updates logic
      packages/db/src/collection/changes.tsEvent emission and subscription handling
      packages/db/src/collection/lifecycle.tsStatus management and garbage collection
      packages/db/src/collection/sync.tsSynchronization logic and preloading
      packages/db/src/collection/indexes.tsIndex creation and management
      packages/db/src/collection/mutations.tsInsert, update, delete operations
      All test filesUpdated import paths and state access patterns
      Files not reviewed (1)
      • pnpm-lock.yaml: Language not supported

    6. **Tip:** Customize your code reviews with copilot-instructions.md. Create the file or learn how to get started.
OutdatedResolvedby samwillis(GitHub profile, opens in a new tab)
OutdatedResolvedby samwillis(GitHub profile, opens in a new tab)
OutdatedResolvedby samwillis(GitHub profile, opens in a new tab)
OutdatedResolvedby samwillis(GitHub profile, opens in a new tab)
OutdatedResolvedby samwillis(GitHub profile, opens in a new tab)
OutdatedResolvedby samwillis(GitHub profile, opens in a new tab)
OutdatedResolvedby samwillis(GitHub profile, opens in a new tab)
  • Copilots review complained of the circular dependancy between CollecitonImpl and the manager classes. I don't think this is an issue as its a type only circular dependency (there is no runtime circular dependency), and there isn't really a clean way around it without making the significantly code much more complex (a separate Collection interface would still have a circular dependency as it would need to have the manager class instances).

  • @kevin-dp either as part of your PR, a followup, or this one if it goes in second, we should consider moving change-events.ts into colleciton/changes.ts along with your refactored stuff.

  • Splitting the Collection class into helper classes was absolutely necessary, great job with that, it is way more understandable and maintainable now!

    The main Collection class (in index.ts) still has quite a bit of boilerplate code because we need to define public methods that just forward the call to the right nested instance and every of these new classes needs to keep a reference back to the collection...

    I really wish Typescript had support for traits or multiple inheritance. It would be nicer and more concise if we could simply extend each behavior and we would inherit these methods instead of having to define them and forward the calls. That way the classes also wouldn't need to keep a reference back to the collection.

    There is a way to achieve mixins in TS, here's a snippet from chatgpt showing how it can be done:

    type Ctor<T = {}> = new (...args: any[]) => T;
    
    function Auth<TBase extends Ctor>(Base: TBase) {
      return class extends Base {
        login(u: string, p: string) {/*...*/}
        logout() {/*...*/}
      };
    }
    
    function Sync<TBase extends Ctor>(Base: TBase) {
      return class extends Base {
        syncNow() {/*...*/}
        scheduleSync(ms: number) {/*...*/}
      };
    }
    
    class Core { /* shared state */ }
    
    class Big extends Sync(Auth(Core)) {
      ping() { return "pong"; }
    }
    
    const big = new Big();
    big.login("k","pw");
    big.syncNow();

    I do think it's worth rewriting this to use this mixins pattern. Wdyt?

  • @kevin-dp thanks! Yep that mix-in pattern is nice. The one drawback is that the inner mix-in doesn't see the types of the props on the outer mix-in (I don't think?). We would need to central abstract interface still. We have a quite a lot of two way dependancies through the separate functionality.

    very open to suggestions on this

  • @samwillis You're right that inner ones don't know about state and methods introduced by outer ones. In typical OOP with traits (or abstract classes with multiple inheritance) each trait can define abstract state/methods it requires. We can also leverage abstract properties/methods in the mixin pattern. So, we can have the base class define abstract state/methods and each mixin can be defined as an abstract class (such that TS doesn't complain if they don't implement (all) the abstract properties/methods):

    type Ctor<T = {}> = abstract new (...args: any[]) => T;
    
    abstract class Core {
      abstract hookSchedule(ms: number, f: () => void): void;
    }
    
    function Scheduler<TBase extends Ctor<Core>>(Base: TBase) {
      abstract class Scheduler extends Base {
        hookSchedule(ms: number, f: () => void) {
          setTimeout(f, ms);
        }
      };
      return Scheduler
    }
    
    function NeedsScheduler<TBase extends Ctor<Core>>(Base: TBase) {
      abstract class NeedsScheduler extends Base {
        doThing() {
          this.hookSchedule(10, () => {/* ... */});
        }
      };
      return NeedsScheduler
    }
    
    class Big extends NeedsScheduler(Scheduler(Core)) {}
  • kevin-dp(GitHub profile, opens in a new tab) approved these changes

    LGTM! Thanks for taking the time to improve this

    OutdatedResolvedby samwillis(GitHub profile, opens in a new tab)
    packages/db/src/collection/sync.ts line 22
    @@ -0,0 +2,21 @@
    2+ CollectionIsInErrorStateError,
    3+ DuplicateKeySyncError,
    4+ NoPendingSyncTransactionCommitError,
    5+ NoPendingSyncTransactionWriteError,
    6+ SyncCleanupError,
    7+ SyncTransactionAlreadyCommittedError,
    8+ SyncTransactionAlreadyCommittedWriteError,
    9+} from "../errors"
    10+import type { StandardSchemaV1 } from "@standard-schema/spec"
    11+import type { ChangeMessage, CollectionConfig } from "../types"
    12+import type { CollectionImpl } from "./index.js"
    13+import type { CollectionStateManager } from "./state"
    14+import type { CollectionLifecycleManager } from "./lifecycle"
    15+
    16+export class CollectionSyncManager<
    17+ TOutput extends object = Record<string, unknown>,
    18+ TKey extends string | number = string | number,
    19+ TSchema extends StandardSchemaV1 = StandardSchemaV1,
    20+ TInput extends object = TOutput,
    21+> {
    22+ private collection!: CollectionImpl<TOutput, TKey, any, TSchema, TInput>

    What is the ! at the end of the name? Is this a TS thing or some convention i don't know?

    These don't have an initialiser in the constructor - they are set in setDeps, so we either have to mark them as optional (then check in every method), or do this which says "I know I don't have an initialiser but I promise it will be there before I access it".

    We can revisit this pattern later if you want and add a check in each method, but I think for this use case in an internal api it's fine.

  • samwillis(GitHub profile, opens in a new tab)replied on packages/db/src/collection/sync.ts line 22

    These don't have an initialiser in the constructor - they are set in setDeps, so we either have to mark them as optional (then check in every method), or do this which says "I know I don't have an initialiser but I promise it will be there before I access it".

    We can revisit this pattern later if you want and add a check in each method, but I think for this use case in an internal api it's fine.

  • Sign in to join this conversation.