Skip to content
Source
Sign inSign up
TanStack/db
Open History to see the last commit

spec.md

288 lines6.8 KB

1# Sync Library Technical Specification
2​
3## Overview
4​
5A general mutation library for sync engines that can write to any API. The library provides automatic optimistic updates, retry handling, and flexible mutation strategies for different types of updates.
6​
7## Core Concepts
8​
9### Collections
10​
11- Collections represent synchronized datasets that can be mutated locally and persisted to a backend
12- Each collection is identified by its sync configuration
13- Multiple components can share the same collection instance
14- Collections maintain both synced state and optimistic updates
15​
16### Mutations
17​
18Two primary mutation strategies:
19​
201. **Ordered** (default)
21 - Mutations queue behind active transactions
22 - Strict ordering of changes
23 - Automatic rollback of dependent transactions
242. **Parallel**
25 - All mutations can fire API calls immediately
26 - Custom merge function to handle sync updates
27 - Optimistic updates reapplied on top of incoming syncs
28 - No queueing or dependency management
29​
30### Transactions
31​
32- Track the state and history of mutations
33- Include retry attempts and error information
34- Support both ordered and parallel mutation patterns
35- Maintain full history of attempts and timing
36- Automatically persist to survive page reloads
37- All-or-nothing atomicity (no partial success)
38​
39## API Design
40​
41### Collection Hook
42​
43```typescript
44const useCollection = (config: {
45 sync: SyncConfig
46 mutationFn?: MutationFn
47}) => {
48 return {
49 data: Record<string, any>
50 update: UpdateFn
51 insert: InsertFn
52 delete: DeleteFn
53 withMutation: WithMutationFn
54 }
56```
57​
58### Sync Configuration
59​
60```typescript
61type SyncConfig = {
62 id: string
63 setup: (params: { onUpdate: (data: any) => void }) => Promise<{
64 data: any
65 }>
67```
68​
69### Mutation Function
70​
71```typescript
72type MutationFn = {
73 // Persist changes to backend
74 persist: (params: {
75 changes: Record<string, any>
76 attempt: number
77 transaction: Transaction
78 }) => Promise<void>
79​
80 // Optional - await sync completion
81 awaitSync?: (params: {
82 changes: Record<string, any>
83 transaction: Transaction
84 sync: SyncInstance
85 }) => Promise<void>
87```
88​
89### Transaction State
90​
91```typescript
92type TransactionState =
93 | "queued" // Waiting for another transaction to complete
94 | "pending" // Created but not yet persisting
95 | "persisting" // Currently running mutationFn.persist()
96 | "persisted_awaiting_sync" // Persist succeeded, waiting for sync
97 | "completed" // Fully completed
98 | "failed" // Failed (includes dependency cancellations)
99​
100type Transaction = {
101 id: string
102 state: TransactionState
103 created_at: Date
104 updated_at: Date
105 mutations: PendingMutation[]
106 attempts: Attempt[]
107 current_attempt: number
108 queued_behind?: string
109 error?: {
110 transaction_id?: string // For dependency failures
111 message: string
112 error: Error
113 }
115​
116type Attempt = {
117 id: string
118 started_at: Date
119 completed_at?: Date
120 error?: Error
121 retry_scheduled_for?: Date
123​
124type PendingMutation = {
125 mutationId: string
126 original: Record<string, any>
127 modified: Record<string, any>
128 changes: Record<string, any>
129 metadata: unknown
130 created_at: Date
131 updated_at: Date
132 state: "created" | "persisting" | "synced"
134```
135​
136### Mutation Strategy
137​
138```typescript
139type MutationStrategy = {
140 type: "ordered" | "parallel"
141 merge?: (syncedData: any, pendingMutations: PendingMutation[]) => any
143```
144​
145### Error Handling
146​
147```typescript
148class NonRetriableError extends Error {
149 constructor(message: string) {
150 super(message)
151 this.name = "NonRetriableError"
152 }
154```
155​
156### Lock Management
157​
158```typescript
159// Helper function to determine locked objects from transaction state
160function getLockedObjects(transactions: Transaction[]): Set<string> {
161 return new Set(
162 transactions
163 .filter(
164 (t) => t.state === "persisting" || t.state === "persisted_awaiting_sync"
165 )
166 .flatMap((t) => t.mutations)
167 .map((m) => m.modified.id)
168 )
170```
171​
172## Implementation Details
173​
174### Transaction Persistence
175​
176- All transactions automatically persist to survive page reloads
177- Uses IndexedDB by default
178- Logs error if persistence setup fails
179- Recovers and retries pending transactions on page load
180- Assumes successful sync if page closed after persist succeeds
181​
182### Retry Handling
183​
184- Retry by default for all errors
185- NonRetriableError to opt out of retries
186- Default 4 retries (5 total attempts) per transaction
187- Exponential backoff with configurable options
188​
189### Optimistic Updates
190​
191- Applied immediately upon mutation
192- Maintained until backend sync completes
193- For parallel mutations, reapplied after syncs via merge function
194- For ordered mutations, queued behind active transactions
195- Dropped after successful persist if page reloads before sync completes
196​
197### Sync Integration
198​
199- Sync mechanism provided by developer
200- Updates to locked objects queue until transaction completes
201- Parallel mutations use merge function to handle concurrent updates
202- Sync instance passed to mutationFn for coordination
203​
204### Transaction Management
205​
2061. Ordered Transactions:
207​
208 - Lock detection via transaction state
209 - Queue dependent mutations
210 - Roll back on failure
211 - Release locks after sync
212​
2132. Parallel Transactions:
214 - No locking or queueing
215 - Independent API calls
216 - Simple merge strategy
217 - No rollback on failure
218​
219### State Management
220​
221- Track all transactions with full history
222- Maintain optimistic updates separately
223- Queue updates to locked objects
224- Expose transaction state for monitoring
225- No automatic pruning of transaction history
226​
227## Extension Points
228​
229The library provides several key extension points for advanced functionality:
230​
231### Error Handling
232​
233- Custom retry strategies
234- Extended error classification beyond NonRetriableError
235- Error transformation and aggregation
236​
237### Merge Strategies
238​
239- Custom merge functions for parallel mutations
240- Advanced conflict resolution
241- Domain-specific merge logic
242​
243### Monitoring
244​
245- Transaction event subscribers
246- Custom logging and metrics
247- Debug tooling and visualization
248- Performance monitoring
249​
250## Testing Utilities
251​
252### Basic Test Harness
253​
254```typescript
255const { collection, syncControl } = createTestCollection({
256 initialData: any
257 sync: SyncConfig
258})
259​
260// Control sync behavior
261syncControl.emitUpdate(data)
262syncControl.setNetworkDelay(ms)
263syncControl.failNextPersist()
264```
265​
266### Future Testing Enhancements
267​
268- Network condition simulation
269- Persistence recovery testing
270- Parallel/ordered mutation interaction testing
271- Transaction state inspection
272- Time-travel debugging
273​
274## State Exposure
275​
276- All transaction state available via useTransactions hook
277- Support for building debugging tools
278- Event subscription for state changes
279​
280## Future Considerations
281​
282- SSR support via Provider pattern
283- RSC support via HydrationBoundary
284- Advanced debugging tools
285- Extended test utilities
286- Transaction history pruning
287- Advanced merge conflict resolution
288- Performance optimizations for large datasets