Open History to see the last commit
1# Sync Library Technical Specification
2
3## Overview
4
5A general mutation library for sync engines that can write to any API. The library provides automatic optimistic updates, retry handling, and flexible mutation strategies for different types of updates.
6
7## Core Concepts
8
9### Collections
10
11- Collections represent synchronized datasets that can be mutated locally and persisted to a backend
12- Each collection is identified by its sync configuration
13- Multiple components can share the same collection instance
14- Collections maintain both synced state and optimistic updates
15
16### Mutations
17
18Two primary mutation strategies:
19
201. **Ordered** (default)
21 - Mutations queue behind active transactions
22 - Strict ordering of changes
23 - Automatic rollback of dependent transactions
242. **Parallel**
25 - All mutations can fire API calls immediately
26 - Custom merge function to handle sync updates
27 - Optimistic updates reapplied on top of incoming syncs
28 - No queueing or dependency management
29
30### Transactions
31
32- Track the state and history of mutations
33- Include retry attempts and error information
34- Support both ordered and parallel mutation patterns
35- Maintain full history of attempts and timing
36- Automatically persist to survive page reloads
37- All-or-nothing atomicity (no partial success)
38
39## API Design
40
41### Collection Hook
42
43```typescript
44const useCollection = (config: {
45 sync: SyncConfig
46 mutationFn?: MutationFn
47}) => {
48 return {
49 data: Record<string, any>
50 update: UpdateFn
51 insert: InsertFn
52 delete: DeleteFn
53 withMutation: WithMutationFn
54 }
55}
56```
57
58### Sync Configuration
59
60```typescript
61type SyncConfig = {
62 id: string
63 setup: (params: { onUpdate: (data: any) => void }) => Promise<{
64 data: any
65 }>
66}
67```
68
69### Mutation Function
70
71```typescript
72type MutationFn = {
73 // Persist changes to backend
74 persist: (params: {
75 changes: Record<string, any>
76 attempt: number
77 transaction: Transaction
78 }) => Promise<void>
79
80 // Optional - await sync completion
81 awaitSync?: (params: {
82 changes: Record<string, any>
83 transaction: Transaction
84 sync: SyncInstance
85 }) => Promise<void>
86}
87```
88
89### Transaction State
90
91```typescript
92type TransactionState =
93 | "queued" // Waiting for another transaction to complete
94 | "pending" // Created but not yet persisting
95 | "persisting" // Currently running mutationFn.persist()
96 | "persisted_awaiting_sync" // Persist succeeded, waiting for sync
97 | "completed" // Fully completed
98 | "failed" // Failed (includes dependency cancellations)
99
100type Transaction = {
101 id: string
102 state: TransactionState
103 created_at: Date
104 updated_at: Date
105 mutations: PendingMutation[]
106 attempts: Attempt[]
107 current_attempt: number
108 queued_behind?: string
109 error?: {
110 transaction_id?: string // For dependency failures
111 message: string
112 error: Error
113 }
114}
115
116type Attempt = {
117 id: string
118 started_at: Date
119 completed_at?: Date
120 error?: Error
121 retry_scheduled_for?: Date
122}
123
124type PendingMutation = {
125 mutationId: string
126 original: Record<string, any>
127 modified: Record<string, any>
128 changes: Record<string, any>
129 metadata: unknown
130 created_at: Date
131 updated_at: Date
132 state: "created" | "persisting" | "synced"
133}
134```
135
136### Mutation Strategy
137
138```typescript
139type MutationStrategy = {
140 type: "ordered" | "parallel"
141 merge?: (syncedData: any, pendingMutations: PendingMutation[]) => any
142}
143```
144
145### Error Handling
146
147```typescript
148class NonRetriableError extends Error {
149 constructor(message: string) {
150 super(message)
151 this.name = "NonRetriableError"
152 }
153}
154```
155
156### Lock Management
157
158```typescript
159// Helper function to determine locked objects from transaction state
160function getLockedObjects(transactions: Transaction[]): Set<string> {
161 return new Set(
162 transactions
163 .filter(
164 (t) => t.state === "persisting" || t.state === "persisted_awaiting_sync"
165 )
166 .flatMap((t) => t.mutations)
167 .map((m) => m.modified.id)
168 )
169}
170```
171
172## Implementation Details
173
174### Transaction Persistence
175
176- All transactions automatically persist to survive page reloads
177- Uses IndexedDB by default
178- Logs error if persistence setup fails
179- Recovers and retries pending transactions on page load
180- Assumes successful sync if page closed after persist succeeds
181
182### Retry Handling
183
184- Retry by default for all errors
185- NonRetriableError to opt out of retries
186- Default 4 retries (5 total attempts) per transaction
187- Exponential backoff with configurable options
188
189### Optimistic Updates
190
191- Applied immediately upon mutation
192- Maintained until backend sync completes
193- For parallel mutations, reapplied after syncs via merge function
194- For ordered mutations, queued behind active transactions
195- Dropped after successful persist if page reloads before sync completes
196
197### Sync Integration
198
199- Sync mechanism provided by developer
200- Updates to locked objects queue until transaction completes
201- Parallel mutations use merge function to handle concurrent updates
202- Sync instance passed to mutationFn for coordination
203
204### Transaction Management
205
2061. Ordered Transactions:
207
208 - Lock detection via transaction state
209 - Queue dependent mutations
210 - Roll back on failure
211 - Release locks after sync
212
2132. Parallel Transactions:
214 - No locking or queueing
215 - Independent API calls
216 - Simple merge strategy
217 - No rollback on failure
218
219### State Management
220
221- Track all transactions with full history
222- Maintain optimistic updates separately
223- Queue updates to locked objects
224- Expose transaction state for monitoring
225- No automatic pruning of transaction history
226
227## Extension Points
228
229The library provides several key extension points for advanced functionality:
230
231### Error Handling
232
233- Custom retry strategies
234- Extended error classification beyond NonRetriableError
235- Error transformation and aggregation
236
237### Merge Strategies
238
239- Custom merge functions for parallel mutations
240- Advanced conflict resolution
241- Domain-specific merge logic
242
243### Monitoring
244
245- Transaction event subscribers
246- Custom logging and metrics
247- Debug tooling and visualization
248- Performance monitoring
249
250## Testing Utilities
251
252### Basic Test Harness
253
254```typescript
255const { collection, syncControl } = createTestCollection({
256 initialData: any
257 sync: SyncConfig
258})
259
260// Control sync behavior
261syncControl.emitUpdate(data)
262syncControl.setNetworkDelay(ms)
263syncControl.failNextPersist()
264```
265
266### Future Testing Enhancements
267
268- Network condition simulation
269- Persistence recovery testing
270- Parallel/ordered mutation interaction testing
271- Transaction state inspection
272- Time-travel debugging
273
274## State Exposure
275
276- All transaction state available via useTransactions hook
277- Support for building debugging tools
278- Event subscription for state changes
279
280## Future Considerations
281
282- SSR support via Provider pattern
283- RSC support via HydrationBoundary
284- Advanced debugging tools
285- Extended test utilities
286- Transaction history pruning
287- Advanced merge conflict resolution
288- Performance optimizations for large datasets