fix(ai-sandbox): delete stale native snapshots and don't leak a sandbox on snapshot failure #2131
Closedcodyde(GitHub profile, opens in a new tab)wants to merge 1 commit into
main fromrefs/pull/2131/headopened View on GitHubtanstack-source added labelsbug
tanstack-source reviewed
packages/ai-sandbox/src/middleware.ts line 1166@@ -1160,3 +1162,5 @@11601162 latestSnapshotId: snapshot.id,11611163 updatedAt: Date.now(),11621164 })1165+ // The previous snapshot is no longer referenced by anything.1166+ if (existing.latestSnapshotId !== snapshot.id)- packages/ai-sandbox-railway/README.md line 162@@ -146,6 +146,17 @@146146147147 ### Checkpoint retention148148149-Checkpoints count against a per-plan quota, and the TanStack provider contract150-has no delete hook. A checkpoint is not deleted when its source sandbox is151-destroyed. Prune `tsai-*` checkpoints you no longer reference:149+Checkpoints outlive their sandbox and count against a per-environment cap set by150+your plan (10 on Trial and Free). Past the cap, a new checkpoint fails with "Too151+many sandbox checkpoints in environment".152+153+`defineSandbox` keeps one checkpoint per sandbox record (the `after-setup` one by154+default, or the latest `after-run` one) and deletes it through the provider's155+`deleteSnapshot` once nothing references it:156+157+when a newer checkpoint replaces it on the record, and158+when the sandbox is torn down (`destroy`, `destroyOnComplete`, an aborted run,159+ or the reaper's `reclaimSandbox`).160+161+Deletes are best effort: a failure is logged and never fails the run. Only162+`tsai-*` names are ever deleted, so a base `checkpoint` you configured is safe.packages/ai-sandbox-railway/README.md lines 167 to 168@@ -151 +149,20 @@151-destroyed. Prune `tsai-*` checkpoints you no longer reference:149+Checkpoints outlive their sandbox and count against a per-environment cap set by150+your plan (10 on Trial and Free). Past the cap, a new checkpoint fails with "Too151+many sandbox checkpoints in environment".152+153+`defineSandbox` keeps one checkpoint per sandbox record (the `after-setup` one by154+default, or the latest `after-run` one) and deletes it through the provider's155+`deleteSnapshot` once nothing references it:156+157+when a newer checkpoint replaces it on the record, and158+when the sandbox is torn down (`destroy`, `destroyOnComplete`, an aborted run,159+ or the reaper's `reclaimSandbox`).160+161+Deletes are best effort: a failure is logged and never fails the run. Only162+`tsai-*` names are ever deleted, so a base `checkpoint` you configured is safe.163+164+A record that is never destroyed keeps its checkpoint, so one long-lived165+sandbox per thread still holds one checkpoint per thread. If that would exceed166+your cap, set `lifecycle.snapshot: 'none'` or tear sandboxes down when threads167+end. When a checkpoint does fail, the run continues without one and the next168+run re-creates the sandbox instead of restoring it.packages/ai-sandbox/src/sandbox.ts lines 358 to 361@@ -324,7 +348,14 @@324- provider: config.provider.name,325- providerSandboxId: created.id,326- latestSnapshotId,327- threadId: ctx.threadId,328- latestRunId: ctx.runId,329- updatedAt: Date.now(),330- })348+ try {349+ await store.upsert({350+ key,351+ provider: config.provider.name,352+ providerSandboxId: created.id,353+ latestSnapshotId,354+ threadId: ctx.threadId,355+ latestRunId: ctx.runId,356+ updatedAt: Date.now(),357+ })358+ } catch (error) {359+ // Unrecorded, so unreachable: tear it down rather than leak it. Keep a360+ // snapshot whose (deterministic) name the old record still points at.361+ await created.destroy().catch(() => {})packages/ai-sandbox/src/snapshot-cleanup.ts lines 37 to 39@@ -0,0 +19,21 @@19+ */20+const DELETE_SNAPSHOT_TIMEOUT_MS = 60 * 100021+22+/**23+ * Delete `snapshotId` through `provider.deleteSnapshot`, if the provider has24+ * one. Never throws: a failure is logged at warn level and swallowed, because25+ * a leaked snapshot must not fail the run or the teardown that dropped it.26+ */27+export async function deleteSnapshotQuietly(28+ provider: SandboxProvider,29+ snapshotId: string | undefined,30+ options: {31+ logger?: InternalLogger32+ meta?: Record<string, unknown>33+ } = {},34+): Promise<void> {35+ if (snapshotId === undefined || !provider.deleteSnapshot) return36+ const controller = new AbortController()37+ const timer = setTimeout(() => controller.abort(), DELETE_SNAPSHOT_TIMEOUT_MS)38+ try {39+ await provider.deleteSnapshot({ snapshotId, signal: controller.signal })
codyde(GitHub profile, opens in a new tab) closed this
Sign in to join this conversation.
🦋 Changeset detected#
Latest commit: 3687cb5f75070b428ea010cce0d5d56c863ff90c
The changes in this PR will be included in the next version bump.
This PR includes changesets to release 4 packages
Not sure what this means? Click here to learn what changesets are.
Click here if you're a maintainer who wants to add another changeset to this PR