Skip to content
Source
Sign inSign up

[Bug]: Storage cleanup removes an active thread's worktree, with its ignored files, hours after creation #33881

Send to my agent
Closedsmonev(GitHub profile, opens in a new tab) opened this issue 1 comment

Before submitting

  • I searched existing issues and did not find a duplicate.
  • I included enough detail to reproduce or investigate the problem.

Area

apps/web

Steps to reproduce

Summary

The hourly storage cleanup ran git worktree remove on the worktree of a thread that was 10 hours old, not archived and not settled. The thread was idle between user messages. A long-running detached process was still writing into the worktree. Everything git ignores was lost with it: about 4.6 GB of run output, node_modules and a generated build project. On the thread's next message T3 logged "recreating missing worktree" and gave it an empty fresh checkout, with no notice to the user or the agent.

Version and settings

  • T3 server 0.1.8 (t3 serve), Linux, Claude agent provider.
  • settings.json:
    "storageCleanup": {
      "worktreeAfterDays": 15,
      "worktreeUnchanged": true
    }
  • worktreeKeepWhen not set (default uncommitted-changes).

Timeline (local time)

TimeEvent
~05:30Thread created in a new worktree from origin/master. The agent never commits in it; its PR work is in a second worktree.
05:30 to 15:17The agent builds there and runs long jobs that write about 4.6 GB into the git-ignored logs/ folder.
~15:17The agent ends its turn. A detached job it started keeps running and writing into logs/.
15:19:44Storage cleanup sweep starts .
15:20:19INFO: storage cleanup removed worktree { threadId: '<thread-id>' }. The detached job dies mid-write.
15:24:39The user sends a message. WARN: provider turn start recreating missing worktree { threadId: '<thread-id>', worktreePath: …, branch: '<branch>' }. The agent finds an empty checkout.

Why it was picked

From the bundled server code (the StorageCleanup worktree pass):

eligible = integrated && settings.worktreeUnchanged;
if (eligible) removalReason = remote === null ? "no commits beyond the project branch" : "no commits beyond the default branch";
  1. worktreeUnchanged: true makes a worktree whose HEAD is an ancestor of the default branch eligible right away. worktreeAfterDays: 15 doesn't hold it back, so a worktree created this morning can go at the next sweep.
  2. "Changes" means uncommitted tracked or untracked files only. Ignored files (logs/, node_modules, build output) don't count, so gigabytes of results look like an empty worktree.
  3. The liveness checks look at the thread's idle state, open terminals and live provider sessions. They don't look at processes whose working directory is inside the worktree. A detached build or test job started by the agent doesn't protect it.
  4. Nothing tells the thread. The next turn silently recreates the worktree, so the agent and the user only find out when files are missing.

Expected

At least one of these:

  • worktreeAfterDays applies to every cleanup reason, including "no commits beyond the default branch", so a worktree under 15 days old is never removed for being unchanged.
  • A thread that isn't archived or settled keeps its worktree, whatever its branch state.
  • A worktree with any process running inside it (cwd or open files) is skipped, like an open terminal.
  • If a worktree that holds ignored files is about to be removed, it is skipped, or at least the ignored size is shown and the user is asked first.
  • When a removed worktree is recreated, the thread gets a visible notice saying when and why it was removed.

Impact

Lost: the outputs of four multi-hour device test runs (per-run results, database copies, grading files). One run was killed mid-way and its partial results are gone. The phone time can't be recovered: each run takes 2.5 to 3 hours on a shared device.

Steps to reproduce

  1. Set storageCleanup.worktreeUnchanged: true and leave worktreeAfterDays at any value.
  2. Start a thread in a new worktree on a branch with no commits beyond the default branch.
  3. In the worktree, write large output into a git-ignored folder, and start a detached process (setsid sleep 3600 &) with that folder as its cwd.
  4. Let the thread go idle and wait for the next storage cleanup sweep (hourly).
  5. The worktree is removed, and the process loses its cwd. The next message to the thread logs "recreating missing worktree".

Expected behavior

.

Actual behavior

.

Impact

Minor bug or occasional failure

Version or commit

No response

Environment

No response

Logs or stack traces

Screenshots, recordings, or supporting files

No response

Workaround

No response

  1. smonev(GitHub profile, opens in a new tab) added labelsneeds-triage

  2. it was a t3 setting - "Delete unchanged worktrees" and it was ON.

Sign in to join this conversation.