Open History to see the last commit
1# Server updates
2
3A [stable launcher](../../apps/server/src/serviceLauncher.ts) owns the runtime
4selected by systemd, launchd, or Task Scheduler. It is the only runtime writer of durable service
5state. Server children request updates over inherited IPC; they never rewrite
6their service definition or select their own replacement. Local service commands
7may replace the launcher and state, optionally deferring the restart. Foreground
8CLI processes do not self-update.
9
10Both writers serialize state and unit writes through
11[file leases](../../apps/server/src/cloud/serviceSelectionLock.ts). The installer
12takes the state lease before the unit lease; the launcher never holds both, so a
13unit failure cannot block a durable commit. Manager stop/start commands run outside
14both leases, since stopping a launcher that waits on a lease would deadlock. A
15running launcher refuses further updates once a local command changes its selection.
16A protocol-3 launcher takes no lease, so a local command can still race its updates.
17
18Exact-version installs keep restarts independent of npm cache eviction or a moving
19release tag. Installation and preflight happen in staging before publishing an
20immutable runtime. Preflight checks the launcher protocol because a target that
21needs new rollback guarantees cannot safely run under an older launcher. A remote
22update replaces only the child, so upgrading the launcher requires a local service
23update with a restart. Every installed launcher is therefore one protocol behind
24after a bump, and a target that refuses the previous protocol stops remote updates
25for every existing service. Refuse it only when the child cannot work under it.
26Protocol 4 changed the launcher alone, so its runtime still runs under protocol 3.
27The server names its launcher's protocol to preflight, not its own, since the two differ
28after the first such update.
29
30A client can stage the target early with `stageOnly`, which installs and preflights the
31runtime and restarts nothing. "Update everything" does this while provider installs run,
32because a restart would kill them. A server without the `serverUpdateStaging` capability
33ignores the flag and runs the whole update, so send it only to a server that advertises it.
34
35After a commit or rollback the launcher points the unit at the selected runtime and
36reloads the manager, so the next service start runs that version's launcher; the
37protocol check is what makes that safe. Each start after a finished update retries this, because
38a previous reload may have failed, and leaves a unit naming a newer runtime or another
39T3 home alone. A protocol-3 launcher does none of this: its unit keeps naming the
40runtime that installed it. Until a deferred local upgrade restarts, systemd (before
41daemon-reload) and launchd (before bootstrap) can respawn the old launcher, which
42rejects the state the new installer wrote.
43
44Task Scheduler tracks only the console host it started, not the launcher. `schtasks /End`
45kills that host and nothing else; the launcher takes its closed console as SIGTERM. The task
46reads as stopped at once, and `/Run` then starts a second launcher beside the first, so a
47Windows launcher holds a lock file for as long as it runs. A replacement waits for that lock,
48and the CLI treats its release as the stop having finished.
49
50Task Scheduler also runs its own copy of the task, and registering one enables it. So every
51write of the task file registers it, including an install that defers its restart. The CLI
52disables the task under the unit lease, and a registration under that lease keeps a disabled
53task disabled, so the repeating trigger cannot restart a service the CLI is holding stopped.
54
55## Commit boundary
56
57The launcher durably records the pending update before acknowledging it, then
58stops the old child and starts the target as a trial. Service-state writes use
59same-directory replacement with file and directory fsync. Invalid state stops
60startup rather than guessing which runtime to boot.
61
62The trial must finish migrations, acquire dependencies, bind HTTP, and park every
63long-running root at the activation gate before reporting `prepared`. The launcher
64then commits the target version durably and replies `committed`. Only then may the
65child release its gates, accept commands, and publish ready. Keep fallible startup
66acquisitions before this boundary. A listener alone does not prove the runtime is
67ready to commit.
68
69A failed or timed-out trial returns to the old version. After commit, the target
70is authoritative and the service manager's ordinary restart policy applies.
71
72The prepared deadline is long because migrations can scale with the database
73and block the trial's main thread, so it cannot report progress. A missed
74deadline restores the snapshot, and the retry pays the same cost again. A
75crashed trial is caught when it exits; the deadline bounds only a stuck one.
76Client update waits must outlast the deadline, or a slow update shows as failed.
77
78## Database rollback
79
80Before stopping the old child, the launcher refuses an update whose snapshot will
81not fit on disk, so a full disk does not cause an outage. After the old child
82exits, the launcher snapshots SQLite's main file, WAL, and shared-memory file. This makes trial migrations reversible without down
83migrations. The snapshot is made once per update and survives launcher restarts;
84replacing it during a retry could capture changes from the failed trial.
85
86Rollback stops the trial before restoring. A durable restore marker makes an
87interrupted restore finish before either version boots. Keep the snapshot until
88commit, or until both restoration and the terminal rollback state are durable.
89Attachments and other files outside SQLite are outside this rollback boundary.
90
91## Client acknowledgement
92
93An accepted update is still pending. Clients correlate the launcher's update ID
94with the ready event after reconnecting, then check the outcome and target version.
95A reconnect alone cannot distinguish successful replacement from rollback. Older
96servers without an update ID retain version-only correlation.
97
98Desktop updates have a separate two-phase handoff because installing the app stops
99its bundled backend. Preparation returns a token while the connection is alive;
100the client commits that token only after receiving it. Otherwise backend shutdown
101could lose the only successful RPC result. The client must then observe the
102prepared version after reconnecting. If installation fails, desktop restarts the
103stopped backends and replays the failure for the same token.
104
105## Recovering interrupted threads
106
107Restart continuation is an environment-owned preference, on by default. A
108`settings.json` that fails to decode turns it off, since the file may hold an
109opt-out. The
110[v2 recovery service](../../apps/server/src/orchestration-v2/ProviderRuntimeRecoveryService.ts)
111requires matching durable run, provider thread, session, and native resume identity.
112Queued runs never started, so recovery holds them and continues the run they wait
113behind. A settled run qualifies only when the restart cancels work the agent waits on
114(a subagent, a background task, or a shell or monitor the agent marked with
115`t3_background_wait`) and its session was live, so stale records from an older crash
116never wake it. Its continuation prompt is the note of what will not report back. An
117unmarked shell or monitor does not wake a settled run: most are dev servers and their
118watchdogs that nobody waits on, so that note waits for the next user message.
119
120A run is marked running before its provider accepts the prompt, so recovery treats a run
121without a provider turn as never started. A user's prompt the provider never received
122joins the held queue. With restart continuation on, a delegated, agent-sent, or scheduled
123prompt has nobody to review it, so recovery starts it again under the same run. The resend
124skips every thread and Stop that a continuation skips, and gives up after three tries so a
125start that crashes the server cannot loop. A crash after the provider received the prompt
126but before its turn was recorded can repeat that prompt.
127
128Recovery retires effects tied to the lost process and records continuation intent
129in the durable outbox. That intent survives another restart before provider startup.
130Continuation effects wait for activation; a slow provider must not delay the server's
131readiness or the launcher's commit boundary. Graceful shutdown captures intent before
132closing providers, then reconciles after ingestion has stopped so a late completion
133cannot be overwritten by a stale cancellation.
134
135The [continuation handler](../../apps/server/src/orchestration-v2/RestartContinuation.ts)
136rechecks the preference, provider selection, newer user work, a stop the user requested,
137and maintenance turns such as `/compact` before dispatching. Intent and delivery both
138refuse a thread that is archived, settled, or still snoozed, a cut turn whose provider
139turn already ended, and a run waiting on the user: an approval, question, or secret that was pending,
140or that recovery closed on the cut turn. Message-capable questions stay answerable, so
141they do not block. New continuation intents persist a cumulative process-loss count;
142delivery stops before the fourth crash recovery of an unfinished turn, regardless
143of elapsed time. Planned shutdowns and completed turns reset it. A live continuation
144retains its preceding intent past outbox retention so cleanup cannot erase the count.
145Legacy intents without a count keep the five-minute chain guard. Stable
146command and message IDs prevent duplicate submissions after an outbox retry. Codex
147resumes without adding provider prompt text unless the turn lost background work;
148other adapters receive the continuation message through their normal turn path.
149
150Delegated tasks (`delegate_task` child threads) are reconciled as their own threads,
151never as the parent's background work. The orchestrator settles child results and
152completion deliveries in a startup pass after reconciliation, because the terminal-run
153listener ignores reconciliation's cancellations. A cancelled child whose restart
154continuation is still pending in the outbox is not a result yet; the continuation's run
155settles it, or the handler settles it when it declines to continue. Schedulers wait for
156activation so they cannot start runs that reconciliation would then cancel.