Open History to see the last commit
1# Server updates
2
3A [stable launcher](../../apps/server/src/serviceLauncher.ts) owns the runtime
4selected by systemd, launchd, or Task Scheduler. It is the only runtime writer of durable service
5state. Server children request updates over inherited IPC; they never rewrite
6their service definition or select their own replacement. Local service commands
7may replace the launcher and state, optionally deferring the restart. Foreground
8CLI processes do not self-update.
9
10Both writers serialize state and unit writes through
11[file leases](../../apps/server/src/cloud/serviceSelectionLock.ts). The installer
12takes the state lease before the unit lease; the launcher never holds both, so a
13unit failure cannot block a durable commit. Manager stop/start commands run outside
14both leases, since stopping a launcher that waits on a lease would deadlock. A
15running launcher refuses further updates once a local command changes its selection.
16A protocol-3 launcher takes no lease, so a local command can still race its updates.
17
18Exact-version installs keep restarts independent of npm cache eviction or a moving
19release tag. Installation and preflight happen in staging before publishing an
20immutable runtime. Preflight checks the launcher protocol because a target that
21needs new rollback guarantees cannot safely run under an older launcher. A remote
22update replaces only the child, so upgrading the launcher requires a local service
23update with a restart. Every installed launcher is therefore one protocol behind
24after a bump, and a target that refuses the previous protocol stops remote updates
25for every existing service. Refuse it only when the child cannot work under it.
26Protocol 4 changed the launcher alone, so its runtime still runs under protocol 3.
27The server names its launcher's protocol to preflight, not its own, since the two differ
28after the first such update.
29
30After a commit or rollback the launcher points the unit at the selected runtime and
31reloads the manager, so the next service start runs that version's launcher; the
32protocol check is what makes that safe. Each start after a finished update retries this, because
33a previous reload may have failed, and leaves a unit naming a newer runtime or another
34T3 home alone. A protocol-3 launcher does none of this: its unit keeps naming the
35runtime that installed it. Until a deferred local upgrade restarts, systemd (before
36daemon-reload) and launchd (before bootstrap) can respawn the old launcher, which
37rejects the state the new installer wrote.
38
39Task Scheduler tracks only the console host it started, not the launcher. `schtasks /End`
40kills that host and nothing else; the launcher takes its closed console as SIGTERM. The task
41reads as stopped at once, and `/Run` then starts a second launcher beside the first, so a
42Windows launcher holds a lock file for as long as it runs. A replacement waits for that lock,
43and the CLI treats its release as the stop having finished.
44
45Task Scheduler also runs its own copy of the task, and registering one enables it. So every
46write of the task file registers it, including an install that defers its restart. The CLI
47disables the task under the unit lease, and a registration under that lease keeps a disabled
48task disabled, so the repeating trigger cannot restart a service the CLI is holding stopped.
49
50## Commit boundary
51
52The launcher durably records the pending update before acknowledging it, then
53stops the old child and starts the target as a trial. Service-state writes use
54same-directory replacement with file and directory fsync. Invalid state stops
55startup rather than guessing which runtime to boot.
56
57The trial must finish migrations, acquire dependencies, bind HTTP, and park every
58long-running root at the activation gate before reporting `prepared`. The launcher
59then commits the target version durably and replies `committed`. Only then may the
60child release its gates, accept commands, and publish ready. Keep fallible startup
61acquisitions before this boundary. A listener alone does not prove the runtime is
62ready to commit.
63
64A failed or timed-out trial returns to the old version. After commit, the target
65is authoritative and the service manager's ordinary restart policy applies.
66
67The prepared deadline is long because migrations can scale with the database
68and block the trial's main thread, so it cannot report progress. A missed
69deadline restores the snapshot, and the retry pays the same cost again. A
70crashed trial is caught when it exits; the deadline bounds only a stuck one.
71Client update waits must outlast the deadline, or a slow update shows as failed.
72
73## Database rollback
74
75Before stopping the old child, the launcher refuses an update whose snapshot will
76not fit on disk, so a full disk does not cause an outage. After the old child
77exits, the launcher snapshots SQLite's main file, WAL, and shared-memory file. This makes trial migrations reversible without down
78migrations. The snapshot is made once per update and survives launcher restarts;
79replacing it during a retry could capture changes from the failed trial.
80
81Rollback stops the trial before restoring. A durable restore marker makes an
82interrupted restore finish before either version boots. Keep the snapshot until
83commit, or until both restoration and the terminal rollback state are durable.
84Attachments and other files outside SQLite are outside this rollback boundary.
85
86## Client acknowledgement
87
88An accepted update is still pending. Clients correlate the launcher's update ID
89with the ready event after reconnecting, then check the outcome and target version.
90A reconnect alone cannot distinguish successful replacement from rollback. Older
91servers without an update ID retain version-only correlation.
92
93Desktop updates have a separate two-phase handoff because installing the app stops
94its bundled backend. Preparation returns a token while the connection is alive;
95the client commits that token only after receiving it. Otherwise backend shutdown
96could lose the only successful RPC result. The client must then observe the
97prepared version after reconnecting. If installation fails, desktop restarts the
98stopped backends and replays the failure for the same token.
99
100## Recovering interrupted threads
101
102Restart continuation is an environment-owned preference, on by default. A
103`settings.json` that fails to decode turns it off, since the file may hold an
104opt-out. The
105[v2 recovery service](../../apps/server/src/orchestration-v2/ProviderRuntimeRecoveryService.ts)
106requires matching durable run, provider thread, session, and native resume identity.
107Queued runs never started, so recovery holds them and continues the run they wait
108behind. A settled run qualifies only when the restart cancels work the agent waits on
109(a subagent, a background task, or a shell or monitor the agent marked with
110`t3_background_wait`) and its session was live, so stale records from an older crash
111never wake it. Its continuation prompt is the note of what will not report back. An
112unmarked shell or monitor does not wake a settled run: most are dev servers and their
113watchdogs that nobody waits on, so that note waits for the next user message.
114
115Recovery retires effects tied to the lost process and records continuation intent
116in the durable outbox. That intent survives another restart before provider startup.
117Continuation effects wait for activation; a slow provider must not delay the server's
118readiness or the launcher's commit boundary. Graceful shutdown captures intent before
119closing providers, then reconciles after ingestion has stopped so a late completion
120cannot be overwritten by a stale cancellation.
121
122The [continuation handler](../../apps/server/src/orchestration-v2/RestartContinuation.ts)
123rechecks the preference, provider selection, newer user work, a stop the user requested,
124and maintenance turns such as `/compact` before dispatching. Intent and delivery both
125refuse a thread that is archived, settled, or still snoozed, a cut turn whose provider
126turn already ended, and a run waiting on the user: an approval, question, or secret that was pending,
127or that recovery closed on the cut turn. Message-capable questions stay answerable, so
128they do not block. Delivery also stops after three consecutive continuations in one
129`restartContinuationOfRunId` chain, so a turn that crashes the server cannot loop. Stable
130command and message IDs prevent duplicate submissions after an outbox retry. Codex
131resumes without adding provider prompt text unless the turn lost background work;
132other adapters receive the continuation message through their normal turn path.
133
134Delegated tasks (`delegate_task` child threads) are reconciled as their own threads,
135never as the parent's background work. The orchestrator settles child results and
136completion deliveries in a startup pass after reconciliation, because the terminal-run
137listener ignores reconciliation's cancellations. A cancelled child whose restart
138continuation is still pending in the outbox is not a result yet; the continuation's run
139settles it, or the handler settles it when it declines to continue. Schedulers wait for
140activation so they cannot start runs that reconciliation would then cancel.