5e4209553a
The third of the kernel's pressure-stall figures, /proc/pressure/io, beside the two already sampled. It is the one resource a build spends its time on that neither the processor nor the memory figure explains. Collected, kept in the history, shown on the System page and the home rail -- and read by nothing: no health verdict, no sizing advice, no new threshold. That is RFC 0004's order, the one oomKills was added under: the series has to exist before anyone can say what a number means. On a shared hive the disks are what is shared hardest, so the figure will say as much about the neighbours as about this instance until it has been watched for a while. Also fills the placeholder in PR#65's doc with the production run it promised: the journal of the first deployment in which the instance ended itself. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Co-authored-by: mhoennig <michael@hoennig.de> Reviewed-on: #69
233 lines
13 KiB
Markdown
233 lines
13 KiB
Markdown
> **WARNING:** This document describes only the change applied in this PR.
|
|
> It may already be outdated once the next PR is merged.
|
|
> Historic PR-documentation is not maintained along with new PRs — treat it as a snapshot, not as current documentation.
|
|
|
|
## Related Links
|
|
|
|
- [RFC 0005 — Werkator deploys itself](../rfcs/0005-werkator-deploys-itself.md), step 2
|
|
- [PR#61](2026-09-14-PR%2361-werkator-deploys-itself.md) — step 1, the deployment itself
|
|
- [PR#64](2026-09-15-PR%2364-swap-while-the-service-is-down.md) — the swap moved into the moment the service is down
|
|
|
|
## The Problem
|
|
|
|
Since PR#61 Werkator installs its own merged commits, and since PR#64 the swap happens while the service is down.
|
|
What still decides *when* that moment has come is a script outside the instance, and it decides by asking.
|
|
The transient unit polls `/api/repos` until the running and pending counts are zero, and then stops the service.
|
|
|
|
Asking is not the same as telling.
|
|
A build that starts between the answer and the stop is cut short, recorded `INTERRUPTED`, and re-run by the startup recovery after the restart.
|
|
On mih09 with `executor.maxConcurrent: 1` the window is small and the cost is one repeated build, which is why step 1 accepted it.
|
|
It is still a race, and it is one the instance itself never has to lose: it knows what is executing, and it is the only party that does.
|
|
|
|
The second half of the problem is that the outside view is coarse.
|
|
"Nothing is running" is the closest a caller can get to "the deployment is safely recorded", and it happens to be true because a build leaves the running list only after its status and its artifacts are written.
|
|
That invariant is load-bearing for a script that cannot see it.
|
|
|
|
## Non-Goals
|
|
|
|
- The rollback stays outside Werkator, in the transient systemd unit: an instance that does not start cannot put the previous runtime back itself.
|
|
- The swap stays outside too, for the reason PR#64 established — a JVM must not have its own runtime replaced underneath it, so someone who is not that JVM has to do it.
|
|
- No self-deployment on a host without a systemd user manager; that is still out of scope and still fails with a message saying so.
|
|
- The two measurements RFC 0005 and PR#64 left open — IO wait time, and CPU time per build — are not part of this PR.
|
|
|
|
## The Scenarios
|
|
|
|
### Feature: a build that replaces the instance running it
|
|
|
|
#### Background
|
|
|
|
- `restartsInstance: true` on a build definition says that this build installs the version the instance is to run next.
|
|
- The key is *pinned* like the sandbox policy: stripped from the branch layer and written back from the host's definition of the same name, or from the host's `builds.default`.
|
|
- *Draining* is the state the executor enters after such a build succeeds: no further build starts, everything still executing is finished and recorded, and then the server context is closed.
|
|
|
|
#### Scenario#65.01: The merge after this one deploys itself without being stopped
|
|
|
|
So that a deployment never cuts short a build that started a moment earlier.
|
|
|
|
- **Given** mih09's instance fragment marks `deploy` with `restartsInstance: true`
|
|
- **When** a commit is merged to `main` and the deploy build succeeds
|
|
- **Then** the instance drains and ends itself
|
|
- **and** the transient unit swaps the staged runtime in and starts the new version
|
|
- **and** its journal shows that it waited for the instance to end rather than stopping it
|
|
|
|
##### Verified by
|
|
|
|
- recorded from the production run on mih09, see [Attachments](#the-production-run-on-mih09).
|
|
|
|
#### Scenario#65.02: The restart is asked for only after the result is recorded
|
|
|
|
So that the restart cannot turn the deployment into an interrupted build that deploys all over again.
|
|
|
|
- **Given** a build definition with `restartsInstance: true` and a `trigger.branches` selector
|
|
- **When** a run of it ends `SUCCESS`
|
|
- **Then** its result is persisted and published before anything closes
|
|
- **and** the build has left the executor's current builds when the restart is requested
|
|
|
|
##### Verified by
|
|
|
|
- [BuildExecutorTest — a build that restarts the instance asks for the restart only after its result is recorded](../../src/test/kotlin/de/hoennig/werkator/build/BuildExecutorTest.kt)
|
|
|
|
#### Scenario#65.03: The deploying build knows that the instance restarts itself
|
|
|
|
So that a deployment script need not check the version of the Werkator it runs under.
|
|
|
|
- **Given** the same definition
|
|
- **When** its build command runs
|
|
- **Then** `WERKATOR_RESTARTS_INSTANCE=true` is in its environment
|
|
|
|
##### Verified by
|
|
|
|
- [BuildExecutorTest — the deploying build sees WERKATOR_RESTARTS_INSTANCE in its environment](../../src/test/kotlin/de/hoennig/werkator/build/BuildExecutorTest.kt)
|
|
|
|
#### Scenario#65.04: A build still executing is finished, not interrupted
|
|
|
|
So that a deployment costs no other build's work.
|
|
|
|
- **Given** another build is executing when the deployment ends `SUCCESS`
|
|
- **When** the executor drains
|
|
- **Then** nothing is terminated and no restart is requested yet
|
|
- **and** the other build finishes and is recorded
|
|
- **and** only then is the restart requested
|
|
|
|
##### Verified by
|
|
|
|
- [BuildExecutorTest — a build still executing when the deployment ends is finished and recorded before the restart](../../src/test/kotlin/de/hoennig/werkator/build/BuildExecutorTest.kt)
|
|
|
|
#### Scenario#65.05: A queued build stays pending and starts nothing
|
|
|
|
So that the outgoing version does not begin work the incoming one will have to repeat.
|
|
|
|
- **Given** a build is queued when the deployment ends `SUCCESS`
|
|
- **When** the executor drains
|
|
- **Then** that build stays `PENDING`
|
|
- **and** no workspace is prepared for it
|
|
- **and** the startup recovery re-enqueues it after the restart
|
|
|
|
##### Verified by
|
|
|
|
- [BuildExecutorTest — a build queued when the deployment ends stays pending and starts nothing](../../src/test/kotlin/de/hoennig/werkator/build/BuildExecutorTest.kt)
|
|
|
|
#### Scenario#65.06: A deployment that did not succeed restarts nothing
|
|
|
|
So that a failed deployment leaves the working version running.
|
|
|
|
- **Given** the same definition
|
|
- **When** its run ends `FAILED` (or timed out, cancelled, interrupted)
|
|
- **Then** the executor does not drain and no restart is requested
|
|
|
|
##### Verified by
|
|
|
|
- [BuildExecutorTest — a deployment that fails restarts nothing](../../src/test/kotlin/de/hoennig/werkator/build/BuildExecutorTest.kt)
|
|
|
|
#### Scenario#65.07: A branch cannot end the instance
|
|
|
|
So that a push to a feature branch can never take the service down.
|
|
|
|
- **Given** a host that marks only its `deploy` build with `restartsInstance: true`
|
|
- **When** a branch's committed `.werkator.yml` sets `restartsInstance: true` on `default` and on a build it invented itself
|
|
- **Then** both are `false` — written back from the host's definition and from the host's `builds.default`
|
|
- **and** the host's own `deploy` keeps the setting
|
|
|
|
##### Verified by
|
|
|
|
- [ConfigLoaderTest — a branch cannot make any of its builds restart the instance](../../src/test/kotlin/de/hoennig/werkator/config/ConfigLoaderTest.kt)
|
|
|
|
#### Scenario#65.08: A host configuration that would restart for any branch is refused
|
|
|
|
So that a restart never comes from whatever branch went green last.
|
|
|
|
- **Given** a configuration that sets `restartsInstance: true` on `builds.default`, or on a definition without `trigger.branches`
|
|
- **When** the configuration is loaded
|
|
- **Then** the start is refused with a message naming the definition and the key
|
|
- **And given** the same definition with `branches: ["main"]`
|
|
- **Then** it loads
|
|
|
|
##### Verified by
|
|
|
|
- [ConfigLoaderTest — a host that would restart the instance for any branch is refused](../../src/test/kotlin/de/hoennig/werkator/config/ConfigLoaderTest.kt)
|
|
|
|
#### Scenario#65.09: The pages say why nothing starts
|
|
|
|
So that a queue standing still is not read as an outage.
|
|
|
|
- **Given** the executor is draining
|
|
- **When** the UI asks `/api/watcher`
|
|
- **Then** the answer says `restarting: true` while the watcher itself reports no error
|
|
- **and** the banner explains that a new version was installed and no build starts until the handover is done
|
|
|
|
##### Verified by
|
|
|
|
- [WatcherApiControllerTest — a draining instance says it is restarting](../../src/test/kotlin/de/hoennig/werkator/server/WatcherApiControllerTest.kt)
|
|
|
|
## The Solution
|
|
|
|
`restartsInstance` joins the pinned settings.
|
|
It is a `BranchConfig` key with a nullable override in `BuildDefinition`, so it inherits like every setting, and adding it to `PINNED_SETTING_KEYS` gives it both halves of the pinning at once: stripped from the branch layer, written back from the host's layers after merging.
|
|
The two refusals live in `checkRestartsInstance`, next to the follow-up checks and scoped the same way — only the host's own configuration is checked, because a branch layer cannot carry the key past the pinning anyway, and a fragment checked on its own may well find its selector in the configuration it is merged with later.
|
|
|
|
In the executor the state is one `AtomicBoolean` and one thread.
|
|
`beginDrain` is called at the very end of a build's `finally`, after the terminal status, the artifacts, and the removal from `builds` — the order that was already an invariant now has a second reader.
|
|
While draining, a worker that reaches its slot returns without starting anything and leaves its build `PENDING`, exactly as it does during a shutdown.
|
|
The drain thread waits for the executing builds and then publishes `InstanceRestartRequestedEvent`.
|
|
|
|
The event, rather than a call, is what keeps the executor from having to know whether it runs inside a server at all.
|
|
`ServerRestartLifecycle` listens in the `server` profile and closes the context from a thread of its own; `ServerCommand.awaitShutdown` already falls through a programmatic close, so `main()` returns and `exitProcess` stays where the invariant puts it.
|
|
A CLI run publishes the event too and nobody listens, which is right: that process ends with its command.
|
|
|
|
`tools/self-deploy` learns nothing new about versions.
|
|
The executor exports `WERKATOR_RESTARTS_INSTANCE=true` into the build's environment, and the script reads the service's `MainPID` while the instance is still the running one and hands it to the transient unit, which waits for that pid to go instead of polling for idleness.
|
|
The `systemctl stop` afterwards stays, and its job has changed: it is no longer what ends the instance, it is what keeps `Restart=always` from bringing the old runtime back into the middle of the swap.
|
|
`RestartSec=30` is the gap the swap fits into.
|
|
Against an instance without the setting the script falls back to the old condition, so the two versions can deploy each other in either direction.
|
|
|
|
## Open Questions
|
|
|
|
- The drain has a bound of ten minutes, after which it closes the context regardless and says so in the log; a build still executing is then interrupted like in any shutdown. A build that outlives its own `executor.buildTimeout` is the only way to reach it.
|
|
- The watcher keeps polling during the drain and may enqueue builds that stay `PENDING`. They cost a recovery run after the restart and nothing else, so stopping the watcher first would be more machinery than it saves.
|
|
|
|
## Additional Changes
|
|
|
|
- `AGENTS.md` names the new pinned key in both places the pinned set is listed.
|
|
|
|
## Prerequisite PRs
|
|
|
|
- [PR#61](2026-09-14-PR%2361-werkator-deploys-itself.md), [PR#62](2026-09-15-PR%2362-a-runtime-the-host-can-start.md), [PR#64](2026-09-15-PR%2364-swap-while-the-service-is-down.md)
|
|
|
|
## Follow-up PRs
|
|
|
|
- Record the CPU time a build consumed beside its wall-clock duration — the number that would have made the spawn outage of PR#64 show itself.
|
|
- Sample `/proc/pressure/io` beside CPU and memory pressure, in the spirit of RFC 0004: collect first, judge later.
|
|
|
|
## Attachments
|
|
|
|
### The production run on mih09
|
|
|
|
The host fragment gained `restartsInstance: true` after this PR was merged, applied with `init --apply`.
|
|
This PR's own merge (`2944db7`) was still deployed by step 1's mechanism, as were `d48feb3` and `008cd3c`.
|
|
The first deployment under step 2 was `762afb5` (v1.16.1), on 2026-09-15 at 13:31 local time: the default build took 105 s, the deploy build 7 s.
|
|
|
|
The instance's own log shows the drain:
|
|
|
|
```
|
|
13:31:56 BuildExecutor build deploy of branch main restarts this instance; starting no further build
|
|
13:31:56 BuildExecutor nothing is executing any more; closing the context so the new version starts
|
|
13:31:56 ServerRestartLifecycle build deploy of werkator deployed a new version; shutting down so it starts
|
|
```
|
|
|
|
The transient unit's journal shows what it did with that, and it is the line that separates step 2 from step 1:
|
|
|
|
```
|
|
13:32:24 Started werkator-self-restart-werkator.service … --self-restart-pid 2667639
|
|
13:32:24 the instance ended itself after 0s (was pid 2667639)
|
|
13:32:24 stopping werkator-werkator.service to swap in the staged runtime
|
|
13:32:24 ==> Swapping in the staged runtime (the replaced one stays as werkator.prev)
|
|
13:32:24 starting werkator-werkator.service into the newly installed runtime
|
|
13:32:29 the new runtime answers on port 18089
|
|
```
|
|
|
|
Every earlier deployment began with `stopping … to swap in the staged runtime`; this one begins with the instance having ended itself.
|
|
Zero seconds of waiting, because the drain took a millisecond and the transient unit only fires five seconds after the build.
|
|
The `stopping` that follows ends nothing — it keeps `Restart=always` from bringing the old runtime back into the middle of the swap.
|
|
|
|
No `Failed to exec spawn helper` in the service log, and the installation carries `werkator` at v1.16.1 with `werkator.prev` beside it.
|