werkator/docs/prs/2026-09-15-PR#65-the-instance-ends-itself.md

WARNING: This document describes only the change applied in this PR. It may already be outdated once the next PR is merged. Historic PR-documentation is not maintained along with new PRs — treat it as a snapshot, not as current documentation.

Related Links

• RFC 0005 — Werkator deploys itself, step 2
• PR#61 — step 1, the deployment itself
• PR#64 — the swap moved into the moment the service is down

The Problem

Since PR#61 Werkator installs its own merged commits, and since PR#64 the swap happens while the service is down. What still decides when that moment has come is a script outside the instance, and it decides by asking. The transient unit polls /api/repos until the running and pending counts are zero, and then stops the service.

Asking is not the same as telling. A build that starts between the answer and the stop is cut short, recorded INTERRUPTED, and re-run by the startup recovery after the restart. On mih09 with executor.maxConcurrent: 1 the window is small and the cost is one repeated build, which is why step 1 accepted it. It is still a race, and it is one the instance itself never has to lose: it knows what is executing, and it is the only party that does.

The second half of the problem is that the outside view is coarse. "Nothing is running" is the closest a caller can get to "the deployment is safely recorded", and it happens to be true because a build leaves the running list only after its status and its artifacts are written. That invariant is load-bearing for a script that cannot see it.

Non-Goals

• The rollback stays outside Werkator, in the transient systemd unit: an instance that does not start cannot put the previous runtime back itself.
• The swap stays outside too, for the reason PR#64 established — a JVM must not have its own runtime replaced underneath it, so someone who is not that JVM has to do it.
• No self-deployment on a host without a systemd user manager; that is still out of scope and still fails with a message saying so.
• The two measurements RFC 0005 and PR#64 left open — IO wait time, and CPU time per build — are not part of this PR.

The Scenarios

Feature: a build that replaces the instance running it

Background

• restartsInstance: true on a build definition says that this build installs the version the instance is to run next.
• The key is pinned like the sandbox policy: stripped from the branch layer and written back from the host's definition of the same name, or from the host's builds.default.
• Draining is the state the executor enters after such a build succeeds: no further build starts, everything still executing is finished and recorded, and then the server context is closed.

Scenario#65.01: The merge after this one deploys itself without being stopped

So that a deployment never cuts short a build that started a moment earlier.

• Given mih09's instance fragment marks deploy with restartsInstance: true
• When a commit is merged to main and the deploy build succeeds
• Then the instance drains and ends itself
  • and the transient unit swaps the staged runtime in and starts the new version
  • and its journal shows that it waited for the instance to end rather than stopping it

Verified by

• recorded from the production run on mih09, see Attachments.

Scenario#65.02: The restart is asked for only after the result is recorded

So that the restart cannot turn the deployment into an interrupted build that deploys all over again.

• Given a build definition with restartsInstance: true and a trigger.branches selector
• When a run of it ends SUCCESS
• Then its result is persisted and published before anything closes
  • and the build has left the executor's current builds when the restart is requested

Verified by

• BuildExecutorTest — a build that restarts the instance asks for the restart only after its result is recorded

Scenario#65.03: The deploying build knows that the instance restarts itself

So that a deployment script need not check the version of the Werkator it runs under.

• Given the same definition
• When its build command runs
• Then WERKATOR_RESTARTS_INSTANCE=true is in its environment

Verified by

• BuildExecutorTest — the deploying build sees WERKATOR_RESTARTS_INSTANCE in its environment

Scenario#65.04: A build still executing is finished, not interrupted

So that a deployment costs no other build's work.

• Given another build is executing when the deployment ends SUCCESS
• When the executor drains
• Then nothing is terminated and no restart is requested yet
  • and the other build finishes and is recorded
  • and only then is the restart requested

Verified by

• BuildExecutorTest — a build still executing when the deployment ends is finished and recorded before the restart

Scenario#65.05: A queued build stays pending and starts nothing

So that the outgoing version does not begin work the incoming one will have to repeat.

• Given a build is queued when the deployment ends SUCCESS
• When the executor drains
• Then that build stays PENDING
  • and no workspace is prepared for it
  • and the startup recovery re-enqueues it after the restart

Verified by

• BuildExecutorTest — a build queued when the deployment ends stays pending and starts nothing

Scenario#65.06: A deployment that did not succeed restarts nothing

So that a failed deployment leaves the working version running.

• Given the same definition
• When its run ends FAILED (or timed out, cancelled, interrupted)
• Then the executor does not drain and no restart is requested

Verified by

• BuildExecutorTest — a deployment that fails restarts nothing

Scenario#65.07: A branch cannot end the instance

So that a push to a feature branch can never take the service down.

• Given a host that marks only its deploy build with restartsInstance: true
• When a branch's committed .werkator.yml sets restartsInstance: true on default and on a build it invented itself
• Then both are false — written back from the host's definition and from the host's builds.default
  • and the host's own deploy keeps the setting

Verified by

• ConfigLoaderTest — a branch cannot make any of its builds restart the instance

Scenario#65.08: A host configuration that would restart for any branch is refused

So that a restart never comes from whatever branch went green last.

• Given a configuration that sets restartsInstance: true on builds.default, or on a definition without trigger.branches
• When the configuration is loaded
• Then the start is refused with a message naming the definition and the key
• And given the same definition with branches: ["main"]
• Then it loads

Verified by

• ConfigLoaderTest — a host that would restart the instance for any branch is refused

Scenario#65.09: The pages say why nothing starts

So that a queue standing still is not read as an outage.

• Given the executor is draining
• When the UI asks /api/watcher
• Then the answer says restarting: true while the watcher itself reports no error
  • and the banner explains that a new version was installed and no build starts until the handover is done

Verified by

• WatcherApiControllerTest — a draining instance says it is restarting

The Solution

restartsInstance joins the pinned settings. It is a BranchConfig key with a nullable override in BuildDefinition, so it inherits like every setting, and adding it to PINNED_SETTING_KEYS gives it both halves of the pinning at once: stripped from the branch layer, written back from the host's layers after merging. The two refusals live in checkRestartsInstance, next to the follow-up checks and scoped the same way — only the host's own configuration is checked, because a branch layer cannot carry the key past the pinning anyway, and a fragment checked on its own may well find its selector in the configuration it is merged with later.

In the executor the state is one AtomicBoolean and one thread. beginDrain is called at the very end of a build's finally, after the terminal status, the artifacts, and the removal from builds — the order that was already an invariant now has a second reader. While draining, a worker that reaches its slot returns without starting anything and leaves its build PENDING, exactly as it does during a shutdown. The drain thread waits for the executing builds and then publishes InstanceRestartRequestedEvent.

The event, rather than a call, is what keeps the executor from having to know whether it runs inside a server at all. ServerRestartLifecycle listens in the server profile and closes the context from a thread of its own; ServerCommand.awaitShutdown already falls through a programmatic close, so main() returns and exitProcess stays where the invariant puts it. A CLI run publishes the event too and nobody listens, which is right: that process ends with its command.

tools/self-deploy learns nothing new about versions. The executor exports WERKATOR_RESTARTS_INSTANCE=true into the build's environment, and the script reads the service's MainPID while the instance is still the running one and hands it to the transient unit, which waits for that pid to go instead of polling for idleness. The systemctl stop afterwards stays, and its job has changed: it is no longer what ends the instance, it is what keeps Restart=always from bringing the old runtime back into the middle of the swap. RestartSec=30 is the gap the swap fits into. Against an instance without the setting the script falls back to the old condition, so the two versions can deploy each other in either direction.

Open Questions

• The drain has a bound of ten minutes, after which it closes the context regardless and says so in the log; a build still executing is then interrupted like in any shutdown. A build that outlives its own executor.buildTimeout is the only way to reach it.
• The watcher keeps polling during the drain and may enqueue builds that stay PENDING. They cost a recovery run after the restart and nothing else, so stopping the watcher first would be more machinery than it saves.

Additional Changes

• AGENTS.md names the new pinned key in both places the pinned set is listed.

Prerequisite PRs

• PR#61, PR#62, PR#64

Follow-up PRs

• Record the CPU time a build consumed beside its wall-clock duration — the number that would have made the spawn outage of PR#64 show itself.
• Sample /proc/pressure/io beside CPU and memory pressure, in the spirit of RFC 0004: collect first, judge later.

Attachments

The production run on mih09

The host fragment gained restartsInstance: true after this PR was merged, applied with init --apply. This PR's own merge (2944db7) was still deployed by step 1's mechanism, as were d48feb3 and 008cd3c. The first deployment under step 2 was 762afb5 (v1.16.1), on 2026-09-15 at 13:31 local time: the default build took 105 s, the deploy build 7 s.

The instance's own log shows the drain:

13:31:56  BuildExecutor  build deploy of branch main restarts this instance; starting no further build
13:31:56  BuildExecutor  nothing is executing any more; closing the context so the new version starts
13:31:56  ServerRestartLifecycle  build deploy of werkator deployed a new version; shutting down so it starts

The transient unit's journal shows what it did with that, and it is the line that separates step 2 from step 1:

13:32:24  Started werkator-self-restart-werkator.service … --self-restart-pid 2667639
13:32:24  the instance ended itself after 0s (was pid 2667639)
13:32:24  stopping werkator-werkator.service to swap in the staged runtime
13:32:24  ==> Swapping in the staged runtime (the replaced one stays as werkator.prev)
13:32:24  starting werkator-werkator.service into the newly installed runtime
13:32:29  the new runtime answers on port 18089

Every earlier deployment began with stopping … to swap in the staged runtime; this one begins with the instance having ended itself. Zero seconds of waiting, because the drain took a millisecond and the transient unit only fires five seconds after the build. The stopping that follows ends nothing — it keeps Restart=always from bringing the old runtime back into the middle of the swap.

No Failed to exec spawn helper in the service log, and the installation carries werkator at v1.16.1 with werkator.prev beside it.