eaa6116cb2
A self-deployment replaced the running instance's runtime directory
underneath it, and for the thirty seconds until the restart that
instance could not start a single child process:
Cannot run program "git": Failed to exec spawn helper
The JVM resolves jspawnhelper under its own java.home, which is a path
inside the installed runtime, and the deployment had just put a
different runtime there. All six repositories reported their origin as
unreachable at that instant, and the metrics sample of that moment was
wrong for a month afterwards (PR#63 stopped that consequence; this is
its cause).
The build therefore installs nothing now. It stages the new runtime as
werkator.new and proves it starts here, and the transient unit does the
rest in the order that matters: stop, swap, start. The JVM being
replaced never sees its own runtime change.
The fixed delay goes with it. Thirty seconds was a guess at when the
deploy build would be recorded — too short would recover it as
interrupted and deploy again, so it was chosen long. The unit now asks
the instance instead: a build leaves the running list only after its
status and artifacts are written, so an idle instance is the condition
the delay was guessing at, and it is usually true within seconds. After
ten minutes it restarts regardless and says so.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Co-authored-by: mhoennig <michael@hoennig.de>
Reviewed-on: #64
9.4 KiB
WARNING: This document describes only the change applied in this PR. It may already be outdated once the next PR is merged. Historic PR-documentation is not maintained along with new PRs — treat it as a snapshot, not as current documentation.
Related Links
- RFC 0005: Werkator Deploys Itself — this PR repairs the step 1 it describes.
- Self-Deployment — the whole mechanism, updated by this PR.
The Problem
A self-deployment replaced the running instance's runtime directory while that instance was running, and for the thirty seconds until the restart the instance could not start a single child process.
The symptom was six repositories reporting their origin as unreachable at the same instant:
werkator: Cannot run program "git" (in directory "…/werkator"):
error=0, Failed to exec spawn helper: pid: 2519163, exit value: 1
jspawnhelper is the small program a Java virtual machine executes to spawn anything at all.
The JVM resolves it under its own java.home, which is a path inside the installed runtime — and the deployment had just put a different runtime at that path.
The old process then executed the new runtime's helper, the handshake between the two failed, and every ProcessBuilder in the instance failed with it.
That is the watcher's git fetch, a build's command, and anything else Werkator shells out to.
It was not a harmless half minute.
The metrics collector samples in that window too, its quota call could not run, and the disk measurement silently fell back to the whole shared volume: a peak of 447 GiB against a 20 GiB quota, which then recommended an upgrade for thirty days.
That second defect is fixed in PR#63; this one is the reason it was triggered at all, and it would keep triggering things.
The delay itself was a guess.
PR#61 chose thirty seconds so that the deploy build would be recorded SUCCESS before the service went down — a build interrupted before that is recovered as INTERRUPTED, re-enqueued, and deploys again, which is the loop the whole design avoids.
Thirty seconds was picked because too short is the dangerous direction, not because anything measured it.
Both observed deployments recorded their deploy build in seven seconds.
Non-Goals
- No executor change. Werkator still does not know that a build restarts it; asking the instance whether it is busy is done from outside. Making the executor drain its own queue is step 2 of RFC 0005.
- No change to what is verified. The new runtime is still started natively before anything is installed, and still in the build, so a bundle that cannot run here still costs a red build and no downtime.
- No shorter outage. The service is down for the same handful of seconds it always was; what changes is what the instance can do while it is up.
The Scenarios
Feature: the running instance keeps its own runtime
Background
- The staged runtime is
werkator.newbeside the installed one; the build creates and verifies it and installs nothing. - The transient unit
werkator-self-restart-<repo-name>is started by the user manager, so stopping the service does not end it. - A build leaves
runninginGET /api/reposonly after its status and its artifacts are written — an invariant ofBuildExecutor, commented there as one.
Scenario#64.01: The runtime is replaced only while the service is down
So that a deployment cannot take away the running instance's ability to start programs.
- Given a deploy build that staged and verified a new runtime
- When the transient unit takes over
- Then it stops the service first, swaps second, and starts third
- and the instance that was running never sees its own runtime directory change
- and the replaced runtime is kept as
werkator.prev
Verified by
- Not verifiable by a test; the failure it removes needs two runtimes, a systemd user manager and a running instance. The swap and its verification are exercised by hand, recorded in Manual verification, and the whole sequence by the deploy build of this branch's merge, see Open Questions.
Scenario#64.02: The restart waits for the deployment to be recorded
So that a restart never turns the deploy build into an interrupted one that deploys again.
- Given the transient unit has come up while the deploy build is still finishing
- When it asks the instance what is building
- Then it waits while any repository reports a running or queued build
- and it proceeds as soon as none does, which is when the deploy build is recorded with its artifacts
- and it proceeds anyway after ten minutes, saying so in the journal, rather than leaving a staged runtime uninstalled
Verified by
- The condition itself, checked against the production instance's own answer and against constructed ones, in Manual verification.
Scenario#64.03: A staged runtime that does not start is never installed
So that the verification keeps its meaning after the swap moved.
- Given a bundle whose launcher fails on this host
- When the build stages and verifies it
- Then the staged directory is removed, the build fails, and no transient unit is scheduled
- and the installation is exactly as it was
Verified by
- Manual run, recorded in Manual verification. This is the production incident of PR#62 in its new shape: there it was caught before the swap, here there is no swap to be caught before.
The Solution
stage_and_swap becomes two: stage_and_verify, which the build runs, and swap_staged, which the transient unit runs between the stop and the start.
The build therefore installs nothing.
It leaves werkator.new and a scheduled unit, and its own success means only that the new runtime exists and starts here.
The unit's sequence is: wait for idle, stop, swap, start, health check, roll back if the instance stays silent.
Waiting for idle replaces the fixed delay with the condition the delay was guessing at.
GET /api/repos reports running and queued builds for every served repository, including a private one — the counts are metadata, not content — and an instance with none of them has recorded whatever it was doing, artifacts included.
An unreachable instance counts as idle, because an instance that is not running is running no builds.
tools/remote keeps its behaviour: it owns the stop and the start, so its path stages, verifies and swaps in one go, as before.
Open Questions
- The sequence is proven by the merge of this branch and by nothing before it: the deploy build runs only on
main. The run belongs here afterwards, with its durations and the journal of the transient unit — which should this time showgitworking throughout. - The race the idle check cannot close: a build that starts between the answer and the stop is still interrupted. The instance is asked, not told to hold, and telling it is step 2 of RFC 0005.
- Ten minutes is a bound, not a measurement. No deployment has come close to it; it exists so a busy instance cannot leave a verified runtime uninstalled forever.
Additional Changes
- The release notes for v1.14.3 describe both halves in the terms a reader of the release page needs.
Prerequisite PRs
- PR#61, which introduced the self-deployment, and PR#63, which stopped the disk measurement from turning this window into a month of false advice.
Follow-up PRs
- Step 2 of RFC 0005:
restartsInstanceas a pinned build setting, so the executor stops taking work itself instead of being asked from outside. - Sampling
/proc/pressure/io, and recording the CPU time a build consumed beside its wall-clock duration — both decided on 2026-09-14, both still open.
Attachments
Manual verification
tools/self-deploy has no automated tests; the repository has no harness for shell scripts.
shellcheck reports nothing for it.
The path tools/remote uses — stage, verify and swap in one go — against a prepared installation directory:
==> Verifying the staged runtime natively
Werkator 9.9.9 (neu)
==> Swapping in the staged runtime (the replaced one stays as werkator.prev)
afterwards the installation holds werkator and werkator.prev, and the installed launcher reports the new version.
The same with a launcher that fails (Scenario#64.03):
boom
ERROR: the staged runtime does not run on this host — nothing was swapped
afterwards the installed launcher still reports the version it had.
The idle condition (Scenario#64.02), against the production instance's own answer and against constructed ones:
| input | verdict |
|---|---|
| the live answer from the production instance, nothing building | idle |
one repository with "running":1 |
busy |
one repository with "pending":2 |
busy |
one repository with "running":12 |
busy |
| every count zero | idle |
every count zero, but a repository named repo1 |
idle |
The last row is the trap the check has to survive: a digit in a repository's name must not read as a build.