eaa6116cb2
A self-deployment replaced the running instance's runtime directory
underneath it, and for the thirty seconds until the restart that
instance could not start a single child process:
Cannot run program "git": Failed to exec spawn helper
The JVM resolves jspawnhelper under its own java.home, which is a path
inside the installed runtime, and the deployment had just put a
different runtime there. All six repositories reported their origin as
unreachable at that instant, and the metrics sample of that moment was
wrong for a month afterwards (PR#63 stopped that consequence; this is
its cause).
The build therefore installs nothing now. It stages the new runtime as
werkator.new and proves it starts here, and the transient unit does the
rest in the order that matters: stop, swap, start. The JVM being
replaced never sees its own runtime change.
The fixed delay goes with it. Thirty seconds was a guess at when the
deploy build would be recorded — too short would recover it as
interrupted and deploy again, so it was chosen long. The unit now asks
the instance instead: a build leaves the running list only after its
status and artifacts are written, so an idle instance is the condition
the delay was guessing at, and it is usually true within seconds. After
ten minutes it restarts regardless and says so.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Co-authored-by: mhoennig <michael@hoennig.de>
Reviewed-on: #64
163 lines
9.4 KiB
Markdown
163 lines
9.4 KiB
Markdown
> **WARNING:** This document describes only the change applied in this PR.
|
|
> It may already be outdated once the next PR is merged.
|
|
> Historic PR-documentation is not maintained along with new PRs — treat it as a snapshot, not as current documentation.
|
|
|
|
## Related Links
|
|
|
|
- [RFC 0005: Werkator Deploys Itself](../rfcs/0005-werkator-deploys-itself.md) — this PR repairs the step 1 it describes.
|
|
- [Self-Deployment](../deployment.md#self-deployment) — the whole mechanism, updated by this PR.
|
|
|
|
## The Problem
|
|
|
|
A self-deployment replaced the running instance's runtime directory while that instance was running, and for the thirty seconds until the restart the instance could not start a single child process.
|
|
|
|
The symptom was six repositories reporting their origin as unreachable at the same instant:
|
|
|
|
```
|
|
werkator: Cannot run program "git" (in directory "…/werkator"):
|
|
error=0, Failed to exec spawn helper: pid: 2519163, exit value: 1
|
|
```
|
|
|
|
`jspawnhelper` is the small program a Java virtual machine executes to spawn anything at all.
|
|
The JVM resolves it under its own `java.home`, which is a path inside the installed runtime — and the deployment had just put a different runtime at that path.
|
|
The old process then executed the new runtime's helper, the handshake between the two failed, and every `ProcessBuilder` in the instance failed with it.
|
|
That is the watcher's `git fetch`, a build's command, and anything else Werkator shells out to.
|
|
|
|
It was not a harmless half minute.
|
|
The metrics collector samples in that window too, its `quota` call could not run, and the disk measurement silently fell back to the whole shared volume: a peak of 447 GiB against a 20 GiB quota, which then recommended an upgrade for thirty days.
|
|
That second defect is fixed in PR#63; this one is the reason it was triggered at all, and it would keep triggering things.
|
|
|
|
The delay itself was a guess.
|
|
PR#61 chose thirty seconds so that the deploy build would be recorded `SUCCESS` before the service went down — a build interrupted before that is recovered as `INTERRUPTED`, re-enqueued, and deploys again, which is the loop the whole design avoids.
|
|
Thirty seconds was picked because too short is the dangerous direction, not because anything measured it.
|
|
Both observed deployments recorded their deploy build in seven seconds.
|
|
|
|
## Non-Goals
|
|
|
|
- **No executor change.** Werkator still does not know that a build restarts it; asking the instance whether it is busy is done from outside. Making the executor drain its own queue is [step 2 of RFC 0005](../rfcs/0005-werkator-deploys-itself.md#step-2--restartsinstance-as-a-build-setting-option-b).
|
|
- **No change to what is verified.** The new runtime is still started natively before anything is installed, and still in the build, so a bundle that cannot run here still costs a red build and no downtime.
|
|
- **No shorter outage.** The service is down for the same handful of seconds it always was; what changes is what the instance can do while it is *up*.
|
|
|
|
## The Scenarios
|
|
|
|
### Feature: the running instance keeps its own runtime
|
|
|
|
#### Background
|
|
|
|
- The **staged runtime** is `werkator.new` beside the installed one; the build creates and verifies it and installs nothing.
|
|
- The **transient unit** `werkator-self-restart-<repo-name>` is started by the user manager, so stopping the service does not end it.
|
|
- A build leaves `running` in `GET /api/repos` only after its status and its artifacts are written — an invariant of `BuildExecutor`, commented there as one.
|
|
|
|
#### Scenario#64.01: The runtime is replaced only while the service is down
|
|
|
|
So that a deployment cannot take away the running instance's ability to start programs.
|
|
|
|
- **Given** a deploy build that staged and verified a new runtime
|
|
- **When** the transient unit takes over
|
|
- **Then** it stops the service first, swaps second, and starts third
|
|
- **and** the instance that was running never sees its own runtime directory change
|
|
- **and** the replaced runtime is kept as `werkator.prev`
|
|
|
|
##### Verified by
|
|
|
|
- Not verifiable by a test; the failure it removes needs two runtimes, a systemd user manager and a running instance. The swap and its verification are exercised by hand, recorded in [Manual verification](#manual-verification), and the whole sequence by the deploy build of this branch's merge, see [Open Questions](#open-questions).
|
|
|
|
#### Scenario#64.02: The restart waits for the deployment to be recorded
|
|
|
|
So that a restart never turns the deploy build into an interrupted one that deploys again.
|
|
|
|
- **Given** the transient unit has come up while the deploy build is still finishing
|
|
- **When** it asks the instance what is building
|
|
- **Then** it waits while any repository reports a running or queued build
|
|
- **and** it proceeds as soon as none does, which is when the deploy build is recorded with its artifacts
|
|
- **and** it proceeds anyway after ten minutes, saying so in the journal, rather than leaving a staged runtime uninstalled
|
|
|
|
##### Verified by
|
|
|
|
- The condition itself, checked against the production instance's own answer and against constructed ones, in [Manual verification](#manual-verification).
|
|
|
|
#### Scenario#64.03: A staged runtime that does not start is never installed
|
|
|
|
So that the verification keeps its meaning after the swap moved.
|
|
|
|
- **Given** a bundle whose launcher fails on this host
|
|
- **When** the build stages and verifies it
|
|
- **Then** the staged directory is removed, the build fails, and no transient unit is scheduled
|
|
- **and** the installation is exactly as it was
|
|
|
|
##### Verified by
|
|
|
|
- Manual run, recorded in [Manual verification](#manual-verification). This is the production incident of PR#62 in its new shape: there it was caught before the swap, here there is no swap to be caught before.
|
|
|
|
## The Solution
|
|
|
|
`stage_and_swap` becomes two: `stage_and_verify`, which the build runs, and `swap_staged`, which the transient unit runs between the stop and the start.
|
|
|
|
The build therefore installs nothing.
|
|
It leaves `werkator.new` and a scheduled unit, and its own success means only that the new runtime exists and starts here.
|
|
|
|
The unit's sequence is: wait for idle, stop, swap, start, health check, roll back if the instance stays silent.
|
|
Waiting for idle replaces the fixed delay with the condition the delay was guessing at.
|
|
`GET /api/repos` reports running and queued builds for every served repository, including a private one — the counts are metadata, not content — and an instance with none of them has recorded whatever it was doing, artifacts included.
|
|
An unreachable instance counts as idle, because an instance that is not running is running no builds.
|
|
|
|
`tools/remote` keeps its behaviour: it owns the stop and the start, so its path stages, verifies and swaps in one go, as before.
|
|
|
|
## Open Questions
|
|
|
|
- **The sequence is proven by the merge of this branch** and by nothing before it: the deploy build runs only on `main`. The run belongs here afterwards, with its durations and the journal of the transient unit — which should this time show `git` working throughout.
|
|
- **The race the idle check cannot close**: a build that starts between the answer and the stop is still interrupted. The instance is asked, not told to hold, and telling it is step 2 of RFC 0005.
|
|
- **Ten minutes is a bound, not a measurement.** No deployment has come close to it; it exists so a busy instance cannot leave a verified runtime uninstalled forever.
|
|
|
|
## Additional Changes
|
|
|
|
- The release notes for v1.14.3 describe both halves in the terms a reader of the release page needs.
|
|
|
|
## Prerequisite PRs
|
|
|
|
- PR#61, which introduced the self-deployment, and PR#63, which stopped the disk measurement from turning this window into a month of false advice.
|
|
|
|
## Follow-up PRs
|
|
|
|
- [Step 2 of RFC 0005](../rfcs/0005-werkator-deploys-itself.md#step-2--restartsinstance-as-a-build-setting-option-b): `restartsInstance` as a pinned build setting, so the executor stops taking work itself instead of being asked from outside.
|
|
- Sampling `/proc/pressure/io`, and recording the CPU time a build consumed beside its wall-clock duration — both decided on 2026-09-14, both still open.
|
|
|
|
## Attachments
|
|
|
|
### Manual verification
|
|
|
|
`tools/self-deploy` has no automated tests; the repository has no harness for shell scripts.
|
|
`shellcheck` reports nothing for it.
|
|
|
|
The path `tools/remote` uses — stage, verify and swap in one go — against a prepared installation directory:
|
|
|
|
```
|
|
==> Verifying the staged runtime natively
|
|
Werkator 9.9.9 (neu)
|
|
==> Swapping in the staged runtime (the replaced one stays as werkator.prev)
|
|
```
|
|
|
|
afterwards the installation holds `werkator` and `werkator.prev`, and the installed launcher reports the new version.
|
|
|
|
The same with a launcher that fails ([Scenario#64.03](#scenario6403-a-staged-runtime-that-does-not-start-is-never-installed)):
|
|
|
|
```
|
|
boom
|
|
ERROR: the staged runtime does not run on this host — nothing was swapped
|
|
```
|
|
|
|
afterwards the installed launcher still reports the version it had.
|
|
|
|
The idle condition ([Scenario#64.02](#scenario6402-the-restart-waits-for-the-deployment-to-be-recorded)), against the production instance's own answer and against constructed ones:
|
|
|
|
| input | verdict |
|
|
| --- | --- |
|
|
| the live answer from the production instance, nothing building | idle |
|
|
| one repository with `"running":1` | busy |
|
|
| one repository with `"pending":2` | busy |
|
|
| one repository with `"running":12` | busy |
|
|
| every count zero | idle |
|
|
| every count zero, but a repository named `repo1` | idle |
|
|
|
|
The last row is the trap the check has to survive: a digit in a repository's name must not read as a build.
|