> **WARNING:** This document describes only the change applied in this PR. > It may already be outdated once the next PR is merged. > Historic PR-documentation is not maintained along with new PRs — treat it as a snapshot, not as current documentation. ## Related Links - [RFC 0005: Werkator Deploys Itself](../rfcs/0005-werkator-deploys-itself.md) — this PR is its step 1. - [Follow-up builds](../configuration.md#follow-up-builds) — the mechanism every other repository deploys with. ## The Problem Werkator is the one repository its own instance builds but cannot deploy. Every other repository on the production instance deploys with a follow-up build: `afterSuccessOf` a green build, at that build's commit, in that branch's worktree (PR#23). Werkator's rollout is still `tools/remote werkator instance-update` from a developer machine. So a merged commit runs on the instance when somebody remembers, and v1.13.1 sat on main for hours while the instance kept running v1.12.0 — with the very bug v1.13.0 had fixed, eating the history of every branch merged in between. A deployment definition would not work here, and not because restarting is hard. The deploy build is a child process of the running Werkator, and a restart has three effects on it: - **The build dies with the service.** `systemctl --user stop` ends the whole control group, and `BuildExecutor.shutdown()` destroys every executing process tree anyway and records the build as `INTERRUPTED`. - **It runs again after the restart.** `Watcher.recoverOnStartup` re-enqueues every `INTERRUPTED` build whose branch still exists. The re-run deploys again and restarts again: a loop, not a one-time glitch. - **The green status never lands.** Result persistence, the Gitea status and the artifacts all happen after the process ends, which never happens regularly. So the restart must be *caused* by the build without *happening inside* it. A second problem sat in the existing rollout. `swap_instance_runtime` moved the running runtime aside, unpacked the new one over it, and only then asked the new launcher for its version. A bundle whose jlinked JRE does not start on the host therefore left the installation with no working runtime at all, discovered after the service had already been stopped. ## Non-Goals - **No executor change.** Werkator learns nothing about restarting itself in this PR; the build ends green and systemd does the rest. Making the restart a property of the build definition is [step 2](../rfcs/0005-werkator-deploys-itself.md#step-2--restartsinstance-as-a-build-setting-option-b). - **No new configuration key.** The deployment is a follow-up build like any other; what makes it special is only that its command talks to systemd. - **No rollout in this PR.** Applying the host fragment on mih09 and running `instance-update` once more happens after the merge, and that manual run is the last one. - **No self-deployment on a host without a systemd user manager.** A Docker host with a system unit, or none at all, keeps deploying the way it does today; the script says so and fails. ## The Scenarios ### Feature: Werkator installs its own merged commits #### Background - The **deploy build** is a follow-up of the test build, restricted to `main`, running natively — it has to reach the installation directory, which the sandbox cannot. - The **bundle** is `build/runtime-bundle/werkator` (91 MiB), built by the test build, because only the sandbox has a JDK and therefore `jlink`. It reaches the deployment through the worktree the two builds share. `runtimeBundle` packs the same directory as a tarball for a transfer to another host, which is what `tools/remote` uses. - The **installation** is a runtime bundle unpacked at `/werkator/`, which a systemd user unit runs. `/werkator.prev` is the runtime it replaced: the last known good version. - The **transient unit** is `werkator-self-restart-`, scheduled with `systemd-run --user` and named after the service it restarts, since transient unit names are global to the user manager. It is started by the user manager, not by the build, so stopping the service does not end it. #### Scenario#61.01: A merged commit installs itself So that main reaches the instance without anybody remembering to deploy it. - **Given** the test build of `main` ended `SUCCESS` and left a runtime bundle in the worktree - **When** the deploy build runs `tools/self-deploy` - **Then** the bundle is staged, verified and swapped in, and the previous runtime is kept as `werkator.prev` - **and** the build is recorded `SUCCESS` and published to Gitea before the service goes down - **and** the transient unit restarts the service afterwards, so the startup recovery finds nothing to re-run - **and** the restarted instance reports the new version in its footer ##### Verified by - Not verifiable by a test: it needs the production instance deploying itself. To be verified on mih09 after the merge, see [Open Questions](#open-questions). #### Scenario#61.02: A deploy without its predecessor installs nothing So that a restart of the deployment alone cannot put an older version back. - **Given** the deploy build is restarted from the UI, so it runs without a preceding test build - **and** the worktree therefore holds no runtime bundle - **When** `tools/self-deploy` looks for it - **Then** it fails naming the missing bundle and the build that produces it - **and** the installation is untouched ##### Verified by - Manual run, recorded in [Manual verification of the script](#manual-verification-of-the-script). #### Scenario#61.03: A runtime that does not start here is never installed So that a bundle built for another host costs a red build and no downtime. - **Given** a bundle whose launcher fails on this host - **When** `tools/self-deploy` verifies the staged runtime natively - **Then** the staged directory is removed and nothing is swapped - **and** the running installation still answers with its own version - **and** the build fails ##### Verified by - Manual run, recorded in [Manual verification of the script](#manual-verification-of-the-script). #### Scenario#61.04: A new version that does not answer is rolled back So that a deployment cannot leave the instance down unattended. - **Given** the new runtime was installed and the service restarted - **When** the transient unit finds `http://127.0.0.1:/api/watcher` unanswered for two minutes - **Then** the new runtime is moved aside as `werkator.failed` and `werkator.prev` takes its place again - **and** the service is restarted once more - **and** the journal of `werkator-self-restart-` says which of the two happened ##### Verified by - Not verifiable by a test; the rollback path is exercised on mih09 only by an actually broken release. The staging half of it — nothing swapped when the runtime does not start — is [Scenario#61.03](#scenario6103-a-runtime-that-does-not-start-here-is-never-installed). #### Scenario#61.05: A branch cannot deploy itself So that a pinned deployment stays the host's decision. - **Given** the host defines `deploy` as a follow-up restricted to `main` - **When** a feature branch commits `builds.deploy.trigger.branches: ["*"]`, or adds `afterSuccessOf` to a build of its own - **Then** the host's trigger block is used unchanged and the branch is not selected - **and** the branch's own build gets no `afterSuccessOf` at all ##### Verified by - [ConfigLoaderTest: "a follow-up trigger is pinned to the host, a branch cannot add or widen one"](../../src/test/kotlin/de/hoennig/werkator/config/ConfigLoaderTest.kt) ## The Solution `tools/self-deploy` is the deploy build's command, and it restarts nothing itself. It takes no parameters; everything it needs it derives from where it runs. The primary checkout comes from `git rev-parse --git-common-dir`, the unit name from that directory's basename exactly as `SystemdServiceFiles.unitName` derives it, the installation directory from that unit's `ExecStart` — whose jar sits in `/werkator/lib/` — and the port from the installed binary's `config:print`. Deriving rather than configuring is what lets the same script serve a developer machine's rollout and the instance's own, with nothing to keep in sync. It refuses, before touching anything, to run in the primary checkout, to install into an instance that watches a different repository, to treat a plain-jar installation as a bundle, and to install when the worktree holds no bundle. The swap is staged. The bundle is put beside the running runtime as `werkator.new`, its launcher is run natively — the one check that catches a jlinked JRE that will not start here — and only then does it take the place of `werkator`, with the replaced one kept as `werkator.prev`. The restart is scheduled, not performed: `systemd-run --user --collect --on-active=30 --unit=werkator-self-restart-` runs a copy of the script that the deployment installs next to the runtime, so it does not depend on a worktree that the next build will reuse. That unit restarts the service, waits for `/api/watcher`, and rolls back to `werkator.prev` when it stays silent. Because it fires after the build has been recorded, the recovery has nothing to re-run, and the loop the RFC describes never starts. `tools/remote` now performs its swap by running the very same script on the host, in its swap-only form. That removes the second implementation of the layout, and it fixes the hole described above: `instance-update` no longer moves the running runtime aside before knowing that the new one starts. The repository's `.werkator.yml` carries what the deployment *does*; when, for which branches and outside the sandbox stays the host's, in the instance fragment, where it is pinned. ## Open Questions - **Scenario#61.01 is verified after the merge, not in this PR.** It needs the production instance to deploy itself, which cannot happen before this branch is on main and the host fragment carries the `deploy` block. The sequence is: `instance-update` once more by hand (the last manual one), `init --apply` with the fragment below, then the next merge to main deploys itself. The run is to be recorded here with its commit and the two build durations. - **The host fragment is not in this repository**, because it is the host's part. For mih09 it belongs in `.env.mih09.yml`: ```yaml builds: deploy: trigger: afterSuccessOf: default branches: ["main"] statusContext: werkator/deploy werkdock: enabled: false ``` - **The restart delay is fixed at 30 seconds**, so a build enqueued between the deployment and the restart is interrupted and re-run by the recovery. With `executor.maxConcurrent: 1` on mih09 nothing else runs while the deployment does, so this is a narrow window; step 2 closes it by draining the queue instead of waiting a fixed time. - **The test build got longer on every branch** by the bundle task, for a bundle only `main` installs: on the CI instance from around 95 to 130 seconds. Restricting it to `main` would mean a branch's own deployment could never be tested, and that testing has already paid off once, when it caught the missing `objcopy` before anything was merged. Leaving it on every branch is the deliberate choice; revisiting it after the first successful self-deployment is the obvious moment. ## Additional Changes - The restart delay is 30 seconds where [RFC 0005](../rfcs/0005-werkator-deploys-itself.md) drafted 10. Too short is the dangerous direction: a build not yet recorded `SUCCESS` when the service goes down is recovered as `INTERRUPTED` and deploys again, which is the loop this whole design exists to avoid. Thirty seconds leaves room for a slow artifact copy or a Gitea call waiting for its timeout. - The bundle task was split in two. `runtimeBundleDir` produces the directory, `runtimeBundle` packs that directory as a tarball. A self-deployment unpacks the bundle on the machine that built it, so compressing 64 MiB and decompressing it seconds later is work for nobody: the CI build now runs `runtimeBundleDir` and `tools/self-deploy` takes the directory, while `tools/remote`, which really does transfer the bundle, still builds and uploads the archive. Measured locally, the archive costs 5 seconds to write and 0.8 to read again, against 0.1 seconds to copy the directory; the CI instance is around twice as slow at this, and it also saves 64 MiB of SSD per build. - `runtimeBundle` no longer passes `--strip-debug` to `jlink` but `--strip-java-debug-attributes`. On Linux the former also strips native debug symbols, for which `jlink` shells out to `objcopy`, and the werkdock rootfs carries no `binutils` — so the very first CI build of this branch that ran the task failed with `Cannot run program "objcopy"` after 45 seconds. The replacement needs no external tool and costs 671 bytes of bundle size, because a headless JDK's native libraries carry almost no debug symbols anyway. Adding `binutils` to the rootfs would have worked too, but it would have required rebuilding and uploading the rootfs before this branch could go green. - The predecessor is `default`, where the RFC wrote `test`: this repository's only triggered definition is the implicit `default` build, and the RFC named a generic one. - `tools/remote`'s install-layout comment now names `werkator-self-deploy` beside the werkdock binary. - `docs/configuration.md` names this deployment in "Follow-up builds" as the one case where a deployment restarts the very instance running it. ## Prerequisite PRs - PR#23 — follow-up builds (`afterSuccessOf`), the mechanism this deployment is an instance of. - The RFC itself, committed to main as `6256e5b`. ## Follow-up PRs - [Step 2 of RFC 0005](../rfcs/0005-werkator-deploys-itself.md#step-2--restartsinstance-as-a-build-setting-option-b): `restartsInstance` as a pinned build setting, so the executor drains its queue and closes its own context instead of a fixed delay deciding when the restart is safe. - Sample `/proc/pressure/io` beside the CPU and memory pressure, the way RFC 0004 introduced the OOM-kill counter: collected first, judged later. It is the one number that would say whether a build waited for storage, which is the open question behind the bundle's cost on the CI instance. On a shared hive it measures the whole machine, so it names the waiting without naming who waited. - Record the CPU time a build consumed beside its wall-clock duration. That is the measurement that attributes the waiting to a single build rather than to the host, and it would settle whether the bundle steps are slow because they compute or because they wait. ## Attachments ### The effective deploy definition `config:print --full` on this branch, which is the proof that the repository's half alone deploys nothing: ```yaml deploy: trigger: onPush: false atTimes: [] branches: [] activeWithin: "" afterSuccessOf: "" isFollowUp: false buildCommand: "tools/self-deploy" cleanCommand: "" artifactDirs: [] stdoutLog: "build.stdout.log" stderrLog: "build.stderr.log" ``` Three things are worth reading off it. The trigger is empty in every field, so the definition never runs until the host's fragment gives it one. The empty `artifactDirs` wins over the `build/reports` it would otherwise inherit from `builds.default`, so the deployment stores no artifacts — while `stdoutLog` and `stderrLog` are inherited, so its log is kept like any other build's. `cleanCommand` is empty, which is what leaves the bundle of the preceding test build in the shared worktree. ### Manual verification of the script `tools/self-deploy` has no automated tests — the repository has no harness for shell scripts, and adding one is outside this PR. The guards and the staged swap were exercised by hand on 2026-09-13; `shellcheck` reports nothing for the script, and reports for `tools/remote` exactly what it reported before this PR. Refusing the primary checkout, and refusing to install without a bundle ([Scenario#61.02](#scenario6102-a-deploy-without-its-predecessor-installs-nothing)): ``` ERROR: this is the primary checkout — the deploy build runs in a worktree under .git/werkator/worktrees/ ERROR: no runtime bundle at …/build/distributions/werkator-runtime-linux-x64.tar.gz — the deploy build runs after a green build that built it; restart that one ``` A real bundle swapped into a prepared installation directory: ``` ==> Staging werkator-runtime-linux-x64.tar.gz in …/realswap ==> Verifying the staged runtime natively Werkator v1.13.1 ==> Swapping it in (the replaced runtime stays as werkator.prev) ``` afterwards `werkator/bin/werkator --version` reports `Werkator v1.13.1` and `werkator.prev` still holds the runtime it replaced. A bundle whose launcher fails ([Scenario#61.03](#scenario6103-a-runtime-that-does-not-start-here-is-never-installed)): ``` ==> Verifying the staged runtime natively boom: wrong glibc ERROR: the staged runtime does not run on this host — nothing was swapped ``` afterwards the installed runtime still answers with its own version, unchanged.