werkator/docs/prs/2026-09-14-PR#61-werkator-deploys-itself.md

WARNING: This document describes only the change applied in this PR. It may already be outdated once the next PR is merged. Historic PR-documentation is not maintained along with new PRs — treat it as a snapshot, not as current documentation.

Related Links

• RFC 0005: Werkator Deploys Itself — this PR is its step 1.
• Follow-up builds — the mechanism every other repository deploys with.

The Problem

Werkator is the one repository its own instance builds but cannot deploy.

Every other repository on the production instance deploys with a follow-up build: afterSuccessOf a green build, at that build's commit, in that branch's worktree (PR#23). Werkator's rollout is still tools/remote werkator instance-update from a developer machine. So a merged commit runs on the instance when somebody remembers, and v1.13.1 sat on main for hours while the instance kept running v1.12.0 — with the very bug v1.13.0 had fixed, eating the history of every branch merged in between.

A deployment definition would not work here, and not because restarting is hard. The deploy build is a child process of the running Werkator, and a restart has three effects on it:

• The build dies with the service. systemctl --user stop ends the whole control group, and BuildExecutor.shutdown() destroys every executing process tree anyway and records the build as INTERRUPTED.
• It runs again after the restart. Watcher.recoverOnStartup re-enqueues every INTERRUPTED build whose branch still exists. The re-run deploys again and restarts again: a loop, not a one-time glitch.
• The green status never lands. Result persistence, the Gitea status and the artifacts all happen after the process ends, which never happens regularly.

So the restart must be caused by the build without happening inside it.

A second problem sat in the existing rollout. swap_instance_runtime moved the running runtime aside, unpacked the new one over it, and only then asked the new launcher for its version. A bundle whose jlinked JRE does not start on the host therefore left the installation with no working runtime at all, discovered after the service had already been stopped.

Non-Goals

• No executor change. Werkator learns nothing about restarting itself in this PR; the build ends green and systemd does the rest. Making the restart a property of the build definition is step 2.
• No new configuration key. The deployment is a follow-up build like any other; what makes it special is only that its command talks to systemd.
• No rollout in this PR. Applying the host fragment on mih09 and running instance-update once more happens after the merge, and that manual run is the last one.
• No self-deployment on a host without a systemd user manager. A Docker host with a system unit, or none at all, keeps deploying the way it does today; the script says so and fails.

The Scenarios

Feature: Werkator installs its own merged commits

Background

• The deploy build is a follow-up of the test build, restricted to main, running natively — it has to reach the installation directory, which the sandbox cannot.
• The bundle is build/runtime-bundle/werkator (91 MiB), built by the test build, because only the sandbox has a JDK and therefore jlink. It reaches the deployment through the worktree the two builds share. runtimeBundle packs the same directory as a tarball for a transfer to another host, which is what tools/remote uses.
• The installation is a runtime bundle unpacked at <install>/werkator/, which a systemd user unit runs. <install>/werkator.prev is the runtime it replaced: the last known good version.
• The transient unit is werkator-self-restart-<repo-name>, scheduled with systemd-run --user and named after the service it restarts, since transient unit names are global to the user manager. It is started by the user manager, not by the build, so stopping the service does not end it.

Scenario#61.01: A merged commit installs itself

So that main reaches the instance without anybody remembering to deploy it.

• Given the test build of main ended SUCCESS and left a runtime bundle in the worktree
• When the deploy build runs tools/self-deploy
• Then the bundle is staged, verified and swapped in, and the previous runtime is kept as werkator.prev
  • and the build is recorded SUCCESS and published to Gitea before the service goes down
  • and the transient unit restarts the service afterwards, so the startup recovery finds nothing to re-run
  • and the restarted instance reports the new version in its footer

Verified by

• Not verifiable by a test: it needs the production instance deploying itself. To be verified on mih09 after the merge, see Open Questions.

Scenario#61.02: A deploy without its predecessor installs nothing

So that a restart of the deployment alone cannot put an older version back.

• Given the deploy build is restarted from the UI, so it runs without a preceding test build
  • and the worktree therefore holds no runtime bundle
• When tools/self-deploy looks for it
• Then it fails naming the missing bundle and the build that produces it
  • and the installation is untouched

Verified by

• Manual run, recorded in Manual verification of the script.

Scenario#61.03: A runtime that does not start here is never installed

So that a bundle built for another host costs a red build and no downtime.

• Given a bundle whose launcher fails on this host
• When tools/self-deploy verifies the staged runtime natively
• Then the staged directory is removed and nothing is swapped
  • and the running installation still answers with its own version
  • and the build fails

Verified by

• Manual run, recorded in Manual verification of the script.

Scenario#61.04: A new version that does not answer is rolled back

So that a deployment cannot leave the instance down unattended.

• Given the new runtime was installed and the service restarted
• When the transient unit finds http://127.0.0.1:<port>/api/watcher unanswered for two minutes
• Then the new runtime is moved aside as werkator.failed and werkator.prev takes its place again
  • and the service is restarted once more
  • and the journal of werkator-self-restart-<repo-name> says which of the two happened

Verified by

• Not verifiable by a test; the rollback path is exercised on mih09 only by an actually broken release. The staging half of it — nothing swapped when the runtime does not start — is Scenario#61.03.

Scenario#61.05: A branch cannot deploy itself

So that a pinned deployment stays the host's decision.

• Given the host defines deploy as a follow-up restricted to main
• When a feature branch commits builds.deploy.trigger.branches: ["*"], or adds afterSuccessOf to a build of its own
• Then the host's trigger block is used unchanged and the branch is not selected
  • and the branch's own build gets no afterSuccessOf at all

Verified by

• ConfigLoaderTest: "a follow-up trigger is pinned to the host, a branch cannot add or widen one"

The Solution

tools/self-deploy is the deploy build's command, and it restarts nothing itself.

It takes no parameters; everything it needs it derives from where it runs. The primary checkout comes from git rev-parse --git-common-dir, the unit name from that directory's basename exactly as SystemdServiceFiles.unitName derives it, the installation directory from that unit's ExecStart — whose jar sits in <install>/werkator/lib/ — and the port from the installed binary's config:print. Deriving rather than configuring is what lets the same script serve a developer machine's rollout and the instance's own, with nothing to keep in sync.

It refuses, before touching anything, to run in the primary checkout, to install into an instance that watches a different repository, to treat a plain-jar installation as a bundle, and to install when the worktree holds no bundle.

The swap is staged. The bundle is put beside the running runtime as werkator.new, its launcher is run natively — the one check that catches a jlinked JRE that will not start here — and only then does it take the place of werkator, with the replaced one kept as werkator.prev.

The restart is scheduled, not performed: systemd-run --user --collect --on-active=30 --unit=werkator-self-restart-<repo-name> runs a copy of the script that the deployment installs next to the runtime, so it does not depend on a worktree that the next build will reuse. That unit restarts the service, waits for /api/watcher, and rolls back to werkator.prev when it stays silent. Because it fires after the build has been recorded, the recovery has nothing to re-run, and the loop the RFC describes never starts.

tools/remote now performs its swap by running the very same script on the host, in its swap-only form. That removes the second implementation of the layout, and it fixes the hole described above: instance-update no longer moves the running runtime aside before knowing that the new one starts.

The repository's .werkator.yml carries what the deployment does; when, for which branches and outside the sandbox stays the host's, in the instance fragment, where it is pinned.

Open Questions

• Scenario#61.01 is verified after the merge, not in this PR. It needs the production instance to deploy itself, which cannot happen before this branch is on main and the host fragment carries the deploy block. The sequence is: instance-update once more by hand (the last manual one), init --apply with the fragment below, then the next merge to main deploys itself. The run is to be recorded here with its commit and the two build durations.

• The host fragment is not in this repository, because it is the host's part. For mih09 it belongs in .env.mih09.yml:

  builds:
    deploy:
      trigger:
        afterSuccessOf: default
        branches: ["main"]
      statusContext: werkator/deploy
      werkdock:
        enabled: false
  

• The restart delay is fixed at 30 seconds, so a build enqueued between the deployment and the restart is interrupted and re-run by the recovery. With executor.maxConcurrent: 1 on mih09 nothing else runs while the deployment does, so this is a narrow window; step 2 closes it by draining the queue instead of waiting a fixed time.

• The test build got longer on every branch by the bundle task, for a bundle only main installs: on the CI instance from around 95 to 130 seconds. Restricting it to main would mean a branch's own deployment could never be tested, and that testing has already paid off once, when it caught the missing objcopy before anything was merged. Leaving it on every branch is the deliberate choice; revisiting it after the first successful self-deployment is the obvious moment.

Additional Changes

• The restart delay is 30 seconds where RFC 0005 drafted 10. Too short is the dangerous direction: a build not yet recorded SUCCESS when the service goes down is recovered as INTERRUPTED and deploys again, which is the loop this whole design exists to avoid. Thirty seconds leaves room for a slow artifact copy or a Gitea call waiting for its timeout.
• The bundle task was split in two. runtimeBundleDir produces the directory, runtimeBundle packs that directory as a tarball. A self-deployment unpacks the bundle on the machine that built it, so compressing 64 MiB and decompressing it seconds later is work for nobody: the CI build now runs runtimeBundleDir and tools/self-deploy takes the directory, while tools/remote, which really does transfer the bundle, still builds and uploads the archive. Measured locally, the archive costs 5 seconds to write and 0.8 to read again, against 0.1 seconds to copy the directory; the CI instance is around twice as slow at this, and it also saves 64 MiB of SSD per build.
• runtimeBundle no longer passes --strip-debug to jlink but --strip-java-debug-attributes. On Linux the former also strips native debug symbols, for which jlink shells out to objcopy, and the werkdock rootfs carries no binutils — so the very first CI build of this branch that ran the task failed with Cannot run program "objcopy" after 45 seconds. The replacement needs no external tool and costs 671 bytes of bundle size, because a headless JDK's native libraries carry almost no debug symbols anyway. Adding binutils to the rootfs would have worked too, but it would have required rebuilding and uploading the rootfs before this branch could go green.
• The predecessor is default, where the RFC wrote test: this repository's only triggered definition is the implicit default build, and the RFC named a generic one.
• tools/remote's install-layout comment now names werkator-self-deploy beside the werkdock binary.
• docs/configuration.md names this deployment in "Follow-up builds" as the one case where a deployment restarts the very instance running it.

Prerequisite PRs

• PR#23 — follow-up builds (afterSuccessOf), the mechanism this deployment is an instance of.
• The RFC itself, committed to main as 6256e5b.

Follow-up PRs

• Step 2 of RFC 0005: restartsInstance as a pinned build setting, so the executor drains its queue and closes its own context instead of a fixed delay deciding when the restart is safe.
• Sample /proc/pressure/io beside the CPU and memory pressure, the way RFC 0004 introduced the OOM-kill counter: collected first, judged later. It is the one number that would say whether a build waited for storage, which is the open question behind the bundle's cost on the CI instance. On a shared hive it measures the whole machine, so it names the waiting without naming who waited.
• Record the CPU time a build consumed beside its wall-clock duration. That is the measurement that attributes the waiting to a single build rather than to the host, and it would settle whether the bundle steps are slow because they compute or because they wait.

Attachments

The effective deploy definition

config:print --full on this branch, which is the proof that the repository's half alone deploys nothing:

  deploy:
    trigger:
      onPush: false
      atTimes: []
      branches: []
      activeWithin: ""
      afterSuccessOf: ""
      isFollowUp: false
    buildCommand: "tools/self-deploy"
    cleanCommand: ""
    artifactDirs: []
    stdoutLog: "build.stdout.log"
    stderrLog: "build.stderr.log"

Three things are worth reading off it. The trigger is empty in every field, so the definition never runs until the host's fragment gives it one. The empty artifactDirs wins over the build/reports it would otherwise inherit from builds.default, so the deployment stores no artifacts — while stdoutLog and stderrLog are inherited, so its log is kept like any other build's. cleanCommand is empty, which is what leaves the bundle of the preceding test build in the shared worktree.

Manual verification of the script

tools/self-deploy has no automated tests — the repository has no harness for shell scripts, and adding one is outside this PR. The guards and the staged swap were exercised by hand on 2026-09-13; shellcheck reports nothing for the script, and reports for tools/remote exactly what it reported before this PR.

Refusing the primary checkout, and refusing to install without a bundle (Scenario#61.02):

ERROR: this is the primary checkout — the deploy build runs in a worktree under .git/werkator/worktrees/
ERROR: no runtime bundle at …/build/distributions/werkator-runtime-linux-x64.tar.gz — the deploy build runs
       after a green build that built it; restart that one

A real bundle swapped into a prepared installation directory:

==> Staging werkator-runtime-linux-x64.tar.gz in …/realswap
==> Verifying the staged runtime natively
    Werkator v1.13.1
==> Swapping it in (the replaced runtime stays as werkator.prev)

afterwards werkator/bin/werkator --version reports Werkator v1.13.1 and werkator.prev still holds the runtime it replaced.

A bundle whose launcher fails (Scenario#61.03):

==> Verifying the staged runtime natively
boom: wrong glibc
ERROR: the staged runtime does not run on this host — nothing was swapped

afterwards the installed runtime still answers with its own version, unchanged.