werkator/docs/rfcs/0005-werkator-deploys-itself.md

RFC 0005: Werkator Deploys Itself

Status:

• proposed: 2026-09-13
• accepted: 2026-09-13 (A first, then B on top; see Decisions)
• rejected: -
• implemented: step 1 on 2026-09-14 (PR#61); step 2 on 2026-09-15 (PR#65)

Proposal: Werkator's own repository gets a deployment like every other repository it builds: a follow-up build on main that installs the freshly built runtime bundle on the host and restarts the instance. The one thing that sets this repository apart — the build that deploys runs inside the process it replaces — is solved in two steps that can be executed independently, each as one pull request. Step 1 needs no Werkator code: the build ends green first and asks systemd for the restart afterwards. Step 2 makes the restart a feature of the build definition, so Werkator drains its queue and ends itself; the deployment no longer decides the moment, it only stages the runtime and leaves the swap and the rollback to the transient unit.

Each step is written so that it can be handed to an agent as a /goal on its own: Step 1 and Step 2 each list their deliverables and their completion conditions.

Context and Problem Statement

Every repository the production instance on mih09 watches can deploy itself with a follow-up build (afterSuccessOf, PR#23): a definition on the host that runs after a green build and ships what it built. Werkator's own repository is the exception. It is built on mih09 like the others, but its rollout is still tools/remote werkator instance-update from a developer machine: upload the bundle, stop the unit, swap the runtime, start the unit. Merged main commits therefore run on the instance only when somebody remembers — v1.13.1 was on main for a while before it replaced v1.12.0, and every merge in between paid the price of the bug it fixed.

A deployment definition would not work for this repository today, and not because of the restart itself. The deploy build is a child process of the running Werkator, and a restart of the instance has three effects on it:

• The build dies with the service. systemctl --user stop ends the whole control group, and BuildExecutor.shutdown() destroys every executing process tree anyway and records the build as INTERRUPTED.
• It runs again after the restart. Watcher.recoverOnStartup re-enqueues every INTERRUPTED build whose branch still exists. The re-run deploys again, restarts again, and so on: an endless loop, not a one-time glitch.
• The green status never lands. Result persistence, the Gitea status and the artifacts happen after the process ends, which never happens regularly.

So the restart must be caused by the build but must not happen inside it. The build ends SUCCESS, and only then does something else restart the service.

Two facts of the mih09 installation shape the solution:

• The instance is a systemd user unit (werkator-werkator.service) with Restart=always, running the jlink bundle from ~/.werkator/werkator/; tools/remote swaps that directory as a whole (swap_instance_runtime: werkator → werkator.prev, untar, werkator --version). A build of this repository runs on the same host as the instance, so no ssh is involved in a self-deployment — the swap is a local mv.
• Builds run in the werkdock sandbox, which has a JDK 21 and therefore jlink; the host itself has no JDK. The bundle must be built inside the sandbox, but the swap and the restart must happen natively, on the host's filesystem and with the user's systemd instance.

The Options

• A — a transient systemd job restarts the service. The deploy definition runs natively, stages and swaps the bundle, and as its last step schedules the restart with systemd-run --user --on-active=…. The transient timer lives outside the service's control group and survives the stop; the build is already SUCCESS when the restart happens, so the recovery leaves it alone. No Werkator code changes; the deployment is a script in the repository plus the host's part of the definition.
• B — restartsInstance: true on the build definition. Werkator knows that this build replaces the service: after the result is persisted and published it stops taking builds, waits until nothing executes, and closes the server context; Restart=always brings the new version up. Works the same on werkdock and Docker hosts, and the deployment script no longer talks to systemd. Costs a new setting in the executor and the config, in the three places that must stay in sync.
• C — a systemd path unit as the deployer. init --systemd generates werkator-deploy.path and .service; the build only drops the bundle and a marker file; the path unit swaps and restarts. Robust and extensible towards rollback, but two more units on every host, including those that never deploy themselves.

Recommendation

A first, because it works today without touching the executor and cleanly avoids the loop; B on top once A has run on mih09, because a restart is then a property of the build and not a trick of one script. C is not needed once B exists.

Why the key is pinned

The branch layer wins over the whole builds section. Without pinning, a branch could write restartsInstance: true into any definition of its committed .werkator.yml, say into test, and every push would end the instance and abort every other running build. That is the reason afterSuccessOf and the trigger block of a follow-up are pinned: a branch describes its own build, but it never decides what happens to the host. A restart is a host decision, like the sandbox, the network and the socket.

Pinning also means a feature branch cannot deploy itself through a pinned key that "also applies" there:

• Whether the deploy build runs at all is decided by its trigger, and the trigger of a follow-up is pinned as a whole: afterSuccessOf: test and branches: ["main"] come from the host, a branch can neither change nor remove them. On a feature branch deploy is never enqueued, and a key of a build that does not run has no effect.
• A branch cannot equip another build with the key: pinning writes the host's value back. For test that is false, because the host does not set it there; for a build the branch invents the value comes from the host's builds.default, also false — exactly how docker.enabled behaves today.

The condition is that the host restricts its deploy definition to main itself. A definition with restartsInstance and no branches selector would replace the instance with whatever branch went green last. That is a host misconfiguration, not a hole a branch can open, and Werkator refuses it at start (Step 2).

Where the configuration lives

The split is the one docs/configuration.md already prescribes for follow-up builds: what the deployment does comes with the repository, when, where and with what belongs to the host.

In the repository, .werkator.yml: a definition deploy that has no trigger without the host's part and therefore never runs anywhere else. test builds the runtime bundle alongside, since only the sandbox has a JDK; deploy takes it from the same worktree, which is why it sets cleanCommand: "".

builds:
  default:
    buildCommand: ./gradlew --console=plain --no-daemon test runtimeBundle
  deploy:
    cleanCommand: ""                  # keeps the bundle test just built in this worktree
    buildCommand: tools/self-deploy   # stage, verify, swap, schedule the restart
    artifactDirs: []

On the host, the fragment applied with init --apply: the trigger, the branch restriction and the decision that this build runs natively, since it has to reach the installation directory. All three are pinned; a branch reaches none of them.

builds:
  deploy:
    trigger:
      afterSuccessOf: test
      branches: ["main"]
    statusContext: werkator/deploy
    werkdock:
      enabled: false

With Step 2, restartsInstance: true joins this host block.

The bundle is not an artifact: it is around 60 MB per build and would be kept retentionPerBranch times per branch on the SSD for no reader. It stays in the worktree, where the follow-up finds it and the next cleanCommand of test removes it.

Step 1 — Restart from outside the build (Option A)

One pull request; minor version bump, since the repository gains a deployment.

Deliverables

1. tools/self-deploy, a bash script run by the deploy build natively in the branch worktree of main. It takes no parameters and reads no env file; everything it needs it derives from where it runs:
   • the primary checkout from git rev-parse --git-common-dir, and from its basename the unit name exactly as SystemdServiceFiles.unitName derives it (werkator-<dir>.service, characters outside [A-Za-z0-9_.-] replaced by -);
   • the installation directory from the unit's ExecStart (systemctl --user show -p ExecStart), whose jar path lies inside the bundle directory <install>/werkator/;
   • the instance's port from <install>/werkator/bin/werkator config:print, the way tools/remote require_idle does. It fails clearly, without touching anything, when build/distributions/werkator-runtime-linux-x64.tar.gz is missing — a restart of deploy alone runs without its predecessor and must not install an old bundle — and when the running unit is not the one the checkout belongs to.
2. The staged swap. Unpack into <install>/werkator.new, run werkator.new/bin/werkator --version natively — this is what catches a bundle whose jlinked JRE does not run on the host — and only then rm -rf werkator.prev; mv werkator werkator.prev; mv werkator.new werkator. The unit's ExecStart path stays valid because the directory name does not change.
3. The restart and the watchdog, one transient unit scheduled with systemd-run --user --collect --on-active=10 --unit=werkator-self-restart: restart the service, wait, probe http://127.0.0.1:<port>/api/watcher; on failure swap werkator.prev back and restart once more, and leave that fact in the journal. The transient unit is not part of the service's control group, so the stop does not end it. XDG_RUNTIME_DIR and DBUS_SESSION_BUS_ADDRESS are set in the build's environment because the build is a child of the service, which the user manager started; the script must not assume a login shell.
4. tools/remote reuses the swap. swap_instance_runtime and instance-update do the same staged swap with the same verification, by uploading and running the same script with a flag that skips the restart scheduling (the remote script does the stop/start itself, as today). Two implementations of the swap would drift apart at the first change of the layout.
5. .werkator.yml as shown above: runtimeBundle in the default build command, the deploy definition with cleanCommand: "" and an empty artifactDirs.
6. docs/deployment.md gets a section "Self-Deployment" under "Updating an Existing Installation": the two parts of the configuration, the staged swap, the watchdog and its rollback, and that werkator.prev is the last known good version. docs/configuration.md gets one sentence in "Follow-up builds" naming this RFC as the example of a deployment that restarts the very instance running it.
7. The PR-doc with the scenarios below, the version bump and the release-notes entry.

Scenarios

• Given test went green on main, when deploy runs, then the new bundle is installed, the build is recorded as SUCCESS before the service restarts, and the restarted instance reports the new version in its footer.
• Given deploy is restarted from the UI without a preceding test, when the script finds no bundle in the worktree, then it fails with a message saying so and the installation is untouched.
• Given the new bundle's werkator --version fails natively, when the script verifies it, then nothing is swapped and the build fails.
• Given the restarted service does not answer within the watchdog's wait, when the watchdog probes it, then werkator.prev is swapped back, the service is restarted again, and the journal says so.
• Given a feature branch commits builds.deploy.trigger.branches: ["*"], when the watcher evaluates the branch, then deploy is not enqueued for it (the trigger of a follow-up is pinned).

The first scenario is verified on mih09, not by a test: the PR-doc records the run. The last is a ConfigLoaderTest case if one does not cover it yet.

Completion conditions

• ./gradlew build is green and the CI build of the branch on werkator.javagil.de is green.
• tools/self-deploy passes shellcheck and refuses to run in the primary checkout (it must be in a worktree of the repository it deploys).
• tools/remote --env-file .env.mih09 werkator instance-update still works and now goes through the staged swap; this is the rollout of the PR itself and the last manual one.
• After the merge, the host fragment on mih09 carries the deploy block, applied with init --apply, and the next merge to main deploys itself: test green, deploy green, footer shows the new version, History shows both builds.
• The PR-doc's first scenario records that run with its commit and the two build durations.

What Step 1 leaves open

• The restart happens at a fixed delay after the build, whether or not another build has started in the meantime. With executor.maxConcurrent: 1 on mih09 nothing else runs while deploy runs, but a build enqueued right after it can be interrupted by the restart and is then re-run by the recovery. Acceptable once, not as a rule; Step 2 closes it.
• The script knows systemd. On a host without a user manager (a Docker host with a system unit, or none) Step 1 does not apply; the script says so and fails.

Step 2 — restartsInstance as a build setting (Option B)

One pull request on top of Step 1; minor version bump.

Deliverables

1. restartsInstance: Boolean = false as a build setting in WerkatorConfig, inherited like every setting, pinned like docker.enabled: stripped from the branch layer, written back from the host's definition of the same name or from the host's builds.default. Refused at start, naming the file, when set on builds.default or on a definition without a trigger.branches selector — a restart must never come from "whatever branch went green". The three places stay in sync: the data class, the InitCommand template (commented out, with the warning), docs/configuration.md.
2. The executor's part. When a build with the setting ends SUCCESS — and only then; a failed or interrupted deploy restarts nothing — the executor, after persisting the result and publishing the status, enters a draining state: it starts no further build (queued builds stay PENDING, which the recovery re-enqueues), waits until nothing executes, and then asks the server to close its context. ServerCommand already blocks on ContextClosedEvent; the programmatic close falls through awaitShutdown, and main() exits — exitProcess stays in main(), the invariant holds. Restart=always starts the new version. The watcher's state and /api/watcher say "restarting" during the drain, so the UI's banner explains why nothing starts.
3. tools/self-deploy stops scheduling the restart when it runs under a Werkator that restarts itself; it keeps scheduling the watchdog. How it knows: the executor exports WERKATOR_RESTARTS_INSTANCE=true into the environment of a build with the setting, so the script needs no version check. The rollback stays with the watchdog: an instance that does not start cannot swap itself back.
4. Docs and PR-doc. docs/deployment.md "Self-Deployment" describes the drain; docs/configuration.md documents the key with its two refusals; AGENTS.md's invariant on pinned keys names it; the PR-doc carries the scenarios.

Scenarios

• Given a host definition deploy with restartsInstance: true and branches: ["main"], when its run ends SUCCESS, then the result is persisted and published before the context closes, and after the restart the recovery re-runs nothing for it.
• Given the same and a second build is executing when deploy ends, when the executor drains, then the second build finishes and is recorded before the context closes, and the restart happens afterwards.
• Given the same and a build is queued when deploy ends, when the executor drains, then the queued build stays PENDING, starts nothing, and is re-enqueued by the recovery after the restart.
• Given deploy ends FAILURE, TIMED_OUT or INTERRUPTED, when the executor looks at the setting, then nothing is drained and the instance keeps running.
• Given a feature branch commits builds.test.restartsInstance: true, when the config is loaded for that branch, then the value is false (written back from the host).
• Given a host writes restartsInstance: true on builds.default, or on a definition without trigger.branches, when the instance starts, then the start is refused naming the file and the key.

All but the first are BuildExecutorTest and ConfigLoaderTest cases; the first is recorded from mih09 in the PR-doc.

Completion conditions

• ./gradlew build green, CI green.
• The host fragment on mih09 adds restartsInstance: true to the deploy block; the merge of this PR is deployed by Step 1's mechanism and the next merge after it by Step 2's — the PR-doc records both.
• tools/self-deploy no longer schedules a restart under the new instance, verified by the journal: only the watchdog unit appears.

Decisions

Taken on 2026-09-13:

1. A, then B — Step 1 ships the deployment without executor changes; Step 2 makes the restart a build setting. C is not pursued.
2. The bundle is built by test inside the sandbox and taken from the worktree by deploy (cleanCommand: ""); it is not an artifact.
3. The swap is staged and verified natively before it replaces anything; werkator.prev is the last known good version, and the watchdog outside the service swaps it back when the new version does not answer.
4. restartsInstance is pinned like the sandbox policy and refused on builds.default and on a definition without a branches selector.
5. The rollback stays outside Werkator even in Step 2: a transient systemd unit, since an instance that does not start cannot help itself.

Consequences

• Merges to main reach mih09 within one poll cycle plus two builds; the manual instance-update remains for the first installation and for emergencies.
• test takes longer on every branch by the runtimeBundle task; the bundle is not kept.
• BuildExecutor gains a draining state (Step 2), and the invariant "never assume a single running build" gains a sibling: never assume the instance keeps running after a build.
• Not touched: the trigger semantics of follow-ups, the pull-request gate, the Docker runner, and the werkdock sandbox.
• Open, for a later RFC: a self-deployment on a Docker host with a system unit, and whether init --systemd should generate the watchdog as a permanent unit (Option C's half) instead of systemd-run.