werkator/docs/rfcs/0005-werkator-deploys-itself.md

# RFC 0005: Werkator Deploys Itself

**Status:**
- proposed: 2026-09-13
- accepted: 2026-09-13 (A first, then B on top; see Decisions)
- rejected: -
- implemented: step 1 on 2026-09-14 (PR#61); step 2 on 2026-09-15 (PR#65)

**Proposal:** Werkator's own repository gets a deployment like every other repository it builds: a follow-up build on `main` that installs the freshly built runtime bundle on the host and restarts the instance.
The one thing that sets this repository apart — the build that deploys runs *inside* the process it replaces — is solved in two steps that can be executed independently, each as one pull request.
Step 1 needs no Werkator code: the build ends green first and asks systemd for the restart afterwards.
Step 2 makes the restart a feature of the build definition, so Werkator drains its queue and ends itself; the deployment no longer decides the moment, it only stages the runtime and leaves the swap and the rollback to the transient unit.

Each step is written so that it can be handed to an agent as a `/goal` on its own: [Step 1](#step-1--restart-from-outside-the-build-option-a) and [Step 2](#step-2--restartsinstance-as-a-build-setting-option-b) each list their deliverables and their completion conditions.

## Context and Problem Statement

Every repository the production instance on mih09 watches can deploy itself with a follow-up build (`afterSuccessOf`, PR#23): a definition on the host that runs after a green build and ships what it built.
Werkator's own repository is the exception.
It is built on mih09 like the others, but its rollout is still `tools/remote werkator instance-update` from a developer machine: upload the bundle, stop the unit, swap the runtime, start the unit.
Merged main commits therefore run on the instance only when somebody remembers — v1.13.1 was on main for a while before it replaced v1.12.0, and every merge in between paid the price of the bug it fixed.

A deployment definition would not work for this repository today, and not because of the restart itself.
The deploy build is a child process of the running Werkator, and a restart of the instance has three effects on it:

- **The build dies with the service.** `systemctl --user stop` ends the whole control group, and `BuildExecutor.shutdown()` destroys every executing process tree anyway and records the build as `INTERRUPTED`.
- **It runs again after the restart.** `Watcher.recoverOnStartup` re-enqueues every `INTERRUPTED` build whose branch still exists.
  The re-run deploys again, restarts again, and so on: an endless loop, not a one-time glitch.
- **The green status never lands.** Result persistence, the Gitea status and the artifacts happen after the process ends, which never happens regularly.

So the restart must be *caused* by the build but must not *happen* inside it.
The build ends `SUCCESS`, and only then does something else restart the service.

Two facts of the mih09 installation shape the solution:

- The instance is a systemd user unit (`werkator-werkator.service`) with `Restart=always`, running the jlink bundle from `~/.werkator/werkator/`; `tools/remote` swaps that directory as a whole (`swap_instance_runtime`: `werkator` → `werkator.prev`, untar, `werkator --version`).
  A build of this repository runs on the same host as the instance, so no ssh is involved in a self-deployment — the swap is a local `mv`.
- Builds run in the werkdock sandbox, which has a JDK 21 and therefore `jlink`; the host itself has no JDK.
  The bundle must be built inside the sandbox, but the swap and the restart must happen natively, on the host's filesystem and with the user's systemd instance.

## The Options

- **A — a transient systemd job restarts the service.**
  The deploy definition runs natively, stages and swaps the bundle, and as its last step schedules the restart with `systemd-run --user --on-active=…`.
  The transient timer lives outside the service's control group and survives the stop; the build is already `SUCCESS` when the restart happens, so the recovery leaves it alone.
  No Werkator code changes; the deployment is a script in the repository plus the host's part of the definition.
- **B — `restartsInstance: true` on the build definition.**
  Werkator knows that this build replaces the service: after the result is persisted and published it stops taking builds, waits until nothing executes, and closes the server context; `Restart=always` brings the new version up.
  Works the same on werkdock and Docker hosts, and the deployment script no longer talks to systemd.
  Costs a new setting in the executor and the config, in the three places that must stay in sync.
- **C — a systemd path unit as the deployer.**
  `init --systemd` generates `werkator-deploy.path` and `.service`; the build only drops the bundle and a marker file; the path unit swaps and restarts.
  Robust and extensible towards rollback, but two more units on every host, including those that never deploy themselves.

### Recommendation

A first, because it works today without touching the executor and cleanly avoids the loop; B on top once A has run on mih09, because a restart is then a property of the build and not a trick of one script.
C is not needed once B exists.

## Why the key is pinned

The branch layer wins over the whole `builds` section.
Without pinning, a branch could write `restartsInstance: true` into any definition of its committed `.werkator.yml`, say into `test`, and every push would end the instance and abort every other running build.
That is the reason `afterSuccessOf` and the trigger block of a follow-up are pinned: a branch describes its own build, but it never decides what happens to the host.
A restart is a host decision, like the sandbox, the network and the socket.

Pinning also means a feature branch cannot deploy itself through a pinned key that "also applies" there:

- Whether the deploy build runs at all is decided by its trigger, and the trigger of a follow-up is pinned as a whole: `afterSuccessOf: test` and `branches: ["main"]` come from the host, a branch can neither change nor remove them.
  On a feature branch `deploy` is never enqueued, and a key of a build that does not run has no effect.
- A branch cannot equip another build with the key: pinning writes the host's value back.
  For `test` that is `false`, because the host does not set it there; for a build the branch invents the value comes from the host's `builds.default`, also `false` — exactly how `docker.enabled` behaves today.

The condition is that the host restricts its deploy definition to `main` itself.
A definition with `restartsInstance` and no `branches` selector would replace the instance with whatever branch went green last.
That is a host misconfiguration, not a hole a branch can open, and Werkator refuses it at start (Step 2).

## Where the configuration lives

The split is the one `docs/configuration.md` already prescribes for follow-up builds: what the deployment *does* comes with the repository, *when, where and with what* belongs to the host.

**In the repository, `.werkator.yml`:** a definition `deploy` that has no trigger without the host's part and therefore never runs anywhere else.
`test` builds the runtime bundle alongside, since only the sandbox has a JDK; `deploy` takes it from the same worktree, which is why it sets `cleanCommand: ""`.

```yaml
builds:
  default:
    buildCommand: ./gradlew --console=plain --no-daemon test runtimeBundle
  deploy:
    cleanCommand: ""                  # keeps the bundle test just built in this worktree
    buildCommand: tools/self-deploy   # stage, verify, swap, schedule the restart
    artifactDirs: []
```

**On the host, the fragment applied with `init --apply`:** the trigger, the branch restriction and the decision that this build runs natively, since it has to reach the installation directory.
All three are pinned; a branch reaches none of them.

```yaml
builds:
  deploy:
    trigger:
      afterSuccessOf: test
      branches: ["main"]
    statusContext: werkator/deploy
    werkdock:
      enabled: false
```

With Step 2, `restartsInstance: true` joins this host block.

The bundle is not an artifact: it is around 60 MB per build and would be kept `retentionPerBranch` times per branch on the SSD for no reader.
It stays in the worktree, where the follow-up finds it and the next `cleanCommand` of `test` removes it.

## Step 1 — Restart from outside the build (Option A)

One pull request; minor version bump, since the repository gains a deployment.

### Deliverables

1. **`tools/self-deploy`**, a bash script run by the deploy build natively in the branch worktree of `main`.
   It takes no parameters and reads no env file; everything it needs it derives from where it runs:
   - the primary checkout from `git rev-parse --git-common-dir`, and from its basename the unit name exactly as `SystemdServiceFiles.unitName` derives it (`werkator-<dir>.service`, characters outside `[A-Za-z0-9_.-]` replaced by `-`);
   - the installation directory from the unit's `ExecStart` (`systemctl --user show -p ExecStart`), whose jar path lies inside the bundle directory `<install>/werkator/`;
   - the instance's port from `<install>/werkator/bin/werkator config:print`, the way `tools/remote require_idle` does.
   It fails clearly, without touching anything, when `build/distributions/werkator-runtime-linux-x64.tar.gz` is missing — a restart of `deploy` alone runs without its predecessor and must not install an old bundle — and when the running unit is not the one the checkout belongs to.
2. **The staged swap.** Unpack into `<install>/werkator.new`, run `werkator.new/bin/werkator --version` natively — this is what catches a bundle whose jlinked JRE does not run on the host — and only then `rm -rf werkator.prev; mv werkator werkator.prev; mv werkator.new werkator`.
   The unit's `ExecStart` path stays valid because the directory name does not change.
3. **The restart and the watchdog**, one transient unit scheduled with `systemd-run --user --collect --on-active=10 --unit=werkator-self-restart`:
   restart the service, wait, probe `http://127.0.0.1:<port>/api/watcher`; on failure swap `werkator.prev` back and restart once more, and leave that fact in the journal.
   The transient unit is not part of the service's control group, so the stop does not end it.
   `XDG_RUNTIME_DIR` and `DBUS_SESSION_BUS_ADDRESS` are set in the build's environment because the build is a child of the service, which the user manager started; the script must not assume a login shell.
4. **`tools/remote` reuses the swap.** `swap_instance_runtime` and `instance-update` do the same staged swap with the same verification, by uploading and running the same script with a flag that skips the restart scheduling (the remote script does the stop/start itself, as today).
   Two implementations of the swap would drift apart at the first change of the layout.
5. **`.werkator.yml`** as shown above: `runtimeBundle` in the default build command, the `deploy` definition with `cleanCommand: ""` and an empty `artifactDirs`.
6. **`docs/deployment.md`** gets a section "Self-Deployment" under "Updating an Existing Installation": the two parts of the configuration, the staged swap, the watchdog and its rollback, and that `werkator.prev` is the last known good version.
   `docs/configuration.md` gets one sentence in "Follow-up builds" naming this RFC as the example of a deployment that restarts the very instance running it.
7. **The PR-doc** with the scenarios below, the version bump and the release-notes entry.

### Scenarios

- Given `test` went green on `main`, when `deploy` runs, then the new bundle is installed, the build is recorded as `SUCCESS` before the service restarts, and the restarted instance reports the new version in its footer.
- Given `deploy` is restarted from the UI without a preceding `test`, when the script finds no bundle in the worktree, then it fails with a message saying so and the installation is untouched.
- Given the new bundle's `werkator --version` fails natively, when the script verifies it, then nothing is swapped and the build fails.
- Given the restarted service does not answer within the watchdog's wait, when the watchdog probes it, then `werkator.prev` is swapped back, the service is restarted again, and the journal says so.
- Given a feature branch commits `builds.deploy.trigger.branches: ["*"]`, when the watcher evaluates the branch, then `deploy` is not enqueued for it (the trigger of a follow-up is pinned).

The first scenario is verified on mih09, not by a test: the PR-doc records the run.
The last is a `ConfigLoaderTest` case if one does not cover it yet.

### Completion conditions

- `./gradlew build` is green and the CI build of the branch on werkator.javagil.de is green.
- `tools/self-deploy` passes `shellcheck` and refuses to run in the primary checkout (it must be in a worktree of the repository it deploys).
- `tools/remote --env-file .env.mih09 werkator instance-update` still works and now goes through the staged swap; this is the rollout of the PR itself and the last manual one.
- After the merge, the host fragment on mih09 carries the `deploy` block, applied with `init --apply`, and the next merge to `main` deploys itself: `test` green, `deploy` green, footer shows the new version, History shows both builds.
- The PR-doc's first scenario records that run with its commit and the two build durations.

### What Step 1 leaves open

- The restart happens at a fixed delay after the build, whether or not another build has started in the meantime.
  With `executor.maxConcurrent: 1` on mih09 nothing else runs while `deploy` runs, but a build enqueued right after it can be interrupted by the restart and is then re-run by the recovery.
  Acceptable once, not as a rule; Step 2 closes it.
- The script knows systemd.
  On a host without a user manager (a Docker host with a system unit, or none) Step 1 does not apply; the script says so and fails.

## Step 2 — `restartsInstance` as a build setting (Option B)

One pull request on top of Step 1; minor version bump.

### Deliverables

1. **`restartsInstance: Boolean = false`** as a build setting in `WerkatorConfig`, inherited like every setting, pinned like `docker.enabled`: stripped from the branch layer, written back from the host's definition of the same name or from the host's `builds.default`.
   Refused at start, naming the file, when set on `builds.default` or on a definition without a `trigger.branches` selector — a restart must never come from "whatever branch went green".
   The three places stay in sync: the data class, the `InitCommand` template (commented out, with the warning), `docs/configuration.md`.
2. **The executor's part.** When a build with the setting ends `SUCCESS` — and only then; a failed or interrupted deploy restarts nothing — the executor, after persisting the result and publishing the status, enters a *draining* state: it starts no further build (queued builds stay `PENDING`, which the recovery re-enqueues), waits until nothing executes, and then asks the server to close its context.
   `ServerCommand` already blocks on `ContextClosedEvent`; the programmatic close falls through `awaitShutdown`, and `main()` exits — `exitProcess` stays in `main()`, the invariant holds.
   `Restart=always` starts the new version.
   The watcher's `state` and `/api/watcher` say "restarting" during the drain, so the UI's banner explains why nothing starts.
3. **`tools/self-deploy` stops scheduling the restart** when it runs under a Werkator that restarts itself; it keeps scheduling the watchdog.
   How it knows: the executor exports `WERKATOR_RESTARTS_INSTANCE=true` into the environment of a build with the setting, so the script needs no version check.
   The rollback stays with the watchdog: an instance that does not start cannot swap itself back.
4. **Docs and PR-doc.** `docs/deployment.md` "Self-Deployment" describes the drain; `docs/configuration.md` documents the key with its two refusals; AGENTS.md's invariant on pinned keys names it; the PR-doc carries the scenarios.

### Scenarios

- Given a host definition `deploy` with `restartsInstance: true` and `branches: ["main"]`, when its run ends `SUCCESS`, then the result is persisted and published before the context closes, and after the restart the recovery re-runs nothing for it.
- Given the same and a second build is executing when `deploy` ends, when the executor drains, then the second build finishes and is recorded before the context closes, and the restart happens afterwards.
- Given the same and a build is queued when `deploy` ends, when the executor drains, then the queued build stays `PENDING`, starts nothing, and is re-enqueued by the recovery after the restart.
- Given `deploy` ends `FAILURE`, `TIMED_OUT` or `INTERRUPTED`, when the executor looks at the setting, then nothing is drained and the instance keeps running.
- Given a feature branch commits `builds.test.restartsInstance: true`, when the config is loaded for that branch, then the value is `false` (written back from the host).
- Given a host writes `restartsInstance: true` on `builds.default`, or on a definition without `trigger.branches`, when the instance starts, then the start is refused naming the file and the key.

All but the first are `BuildExecutorTest` and `ConfigLoaderTest` cases; the first is recorded from mih09 in the PR-doc.

### Completion conditions

- `./gradlew build` green, CI green.
- The host fragment on mih09 adds `restartsInstance: true` to the `deploy` block; the merge of this PR is deployed by Step 1's mechanism and the next merge after it by Step 2's — the PR-doc records both.
- `tools/self-deploy` no longer schedules a restart under the new instance, verified by the journal: only the watchdog unit appears.

## Decisions

Taken on 2026-09-13:

1. **A, then B** — Step 1 ships the deployment without executor changes; Step 2 makes the restart a build setting.
   C is not pursued.
2. **The bundle is built by `test` inside the sandbox** and taken from the worktree by `deploy` (`cleanCommand: ""`); it is not an artifact.
3. **The swap is staged and verified natively** before it replaces anything; `werkator.prev` is the last known good version, and the watchdog outside the service swaps it back when the new version does not answer.
4. **`restartsInstance` is pinned** like the sandbox policy and refused on `builds.default` and on a definition without a `branches` selector.
5. **The rollback stays outside Werkator** even in Step 2: a transient systemd unit, since an instance that does not start cannot help itself.

## Consequences

- Merges to `main` reach mih09 within one poll cycle plus two builds; the manual `instance-update` remains for the first installation and for emergencies.
- `test` takes longer on every branch by the `runtimeBundle` task; the bundle is not kept.
- `BuildExecutor` gains a draining state (Step 2), and the invariant "never assume a single running build" gains a sibling: never assume the instance keeps running after a build.
- Not touched: the trigger semantics of follow-ups, the pull-request gate, the Docker runner, and the werkdock sandbox.
- Open, for a later RFC: a self-deployment on a Docker host with a system unit, and whether `init --systemd` should generate the watchdog as a permanent unit (Option C's half) instead of `systemd-run`.