Merge branch 'main' into 20-build-duration-tracking

2026-09-01 05:38:37 +02:00
parent bc70c77126 71f1fc62c6
commit c61f96e304
21 changed files with 1564 additions and 37 deletions
@@ -34,3 +34,5 @@ replay_pid*
 # Other
 /.local/
 /.env
@@ -39,9 +39,9 @@ All production code lives under `de.hoennig.werkator`, with sub-packages `comman
 - Builds run detached in worktrees under `.git/werkator/worktrees/<branchKey>`; the primary checkout is never used for builds; never assume a single running build.
 - When config keys change, three places must stay in sync: the `WerkatorConfig` data classes, the `InitCommand` templates, and `docs/configuration.md`.
 - Every config file may declare `werkator.version.since`/`below` (the Werkator it is written for, never a format version — no API is involved). `since` is enforced in both directions, using `ConfigVersions.FORMAT_BROKE_IN` for "file predates a breaking change"; `below` only warns. A violation aborts the start for the machine and project config, but fails only that branch's builds for a branch config.
- A branch describes its own CI: its committed `.werkator.yml` is the branch layer (`ConfigLoader.loadWithBranchLayer`, used by the watcher per origin branch and by `loadForWorktree` at build time) and takes precedence over `.git`/project — including the whole `builds` section, so a new configuration can be tried out on a branch without affecting other branches. Only the pinned set is stripped from that layer: secrets (`git`), host/repository sections (`server`, `gitea`, `executor`, `watcher`), the docker sandbox policy (`docker.enabled`, `docker.network`), and the trust gate (`requirePullRequest`). A branch must never reach credentials, disable its container, change its network, raise global concurrency, or bypass its own pull-request gate; a branch's definitions apply to that branch alone.
+- A branch describes its own CI: its committed `.werkator.yml` is the branch layer (`ConfigLoader.loadWithBranchLayer`, used by the watcher per origin branch and by `loadForWorktree` at build time) and takes precedence over `.git`/project — including the whole `builds` section, so a new configuration can be tried out on a branch without affecting other branches. Only the pinned set is stripped from that layer: secrets (`git`), host/repository sections (`server`, `gitea`, `executor`, `watcher`), the docker (`docker.enabled`, `docker.network`) and bubblewrap (`bwrap.enabled`, `bwrap.rootfs`) sandbox policies, and the trust gate (`requirePullRequest`). A branch must never reach credentials, disable its container or sandbox, change its network, substitute a foreign rootfs, raise global concurrency, or bypass its own pull-request gate; a branch's definitions apply to that branch alone.
 - A build definition carries the complete description of its build, split in two: the `trigger` block (`onPush`, `atTimes`, `branches`, `activeWithin`) says when and for which branches it runs, everything else what it does. `builds.default` is the base every other definition inherits its settings — never its `trigger` — from. The split is structural so that a selector added to `TriggerConfig` later is non-inheritable by construction; writing a trigger key flat is refused, never ignored, because ignoring it leaves a build that silently stops running. A `!` prefix in `trigger.branches` excludes and always wins.
- The inheritance is applied after all layers are merged: that order is what makes a build a branch invents inherit the host's sandbox policy instead of the data-class default, so the pinning also holds for a build the host has never heard of. Pinned are `requirePullRequest`, `statusContext`, `docker.enabled`, and `docker.network`.
+- The inheritance is applied after all layers are merged: that order is what makes a build a branch invents inherit the host's sandbox policy instead of the data-class default, so the pinning also holds for a build the host has never heard of. Pinned are `requirePullRequest`, `statusContext`, `docker.enabled`, `docker.network`, `bwrap.enabled`, and `bwrap.rootfs`. Docker and bwrap are mutually exclusive per branch — enabling both is rejected at start.
 - `builds` or the legacy `branches`, never both: `branches` is read only while the merged config defines no build at all (`builds.maxConcurrent` is not one), and ignored with a warning as soon as one exists. The section is deprecated and goes away once the repositories have migrated; then `ConfigVersions.FORMAT_BROKE_IN` gets set and a leftover `branches:` key must be rejected by name — the version check alone cannot catch a file that declares no version.
 - Web UI: server-rendered Thymeleaf plus one hand-written `static/werkator.js` — no SPA framework, no frontend build pipeline; every fetch has a timeout and an explicit error badge; `UiFormats` and `werkator.js` must produce identical display formats.
 - Git and Docker access shells out to the CLIs (`GitCommandRunner`, `docker`) — no JGit, no Docker SDK.
@@ -63,7 +63,7 @@ above, giving the precedence **branch > repo install > project**. It takes prece
 everything that describes how this branch is built: the whole `builds` section — its own
 definitions and its overrides of the definitions from the project config, with
 `buildCommand`, `cleanCommand`, `artifactDirs`, log file names, and
-`docker.image`/`dockerfile`/`context`/`env` inside them. That is how a new configuration is tried out: change it on a branch, and
+`docker.image`/`dockerfile`/`context`/`env` and `bwrap.env` inside them. That is how a new configuration is tried out: change it on a branch, and
 no other branch's builds are affected.
 The branch layer is used in both places where it matters: the watcher reads the committed
@@ -90,7 +90,8 @@ single branch may decide it:
 - the repository-side settings: the whole `gitea`, `executor`, and `watcher` sections;
 - the trust gate: `requirePullRequest`, and the Gitea status context: `statusContext`;
- the container sandbox policy: `docker.enabled` and `docker.network` — host-pinned as
+- the container sandbox policy: `docker.enabled`/`docker.network` and
  `bwrap.enabled`/`bwrap.rootfs` — host-pinned as
  long as only the host's configuration sets them, master-pinned once the committed
  configuration does.
@@ -138,6 +139,12 @@ server:
  bindAddress: 127.0.0.1
  # optional Impressum (legal disclosure) link in the web UI footer; empty hides the link
  impressumUrl: ""
  # Resource limits of the systemd user unit generated by `init --systemd`; empty = directive omitted.
  # Needed where the service shares a memory slice, e.g. Hostsharing Managed Webspaces, where a
  # runaway Gradle build would starve everything else in the package.
  systemd:
    memoryMax: ""   # e.g. 1G — written as `MemoryMax=` into the unit
    tasksMax: ""    # e.g. 512 — written as `TasksMax=` into the unit
  # Opt-in managed nginx+certbot Docker container for HTTPS, for hosts without
  # a usable reverse proxy (ADR 0005; see notes below and deployment.md).
  nginx:
@@ -342,9 +349,9 @@ That is how a branch gets a build of its own without being built by the default
 `activeWithin` (e.g. `24h`) additionally keeps only branches whose origin head commit is younger than the duration — useful to run a nightly deep check over all recently active branches.
 Both parts combine as an intersection.
-Settings: `buildCommand`, `cleanCommand`, `artifactDirs`, `stdoutLog`/`stderrLog`, `requirePullRequest`, `statusContext`, and `docker` with all its keys.
+Settings: `buildCommand`, `cleanCommand`, `artifactDirs`, `stdoutLog`/`stderrLog`, `requirePullRequest`, `statusContext`, and `docker` and `bwrap` with all their keys.
 A definition carries the complete description of its build; unset keys fall back to `builds.default` and then to Werkator's own defaults.
-`requirePullRequest`, `statusContext`, `docker.enabled`, and `docker.network` are pinned (master-pinned, see [the branch layer](#the-branch-layer-a-branch-describes-its-own-ci)): they are read from the repo install/project config even when a branch sets them in its own committed config.
+`requirePullRequest`, `statusContext`, `docker.enabled`, `docker.network`, `bwrap.enabled`, and `bwrap.rootfs` are pinned (master-pinned, see [the branch layer](#the-branch-layer-a-branch-describes-its-own-ci)): they are read from the repo install/project config even when a branch sets them in its own committed config.
 Inheritance from `builds.default` covers the settings only — the `trigger` block says when and where *this* build runs and is never inherited.
 Definitions are part of the branch layer: a branch may add its own and override those from the project config, for its own builds only.
 Because the inheritance is applied after all layers are merged, a build a branch invents still inherits the host's `builds.default` — its sandbox policy included, which is what keeps the pinning effective for a build the host has never heard of.
@@ -400,6 +407,23 @@ Note that the rest of `.git` — including `.git/config` — is visible to build
 The Docker socket is mounted into the container and `DOCKER_HOST`/`TESTCONTAINERS_*` variables are set, so Testcontainers-based builds work inside the container.
 All Werkator containers carry `org.hoennig.werkator` labels; stale build containers of the repository are removed before the first Docker build after a restart.
 ### Notes on `builds.<name>.bwrap`
 With `bwrap.enabled`, Werkator shells out to the `bwrap` CLI (bubblewrap) instead of native execution.
 This is the third runtime, for hosts without root and without a Docker daemon (e.g. Hostsharing managed webspaces); see `docs/plan/17-bwrap-build-runtime.md` and ADR 0007.
 `bwrap` must be on the `PATH`.
 `bwrap.rootfs` names the prepared root filesystem archive — a Debian-base rootfs with the build tools (JDK, git, locales, project-specific tooling) built elsewhere, since `debootstrap` is unavailable on the target.
 It is a local path or an `http(s)` URL; a URL is downloaded once.
 Build the archive with `tools/build-bwrap-rootfs.sh` on any machine with Docker; verify the host's user-namespace capability first with `tools/werkator-build-prerequisites.sh`.
 The archive is unpacked on demand (`tar --no-same-owner`) into `.git/werkator/buildenv/<envKey>/rootfs`, shared across all branch worktrees like the Docker Gradle cache volume; `<envKey>` derives from a hash of the source, so a changed `rootfs` unpacks a fresh environment and stale ones can be pruned.
 Per-branch Gradle caches persist in `.git/werkator/buildenv/home`, bound as `/root`.
 `bwrap.env` adds environment variables inside the sandbox.
 Files created inside the sandbox are owned by the host user, because uid 0 maps back to the unprivileged webspace user.
 `docker` and `bwrap` are mutually exclusive per branch: enabling both is rejected at start, not silently picked.
 Git works inside the sandbox exactly as inside the Docker container: the primary `.git` is mounted read-only with `.git/werkator/` masked, so builds can run read-only git commands but never reach the machine config or the control token.
 ## `.git/werkator/.werkator.yml` (not committed)
 ```yaml
@@ -95,6 +95,17 @@ bwrap --unshare-user --unshare-pid --die-with-parent --uid 0 --gid 0 \
 - No Docker inside the sandbox, so no Testcontainers-based tests; build commands must select a Docker-free test subset.
  For Werkator's own build this means `TestcontainersSmokeTest` must become conditional (`enabledIf` docker present) — that change is part of this step.
 **Done (branch `bwrap-build-runtime`):** `TestcontainersSmokeTest` is now gated with `enabledIf docker available` — it is reported as skipped (never failed) when no Docker daemon is reachable, and runs as before when one is.
 The two manual steps of the workflow have scripts in `tools/`:
 `tools/build-bwrap-rootfs.sh` builds the rootfs archive (debootstrap-minbase Debian + JDK 21 + git + locales) on any Docker machine — the rootfs is *not* built on the target; `tools/werkator-build-prerequisites.sh` re-runs the precondition command line above on the target webspace and checks all three signals.
 It also checks the disk/quota situation: free space via `df` plus group-quota headroom (`quota -g` limit minus usage) against the ~4 GiB build footprint (unpacked rootfs + Gradle cache + artifacts) — an undersized quota fails the check, since a build would otherwise be blocked mid-flight (experienced on h68: 1 GiB group quota).
 **Planned: a central `tools/remote` control script** (same pattern as the user's other repos; first argument is the repo, here `werkator`, then a command):
 - `tools/remote werkator check-prerequisites <user>@<host>` — runs `werkator-build-prerequisites.sh` on the target host (uploaded if missing).
 - `tools/remote werkator install <user>@<host>` — full setup, idempotent: ensure SSH access (`ssh-copy-id`, first login asks for the password), run the prerequisites check, upload runtime bundle + rootfs archive to `~/.werkator/`, unpack, clone the repo, run `werkator init`, write the machine-local bwrap config.
  The one manual dependency remains: the host's public SSH key must be added to GitHub once; the script prints the key and waits.
 ## Web Access under a Domain (no Docker, no managed nginx)
 The managed nginx/TLS container from ADR 0005 is for container hosts without a reverse proxy.
@@ -0,0 +1,135 @@
 > **WARNING:** This document describes only the change applied in this PR.
 > It may already be outdated once the next PR is merged.
 > Historic PR-documentation is not maintained along with new PRs — treat it as a snapshot, not as current documentation.
 ## The Problem
 Werkator runs its builds on the host, either natively or inside a Docker container.
 A Hostsharing **Managed Webspace** has neither root nor a Docker daemon, so neither runtime works there — yet that is exactly where some users want to run a Werkator that builds Werkator itself.
 The only sandboxing primitive available there is `bwrap` (bubblewrap): unprivileged user namespaces with a uid-0 mapping and read-only root binds.
 Step 17 (docs/plan/17-bwrap-build-runtime.md) defines a third build runtime behind the `BuildRunner` interface that is based on it.
 ## Non-Goals
 - No change to the native and Docker runtimes; bwrap is added alongside them and selected per branch via config.
 - No Docker inside the sandbox; Testcontainers-based tests cannot run there and are excluded by a branch's own build command.
 - No overlayfs: the webspace's bubblewrap 0.8.0 predates `--overlay`, so a throwaway writable rootfs per build is out of scope.
 - No web-access deployment; that half of step 17 (Apache `.htaccess` proxy, systemd user unit, Let's Encrypt) needs a real webspace and is written up separately.
 - No per-repo build automation on the webspace; this PR makes it possible, and a follow-up runs it on a real host.
 ## The Scenarios
 ### Feature: bubblewrap as the third build runtime
 #### Background
 - A branch selects exactly one runtime: nothing (native), `docker.enabled`, or `bwrap.enabled`.
 - `bwrap.enabled` and `bwrap.rootfs` are pinned (host-set), so a branch's committed config cannot switch its own sandbox off or substitute a foreign rootfs.
 - `docker.enabled` and `bwrap.enabled` are mutually exclusive per branch; enabling both is rejected, not silently picked.
 #### Scenario#000.01: A bwrap build runs the command inside the sandbox as root
 So that the build is isolated from the host exactly as the native and Docker runtimes intend.
 - **Given** a branch with `bwrap.enabled` and a `bwrap.rootfs` archive
 - **When** a build for that branch starts
 - **Then** Werkator unpacks the rootfs on demand into `.git/werkator/buildenv/<envKey>/rootfs`
  - **and** invokes `bwrap` with a uid-0 mapping, a read-only root bind of the rootfs, the workspace bound at its host path, and `--chdir` into it
  - **and** the returned process is the attached `bwrap` process, so streaming and cancellation behave like native builds
 - **and** the environment plus `bwrap.env` are passed via `--setenv`
 ##### Verified by
 - [BwrapBuildRunnerTest](../../src/test/kotlin/de/hoennig/werkator/build/BwrapBuildRunnerTest.kt)
 #### Scenario#000.02: Git metadata mounts keep secrets out of the sandbox
 So that builds can run read-only git commands but never reach the machine config or the control token.
 - **Given** a workspace that is a worktree of the repository
 - **When** the sandbox is assembled
 - **Then** the primary `.git` is bound read-only
  - **and** `.git/werkator/` is masked by an empty tmpfs
  - **and** the worktree admin directory is bound read-write
 ##### Verified by
 - [BwrapBuildRunnerTest](../../src/test/kotlin/de/hoennig/werkator/build/BwrapBuildRunnerTest.kt)
 #### Scenario#000.03: A branch cannot turn its sandbox off or swap its rootfs
 So that the pinned sandbox policy holds for builds a branch invents as well as for ones the host already knows.
 - **Given** a branch whose committed config sets `bwrap.enabled` or `bwrap.rootfs`
 - **When** that config is resolved into a build
 - **Then** the pinned keys are stripped from the worktree layer
  - **and** enabling both `docker` and `bwrap` on a build is rejected, not picked silently
 ##### Verified by
 - [ConfigLoaderTest](../../src/test/kotlin/de/hoennig/werkator/config/ConfigLoaderTest.kt)
 #### Scenario#000.04: The dispatcher routes builds to the bwrap runtime
 So that a bwrap-explicit branch builds inside the sandbox rather than natively.
 - **Given** a branch with `bwrap.enabled`
 - **When** its build is dispatched
 - **Then** `BwrapBuildRunner` is selected
 ##### Verified by
 - [DispatchingBuildRunnerTest](../../src/test/kotlin/de/hoennig/werkator/build/DispatchingBuildRunnerTest.kt)
 ### Feature: Werkator builds itself without Docker
 #### Background
 - Werkator's own build runs the full test suite, which includes `TestcontainersSmokeTest`.
 - On a Docker-less host (the webspace, and the bwrap sandbox that builds there), that test must not fail the self-build.
 #### Scenario#000.05: The Testcontainers smoke test is skipped, not failed, without Docker
 So that a Docker-less build of Werkator itself stays green.
 - **Given** no reachable Docker daemon
 - **When** the test suite runs
 - **Then** `TestcontainersSmokeTest` is reported as skipped
  - **and** the build is not failed by it
 - **and** with a Docker daemon present the test still runs and verifies a container
 ##### Verified by
 - [TestcontainersSmokeTest](../../src/test/kotlin/de/hoennig/werkator/framework/TestcontainersSmokeTest.kt)
 ## The Solution
 **A third `BuildRunner` by the same shell-out pattern as git and Docker.**
 `BwrapBuildRunner` shells out to the `bwrap` CLI (no library), unpacks a prepared Debian rootfs on demand into `.git/werkator/buildenv/<envKey>/rootfs`, reuses the step-16 git-metadata mounts verbatim, binds a persistent `.git/werkator/buildenv/home` as `/root`, and returns the attached `bwrap` process so streaming and cancellation match the other runtimes.
 `DispatchingBuildRunner` routes by `bwrap.enabled`.
 **Pinning and mutual exclusion hold at the choke point.**
 `bwrap.enabled` and `bwrap.rootfs` join the pinned sandbox-policy set, stripped from the worktree layer so a branch cannot disable its sandbox or substitute a foreign rootfs.
 `docker` and `bwrap` are mutually exclusive per build, rejected in `buildSettings` — the single point every build passes through — instead of picked silently.
 **Tooling makes the two manual steps reproducible.**
 `tools/build-bwrap-rootfs.sh` builds the rootfs archive (debootstrap-minbase Debian + JDK 21 + git + locales) on any machine with Docker, since `debootstrap` is not available on the target.
 `tools/werkator-build-prerequisites.sh` re-runs the exact precondition command line from the plan on the target webspace and checks all three signals.
 `TestcontainersSmokeTest` is gated with `enabledIf docker available`, so a Docker-less self-build skips it.
 ## Open Questions
 - Whether the rootfs archive built by `tools/build-bwrap-rootfs.sh` is complete for `./gradlew build` has not been exercised on a real webspace; the JDK 21, git and locales package set is a good baseline but project-specific tooling must be added.
 - The `systemd` user unit for a webspace should gain `MemoryMax`/`TasksMax` (configurable per the plan); not implemented here.
 ## Additional Changes
 - Config template (`InitCommand`), `docs/configuration.md` and `AGENTS.md` updated so all three config places stay in sync and the pinned set is documented.
 - The plan file 17 records the precondition result and the new tooling.
 ## Follow-up PRs
 - ADR 0007 recording the bubblewrap runtime decision (options: bwrap vs proot/fakechroot vs plain native).
 - Web-access deployment on a real webspace and the `docs/deployment.md` third variant, written once verified there.
 - `MemoryMax`/`TasksMax` on the webspace systemd unit.
@@ -44,13 +44,16 @@ class ProcessBuildRunner : BuildRunner {
 /**
 * Selects the runtime per branch: Docker when `branches.<name>.docker.enabled`,
- * native shell execution otherwise (the unchanged default).
+ * bubblewrap when `branches.<name>.bwrap.enabled`, native shell execution otherwise
 * (the unchanged default). Docker and bwrap are mutually exclusive per branch and are
 * rejected together at config load, so the branch order here never has to "pick".
 */
@Primary
@Component
 class DispatchingBuildRunner(
    private val processBuildRunner: ProcessBuildRunner,
    private val dockerBuildRunner: DockerBuildRunner,
    private val bwrapBuildRunner: BwrapBuildRunner,
 ) : BuildRunner {
    override fun start(
        command: String,
@@ -60,7 +63,12 @@ class DispatchingBuildRunner(
        branchConfig: BranchConfig,
        onAuxProcess: (Process) -> Unit,
    ): Process {
-        val runner = if (branchConfig.docker.enabled) dockerBuildRunner else processBuildRunner
+        val runner =
            when {
                branchConfig.docker.enabled -> dockerBuildRunner
                branchConfig.bwrap.enabled -> bwrapBuildRunner
                else -> processBuildRunner
            }
        return runner.start(command, workingDir, environment, repoDir, branchConfig, onAuxProcess)
    }
 }
@@ -0,0 +1,266 @@
 package de.hoennig.werkator.build
 import de.hoennig.werkator.config.BranchConfig
 import de.hoennig.werkator.config.BwrapConfig
 import de.hoennig.werkator.git.GitCommandRunner
 import org.slf4j.LoggerFactory
 import org.springframework.stereotype.Component
 import java.nio.file.Files
 import java.nio.file.Path
 import java.security.MessageDigest
 /**
 * Runs build commands inside a bubblewrap user-namespace sandbox (Step 17 / ADR 0007),
 * for hosts without root and without a Docker daemon (e.g. Hostsharing managed
 * webspaces). Shells out to the `bwrap` CLI via the generic [GitCommandRunner] process
 * wrapper — no library, consistent with git and docker.
 *
 * The prepared rootfs (a Debian-base archive built elsewhere, since `debootstrap` is not
 * available on the target) is unpacked on demand into `.git/werkator/buildenv/<envKey>/rootfs`,
 * shared across all branch worktrees like the Docker gradle cache volume; `<envKey>` derives
 * from a hash of the archive source, so a changed source unpacks a fresh rootfs and stale
 * ones can be pruned. The returned [Process] is the attached `bwrap` process, so log
 * streaming and cancellation work exactly like native builds (`--die-with-parent` plus
 * `--unshare-pid` tear down the whole tree on cancel). Git works inside the sandbox with
 * the same layered mounts as the Docker runner: the primary `.git` read-only with
 * `.git/werkator/` masked, see [gitMetadataMounts].
 */
@Component
 class BwrapBuildRunner(
    private val commandRunner: GitCommandRunner,
 ) : BuildRunner {
    private val log = LoggerFactory.getLogger(BwrapBuildRunner::class.java)
    /** Replaceable process launcher so unit tests can capture the assembled `bwrap` argv. */
    internal var processStarter: (List<String>, Path) -> Process = { command, dir ->
        ProcessBuilder(command).directory(dir.toFile()).start()
    }
    override fun start(
        command: String,
        workingDir: Path,
        environment: Map<String, String>,
        repoDir: Path,
        branchConfig: BranchConfig,
        onAuxProcess: (Process) -> Unit,
    ): Process {
        val bwrap = branchConfig.bwrap
        require(bwrap.rootfs.isNotBlank()) { "branches.<name>.bwrap.rootfs must be set when bwrap.enabled is true" }
        val buildEnvRoot = buildEnvRoot(repoDir)
        val envKey = envKey(bwrap.rootfs)
        val rootfsDir = buildEnvRoot.resolve(envKey).resolve(ROOTFS_DIR)
        ensureRootfs(bwrap, rootfsDir, repoDir, onAuxProcess)
        val homeDir = buildEnvRoot.resolve(HOME_DIR)
        Files.createDirectories(homeDir)
        val args =
            invocation(command, workingDir, environment, repoDir, bwrap, rootfsDir, homeDir)
        ensureMountpoints(rootfsDir, args)
        return processStarter(args, repoDir)
    }
    /**
     * bwrap creates mountpoint directories for bind destinations inside the sandbox —
     * against the read-only rootfs bind that fails with "Can't mkdir parents ...
     * Read-only file system" for every destination that does not exist in the rootfs
     * (the workspace under the repo, for example). The rootfs directory itself is a
     * plain host directory, so we pre-create the mountpoints there; bwrap then finds
     * them and has nothing left to mkdir.
     */
    private fun ensureMountpoints(
        rootfsDir: Path,
        args: List<String>,
    ) {
        var i = 0
        while (i < args.size) {
            val arg = args[i]
            if (arg == "--bind" || arg == "--ro-bind") {
                val dest = args[i + 2]
                val mountpoint = rootfsDir.resolve(dest.substring(1))
                // Skip anything that already exists in the rootfs (e.g. /etc/resolv.conf
                // is a file the rootfs ships); only missing dirs are created.
                if (dest.startsWith("/") && !Files.exists(mountpoint)) {
                    Files.createDirectories(mountpoint)
                }
                i += 3
            } else if (arg == "--proc" || arg == "--dev" || arg == "--tmpfs") {
                // The rootfs archive ships no /proc, /dev (excluded when packed), so
                // these mountpoints must exist too.
                val dest = args[i + 1]
                if (dest.startsWith("/") && !Files.exists(rootfsDir.resolve(dest.substring(1)))) {
                    Files.createDirectories(rootfsDir.resolve(dest.substring(1)))
                }
                i += 2
            } else {
                i += 1
            }
        }
    }
    /**
     * Unpacks the configured archive into [rootfsDir] once per environment version
     * (identified by [envKey]). Missing means "not yet unpacked"; the environment is a
     * cache like the Docker image and the Gradle volume, and stale ones are pruned with
     * the rest of `.git/werkator`.
     */
    private fun ensureRootfs(
        bwrap: BwrapConfig,
        rootfsDir: Path,
        repoDir: Path,
        onAuxProcess: (Process) -> Unit,
    ) {
        if (Files.isDirectory(rootfsDir)) {
            return
        }
        Files.createDirectories(rootfsDir)
        val archive = localArchive(bwrap.rootfs, rootfsDir.parent, repoDir, onAuxProcess)
        log.info("unpacking build environment {} into {}", bwrap.rootfs, rootfsDir)
        commandRunner.runOrThrow(
            listOf("tar", "--no-same-owner", "-xf", archive, "-C", rootfsDir.toString()),
            repoDir,
            onProcess = onAuxProcess,
        )
    }
    /**
     * Resolves [BwrapConfig.rootfs] to a local archive path: a bare or `file:` path is
     * used as-is; an `http(s)` URL is downloaded once into the buildenv root. GNU tar
     * auto-detects the compression from the archive magic, so a `.tar.gz` or `.tar.zst`
     * needs no extra flag.
     */
    private fun localArchive(
        rootfs: String,
        envDir: Path,
        repoDir: Path,
        onAuxProcess: (Process) -> Unit,
    ): String {
        if (!rootfs.startsWith("http://") && !rootfs.startsWith("https://")) {
            return rootfs.removePrefix("file://")
        }
        val fileName = rootfs.substringAfterLast('/').ifBlank { "buildenv" }
        val target = envDir.resolve(fileName)
        if (!Files.exists(target)) {
            log.info("downloading build environment {} from {}", fileName, rootfs)
            commandRunner.runOrThrow(
                listOf("curl", "-fsSL", "-o", target.toString(), rootfs),
                repoDir,
                onProcess = onAuxProcess,
            )
        }
        return target.toString()
    }
    private fun invocation(
        command: String,
        workspace: Path,
        environment: Map<String, String>,
        repoDir: Path,
        bwrap: BwrapConfig,
        rootfsDir: Path,
        homeDir: Path,
    ): List<String> {
        // bwrap creates mountpoints for bind destinations inside the sandbox; a
        // relative workspace path would resolve there into the read-only rootfs
        // ("Can't mkdir parents ...: Read-only file system"). Bind at absolute
        // host paths instead — same contract as the Docker runner. Relative
        // paths come from the CLI relative to the repo, so resolve them against
        // repoDir, not against the process working directory.
        val repoDirAbs = repoDir.toAbsolutePath().normalize()
        val workspaceAbs =
            if (workspace.isAbsolute) workspace.normalize() else repoDirAbs.resolve(workspace).normalize()
        val homeDirAbs =
            if (homeDir.isAbsolute) homeDir.normalize() else repoDirAbs.resolve(homeDir).normalize()
        val args =
            mutableListOf(
                "bwrap",
                "--unshare-user",
                "--unshare-pid",
                "--die-with-parent",
                "--uid",
                "0",
                "--gid",
                "0",
                "--ro-bind",
                rootfsDir.toString(),
                "/",
            )
        // Bind the repo read-write FIRST so bwrap can create the mountpoints of
        // the later binds (workspace, worktree admin dir) inside it — creating
        // them against the read-only rootfs fails with "Can't mkdir parents ...
        // Read-only file system". The git metadata mounts below then layer the
        // usual isolation on top: read-only .git, tmpfs mask over .git/werkator,
        // read-write worktree admin dir.
        args += listOf("--bind", "$repoDirAbs", "$repoDirAbs")
        // Git metadata mounts BEFORE the workspace bind: the tmpfs mask over
        // .git/werkator must not shadow the workspace, which lives under
        // .git/werkator/worktrees — the later workspace bind shadows the mask
        // at exactly its own path and nothing else.
        args += gitMetadataMounts(workspaceAbs, repoDir)
        args += listOf("--bind", "$workspaceAbs", "$workspaceAbs")
        args += listOf("--bind", "$homeDirAbs", "/root")
        args += listOf("--ro-bind", "/etc/resolv.conf", "/etc/resolv.conf")
        args += listOf("--proc", "/proc", "--dev", "/dev", "--tmpfs", "/tmp")
        args += listOf("--setenv", "HOME", "/root")
        for ((key, value) in environment) {
            args += listOf("--setenv", key, value)
        }
        for ((key, value) in bwrap.env) {
            args += listOf("--setenv", key, value)
        }
        args += listOf("--chdir", "$workspaceAbs", "/bin/sh", "-c", command)
        return args
    }
    /**
     * Makes git work inside the sandbox without exposing Werkator's secrets — the same
     * three layered mounts as the Docker runner, expressed in `bwrap` flags (bwrap nests
     * mounts by target path like Docker): the primary `.git` read-only, an empty tmpfs
     * masking `.git/werkator/` (machine config with `git.token`, control token, build
     * state), and this worktree's admin directory read-write so index-refreshing commands
     * keep working. Object and ref writes stay blocked by the read-only `.git` mount.
     * No mounts are added when the workspace is not a worktree of [repoDir].
     */
    private fun gitMetadataMounts(
        workspace: Path,
        repoDir: Path,
    ): List<String> {
        val gitDir = repoDir.toAbsolutePath().normalize().resolve(".git")
        val workspaceGitFile = workspace.resolve(".git")
        if (!Files.isDirectory(gitDir) || !Files.isRegularFile(workspaceGitFile)) {
            return emptyList()
        }
        val adminDir =
            Files
                .readString(workspaceGitFile)
                .substringAfter("gitdir:", "")
                .trim()
                .takeIf { it.isNotEmpty() }
                ?.let { workspace.resolve(it).normalize() }
                ?: return emptyList()
        if (!adminDir.startsWith(gitDir) || !Files.isDirectory(adminDir)) {
            return emptyList()
        }
        val args = mutableListOf("--ro-bind", "$gitDir", "$gitDir")
        val werkatorDir = gitDir.resolve("werkator")
        if (Files.isDirectory(werkatorDir)) {
            args += listOf("--tmpfs", "$werkatorDir")
        }
        args += listOf("--bind", "$adminDir", "$adminDir")
        return args
    }
    private fun buildEnvRoot(repoDir: Path): Path = repoDir.resolve(BUILDENV_DIR)
    /** A short hash of the archive source, so a changed source unpacks a fresh rootfs. */
    private fun envKey(rootfs: String): String =
        MessageDigest
            .getInstance("SHA-256")
            .digest(rootfs.toByteArray())
            .joinToString("") { "%02x".format(it) }
            .take(12)
    companion object {
        const val BUILDENV_DIR = ".git/werkator/buildenv"
        const val ROOTFS_DIR = "rootfs"
        const val HOME_DIR = "home"
    }
 }
@@ -19,6 +19,7 @@ import java.nio.file.Paths
 )
 class InitCommand(
    private val gitService: GitService,
    private val configLoader: de.hoennig.werkator.config.ConfigLoader,
    /** The version written into the generated config as `werkator.version.since`. */
    private val buildProperties: ObjectProvider<BuildProperties>? = null,
 ) : Runnable {
@@ -157,6 +158,11 @@ class InitCommand(
              bindAddress: 127.0.0.1
              # optional Impressum (legal disclosure) link in the web UI footer; empty hides the link
              impressumUrl: ""
              # Resource limits of the systemd user unit (init --systemd); empty = directive omitted.
              # Needed where the service shares a memory slice, e.g. Hostsharing Managed Webspaces.
              systemd:
                memoryMax: ""           # e.g. 1G — a runaway Gradle build must not starve the package
                tasksMax: ""            # e.g. 512
              # Opt-in managed nginx+certbot Docker container for HTTPS, for hosts without
              # a usable reverse proxy (see docs/deployment.md). Off by default.
              nginx:
@@ -215,6 +221,12 @@ class InitCommand(
                  context: "."          # Docker build context used with dockerfile
                  network: ""           # Docker network mode for the build container; empty = Docker default (pinned)
                  env: {}               # additional environment variables set inside the build container
                # bubblewrap user-namespace sandbox — for hosts without root and without a
                # Docker daemon (e.g. Hostsharing managed webspaces). Mutually exclusive with docker.
                bwrap:
                  enabled: false        # run clean/build in a bwrap sandbox instead of natively (pinned)
                  rootfs: ""            # prepared rootfs archive (path or URL); required when enabled (pinned)
                  env: {}               # additional environment variables set inside the sandbox
                # Gitea check this build reports as; empty uses gitea.statusContext.
                # Two builds of one commit under the same context overwrite each other.
                statusContext: ""
@@ -256,6 +268,19 @@ class InitCommand(
        println("created ${file.toFile().relativeTo(normalizedWorkingDir.toFile())}")
    }
    /**
     * Resource limits for the unit come from the effective configuration when one is
     * already loadable (re-running `init --systemd` on an installed instance); during
     * the very first bootstrap they stay unset and the defaults (no directives) apply.
     */
    private fun loadedSystemdConfig(): de.hoennig.werkator.config.SystemdConfig =
        try {
            configLoader.load(Paths.get(".")).server.systemd
        } catch (_: Exception) {
            de.hoennig.werkator.config
                .SystemdConfig()
        }
    private fun createSystemdFiles(
        root: Path,
        normalizedWorkingDir: Path,
@@ -277,6 +302,8 @@ class InitCommand(
                javaExecutable = javaExecutableResolver(),
                jarPath = jarPath,
                envFile = envFile,
                memoryMax = loadedSystemdConfig().memoryMax,
                tasksMax = loadedSystemdConfig().tasksMax,
            ),
        )
        println("created ${unitFile.toFile().relativeTo(normalizedWorkingDir.toFile())}")
@@ -22,24 +22,33 @@ object SystemdServiceFiles {
        javaExecutable: Path,
        jarPath: Path,
        envFile: Path,
-    ): String =
+        memoryMax: String = "",
-        """
+        tasksMax: String = "",
-        [Unit]
+    ): String {
-        Description=Werkator CI for ${repoRoot.fileName}
+        val limits =
-        Wants=network-online.target
+            listOfNotNull(
-        After=network-online.target docker.service
+                "MemoryMax=$memoryMax".takeIf { memoryMax.isNotBlank() },
                "TasksMax=$tasksMax".takeIf { tasksMax.isNotBlank() },
            ).joinToString("\n")
        val limitsLine = if (limits.isEmpty()) "" else "\n$limits"
        return """
            [Unit]
            Description=Werkator CI for ${repoRoot.fileName}
            Wants=network-online.target
            After=network-online.target docker.service
-        [Service]
+            [Service]
-        Type=simple
+            Type=simple
-        WorkingDirectory=${systemdPath("$repoRoot")}
+            WorkingDirectory=${systemdPath("$repoRoot")}
-        EnvironmentFile=-${systemdPath("$envFile")}
+            EnvironmentFile=-${systemdPath("$envFile")}
-        ExecStart=${systemdQuote("$javaExecutable")} ${'$'}JAVA_OPTS -jar ${systemdQuote("$jarPath")} server
+            ExecStart=${systemdQuote("$javaExecutable")} ${'$'}JAVA_OPTS -jar ${systemdQuote("$jarPath")} server
-        Restart=always
+            Restart=always
-        RestartSec=30
+            RestartSec=30
-        [Install]
+            [Install]
-        WantedBy=default.target
+            WantedBy=default.target
-        """.trimIndent() + "\n"
+            """.trimIndent().replace("\n\n[Install]", "$limitsLine\n\n[Install]") + "\n"
    }
    /**
     * Nightly Docker cleanup like the legacy `docker-prune.service`, but without `--volumes`:
@@ -42,6 +42,8 @@ data class BuildDefinition(
    val statusContext: String? = null,
    /** Overrides of the docker settings; null inherits them. */
    val docker: DockerOverrides? = null,
    /** Overrides of the bwrap settings; null inherits them. */
    val bwrap: BwrapOverrides? = null,
 ) {
    /** The settings this build runs with: [branchConfig] with this definition applied; unset values fall through. */
    fun applyTo(branchConfig: BranchConfig): BranchConfig =
@@ -62,6 +64,12 @@ data class BuildDefinition(
                    network = docker?.network ?: branchConfig.docker.network,
                    env = docker?.env ?: branchConfig.docker.env,
                ),
            bwrap =
                branchConfig.bwrap.copy(
                    enabled = bwrap?.enabled ?: branchConfig.bwrap.enabled,
                    rootfs = bwrap?.rootfs ?: branchConfig.bwrap.rootfs,
                    env = bwrap?.env ?: branchConfig.bwrap.env,
                ),
        )
    companion object {
@@ -159,3 +167,12 @@ data class DockerOverrides(
    val context: String? = null,
    val env: Map<String, String>? = null,
 )
 /** Nullable bubblewrap overrides of a [BuildDefinition]; null values inherit the branch's setting. */
 data class BwrapOverrides(
    /** Run the build in the bwrap sandbox instead of natively. Pinned — a branch must not escape its sandbox. */
    val enabled: Boolean? = null,
    /** Rootfs archive source. Pinned — a branch must not substitute a foreign rootfs. */
    val rootfs: String? = null,
    val env: Map<String, String>? = null,
 )
@@ -159,6 +159,11 @@ class ConfigLoader(
            val strippedDocker = docker.toMutableMap().apply { PINNED_DOCKER_KEYS.forEach { remove(it) } }
            if (strippedDocker.isEmpty()) result.remove("docker") else result["docker"] = strippedDocker
        }
        val bwrap = entry["bwrap"] as? Map<String, Any?>
        if (bwrap != null) {
            val strippedBwrap = bwrap.toMutableMap().apply { PINNED_BWRAP_KEYS.forEach { remove(it) } }
            if (strippedBwrap.isEmpty()) result.remove("bwrap") else result["bwrap"] = strippedBwrap
        }
        return result
    }
@@ -398,6 +403,9 @@ class ConfigLoader(
        /** `docker` keys a branch must never override: the sandbox policy. */
        private val PINNED_DOCKER_KEYS = setOf("enabled", "network")
        /** `bwrap` keys a branch must never override: the sandbox policy (Step 17). */
        private val PINNED_BWRAP_KEYS = setOf("enabled", "rootfs")
        /**
         * The one key of a build definition that says *when* and *for which branches* it
         * runs; never inherited from `builds.default`. A single key on purpose: a selector
@@ -37,7 +37,14 @@ data class WerkatorConfig(
        build: String,
    ): BranchConfig {
        val branchConfig = branches[branch] ?: branches["default"] ?: BranchConfig()
-        return effectiveBuildDefinitions()[build]?.applyTo(branchConfig) ?: branchConfig
+        val settings = effectiveBuildDefinitions()[build]?.applyTo(branchConfig) ?: branchConfig
        if (settings.docker.enabled && settings.bwrap.enabled) {
            throw IllegalArgumentException(
                "builds.$build on '$branch' enables both docker and bwrap; a build runs in exactly one sandbox. " +
                    "Disable one of them.",
            )
        }
        return settings
    }
 }
@@ -57,9 +64,24 @@ data class ServerConfig(
    val bindAddress: String = "127.0.0.1",
    /** Optional Impressum (legal disclosure) link shown in the web UI footer; empty hides the link. */
    val impressumUrl: String = "",
    /**
     * Resource limits for the generated systemd user unit (`init --systemd`); empty
     * means the directive is not written. Needed on platforms where the service runs
     * inside a shared memory slice, e.g. Hostsharing Managed Webspaces, where a
     * runaway Gradle build would starve everything else in the package.
     */
    val systemd: SystemdConfig = SystemdConfig(),
    val nginx: NginxConfig = NginxConfig(),
 )
 /** Resource-limit directives of the systemd user unit (`server.systemd`, see [ServerConfig.systemd]). */
 data class SystemdConfig(
    /** `MemoryMax=` of the unit, e.g. `1G`; empty omits the directive. */
    val memoryMax: String = "",
    /** `TasksMax=` of the unit, e.g. `512`; empty omits the directive. */
    val tasksMax: String = "",
 )
 /**
 * Opt-in managed nginx+certbot Docker container serving Werkator over HTTPS,
 * for hosts without a usable reverse proxy (ADR 0005). Off by default; the
@@ -157,6 +179,26 @@ data class BranchConfig(
    val statusContext: String = "",
    val autoBuild: AutoBuildConfig = AutoBuildConfig(),
    val docker: DockerConfig = DockerConfig(),
    /** bubblewrap user-namespace sandbox; mutually exclusive with [docker]. */
    val bwrap: BwrapConfig = BwrapConfig(),
 )
 /**
 * bubblewrap build sandbox (Step 17): runs the build in an unprivileged user namespace
 * with a prepared Debian root filesystem. For hosts without root and without a Docker
 * daemon (e.g. Hostsharing managed webspaces); see `docs/plan/17-bwrap-build-runtime.md`.
 */
 data class BwrapConfig(
    /** Run the clean and build commands in a bwrap sandbox instead of natively. */
    val enabled: Boolean = false,
    /**
     * Path or URL of the prepared rootfs archive (e.g. `werkator-buildenv-trixie-java21.tar.zst`),
     * unpacked on demand into `.git/werkator/buildenv/<envKey>/rootfs`; required when [enabled].
     * Pinned — a branch must not substitute a foreign rootfs via its committed config.
     */
    val rootfs: String = "",
    /** Additional environment variables set inside the sandbox. */
    val env: Map<String, String> = emptyMap(),
 )
 data class DockerConfig(
@@ -0,0 +1,240 @@
 package de.hoennig.werkator.build
 import de.hoennig.werkator.config.BranchConfig
 import de.hoennig.werkator.config.BwrapConfig
 import de.hoennig.werkator.git.GitCommandResult
 import de.hoennig.werkator.git.GitCommandRunner
 import io.kotest.assertions.throwables.shouldThrow
 import io.kotest.core.spec.style.FunSpec
 import io.kotest.matchers.shouldBe
 import io.kotest.matchers.string.shouldContain
 import io.mockk.clearMocks
 import io.mockk.every
 import io.mockk.mockk
 import io.mockk.verify
 import java.nio.file.Files
 import java.nio.file.Path
 class BwrapBuildRunnerTest : FunSpec() {
    private val commandRunner = mockk<GitCommandRunner>()
    private lateinit var runner: BwrapBuildRunner
    private lateinit var repoDir: Path
    private lateinit var workspace: Path
    private val captured = mutableListOf<List<String>>()
    private fun bwrapBranchConfig(
        rootfs: String = "/srv/buildenv.tar.zst",
        env: Map<String, String> = emptyMap(),
    ): BranchConfig =
        BranchConfig(
            bwrap =
                BwrapConfig(
                    enabled = true,
                    rootfs = rootfs,
                    env = env,
                ),
        )
    private fun rootfsUnpacked(rootfs: String = "/srv/buildenv.tar.zst"): Path =
        repoDir
            .resolve(BwrapBuildRunner.BUILDENV_DIR)
            .resolve(rootfs.sha12())
            .resolve(BwrapBuildRunner.ROOTFS_DIR)
    init {
        beforeEach {
            clearMocks(commandRunner)
            captured.clear()
            repoDir = Files.createTempDirectory("werkator-bwrap-runner")
            workspace = repoDir.resolve("workspace")
            runner = BwrapBuildRunner(commandRunner)
            runner.processStarter = { command, _ ->
                captured += command
                ProcessBuilder("true").start()
            }
        }
        test("unpacks the rootfs on demand and assembles the exact bwrap command") {
            every {
                commandRunner.runOrThrow(
                    listOf("tar", "--no-same-owner", "-xf", "/srv/buildenv.tar.zst", "-C", rootfsUnpacked().toString()),
                    repoDir,
                    any(),
                    any(),
                )
            } returns
                GitCommandResult(0, "", "")
            runner.start("./gradlew test", workspace, mapOf("branch" to "main"), repoDir, bwrapBranchConfig())
            val args = captured.single()
            val rootfsDir = args[args.indexOf("--ro-bind") + 1]
            args shouldBe
                listOf(
                    "bwrap",
                    "--unshare-user",
                    "--unshare-pid",
                    "--die-with-parent",
                    "--uid",
                    "0",
                    "--gid",
                    "0",
                    "--ro-bind",
                    rootfsUnpacked().toString(),
                    "/",
                    "--bind",
                    repoDir.toString(),
                    repoDir.toString(),
                    "--bind",
                    workspace.toString(),
                    workspace.toString(),
                    "--bind",
                    repoDir.resolve(".git/werkator/buildenv/home").toString(),
                    "/root",
                    "--ro-bind",
                    "/etc/resolv.conf",
                    "/etc/resolv.conf",
                    "--proc",
                    "/proc",
                    "--dev",
                    "/dev",
                    "--tmpfs",
                    "/tmp",
                    "--setenv",
                    "HOME",
                    "/root",
                    "--setenv",
                    "branch",
                    "main",
                    "--chdir",
                    workspace.toString(),
                    "/bin/sh",
                    "-c",
                    "./gradlew test",
                )
            Files.isDirectory(rootfsUnpacked()) shouldBe true
        }
        test("binds a relative workspace path at its absolute location") {
            // bwrap creates mountpoints for bind destinations inside the sandbox;
            // a relative path would land in the read-only rootfs and fail with
            // "Can't mkdir parents ...: Read-only file system" (seen on the webspace).
            every {
                commandRunner.runOrThrow(
                    listOf("tar", "--no-same-owner", "-xf", "/srv/buildenv.tar.zst", "-C", rootfsUnpacked().toString()),
                    repoDir,
                    any(),
                    any(),
                )
            } returns
                GitCommandResult(0, "", "")
            val relativeWorkspace = repoDir.relativize(workspace)
            runner.start("./gradlew test", relativeWorkspace, mapOf("branch" to "main"), repoDir, bwrapBranchConfig())
            val args = captured.single()
            val absolute = workspace.toAbsolutePath().normalize().toString()
            val bindIdx = args.withIndex().filter { it.value == "--bind" }.map { it.index }
            // first bind is the repo dir (mountpoint base), second is the workspace
            args[bindIdx[1] + 1] shouldBe absolute
            args[bindIdx[1] + 2] shouldBe absolute
            args[args.indexOf("--chdir") + 1] shouldBe absolute
        }
        test("does not re-unpack an already prepared rootfs") {
            Files.createDirectories(rootfsUnpacked())
            runner.start("./gradlew test", workspace, mapOf("branch" to "main"), repoDir, bwrapBranchConfig())
            verify(exactly = 0) { commandRunner.runOrThrow(match { it.first() == "tar" }, any(), any(), any()) }
        }
        test("adds bwrap env and passes the branch environment through") {
            Files.createDirectories(rootfsUnpacked())
            runner.start(
                "./gradlew test",
                workspace,
                mapOf("branch" to "main"),
                repoDir,
                bwrapBranchConfig(env = mapOf("FOO" to "bar")),
            )
            val args = captured.single()
            args[args.indexOf("branch") - 1] shouldBe "--setenv"
            args[args.indexOf("branch") + 1] shouldBe "main"
            args[args.indexOf("FOO") - 1] shouldBe "--setenv"
            args[args.indexOf("FOO") + 1] shouldBe "bar"
        }
        test("exposes git metadata read-only with the werkator dir masked for a worktree workspace") {
            val gitDir = repoDir.resolve(".git")
            val adminDir = gitDir.resolve("worktrees/workspace")
            Files.createDirectories(adminDir)
            Files.createDirectories(gitDir.resolve("werkator"))
            Files.createDirectories(workspace)
            Files.writeString(workspace.resolve(".git"), "gitdir: $adminDir\n")
            Files.createDirectories(rootfsUnpacked())
            runner.start("./gradlew test", workspace, mapOf("branch" to "main"), repoDir, bwrapBranchConfig())
            val args = captured.single()
            args[args.indexOf(gitDir.toString()) - 1] shouldBe "--ro-bind"
            args[args.indexOf("$gitDir/werkator") - 1] shouldBe "--tmpfs"
            args[args.indexOf(adminDir.toString()) - 1] shouldBe "--bind"
        }
        test("mounts no git metadata when the workspace is not a worktree") {
            Files.createDirectories(rootfsUnpacked())
            Files.createDirectories(workspace)
            runner.start("./gradlew test", workspace, mapOf("branch" to "main"), repoDir, bwrapBranchConfig())
            val args = captured.single()
            val gitDir = repoDir.resolve(".git")
            // the sandbox's own /tmp tmpfs is always present; the point is that no tmpfs
            // masks .git/werkator and no worktree admin dir is bound
            args.none { it == "$gitDir/werkator" } shouldBe true
            args.none { it.contains("worktrees/") } shouldBe true
        }
        test("fails without a configured rootfs") {
            val branchConfig = BranchConfig(bwrap = BwrapConfig(enabled = true))
            val exception =
                shouldThrow<IllegalArgumentException> {
                    runner.start("./gradlew test", workspace, mapOf("branch" to "main"), repoDir, branchConfig)
                }
            exception.message shouldContain "bwrap.rootfs"
        }
        test("downloads a URL rootfs once before unpacking") {
            val url = "https://example.test/buildenv.tar.zst"
            val downloadTarget =
                repoDir
                    .resolve(BwrapBuildRunner.BUILDENV_DIR)
                    .resolve(url.sha12())
                    .resolve("buildenv.tar.zst")
            every { commandRunner.runOrThrow(listOf("curl", "-fsSL", "-o", downloadTarget.toString(), url), repoDir, any(), any()) } returns
                GitCommandResult(0, "", "")
            every { commandRunner.runOrThrow(match { it.first() == "tar" }, any(), any(), any()) } returns
                GitCommandResult(0, "", "")
            runner.start("./gradlew test", workspace, mapOf("branch" to "main"), repoDir, bwrapBranchConfig(rootfs = url))
            verify {
                commandRunner.runOrThrow(listOf("curl", "-fsSL", "-o", downloadTarget.toString(), url), repoDir, any(), any())
            }
            verify { commandRunner.runOrThrow(match { it.first() == "tar" }, repoDir, any(), any()) }
        }
    }
    private fun String.sha12(): String =
        java.security.MessageDigest
            .getInstance("SHA-256")
            .digest(toByteArray())
            .joinToString("") { "%02x".format(it) }
            .take(12)
 }
@@ -1,6 +1,7 @@
 package de.hoennig.werkator.build
 import de.hoennig.werkator.config.BranchConfig
 import de.hoennig.werkator.config.BwrapConfig
 import de.hoennig.werkator.config.DockerConfig
 import io.kotest.core.spec.style.FunSpec
 import io.kotest.matchers.shouldBe
@@ -14,12 +15,13 @@ import java.nio.file.Paths
 class DispatchingBuildRunnerTest : FunSpec() {
    private val processBuildRunner = mockk<ProcessBuildRunner>()
    private val dockerBuildRunner = mockk<DockerBuildRunner>()
-    private val dispatcher = DispatchingBuildRunner(processBuildRunner, dockerBuildRunner)
+    private val bwrapBuildRunner = mockk<BwrapBuildRunner>()
    private val dispatcher = DispatchingBuildRunner(processBuildRunner, dockerBuildRunner, bwrapBuildRunner)
    private val process = mockk<Process>()
    private val dir = Paths.get(".")
    init {
-        beforeEach { clearMocks(processBuildRunner, dockerBuildRunner) }
+        beforeEach { clearMocks(processBuildRunner, dockerBuildRunner, bwrapBuildRunner) }
        test("runs natively by default") {
            val branchConfig = BranchConfig()
@@ -28,6 +30,7 @@ class DispatchingBuildRunnerTest : FunSpec() {
            dispatcher.start("cmd", dir, emptyMap(), dir, branchConfig) shouldBe process
            verify { dockerBuildRunner wasNot Called }
            verify { bwrapBuildRunner wasNot Called }
        }
        test("runs in Docker when the branch enables it") {
@@ -37,6 +40,17 @@ class DispatchingBuildRunnerTest : FunSpec() {
            dispatcher.start("cmd", dir, emptyMap(), dir, branchConfig) shouldBe process
            verify { processBuildRunner wasNot Called }
            verify { bwrapBuildRunner wasNot Called }
        }
        test("runs in bwrap when the branch enables it (and not Docker)") {
            val branchConfig = BranchConfig(bwrap = BwrapConfig(enabled = true, rootfs = "/srv/buildenv.tar.zst"))
            every { bwrapBuildRunner.start("cmd", dir, emptyMap(), dir, branchConfig) } returns process
            dispatcher.start("cmd", dir, emptyMap(), dir, branchConfig) shouldBe process
            verify { processBuildRunner wasNot Called }
            verify { dockerBuildRunner wasNot Called }
        }
    }
 }
@@ -14,7 +14,12 @@ import java.nio.file.attribute.PosixFilePermissions
 class InitCommandTest : FunSpec() {
    private val gitService = mockk<GitService>()
-    private val initCommand = InitCommand(gitService)
+    private val initCommand =
        InitCommand(
            gitService,
            de.hoennig.werkator.config
                .ConfigLoader(mockk(relaxed = true)),
        )
    init {
        test("creates config files with auto-detected values") {
@@ -33,6 +33,26 @@ class SystemdServiceFilesTest : FunSpec() {
            content shouldContain "WantedBy=default.target"
        }
        test("resource limits are written when configured and omitted when unset") {
            fun unit(
                memoryMax: String,
                tasksMax: String,
            ) = SystemdServiceFiles.unitFileContent(
                repoRoot = Paths.get("/srv/repos/my-repo"),
                javaExecutable = Paths.get("/usr/bin/java"),
                jarPath = Paths.get("/srv/repos/my-repo/werkator.jar"),
                envFile = Paths.get("/srv/repos/my-repo/werkator.env"),
                memoryMax = memoryMax,
                tasksMax = tasksMax,
            )
            val with = unit(memoryMax = "1G", tasksMax = "512")
            with shouldContain "MemoryMax=1G"
            with shouldContain "TasksMax=512"
            val without = unit(memoryMax = "", tasksMax = "")
            without shouldNotContain "MemoryMax"
            without shouldNotContain "TasksMax"
        }
        test("percent signs in paths are escaped for systemd") {
            val content =
                SystemdServiceFiles.unitFileContent(
@@ -350,6 +350,39 @@ class ConfigLoaderTest : FunSpec() {
            settings.docker.image shouldBe "attacker-image"
        }
        test("a branch cannot disable its bwrap sandbox or substitute a foreign rootfs through a build definition") {
            val dir = Files.createTempDirectory("werkator-test")
            dir.resolve(".werkator.yml").toFile().writeText(
                """
                builds:
                  default:
                    bwrap:
                      enabled: true
                      rootfs: /host/rootfs.tar.zst
                """.trimIndent(),
            )
            val worktree = Files.createTempDirectory("werkator-test-worktree")
            worktree.resolve(".werkator.yml").toFile().writeText(
                """
                builds:
                  default:
                    bwrap:
                      enabled: false
                      rootfs: /attacker/rootfs.tar.zst
                      env:
                        FOO: from-branch
                """.trimIndent(),
            )
            val settings = loader.loadForWorktree(dir, worktree).buildSettings("any-branch", "default")
            // pinned: the sandbox can neither be switched off nor pointed at a foreign rootfs
            settings.bwrap.enabled shouldBe true
            settings.bwrap.rootfs shouldBe "/host/rootfs.tar.zst"
            // everything that describes the build itself stays the branch's own business
            settings.bwrap.env shouldBe mapOf("FOO" to "from-branch")
        }
        test("a build the branch invents inherits the host's sandbox policy") {
            val dir = Files.createTempDirectory("werkator-test")
            dir.resolve(".werkator.yml").toFile().writeText(
@@ -386,6 +419,31 @@ class ConfigLoaderTest : FunSpec() {
            settings.requirePullRequest shouldBe true
        }
        test("enabling both docker and bwrap on a build is rejected, not picked silently") {
            val dir = Files.createTempDirectory("werkator-test")
            dir.resolve(".werkator.yml").toFile().writeText(
                """
                builds:
                  default:
                    docker:
                      enabled: true
                      image: build-env
                    bwrap:
                      enabled: true
                      rootfs: /srv/rootfs.tar.zst
                """.trimIndent(),
            )
            val config = loader.load(dir)
            val exception =
                shouldThrow<IllegalArgumentException> {
                    config.buildSettings("any-branch", "default")
                }
            exception.message shouldContain "both docker and bwrap"
            exception.message shouldContain "builds.default"
        }
        test("an exclusion pattern takes a branch out of a build that would otherwise select it") {
            val dir = Files.createTempDirectory("werkator-test")
            dir.resolve(".werkator.yml").toFile().writeText(
@@ -2,18 +2,35 @@ package de.hoennig.werkator.framework
 import io.kotest.core.spec.style.FunSpec
 import io.kotest.matchers.shouldBe
 import org.testcontainers.DockerClientFactory
 import org.testcontainers.containers.GenericContainer
 import org.testcontainers.utility.DockerImageName
 /**
 * Probes that Testcontainers can actually start a container on this host.
 * Skipped, not failed, when no Docker is present — that is what lets Werkator
 * build itself on a Docker-less host (e.g. a Hostsharing webspace, where its own
 * build runs in the bubblewrap sandbox); see `tools/werkator-build-prerequisites.sh`.
 */
 class TestcontainersSmokeTest :
    FunSpec({
-        test("Testcontainers starts a container") {
+        test("Testcontainers starts a container")
-            val container =
+            .config(enabledIf = { dockerAvailable() }) {
-                GenericContainer(DockerImageName.parse("alpine:3"))
+                val container =
-                    .withCommand("sh", "-c", "sleep 30")
+                    GenericContainer(DockerImageName.parse("alpine:3"))
-            container.start()
+                        .withCommand("sh", "-c", "sleep 30")
-            container.isRunning shouldBe true
+                container.start()
-            container.stop()
+                container.isRunning shouldBe true
-        }
+                container.stop()
-    })
+            }
    }) {
    companion object {
        private fun dockerAvailable(): Boolean =
            try {
                DockerClientFactory.instance().isDockerAvailable()
            } catch (_: Throwable) {
                false
            }
    }
 }
@@ -0,0 +1,116 @@
 #!/usr/bin/env bash
 #
 # Build the Werkator bwrap build environment (rootfs) archive.
 #
 # The bwrap build runtime (step 17 / ADR 0007) runs each build inside a
 # bubblewrap user namespace, chrooted into a *prepared* Debian root filesystem.
 # That rootfs is NOT built on the target (the webspace has no root and no
 # debootstrap), it is built once on any machine that can — most comfortably a
 # machine with Docker — and distributed as an archive, e.g.
 # `werkator-buildenv-trixie-java21.tar.zst`.
 #
 # This script builds exactly that archive: a debootstrap-minbase Debian release
 # plus the packages Werkator itself needs to run `./gradlew build` inside the
 # sandbox (JDK 21, git, ca-certificates, locales, curl/unzip for the wrapper).
 # The whole build runs inside a throwaway Docker container, so no root is
 # needed on the machine running this script.
 #
 # Why it works this way (each quirk learned the hard way):
 #   - The whole job runs in ONE container whose stdin carries a base64-encoded
 #     script (no file is bind-mounted for the script — a bind-mounted script
 #     hit noexec/tmpfs trouble and vanished inside the container).
 #   - The rootfs is built inside the container's own writable layer, NOT on a
 #     bind-mounted host directory — debootstrap "Tried to extract package, but
 #     tar failed" when its target sat on some bind-mounted/special filesystems.
 #   - Only the final `tar --zstd` writes to stdout; every build step is
 #     redirected to stderr, so the archive coming out of `docker run` is pure.
 #
 # Usage: build-bwrap-rootfs.sh [--release trixie] [--mirror URL] [--out path]
 #   --release  Debian release/architecture tail, default "trixie"
 #   --mirror   apt mirror for debootstrap, default http://deb.debian.org/debian
 #   --out      output archive path, default ./werkator-buildenv-<release>.tar.zst
 #
 set -euo pipefail
 die()  { echo "ERROR: $*" >&2; exit 1; }
 warn() { echo "WARNING: $*" >&2; }
 usage() {
    echo "usage: build-bwrap-rootfs.sh [--release trixie] [--mirror URL] [--out path]" >&2
    exit 1
 }
 # --------------------------------------------------------------- arguments --
 release="trixie"
 out=""
 while [ $# -gt 0 ]; do
    case "$1" in
        --release) release="${2:?missing value for --release}"; shift 2 ;;
        --mirror)  mirror="${2:?missing value for --mirror}"; shift 2 ;;
        --out)     out="${2:?missing value for --out}"; shift 2 ;;
        -*) die "unknown option: $1" ;;
        *) usage ;;
    esac
 done
 mirror="${mirror:-http://deb.debian.org/debian}"
 [ -n "$out" ] || out="$(pwd)/werkator-buildenv-${release}.tar.zst"
 command -v docker >/dev/null 2>&1 || die "docker is required to build the rootfs"
 # Rootfs content: Werkator's own build needs a JDK 21 toolchain (Gradle
 # toolchain resolution), git, ca-certificates for HTTPS, locales for git, and
 # curl/unzip/xz-utils/zstd for the Gradle wrapper and general build hygiene.
 # Keep this list additive — project-specific tooling goes on top of this base.
 PKGS="openjdk-21-jdk git ca-certificates locales procps file curl unzip xz-utils zstd"
 # The chroot step runs inside the freshly debootstrapped rootfs; passed into
 # the container as base64 so no nested heredoc corrupts the piped script.
 inner="$(printf '%s' '#!/bin/bash
 set -euxo pipefail
 mount -t proc none /proc
 apt-get update -qq
 apt-get install -y --no-install-recommends '"${PKGS}"'
 apt-get clean
 rm -f /etc/localtime
 locale-gen en_US.UTF-8 de_DE.UTF-8 >/dev/null 2>&1 || true
 update-locale LANG=en_US.UTF-8 >/dev/null 2>&1 || true
 ' | base64 -w0)"
 # The outer script runs inside the Debian container as root. Build noise goes
 # to stderr (fd 1 is saved on fd 3 and restored only for the final tar), so
 # docker stdout is exactly the archive.
 outer="$(printf '%s' '#!/bin/bash
 set -euo pipefail
 export DEBIAN_FRONTEND=noninteractive
 exec 3>&1
 exec 1>&2
 apt-get update -qq
 apt-get install -y --no-install-recommends debootstrap zstd ca-certificates
 mkdir -p /b/rootfs
 debootstrap --variant=minbase --components=main,contrib --include=apt,ca-certificates '"${release}"' /b/rootfs '"${mirror}"'
 mount --bind /proc /b/rootfs/proc
 mount --bind /sys /b/rootfs/sys
 mount --bind /dev /b/rootfs/dev
 echo '"${inner}"' | base64 -d > /b/rootfs/inner.sh
 chmod +x /b/rootfs/inner.sh
 chroot /b/rootfs /bin/bash /inner.sh
 umount /b/rootfs/proc; umount /b/rootfs/sys; umount /b/rootfs/dev
 exec 1>&3
 tar --zstd --exclude=proc --exclude=sys --exclude=dev -C /b/rootfs -cf - .
 ' | base64 -w0)"
 echo "building ${release} rootfs (downloads packages, takes a while; log below)..."
 echo "archive → $out"
 # Stream the base64-encoded outer script into the container over stdin; the
 # archive lands on stdout (redirected to $out), the build log on stderr.
 docker run --rm -i --privileged debian:"${release}-slim" \
    bash -c 'base64 -d | bash' \
    <<<"$outer" >"$out"
 echo
 echo "OK: build environment written to $out"
 echo "    Configure it as branches.<name>.bwrap.rootfs (a bare path or a URL)"
 echo "    on the target Werkator instance to build in this environment."
@@ -0,0 +1,344 @@
 #!/usr/bin/env bash
 #
 # Central control script for remote Werkator operations (same pattern as the
 # `remote` scripts in the other repos): the first argument is the repo selector,
 # the second the command. All connection and deployment values come from the
 # `.env` file in the repository root — never as command line parameters.
 #
 # Usage:
 #   tools/remote werkator check-prerequisites
 #   tools/remote werkator install
 #   tools/remote werkator build            # WERKATOR_BRANCH to override, default main
 #   tools/remote werkator start
 #   tools/remote port-forward start        # background tunnel to the Werkator UI
 #   tools/remote port-forward stop
 #   tools/remote werkator control-token
 #
 # Required in .env:
 #   WERKATOR_REMOTE   user@host to operate on, e.g. mih34-werkator@mih34.hostsharing.net
 #   WERKATOR_PATH     target directory on that host, e.g. /home/storage/mih34/users/werkator
 #
 # Required for `start`:
 #   WERKATOR_PORT     the localhost port assigned by Hostsharing (eigener Serverdienst)
 #   WERKATOR_DOMAIN   the domain served by the managed Apache, e.g. ci.example.de
 #
 # Required for `port-forward`:
 #   WERKATOR_LOCAL_PORT   the local port the browser uses
 # Optional in .env:
 #   WERKATOR_BRANCH      branch for `build` (default: main)
 #   WERKATOR_MEMORY_MAX  systemd MemoryMax for the unit, e.g. 1G (start)
 #   WERKATOR_TASKS_MAX   systemd TasksMax for the unit, e.g. 512 (start)
 #   WERKATOR_ROOTFS      rootfs archive path
 #                       (default: <repo>/build/werkator-buildenv-trixie.tar.zst)
 #
 # Install layout on the host:
 #   $WERKATOR_PATH/werkator/    the repository clone
 #   $WERKATOR_PATH/.werkator/   runtime bundle + rootfs archive
 #
 # `install` performs, in order:
 #   1. check-prerequisites  (bwrap capability + disk/quota, aborts on FAIL)
 #   2. ensure SSH access    (ssh-copy-id on first use; asks for the password)
 #   3. upload artifacts     (runtime bundle, built locally if missing, + rootfs)
 #   4. clone the repository (needs the host SSH key registered at GitHub once —
 #      the script prints the key and waits)
 #   5. `werkator init` + machine-local bwrap configuration
 #
 set -euo pipefail
 REPO="${1:-}"
 COMMAND="${2:-}"
 REPO_ROOT="$(cd "$(dirname "${BASH_SOURCE[0]}")/.." && pwd)"
 PREREQ_SCRIPT="$REPO_ROOT/tools/werkator-build-prerequisites.sh"
 RUNTIME_BUNDLE="$REPO_ROOT/build/distributions/werkator-runtime-linux-x64.tar.gz"
 PID_FILE="/tmp/werkator-port-forward-$(id -u).pid"
 LOG_FILE="/tmp/werkator-port-forward-$(id -u).log"
 usage() {
    sed -n '3,32p' "${BASH_SOURCE[0]}" | sed 's/^# \{0,1\}//'
    exit 2
 }
 require_env() {
    local missing=0
    for name in "$@"; do
        if [ -z "${!name:-}" ]; then
            echo "ERROR: $name is not set — define it in $REPO_ROOT/.env" >&2
            missing=1
        fi
    done
    [ "$missing" -eq 0 ] || exit 1
 }
 [ -n "$REPO" ] && [ -n "$COMMAND" ] || usage
 # Load the connection and deployment values; explicit environment wins, the
 # .env in the repository root fills the rest.
 set -a
 [ -f "$REPO_ROOT/.env" ] && source "$REPO_ROOT/.env"
 set +a
 require_env WERKATOR_REMOTE WERKATOR_PATH
 HOST="$WERKATOR_REMOTE"
 TARGET_DIR="$WERKATOR_PATH"
 ROOTFS="${WERKATOR_ROOTFS:-$REPO_ROOT/build/werkator-buildenv-trixie.tar.zst}"
 ssh_present() {
    ssh -o BatchMode=yes -o ConnectTimeout=10 "$HOST" true 2>/dev/null
 }
 ensure_ssh() {
    if ssh_present; then
        echo "==> SSH access to $HOST: ok"
    else
        echo "==> No key-based SSH access yet; running ssh-copy-id (password prompt expected)"
        ssh-copy-id "$HOST"
        ssh_present || { echo "ERROR: SSH access still not working after ssh-copy-id" >&2; exit 1; }
    fi
 }
 # Run the prerequisites script remotely by piping it over stdin; TARGET_DIR and
 # ROOTFS_ARCHIVE are passed as arguments to `bash -s --`.
 check_prerequisites() {
    echo "==> Checking prerequisites on $HOST (target dir: $TARGET_DIR)"
    local rootfs_remote="$TARGET_DIR/.werkator/$(basename "$ROOTFS")"
    if ! ssh "$HOST" "WERKATOR_SSH_TARGET='$HOST' bash -s -- '$TARGET_DIR' '$rootfs_remote'" < "$PREREQ_SCRIPT"; then
        echo "ERROR: prerequisites failed on $HOST — install aborted" >&2
        exit 1
    fi
 }
 ensure_local_artifacts() {
    if [ ! -f "$RUNTIME_BUNDLE" ]; then
        echo "==> Runtime bundle not found; building it locally (./gradlew runtimeBundle)"
        (cd "$REPO_ROOT" && ./gradlew runtimeBundle --console=plain -q)
    fi
    [ -f "$RUNTIME_BUNDLE" ] || { echo "ERROR: runtime bundle missing: $RUNTIME_BUNDLE" >&2; exit 1; }
    [ -f "$ROOTFS" ] || {
        echo "ERROR: rootfs archive missing: $ROOTFS" >&2
        echo "       build it with tools/build-bwrap-rootfs.sh or set WERKATOR_ROOTFS" >&2
        exit 1
    }
 }
 ensure_github_access() {
    # `ssh -T git@github.com` exits 1 even on success ("does not provide shell
    # access") — neutralize remotely, then match on the greeting text.
    if ssh "$HOST" 'ssh -o BatchMode=yes -o ConnectTimeout=10 -T git@github.com 2>&1 || true' | grep -q "successfully authenticated"; then
        echo "==> GitHub SSH access from $HOST: ok"
        return 0
    fi
    echo
    echo "==> The host cannot reach GitHub via SSH yet."
    echo "    Add THIS public key to GitHub (Settings > SSH and GPG keys > New SSH key):"
    ssh "$HOST" 'cat ~/.ssh/id_*.pub 2>/dev/null' || {
        echo "ERROR: no public key on the host; create one with ssh-keygen -t ed25519" >&2
        exit 1
    }
    read -r -p "    Press Enter once the key is registered at GitHub... "
    ssh "$HOST" 'ssh -o BatchMode=yes -T git@github.com 2>&1 || true' | grep -q "successfully authenticated" || {
        echo "ERROR: GitHub authentication from $HOST still failing" >&2
        exit 1
    }
    echo "==> GitHub SSH access from $HOST: ok"
 }
 install() {
    ensure_ssh
    check_prerequisites
    ensure_local_artifacts
    echo "==> Uploading runtime bundle and rootfs archive"
    ssh "$HOST" "mkdir -p '$TARGET_DIR/.werkator'"
    scp -q "$RUNTIME_BUNDLE" "$HOST:$TARGET_DIR/.werkator/"
    scp -q "$ROOTFS" "$HOST:$TARGET_DIR/.werkator/"
    echo "==> Unpacking runtime bundle"
    ssh "$HOST" "tar xzf '$TARGET_DIR/.werkator/$(basename "$RUNTIME_BUNDLE")' -C '$TARGET_DIR/.werkator'"
    ssh "$HOST" "'$TARGET_DIR/.werkator/werkator/bin/werkator' --version"
    ensure_github_access
    echo "==> Cloning the repository"
    if ssh "$HOST" "test -d '$TARGET_DIR/werkator/.git'"; then
        echo "    (already cloned, skipping)"
    else
        ssh "$HOST" "git clone git@github.com:mhoennig/werkator.git '$TARGET_DIR/werkator'"
    fi
    echo "==> Running werkator init"
    ssh "$HOST" "cd '$TARGET_DIR/werkator' && '$TARGET_DIR/.werkator/werkator/bin/werkator' init"
    echo "==> Writing machine-local bwrap configuration"
    ssh "$HOST" "grep -q '^  bwrap:' '$TARGET_DIR/werkator/.git/werkator/.werkator.yml' 2>/dev/null" || ssh "$HOST" "cat >> '$TARGET_DIR/werkator/.git/werkator/.werkator.yml' <<'CFG'
 # Build in the bubblewrap sandbox instead of natively (Step 17 / ADR 0007).
 # Both keys are pinned: read from this machine config even if a branch sets
 # its own values in a committed .werkator.yml.
 builds:
  default:
    bwrap:
      enabled: true
      rootfs: $TARGET_DIR/.werkator/$(basename "$ROOTFS")
 CFG"
    echo "==> Verifying the effective configuration"
    ssh "$HOST" "cd '$TARGET_DIR/werkator' && '$TARGET_DIR/.werkator/werkator/bin/werkator' config:print 2>/dev/null | grep -A3 'bwrap:' | head -4"
    echo
    echo "==> Install complete."
    echo "    Repo:     $TARGET_DIR/werkator"
    echo "    Runtime:  $TARGET_DIR/.werkator/werkator/bin/werkator"
    echo "    Next:     tools/remote werkator build"
 }
 build() {
    ensure_ssh
    local branch="${WERKATOR_BRANCH:-main}"
    echo "==> Running one initial build of branch '$branch' on $HOST (in the bwrap sandbox)"
    ssh -t "$HOST" "cd '$TARGET_DIR/werkator' && '$TARGET_DIR/.werkator/werkator/bin/werkator' build '$branch'"
 }
 # Start the server as a systemd user unit behind the managed Apache.
 # WERKATOR_MEMORY_MAX / WERKATOR_TASKS_MAX (optional) are written into the
 # machine config so `init --systemd` bakes them into the unit.
 start() {
    ensure_ssh
    require_env WERKATOR_PORT WERKATOR_DOMAIN
    local machine="$TARGET_DIR/werkator/.git/werkator/.werkator.yml"
    local unit="werkator-$(basename "$TARGET_DIR/werkator").service"
    local htaccess="$TARGET_DIR/doms/$WERKATOR_DOMAIN/subs/www/.htaccess"
    echo "==> Writing server settings to the machine config"
    if ssh "$HOST" "grep -q '^server:' '$machine' 2>/dev/null"; then
        # re-run: update port and publicBaseUrl in place (systemd limits stay as written)
        ssh "$HOST" "sed -i 's/^  port: .*/  port: $WERKATOR_PORT/; s|^  publicBaseUrl: .*|  publicBaseUrl: \"https://$WERKATOR_DOMAIN/\"|' '$machine'"
    else
        ssh "$HOST" "cat >> '$machine' <<'CFG'
 # Web access: the managed Apache terminates TLS and proxies to the localhost
 # port assigned by Hostsharing (eigener Serverdienst); TLS is the domain's
 # Let's Encrypt certificate, so Werkator itself stays on 127.0.0.1.
 server:
  port: $WERKATOR_PORT
  bindAddress: 127.0.0.1
  publicBaseUrl: \"https://$WERKATOR_DOMAIN/\"
  nginx:
    enabled: false
  systemd:
    memoryMax: \"${WERKATOR_MEMORY_MAX:-}\"
    tasksMax: \"${WERKATOR_TASKS_MAX:-}\"
 CFG"
    fi
    echo "==> Writing the Apache reverse proxy to $htaccess"
    ssh "$HOST" "mkdir -p '$TARGET_DIR/doms/$WERKATOR_DOMAIN/subs/www' && cat > '$htaccess' <<'HT'
 DirectoryIndex disabled
 RewriteEngine On
 RewriteBase /
 RewriteRule .* http://127.0.0.1:$WERKATOR_PORT%{REQUEST_URI} [proxy]
 HT"
    echo "==> Generating the systemd user unit (init --systemd)"
    ssh "$HOST" "cd '$TARGET_DIR/werkator' && '$TARGET_DIR/.werkator/werkator/bin/werkator' init --systemd"
    echo "==> Linking the units into ~/.config/systemd/user and enabling the service"
    ssh "$HOST" "mkdir -p ~/.config/systemd/user && \
      ln -sf '$TARGET_DIR/werkator/.git/werkator/$unit' ~/.config/systemd/user/ && \
      ln -sf '$TARGET_DIR/werkator/.git/werkator/werkator-docker-prune.service' ~/.config/systemd/user/ && \
      ln -sf '$TARGET_DIR/werkator/.git/werkator/werkator-docker-prune.timer' ~/.config/systemd/user/ && \
      systemctl --user daemon-reload && systemctl --user restart '$unit' && systemctl --user status '$unit' --no-pager -l | head -12"
    echo
    echo "==> Server started. Verify: https://$WERKATOR_DOMAIN/"
    echo "    Logs: ssh $HOST -- systemctl --user status '$unit'"
 }
 # Background SSH tunnel to the Werkator server, so the browser reaches the UI
 # at http://localhost:<WERKATOR_LOCAL_PORT> without keeping a terminal busy.
 # `start` runs ssh -N -L detached with a pid file; `stop` kills it.
 port_forward() {
    require_env WERKATOR_LOCAL_PORT
    local remote_port
    remote_port="$(ssh "$HOST" "awk '/^server:/{f=1;next} f && /^  port:/{print \$2; exit}' '$TARGET_DIR/werkator/.git/werkator/.werkator.yml'")"
    [ -n "$remote_port" ] || { echo "ERROR: no server.port in the machine config — run 'tools/remote werkator start' first" >&2; exit 1; }
    case "$COMMAND" in
        start)
            if [ -f "$PID_FILE" ] && kill -0 "$(cat "$PID_FILE")" 2>/dev/null; then
                echo "==> Port-forward already running (pid $(cat "$PID_FILE")) — http://localhost:$WERKATOR_LOCAL_PORT"
                exit 0
            fi
            nohup ssh -N -L "$WERKATOR_LOCAL_PORT:127.0.0.1:$remote_port" "$HOST" \
                >"$LOG_FILE" 2>&1 &
            echo $! > "$PID_FILE"
            sleep 1
            if kill -0 "$(cat "$PID_FILE")" 2>/dev/null; then
                echo "==> Forwarding http://localhost:$WERKATOR_LOCAL_PORT -> $HOST:127.0.0.1:$remote_port (pid $(cat "$PID_FILE"))"
            else
                echo "ERROR: port-forward failed to start — see $LOG_FILE" >&2
                rm -f "$PID_FILE"
                exit 1
            fi
            ;;
        stop)
            if [ -f "$PID_FILE" ] && kill -0 "$(cat "$PID_FILE")" 2>/dev/null; then
                kill "$(cat "$PID_FILE")"
                rm -f "$PID_FILE"
                echo "==> Port-forward stopped"
            else
                rm -f "$PID_FILE"
                echo "==> Port-forward is not running"
            fi
            ;;
        *)
            echo "ERROR: unknown port-forward command: $COMMAND (use start or stop)" >&2
            exit 2
            ;;
    esac
 }
 # Print the control token guarding the mutating build endpoints. If the server
 # has not created it yet (it does so on first use), generate one in place — the
 # server reads the file lazily, so a pre-created token is equivalent.
 control_token() {
    ensure_ssh
    local token_file="$TARGET_DIR/werkator/.git/werkator/control-token"
    ssh "$HOST" "if [ -f '$token_file' ]; then cat '$token_file'; else \
        umask 077 && head -c 32 /dev/urandom | od -An -tx1 | tr -d ' \\n' > '$token_file' && cat '$token_file'; fi"
 }
 case "$REPO" in
    port-forward)
        port_forward
        ;;
    werkator)
        case "$COMMAND" in
            check-prerequisites)
                ensure_ssh
                check_prerequisites
                ;;
            install)
                install
                ;;
            build)
                build
                ;;
            start)
                start
                ;;
            control-token)
                control_token
                ;;
            *)
                echo "ERROR: unknown command: $COMMAND" >&2
                usage
                ;;
        esac
        ;;
    *)
        echo "ERROR: unknown repo selector: $REPO" >&2
        usage
        ;;
 esac
@@ -0,0 +1,164 @@
 #!/usr/bin/env bash
 #
 # Verify the bwrap (bubblewrap) build precondition on a target host before
 # running Werkator's bwrap build runtime there (step 17 / ADR 0007).
 #
 # The whole "build Werkator inside bubblewrap on a Managed Webspace" approach
 # hinges on one hard precondition: unprivileged user namespaces with a uid-0
 # mapping and read-only root binds must work. This script runs the exact
 # command line recorded in docs/plan/17-bwrap-build-runtime.md, checks the
 # expected signals, and additionally verifies the disk/quota situation:
 # a bwrap build unpacks the rootfs (a zstd archive expands to several GiB)
 # plus a Gradle distribution and per-branch caches, so the host needs both
 # raw free space and enough group-quota headroom.
 #
 # Run this ON the target host (the webspace), no root needed.
 #
 # Optional: the rootfs archive to size the disk/quota check against, e.g.
 #   werkator-build-prerequisites.sh /path/to/werkator-buildenv-trixie.tar.zst
 # When omitted, the check runs against a conservative default footprint.
 #
 # Usage: werkator-build-prerequisites.sh [TARGET_DIR] [ROOTFS_ARCHIVE]
 #
 # TARGET_DIR is the directory the build workspace will live in (default: $HOME).
 # The check verifies it sits on the home filesystem and has enough free space.
 # ROOTFS_ARCHIVE, when given, is the rootfs archive that will be used there.
 #
 # Output is one PASS/FAIL line per check plus a final RESULT line, e.g.:
 #   PASS: bwrap version: bubblewrap 0.8.0
 #   PASS: build runs as root inside the namespace (uid 0)
 #   PASS: uid_map maps root back to the unprivileged user (uid 120957)
 #   PASS: read-only root bind is enforced
 #   PASS: at least 5 GiB free space on the build working filesystem
 #   FAIL: group quota headroom below the 5 GiB build footprint ...
 #   RESULT: FAIL (4/5) — Werkator bubblewrap builds are not usable on this host.
 #
 set -euo pipefail
 target_dir_arg="${1:-}"
 rootfs_arg="${2:-}"
 die()  { echo "ERROR: $*" >&2; exit 1; }
 # Disk footprint a bwrap build needs headroom for, in 1K blocks: unpacked
 # rootfs (zstd expands roughly 3-4x), Gradle distribution + per-branch cache,
 # build output and artifacts. ~5 GiB.
 MIN_FREE_BLOCKS=$((5 * 1024 * 1024))
 # Reference filesystem: the one the invoking user's home directory lives on.
 # Builds (repo clone, buildenv, caches) must run there — other mounts, such as
 # a slow mass-storage volume, are rejected.
 HOME_FS="$(df -Pk "$HOME" 2>/dev/null | awk 'NR==2 {print $1}')"
 pass=0
 fail=0
 result() {  # result PASS|FAIL "message"
    echo "$1: $2"
    if [ "$1" = "PASS" ]; then pass=$((pass+1)); else fail=$((fail+1)); fi
 }
 command -v bwrap >/dev/null 2>&1 || die "bwrap is not installed on this host"
 output="$(bwrap --unshare-user --unshare-pid --die-with-parent --uid 0 --gid 0 \
      --ro-bind / / --dev /dev --proc /proc --tmpfs /tmp \
      sh -c 'id -u && cat /proc/self/uid_map && (touch /usr/ro-test 2>&1 || true)' 2>&1)" ||
    die "bwrap invocation failed (no user namespace support?): $output"
 # Signal 0: bwrap itself is usable (version as a visible marker).
 result PASS "bwrap version: $(bwrap --version 2>&1)"
 # Signal 1: runs as root (uid 0) inside the namespace.
 first="$(printf '%s\n' "$output" | sed -n '1p')"
 if [ "$first" = "0" ]; then
    result PASS "build runs as root inside the namespace (uid 0)"
 else
    result FAIL "expected uid 0 inside the namespace, got: $first"
 fi
 # Signal 2: uid_map maps root to the invoking unprivileged user.
 uid_line="$(printf '%s\n' "$output" | sed -n '2p')"
 self_uid="$(id -u)"
 if printf '%s\n' "$uid_line" | grep -E "^[[:space:]]*0[[:space:]]+${self_uid}[[:space:]]+1" >/dev/null; then
    result PASS "uid_map maps root back to the unprivileged user (uid $self_uid)"
 else
    result FAIL "expected uid_map '0 $self_uid 1', got: $uid_line"
 fi
 # Signal 3: the read-only root bind is enforced (a write to /usr fails).
 if printf '%s\n' "$output" | grep -qi "read-only file system"; then
    result PASS "read-only root bind is enforced"
 else
    result FAIL "the read-only root bind did not reject a write to /usr"
 fi
 # --- Disk / quota checks ------------------------------------------------
 target_dir="${target_dir_arg:-$HOME}"
 target_dir="$(realpath -m "$target_dir")"
 min_gib=$((MIN_FREE_BLOCKS / 1024 / 1024))
 if [ -n "$rootfs_arg" ] && [ ! -f "$rootfs_arg" ]; then
    echo "WARNING: rootfs archive not found: $rootfs_arg (continuing without it)"
 fi
 target_fs="$(df -Pk "$target_dir" 2>/dev/null | awk 'NR==2 {print $1}')"
 df_output="$(df -Pk "$target_dir" 2>/dev/null | awk 'NR==2 {print int($4) " " $6}')"
 if [ -n "$df_output" ]; then
    avail_k="${df_output%% *}"
    mount="${df_output##* }"
    if [ -n "$HOME_FS" ] && [ "$target_fs" != "$HOME_FS" ]; then
        # An explicitly chosen foreign filesystem is allowed (e.g. for testing)
        # but flagged: builds there will be slow.
        echo "WARNING: target dir is on $target_fs (mounted at $mount), not the home filesystem ($HOME_FS) — builds will run on slower storage"
    fi
    if [ "${avail_k:-0}" -lt "$MIN_FREE_BLOCKS" ]; then
        result FAIL "less than ${min_gib} GiB free space on the build working filesystem ($mount)"
    else
        result PASS "at least ${min_gib} GiB free space on the build working filesystem ($mount, device $target_fs)"
    fi
 else
    echo "WARNING: could not measure free space on $target_dir — only the quota check below applies"
 fi
 if quota_output="$(quota -g 2>/dev/null)" && [ -n "$quota_output" ]; then
    quota_ok=1
    quota_seen=0
    detail=""
    while read -r fs blocks quota_limit; do
        quota_seen=1
        # Only the quota of the target filesystem counts — other volumes may
        # legitimately be full or unquota'd without affecting the build.
        if [ -n "$target_fs" ] && [ "$(basename "$fs")" != "$(basename "$target_fs")" ] && [ "$fs" != "$target_fs" ]; then
            continue
        fi
        headroom=$((quota_limit - blocks))
        if [ "$headroom" -lt "$MIN_FREE_BLOCKS" ]; then
            quota_ok=0
            detail+=" $(basename "$fs"): $(awk -v b="$headroom" 'BEGIN{printf "%.1f", b/1024/1024}') GiB free of quota;"
        fi
    done < <(printf '%s\n' "$quota_output" | awk '
        NF==1 && $1 ~ /^\// { pending_fs=$1; next }
        $1 ~ /^\// && $2 ~ /^[0-9]+$/ { print $1, $2, $4; pending_fs=""; next }
        $1 ~ /^[0-9]+[*]?/ && pending_fs != "" { gsub(/\*/, "", $1); print pending_fs, $1, $3; pending_fs="" }')
    if [ "$quota_seen" -eq 0 ]; then
        echo "WARNING: quota tooling present but no group quota lines could be parsed — only free space was checked"
    elif [ "$quota_ok" -eq 1 ]; then
        result PASS "group quota headroom covers the ${min_gib} GiB build footprint"
    else
        result FAIL "group quota headroom below the ${min_gib} GiB build footprint (rootfs + Gradle cache); raise the quota before building.$detail"
    fi
 else
    echo "WARNING: no readable group quota tooling on this host — only free space was checked"
 fi
 total=$((pass + fail))
 echo
 if [ "$fail" -eq 0 ]; then
    echo "RESULT: PASS ($pass/$total) — Werkator bubblewrap builds are usable on this host."
    echo "Next: install the Werkator instance with: tools/remote werkator install ${WERKATOR_SSH_TARGET:-<user>@<host>} '$target_dir'"
    exit 0
 else
    echo "RESULT: FAIL ($pass/$total) — Werkator bubblewrap builds are not usable on this host."
    exit 1
 fi