Bwrap build runtime und Installation in Hostsharing Managed Webspace (#4)

* Add the bubblewrap build runtime (step 17, ADR 0007)

BwrapBuildRunner: third runtime behind BuildRunner for hosts without root
and without Docker (e.g. Hostsharing managed webspaces). Shells out to the
bwrap CLI, unpacks a prepared rootfs on demand into
.git/werkator/buildenv/<envKey>/rootfs, reuses the Docker runner's git
metadata mounts, and returns the attached bwrap process for streaming and
cancellation.

Config: bwrap.enabled/rootfs/env on BranchConfig and BwrapOverrides on
BuildDefinition; enabled/rootfs are pinned like the docker sandbox policy.
Docker and bwrap are mutually exclusive per build, rejected in
buildSettings instead of picked silently. DispatchingBuildRunner routes
bwrap; InitCommand template, docs/configuration.md and AGENTS.md in sync.

* bwrap rollout tooling: remote script, prerequisites disk/quota check, absolute workspace binds

- tools/remote: central remote control script with check-prerequisites, install and build commands
- tools/werkator-build-prerequisites.sh: compact PASS/FAIL output, target-dir parameter, free-space and
  group-quota headroom checks against the ~5 GiB build footprint, home-filesystem reference
- BwrapBuildRunner: bind workspace and home at absolute paths resolved against repoDir — a relative
  path made bwrap create mountpoints inside the read-only rootfs (seen on the webspace); regression test
- TestcontainersSmokeTest: gated with enabledIf docker available (skip, never fail, without a daemon)
- docs: configuration reference, step-17 plan notes, PR-doc

* bwrap: bind the repo read-write before the workspace so mountpoints are creatable

bwrap creates mountpoints for bind destinations inside the sandbox; with only a
read-only rootfs bound at /, creating them for the workspace under .git/werkator/
worktrees failed with 'Read-only file system' (seen on the webspace). Binding the
repo dir read-write first provides the base; the git metadata mounts then layer
the usual isolation on top (read-only .git, tmpfs mask over .git/werkator,
read-write worktree admin dir).

* bwrap: pre-create bind mountpoints inside the unpacked rootfs

bwrap mkdirs mountpoints for bind destinations against the sandbox view; with the
rootfs ro-bound at / every destination missing from the rootfs (the repo dir under
/home/storage/... on the webspace) fails with 'Read-only file system'. The rootfs
directory is a plain host dir, so create the mountpoints there before launching
bwrap; it then finds them and has nothing left to create.

* bwrap: skip existing rootfs files when pre-creating bind mountpoints

/etc/resolv.conf is a file the rootfs already ships; createDirectories threw on it.
Only missing directories are created now.

* bwrap: pre-create proc/dev/tmpfs mountpoints in the rootfs too

The rootfs archive ships no /proc or /dev (excluded when packed), so bwrap failed
mkdir'ing their mountpoints against the read-only root.

* bwrap: bind the workspace after the git metadata mounts

The tmpfs mask over .git/werkator shadowed the earlier workspace bind, because the
worktree lives under .git/werkator/worktrees — chdir then failed with ENOENT. The
workspace bind now comes last and shadows the mask at exactly its own path.

* systemd resource limits and webspace start command (step 17, web access)

- server.systemd.memoryMax/tasksMax (empty = directive omitted): on platforms where
  the service runs in a shared memory slice (Hostsharing Managed Webspaces) a
  runaway Gradle build must not starve the whole package; init --systemd reads the
  effective config and bakes the values into the generated unit
- tools/remote werkator start: writes server settings (assigned port, loopback
  bind, publicBaseUrl, nginx off) plus the Apache reverse-proxy .htaccess into
  ~/doms/<domain>/subs/www, runs init --systemd and enables the user unit
- docs/configuration.md documents the new keys

* tools/remote: env-based configuration and background port-forward

All connection and deployment values come from .env in the repository root
(WERKATOR_REMOTE, WERKATOR_PATH, WERKATOR_PORT, WERKATOR_DOMAIN,
WERKATOR_LOCAL_PORT, optional WERKATOR_BRANCH/MEMORY_MAX/TASKS_MAX/ROOTFS);
missing values fail with a pointing error instead of positional parameters.

- port-forward is now 'tools/remote port-forward start|stop' with a detached
  ssh tunnel, pid file under /tmp, and idempotent start
- start restarts the systemd unit after updating the machine config
- control-token generates the token in place when the server has not yet
- the rootfs archive default moves to build/ (already gitignored)

src/test/kotlin/de/hoennig/werkator/build/BwrapBuildRunnerTest.kt

@@ -0,0 +1,240 @@
+package de.hoennig.werkator.build
+
+import de.hoennig.werkator.config.BranchConfig
+import de.hoennig.werkator.config.BwrapConfig
+import de.hoennig.werkator.git.GitCommandResult
+import de.hoennig.werkator.git.GitCommandRunner
+import io.kotest.assertions.throwables.shouldThrow
+import io.kotest.core.spec.style.FunSpec
+import io.kotest.matchers.shouldBe
+import io.kotest.matchers.string.shouldContain
+import io.mockk.clearMocks
+import io.mockk.every
+import io.mockk.mockk
+import io.mockk.verify
+import java.nio.file.Files
+import java.nio.file.Path
+
+class BwrapBuildRunnerTest : FunSpec() {
+    private val commandRunner = mockk<GitCommandRunner>()
+    private lateinit var runner: BwrapBuildRunner
+    private lateinit var repoDir: Path
+    private lateinit var workspace: Path
+    private val captured = mutableListOf<List<String>>()
+
+    private fun bwrapBranchConfig(
+        rootfs: String = "/srv/buildenv.tar.zst",
+        env: Map<String, String> = emptyMap(),
+    ): BranchConfig =
+        BranchConfig(
+            bwrap =
+                BwrapConfig(
+                    enabled = true,
+                    rootfs = rootfs,
+                    env = env,
+                ),
+        )
+
+    private fun rootfsUnpacked(rootfs: String = "/srv/buildenv.tar.zst"): Path =
+        repoDir
+            .resolve(BwrapBuildRunner.BUILDENV_DIR)
+            .resolve(rootfs.sha12())
+            .resolve(BwrapBuildRunner.ROOTFS_DIR)
+
+    init {
+        beforeEach {
+            clearMocks(commandRunner)
+            captured.clear()
+            repoDir = Files.createTempDirectory("werkator-bwrap-runner")
+            workspace = repoDir.resolve("workspace")
+            runner = BwrapBuildRunner(commandRunner)
+            runner.processStarter = { command, _ ->
+                captured += command
+                ProcessBuilder("true").start()
+            }
+        }
+
+        test("unpacks the rootfs on demand and assembles the exact bwrap command") {
+            every {
+                commandRunner.runOrThrow(
+                    listOf("tar", "--no-same-owner", "-xf", "/srv/buildenv.tar.zst", "-C", rootfsUnpacked().toString()),
+                    repoDir,
+                    any(),
+                    any(),
+                )
+            } returns
+                GitCommandResult(0, "", "")
+
+            runner.start("./gradlew test", workspace, mapOf("branch" to "main"), repoDir, bwrapBranchConfig())
+
+            val args = captured.single()
+            val rootfsDir = args[args.indexOf("--ro-bind") + 1]
+            args shouldBe
+                listOf(
+                    "bwrap",
+                    "--unshare-user",
+                    "--unshare-pid",
+                    "--die-with-parent",
+                    "--uid",
+                    "0",
+                    "--gid",
+                    "0",
+                    "--ro-bind",
+                    rootfsUnpacked().toString(),
+                    "/",
+                    "--bind",
+                    repoDir.toString(),
+                    repoDir.toString(),
+                    "--bind",
+                    workspace.toString(),
+                    workspace.toString(),
+                    "--bind",
+                    repoDir.resolve(".git/werkator/buildenv/home").toString(),
+                    "/root",
+                    "--ro-bind",
+                    "/etc/resolv.conf",
+                    "/etc/resolv.conf",
+                    "--proc",
+                    "/proc",
+                    "--dev",
+                    "/dev",
+                    "--tmpfs",
+                    "/tmp",
+                    "--setenv",
+                    "HOME",
+                    "/root",
+                    "--setenv",
+                    "branch",
+                    "main",
+                    "--chdir",
+                    workspace.toString(),
+                    "/bin/sh",
+                    "-c",
+                    "./gradlew test",
+                )
+            Files.isDirectory(rootfsUnpacked()) shouldBe true
+        }
+
+        test("binds a relative workspace path at its absolute location") {
+            // bwrap creates mountpoints for bind destinations inside the sandbox;
+            // a relative path would land in the read-only rootfs and fail with
+            // "Can't mkdir parents ...: Read-only file system" (seen on the webspace).
+            every {
+                commandRunner.runOrThrow(
+                    listOf("tar", "--no-same-owner", "-xf", "/srv/buildenv.tar.zst", "-C", rootfsUnpacked().toString()),
+                    repoDir,
+                    any(),
+                    any(),
+                )
+            } returns
+                GitCommandResult(0, "", "")
+
+            val relativeWorkspace = repoDir.relativize(workspace)
+
+            runner.start("./gradlew test", relativeWorkspace, mapOf("branch" to "main"), repoDir, bwrapBranchConfig())
+
+            val args = captured.single()
+            val absolute = workspace.toAbsolutePath().normalize().toString()
+            val bindIdx = args.withIndex().filter { it.value == "--bind" }.map { it.index }
+            // first bind is the repo dir (mountpoint base), second is the workspace
+            args[bindIdx[1] + 1] shouldBe absolute
+            args[bindIdx[1] + 2] shouldBe absolute
+            args[args.indexOf("--chdir") + 1] shouldBe absolute
+        }
+
+        test("does not re-unpack an already prepared rootfs") {
+            Files.createDirectories(rootfsUnpacked())
+
+            runner.start("./gradlew test", workspace, mapOf("branch" to "main"), repoDir, bwrapBranchConfig())
+
+            verify(exactly = 0) { commandRunner.runOrThrow(match { it.first() == "tar" }, any(), any(), any()) }
+        }
+
+        test("adds bwrap env and passes the branch environment through") {
+            Files.createDirectories(rootfsUnpacked())
+
+            runner.start(
+                "./gradlew test",
+                workspace,
+                mapOf("branch" to "main"),
+                repoDir,
+                bwrapBranchConfig(env = mapOf("FOO" to "bar")),
+            )
+
+            val args = captured.single()
+            args[args.indexOf("branch") - 1] shouldBe "--setenv"
+            args[args.indexOf("branch") + 1] shouldBe "main"
+            args[args.indexOf("FOO") - 1] shouldBe "--setenv"
+            args[args.indexOf("FOO") + 1] shouldBe "bar"
+        }
+
+        test("exposes git metadata read-only with the werkator dir masked for a worktree workspace") {
+            val gitDir = repoDir.resolve(".git")
+            val adminDir = gitDir.resolve("worktrees/workspace")
+            Files.createDirectories(adminDir)
+            Files.createDirectories(gitDir.resolve("werkator"))
+            Files.createDirectories(workspace)
+            Files.writeString(workspace.resolve(".git"), "gitdir: $adminDir\n")
+            Files.createDirectories(rootfsUnpacked())
+
+            runner.start("./gradlew test", workspace, mapOf("branch" to "main"), repoDir, bwrapBranchConfig())
+
+            val args = captured.single()
+            args[args.indexOf(gitDir.toString()) - 1] shouldBe "--ro-bind"
+            args[args.indexOf("$gitDir/werkator") - 1] shouldBe "--tmpfs"
+            args[args.indexOf(adminDir.toString()) - 1] shouldBe "--bind"
+        }
+
+        test("mounts no git metadata when the workspace is not a worktree") {
+            Files.createDirectories(rootfsUnpacked())
+            Files.createDirectories(workspace)
+
+            runner.start("./gradlew test", workspace, mapOf("branch" to "main"), repoDir, bwrapBranchConfig())
+
+            val args = captured.single()
+            val gitDir = repoDir.resolve(".git")
+            // the sandbox's own /tmp tmpfs is always present; the point is that no tmpfs
+            // masks .git/werkator and no worktree admin dir is bound
+            args.none { it == "$gitDir/werkator" } shouldBe true
+            args.none { it.contains("worktrees/") } shouldBe true
+        }
+
+        test("fails without a configured rootfs") {
+            val branchConfig = BranchConfig(bwrap = BwrapConfig(enabled = true))
+
+            val exception =
+                shouldThrow<IllegalArgumentException> {
+                    runner.start("./gradlew test", workspace, mapOf("branch" to "main"), repoDir, branchConfig)
+                }
+
+            exception.message shouldContain "bwrap.rootfs"
+        }
+
+        test("downloads a URL rootfs once before unpacking") {
+            val url = "https://example.test/buildenv.tar.zst"
+            val downloadTarget =
+                repoDir
+                    .resolve(BwrapBuildRunner.BUILDENV_DIR)
+                    .resolve(url.sha12())
+                    .resolve("buildenv.tar.zst")
+            every { commandRunner.runOrThrow(listOf("curl", "-fsSL", "-o", downloadTarget.toString(), url), repoDir, any(), any()) } returns
+                GitCommandResult(0, "", "")
+            every { commandRunner.runOrThrow(match { it.first() == "tar" }, any(), any(), any()) } returns
+                GitCommandResult(0, "", "")
+
+            runner.start("./gradlew test", workspace, mapOf("branch" to "main"), repoDir, bwrapBranchConfig(rootfs = url))
+
+            verify {
+                commandRunner.runOrThrow(listOf("curl", "-fsSL", "-o", downloadTarget.toString(), url), repoDir, any(), any())
+            }
+            verify { commandRunner.runOrThrow(match { it.first() == "tar" }, repoDir, any(), any()) }
+        }
+    }
+
+    private fun String.sha12(): String =
+        java.security.MessageDigest
+            .getInstance("SHA-256")
+            .digest(toByteArray())
+            .joinToString("") { "%02x".format(it) }
+            .take(12)
+}

src/test/kotlin/de/hoennig/werkator/build/DispatchingBuildRunnerTest.kt

@@ -1,6 +1,7 @@
 package de.hoennig.werkator.build
 
 import de.hoennig.werkator.config.BranchConfig
+import de.hoennig.werkator.config.BwrapConfig
 import de.hoennig.werkator.config.DockerConfig
 import io.kotest.core.spec.style.FunSpec
 import io.kotest.matchers.shouldBe
@@ -14,12 +15,13 @@ import java.nio.file.Paths
 class DispatchingBuildRunnerTest : FunSpec() {
     private val processBuildRunner = mockk<ProcessBuildRunner>()
     private val dockerBuildRunner = mockk<DockerBuildRunner>()
-    private val dispatcher = DispatchingBuildRunner(processBuildRunner, dockerBuildRunner)
+    private val bwrapBuildRunner = mockk<BwrapBuildRunner>()
+    private val dispatcher = DispatchingBuildRunner(processBuildRunner, dockerBuildRunner, bwrapBuildRunner)
     private val process = mockk<Process>()
     private val dir = Paths.get(".")
 
     init {
-        beforeEach { clearMocks(processBuildRunner, dockerBuildRunner) }
+        beforeEach { clearMocks(processBuildRunner, dockerBuildRunner, bwrapBuildRunner) }
 
         test("runs natively by default") {
             val branchConfig = BranchConfig()
@@ -28,6 +30,7 @@ class DispatchingBuildRunnerTest : FunSpec() {
             dispatcher.start("cmd", dir, emptyMap(), dir, branchConfig) shouldBe process
 
             verify { dockerBuildRunner wasNot Called }
+            verify { bwrapBuildRunner wasNot Called }
         }
 
         test("runs in Docker when the branch enables it") {
@@ -37,6 +40,17 @@ class DispatchingBuildRunnerTest : FunSpec() {
             dispatcher.start("cmd", dir, emptyMap(), dir, branchConfig) shouldBe process
 
             verify { processBuildRunner wasNot Called }
+            verify { bwrapBuildRunner wasNot Called }
+        }
+
+        test("runs in bwrap when the branch enables it (and not Docker)") {
+            val branchConfig = BranchConfig(bwrap = BwrapConfig(enabled = true, rootfs = "/srv/buildenv.tar.zst"))
+            every { bwrapBuildRunner.start("cmd", dir, emptyMap(), dir, branchConfig) } returns process
+
+            dispatcher.start("cmd", dir, emptyMap(), dir, branchConfig) shouldBe process
+
+            verify { processBuildRunner wasNot Called }
+            verify { dockerBuildRunner wasNot Called }
         }
     }
 }

src/test/kotlin/de/hoennig/werkator/commands/InitCommandTest.kt

@@ -14,7 +14,12 @@ import java.nio.file.attribute.PosixFilePermissions
 
 class InitCommandTest : FunSpec() {
     private val gitService = mockk<GitService>()
-    private val initCommand = InitCommand(gitService)
+    private val initCommand =
+        InitCommand(
+            gitService,
+            de.hoennig.werkator.config
+                .ConfigLoader(mockk(relaxed = true)),
+        )
 
     init {
         test("creates config files with auto-detected values") {

src/test/kotlin/de/hoennig/werkator/commands/SystemdServiceFilesTest.kt

@@ -33,6 +33,26 @@ class SystemdServiceFilesTest : FunSpec() {
             content shouldContain "WantedBy=default.target"
         }
 
+        test("resource limits are written when configured and omitted when unset") {
+            fun unit(
+                memoryMax: String,
+                tasksMax: String,
+            ) = SystemdServiceFiles.unitFileContent(
+                repoRoot = Paths.get("/srv/repos/my-repo"),
+                javaExecutable = Paths.get("/usr/bin/java"),
+                jarPath = Paths.get("/srv/repos/my-repo/werkator.jar"),
+                envFile = Paths.get("/srv/repos/my-repo/werkator.env"),
+                memoryMax = memoryMax,
+                tasksMax = tasksMax,
+            )
+            val with = unit(memoryMax = "1G", tasksMax = "512")
+            with shouldContain "MemoryMax=1G"
+            with shouldContain "TasksMax=512"
+            val without = unit(memoryMax = "", tasksMax = "")
+            without shouldNotContain "MemoryMax"
+            without shouldNotContain "TasksMax"
+        }
+
         test("percent signs in paths are escaped for systemd") {
             val content =
                 SystemdServiceFiles.unitFileContent(

src/test/kotlin/de/hoennig/werkator/config/ConfigLoaderTest.kt

@@ -350,6 +350,39 @@ class ConfigLoaderTest : FunSpec() {
             settings.docker.image shouldBe "attacker-image"
         }
 
+        test("a branch cannot disable its bwrap sandbox or substitute a foreign rootfs through a build definition") {
+            val dir = Files.createTempDirectory("werkator-test")
+            dir.resolve(".werkator.yml").toFile().writeText(
+                """
+                builds:
+                  default:
+                    bwrap:
+                      enabled: true
+                      rootfs: /host/rootfs.tar.zst
+                """.trimIndent(),
+            )
+            val worktree = Files.createTempDirectory("werkator-test-worktree")
+            worktree.resolve(".werkator.yml").toFile().writeText(
+                """
+                builds:
+                  default:
+                    bwrap:
+                      enabled: false
+                      rootfs: /attacker/rootfs.tar.zst
+                      env:
+                        FOO: from-branch
+                """.trimIndent(),
+            )
+
+            val settings = loader.loadForWorktree(dir, worktree).buildSettings("any-branch", "default")
+
+            // pinned: the sandbox can neither be switched off nor pointed at a foreign rootfs
+            settings.bwrap.enabled shouldBe true
+            settings.bwrap.rootfs shouldBe "/host/rootfs.tar.zst"
+            // everything that describes the build itself stays the branch's own business
+            settings.bwrap.env shouldBe mapOf("FOO" to "from-branch")
+        }
+
         test("a build the branch invents inherits the host's sandbox policy") {
             val dir = Files.createTempDirectory("werkator-test")
             dir.resolve(".werkator.yml").toFile().writeText(
@@ -386,6 +419,31 @@ class ConfigLoaderTest : FunSpec() {
             settings.requirePullRequest shouldBe true
         }
 
+        test("enabling both docker and bwrap on a build is rejected, not picked silently") {
+            val dir = Files.createTempDirectory("werkator-test")
+            dir.resolve(".werkator.yml").toFile().writeText(
+                """
+                builds:
+                  default:
+                    docker:
+                      enabled: true
+                      image: build-env
+                    bwrap:
+                      enabled: true
+                      rootfs: /srv/rootfs.tar.zst
+                """.trimIndent(),
+            )
+
+            val config = loader.load(dir)
+            val exception =
+                shouldThrow<IllegalArgumentException> {
+                    config.buildSettings("any-branch", "default")
+                }
+
+            exception.message shouldContain "both docker and bwrap"
+            exception.message shouldContain "builds.default"
+        }
+
         test("an exclusion pattern takes a branch out of a build that would otherwise select it") {
             val dir = Files.createTempDirectory("werkator-test")
             dir.resolve(".werkator.yml").toFile().writeText(

src/test/kotlin/de/hoennig/werkator/framework/TestcontainersSmokeTest.kt

@@ -2,18 +2,35 @@ package de.hoennig.werkator.framework
 
 import io.kotest.core.spec.style.FunSpec
 import io.kotest.matchers.shouldBe
+import org.testcontainers.DockerClientFactory
 import org.testcontainers.containers.GenericContainer
 import org.testcontainers.utility.DockerImageName
 
+/**
+ * Probes that Testcontainers can actually start a container on this host.
+ * Skipped, not failed, when no Docker is present — that is what lets Werkator
+ * build itself on a Docker-less host (e.g. a Hostsharing webspace, where its own
+ * build runs in the bubblewrap sandbox); see `tools/werkator-build-prerequisites.sh`.
+ */
 class TestcontainersSmokeTest :
     FunSpec({
 
-        test("Testcontainers starts a container") {
-            val container =
-                GenericContainer(DockerImageName.parse("alpine:3"))
-                    .withCommand("sh", "-c", "sleep 30")
-            container.start()
-            container.isRunning shouldBe true
-            container.stop()
-        }
-    })
+        test("Testcontainers starts a container")
+            .config(enabledIf = { dockerAvailable() }) {
+                val container =
+                    GenericContainer(DockerImageName.parse("alpine:3"))
+                        .withCommand("sh", "-c", "sleep 30")
+                container.start()
+                container.isRunning shouldBe true
+                container.stop()
+            }
+    }) {
+    companion object {
+        private fun dockerAvailable(): Boolean =
+            try {
+                DockerClientFactory.instance().isDockerAvailable()
+            } catch (_: Throwable) {
+                false
+            }
+    }
+}