Step 21 session C: BwrapBuildRunner delegates to the werkdock CLI

The runner no longer assembles raw bwrap argv: it checks image
existence via 'werkdock images', loads the rootfs archive once per
source as image werkator-buildenv-<hash> into werkdock's store (shared
across every repository of the OS user — resolving step 22's
buildenv-sharing question), and runs builds through 'werkdock run --rm'
with the git-metadata mask expressed as ordered -v/--tmpfs flags.

New pinned config key bwrap.werkdock (the executing binary, default via
PATH) — a branch must not substitute the executing binary; synced in
WerkatorConfig, BwrapOverrides, the pinned-key strip, the init
template, docs/configuration.md, and AGENTS.md.

werkdock's --clearenv means the server environment no longer leaks into
builds; the TMPDIR workaround is gone with the raw invocation.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
This commit is contained in:
mhoennig
2026-09-01 15:46:19 +02:00
co-authored by Claude Fable 5
parent f2d685932e
commit 823c8adc36
9 changed files with 202 additions and 247 deletions
+2 -2
View File
@@ -39,9 +39,9 @@ All production code lives under `de.hoennig.werkator`, with sub-packages `comman
- Builds run detached in worktrees under `.git/werkator/worktrees/<branchKey>`; the primary checkout is never used for builds; never assume a single running build. - Builds run detached in worktrees under `.git/werkator/worktrees/<branchKey>`; the primary checkout is never used for builds; never assume a single running build.
- When config keys change, three places must stay in sync: the `WerkatorConfig` data classes, the `InitCommand` templates, and `docs/configuration.md`. - When config keys change, three places must stay in sync: the `WerkatorConfig` data classes, the `InitCommand` templates, and `docs/configuration.md`.
- Every config file may declare `werkator.version.since`/`below` (the Werkator it is written for, never a format version — no API is involved). `since` is enforced in both directions, using `ConfigVersions.FORMAT_BROKE_IN` for "file predates a breaking change"; `below` only warns. A violation aborts the start for the machine and project config, but fails only that branch's builds for a branch config. - Every config file may declare `werkator.version.since`/`below` (the Werkator it is written for, never a format version — no API is involved). `since` is enforced in both directions, using `ConfigVersions.FORMAT_BROKE_IN` for "file predates a breaking change"; `below` only warns. A violation aborts the start for the machine and project config, but fails only that branch's builds for a branch config.
- A branch describes its own CI: its committed `.werkator.yml` is the branch layer (`ConfigLoader.loadWithBranchLayer`, used by the watcher per origin branch and by `loadForWorktree` at build time) and takes precedence over `.git`/project — including the whole `builds` section, so a new configuration can be tried out on a branch without affecting other branches. Only the pinned set is stripped from that layer: secrets (`git`), host/repository sections (`server`, `gitea`, `executor`, `watcher`), the docker (`docker.enabled`, `docker.network`) and bubblewrap (`bwrap.enabled`, `bwrap.rootfs`) sandbox policies, and the trust gate (`requirePullRequest`). A branch must never reach credentials, disable its container or sandbox, change its network, substitute a foreign rootfs, raise global concurrency, or bypass its own pull-request gate; a branch's definitions apply to that branch alone. - A branch describes its own CI: its committed `.werkator.yml` is the branch layer (`ConfigLoader.loadWithBranchLayer`, used by the watcher per origin branch and by `loadForWorktree` at build time) and takes precedence over `.git`/project — including the whole `builds` section, so a new configuration can be tried out on a branch without affecting other branches. Only the pinned set is stripped from that layer: secrets (`git`), host/repository sections (`server`, `gitea`, `executor`, `watcher`), the docker (`docker.enabled`, `docker.network`) and bubblewrap (`bwrap.enabled`, `bwrap.rootfs`, `bwrap.werkdock`) sandbox policies, and the trust gate (`requirePullRequest`). A branch must never reach credentials, disable its container or sandbox, change its network, substitute a foreign rootfs, raise global concurrency, or bypass its own pull-request gate; a branch's definitions apply to that branch alone.
- A build definition carries the complete description of its build, split in two: the `trigger` block (`onPush`, `atTimes`, `branches`, `activeWithin`) says when and for which branches it runs, everything else what it does. `builds.default` is the base every other definition inherits its settings — never its `trigger` — from. The split is structural so that a selector added to `TriggerConfig` later is non-inheritable by construction; writing a trigger key flat is refused, never ignored, because ignoring it leaves a build that silently stops running. A `!` prefix in `trigger.branches` excludes and always wins. - A build definition carries the complete description of its build, split in two: the `trigger` block (`onPush`, `atTimes`, `branches`, `activeWithin`) says when and for which branches it runs, everything else what it does. `builds.default` is the base every other definition inherits its settings — never its `trigger` — from. The split is structural so that a selector added to `TriggerConfig` later is non-inheritable by construction; writing a trigger key flat is refused, never ignored, because ignoring it leaves a build that silently stops running. A `!` prefix in `trigger.branches` excludes and always wins.
- The inheritance is applied after all layers are merged: that order is what makes a build a branch invents inherit the host's sandbox policy instead of the data-class default, so the pinning also holds for a build the host has never heard of. Pinned are `requirePullRequest`, `statusContext`, `docker.enabled`, `docker.network`, `bwrap.enabled`, and `bwrap.rootfs`. Docker and bwrap are mutually exclusive per branch — enabling both is rejected at start. - The inheritance is applied after all layers are merged: that order is what makes a build a branch invents inherit the host's sandbox policy instead of the data-class default, so the pinning also holds for a build the host has never heard of. Pinned are `requirePullRequest`, `statusContext`, `docker.enabled`, `docker.network`, `bwrap.enabled`, `bwrap.rootfs`, and `bwrap.werkdock`. Docker and bwrap are mutually exclusive per branch — enabling both is rejected at start.
- `builds` or the legacy `branches`, never both: `branches` is read only while the merged config defines no build at all (`builds.maxConcurrent` is not one), and ignored with a warning as soon as one exists. The section is deprecated and goes away once the repositories have migrated; then `ConfigVersions.FORMAT_BROKE_IN` gets set and a leftover `branches:` key must be rejected by name — the version check alone cannot catch a file that declares no version. - `builds` or the legacy `branches`, never both: `branches` is read only while the merged config defines no build at all (`builds.maxConcurrent` is not one), and ignored with a warning as soon as one exists. The section is deprecated and goes away once the repositories have migrated; then `ConfigVersions.FORMAT_BROKE_IN` gets set and a leftover `branches:` key must be rejected by name — the version check alone cannot catch a file that declares no version.
- Web UI: server-rendered Thymeleaf plus one hand-written `static/werkator.js` — no SPA framework, no frontend build pipeline; every fetch has a timeout and an explicit error badge; `UiFormats` and `werkator.js` must produce identical display formats. - Web UI: server-rendered Thymeleaf plus one hand-written `static/werkator.js` — no SPA framework, no frontend build pipeline; every fetch has a timeout and an explicit error badge; `UiFormats` and `werkator.js` must produce identical display formats.
- Git and Docker access shells out to the CLIs (`GitCommandRunner`, `docker`) — no JGit, no Docker SDK. - Git and Docker access shells out to the CLIs (`GitCommandRunner`, `docker`) — no JGit, no Docker SDK.
+11 -10
View File
@@ -91,7 +91,7 @@ single branch may decide it:
- the repository-side settings: the whole `gitea`, `executor`, and `watcher` sections; - the repository-side settings: the whole `gitea`, `executor`, and `watcher` sections;
- the trust gate: `requirePullRequest`, and the Gitea status context: `statusContext`; - the trust gate: `requirePullRequest`, and the Gitea status context: `statusContext`;
- the container sandbox policy: `docker.enabled`/`docker.network` and - the container sandbox policy: `docker.enabled`/`docker.network` and
`bwrap.enabled`/`bwrap.rootfs` — host-pinned as `bwrap.enabled`/`bwrap.rootfs`/`bwrap.werkdock` — host-pinned as
long as only the host's configuration sets them, master-pinned once the committed long as only the host's configuration sets them, master-pinned once the committed
configuration does. configuration does.
@@ -353,7 +353,7 @@ Both parts combine as an intersection.
Settings: `buildCommand`, `cleanCommand`, `artifactDirs`, `stdoutLog`/`stderrLog`, `requirePullRequest`, `statusContext`, and `docker` and `bwrap` with all their keys. Settings: `buildCommand`, `cleanCommand`, `artifactDirs`, `stdoutLog`/`stderrLog`, `requirePullRequest`, `statusContext`, and `docker` and `bwrap` with all their keys.
A definition carries the complete description of its build; unset keys fall back to `builds.default` and then to Werkator's own defaults. A definition carries the complete description of its build; unset keys fall back to `builds.default` and then to Werkator's own defaults.
`requirePullRequest`, `statusContext`, `docker.enabled`, `docker.network`, `bwrap.enabled`, and `bwrap.rootfs` are pinned (master-pinned, see [the branch layer](#the-branch-layer-a-branch-describes-its-own-ci)): they are read from the repo install/project config even when a branch sets them in its own committed config. `requirePullRequest`, `statusContext`, `docker.enabled`, `docker.network`, `bwrap.enabled`, `bwrap.rootfs`, and `bwrap.werkdock` are pinned (master-pinned, see [the branch layer](#the-branch-layer-a-branch-describes-its-own-ci)): they are read from the repo install/project config even when a branch sets them in its own committed config.
Inheritance from `builds.default` covers the settings only — the `trigger` block says when and where *this* build runs and is never inherited. Inheritance from `builds.default` covers the settings only — the `trigger` block says when and where *this* build runs and is never inherited.
Definitions are part of the branch layer: a branch may add its own and override those from the project config, for its own builds only. Definitions are part of the branch layer: a branch may add its own and override those from the project config, for its own builds only.
Because the inheritance is applied after all layers are merged, a build a branch invents still inherits the host's `builds.default` — its sandbox policy included, which is what keeps the pinning effective for a build the host has never heard of. Because the inheritance is applied after all layers are merged, a build a branch invents still inherits the host's `builds.default` — its sandbox policy included, which is what keeps the pinning effective for a build the host has never heard of.
@@ -411,16 +411,17 @@ All Werkator containers carry `org.hoennig.werkator` labels; stale build contain
### Notes on `builds.<name>.bwrap` ### Notes on `builds.<name>.bwrap`
With `bwrap.enabled`, Werkator shells out to the `bwrap` CLI (bubblewrap) instead of native execution. With `bwrap.enabled`, Werkator runs the build in a bubblewrap sandbox instead of native execution.
This is the third runtime, for hosts without root and without a Docker daemon (e.g. Hostsharing managed webspaces); see `docs/plan/17-bwrap-build-runtime.md` and ADR 0007. This is the third runtime, for hosts without root and without a Docker daemon (e.g. Hostsharing managed webspaces); see `docs/plan/17-bwrap-build-runtime.md` and ADR 0008.
`bwrap` must be on the `PATH`. Since step 21 session C the sandbox is executed by the `werkdock` CLI (`bwrap.werkdock`, default: resolved via `PATH`) — Werkator no longer invokes `bwrap` itself; `bwrap` must be installed for werkdock.
`werkdock doctor` checks the host's capability, superseding `tools/werkator-build-prerequisites.sh`.
`bwrap.rootfs` names the prepared root filesystem archive — a Debian-base rootfs with the build tools (JDK, git, locales, project-specific tooling) built elsewhere, since `debootstrap` is unavailable on the target. `bwrap.rootfs` names the prepared root filesystem archive — a Debian-base rootfs with the build tools (JDK, git, locales, project-specific tooling) built elsewhere, since `debootstrap` is unavailable on the target.
It is a local path or an `http(s)` URL; a URL is downloaded once. It is a local path or an `http(s)` URL; a URL is downloaded once into `.git/werkator/buildenv/`.
Build the archive with `tools/build-bwrap-rootfs.sh` on any machine with Docker; verify the host's user-namespace capability first with `tools/werkator-build-prerequisites.sh`. Build the archive with `tools/build-bwrap-rootfs.sh` on any machine with Docker.
The archive is unpacked on demand (`tar --no-same-owner`) into `.git/werkator/buildenv/<envKey>/rootfs`, shared across all branch worktrees like the Docker Gradle cache volume; `<envKey>` derives from a hash of the source, so a changed `rootfs` unpacks a fresh environment and stale ones can be pruned. The archive is loaded once per source as the werkdock image `werkator-buildenv-<hash>` into werkdock's store (`$WERKDOCK_HOME`, default `~/.werkdock`) — shared by every repository of this OS user; the hash derives from the source string, so a changed `rootfs` loads a fresh image and stale ones can be removed from the store.
Per-branch Gradle caches persist in `.git/werkator/buildenv/home`, bound as `/root`. Per-repo Gradle caches persist in `.git/werkator/buildenv/home`, bound as `/root`.
`bwrap.env` adds environment variables inside the sandbox. `bwrap.env` adds environment variables inside the sandbox; the environment is otherwise cleared (docker semantics) — the server's environment does not leak in.
Files created inside the sandbox are owned by the host user, because uid 0 maps back to the unprivileged webspace user. Files created inside the sandbox are owned by the host user, because uid 0 maps back to the unprivileged webspace user.
`docker` and `bwrap` are mutually exclusive per branch: enabling both is rejected at start, not silently picked. `docker` and `bwrap` are mutually exclusive per branch: enabling both is rejected at start, not silently picked.
@@ -64,11 +64,14 @@ A docker-like CLI over `bwrap`, filesystem isolation only.
- Own docs, plan, and ADRs under `werkdock/` from the start, so the later repository split is a directory move; the Werkator side only keeps what is Werkator-specific (the git-metadata mounts of step 16 and the config pinning). - Own docs, plan, and ADRs under `werkdock/` from the start, so the later repository split is a directory move; the Werkator side only keeps what is Werkator-specific (the git-metadata mounts of step 16 and the config pinning).
- Keep `werkdock/` self-contained: no imports from Werkator code, no Gradle coupling to the Werkator build — it must build and test on its own. - Keep `werkdock/` self-contained: no imports from Werkator code, no Gradle coupling to the Werkator build — it must build and test on its own.
### C — Werkator consumes Werkdock (this repo, after B) ### C — Werkator consumes Werkdock (this repo, after B; implemented 2026-09-01 on branch `werkator-consumes-werkdock`)
- `BwrapBuildRunner` shells out to `werkdock run` instead of assembling the raw `bwrap` argv — same pattern as git and docker: CLI, no library. - `BwrapBuildRunner` shells out to `werkdock run` instead of assembling the raw `bwrap` argv — same pattern as git and docker: CLI, no library.
- Config keys (`bwrap.enabled`, `bwrap.rootfs`) and their pinning stay as they are; only the executor behind them changes. - Config keys (`bwrap.enabled`, `bwrap.rootfs`) and their pinning stay as they are; only the executor behind them changes.
- Decide in the step: whether the git-metadata mounts stay Werkator-side (passed as extra `--bind`/`--tmpfs` options to `werkdock run`) or become a Werkdock feature; the secrets-masking of `.git/werkator/` must hold either way. One key was added: `bwrap.werkdock` (the executing binary, default via PATH) — pinned like the rest of the sandbox policy, since a branch must not substitute the executing binary.
- Decided: the git-metadata mounts stay Werkator-side, passed as `-v …:ro` / `--tmpfs` / `-v` options whose flag order werkdock preserves (it grew `--tmpfs` and an ordered mount list for exactly this); the secrets-masking of `.git/werkator/` holds unchanged.
- Decided: the rootfs archive becomes a werkdock *image* (`werkator-buildenv-<source-hash>`, checked via `werkdock images`, loaded via `werkdock load`) in werkdock's own store — shared across every repository of the OS user, which resolves step 22's buildenv-sharing question; only the URL download cache and the persistent `/root` toolchain home stay under `.git/werkator/buildenv/`.
- Consequence of werkdock's `--clearenv`: the server environment no longer leaks into builds, and the runner's TMPDIR workaround is gone.
### D — The Managed-Webspace install path (this repo, independent of B/C) ### D — The Managed-Webspace install path (this repo, independent of B/C)
@@ -10,20 +10,30 @@ import java.nio.file.Path
import java.security.MessageDigest import java.security.MessageDigest
/** /**
* Runs build commands inside a bubblewrap user-namespace sandbox (Step 17 / ADR 0007), * Runs build commands inside a bubblewrap user-namespace sandbox (Step 17 / ADR 0008),
* for hosts without root and without a Docker daemon (e.g. Hostsharing managed * for hosts without root and without a Docker daemon (e.g. Hostsharing managed
* webspaces). Shells out to the `bwrap` CLI via the generic [GitCommandRunner] process * webspaces). Since step 21 session C it no longer assembles the raw `bwrap` argv:
* wrapper — no library, consistent with git and docker. * it shells out to the `werkdock` CLI (`bwrap.werkdock`, default via PATH) — the same
* pattern as git and docker, CLI, no library.
* *
* The prepared rootfs (a Debian-base archive built elsewhere, since `debootstrap` is not * The rootfs archive becomes a werkdock *image*, loaded once per source
* available on the target) is unpacked on demand into `.git/werkator/buildenv/<envKey>/rootfs`, * (`werkator-buildenv-<hash>`, the hash over the source string, so a changed source
* shared across all branch worktrees like the Docker gradle cache volume; `<envKey>` derives * loads a fresh image) into werkdock's own store (`$WERKDOCK_HOME`, default
* from a hash of the archive source, so a changed source unpacks a fresh rootfs and stale * `~/.werkdock`) — shared by every repository of this OS user, unlike the old
* ones can be pruned. The returned [Process] is the attached `bwrap` process, so log * per-repo unpack. Only the download cache for URL sources and the persistent
* streaming and cancellation work exactly like native builds (`--die-with-parent` plus * toolchain home (bound to `/root` for Gradle/Go caches) stay under
* `--unshare-pid` tear down the whole tree on cancel). Git works inside the sandbox with * `.git/werkator/buildenv/`.
* the same layered mounts as the Docker runner: the primary `.git` read-only with *
* `.git/werkator/` masked, see [gitMetadataMounts]. * Werkdock clears the environment inside the sandbox (docker semantics), so the
* server's environment no longer leaks in — only the explicit `-e` variables below
* plus werkdock's own `HOME`/`PATH` exist inside; the pam_tmpdir TMPDIR class of
* bugs is gone by construction. Git works inside the sandbox with the same layered
* mounts as the Docker runner, expressed as werkdock flags whose order is
* significant and preserved: read-only `.git`, tmpfs mask over `.git/werkator`,
* read-write worktree admin dir — see [gitMetadataMounts]. The returned [Process]
* is the attached `werkdock run`, whose `bwrap` child dies with it
* (`--die-with-parent`), so log streaming and cancellation work exactly like
* native builds.
*/ */
@Component @Component
class BwrapBuildRunner( class BwrapBuildRunner(
@@ -31,7 +41,7 @@ class BwrapBuildRunner(
) : BuildRunner { ) : BuildRunner {
private val log = LoggerFactory.getLogger(BwrapBuildRunner::class.java) private val log = LoggerFactory.getLogger(BwrapBuildRunner::class.java)
/** Replaceable process launcher so unit tests can capture the assembled `bwrap` argv. */ /** Replaceable process launcher so unit tests can capture the assembled `werkdock` argv. */
internal var processStarter: (List<String>, Path) -> Process = { command, dir -> internal var processStarter: (List<String>, Path) -> Process = { command, dir ->
ProcessBuilder(command).directory(dir.toFile()).start() ProcessBuilder(command).directory(dir.toFile()).start()
} }
@@ -46,76 +56,37 @@ class BwrapBuildRunner(
): Process { ): Process {
val bwrap = branchConfig.bwrap val bwrap = branchConfig.bwrap
require(bwrap.rootfs.isNotBlank()) { "branches.<name>.bwrap.rootfs must be set when bwrap.enabled is true" } require(bwrap.rootfs.isNotBlank()) { "branches.<name>.bwrap.rootfs must be set when bwrap.enabled is true" }
val buildEnvRoot = buildEnvRoot(repoDir) val werkdock = bwrap.werkdock.ifBlank { "werkdock" }
val envKey = envKey(bwrap.rootfs) val image = imageName(bwrap.rootfs)
val rootfsDir = buildEnvRoot.resolve(envKey).resolve(ROOTFS_DIR) ensureImage(werkdock, image, bwrap, repoDir, onAuxProcess)
ensureRootfs(bwrap, rootfsDir, repoDir, onAuxProcess) val homeDir = repoDir.resolve(BUILDENV_DIR).resolve(HOME_DIR)
val homeDir = buildEnvRoot.resolve(HOME_DIR)
Files.createDirectories(homeDir) Files.createDirectories(homeDir)
val args = val args = invocation(command, workingDir, environment, repoDir, bwrap, werkdock, image, homeDir)
invocation(command, workingDir, environment, repoDir, bwrap, rootfsDir, homeDir)
ensureMountpoints(rootfsDir, args)
return processStarter(args, repoDir) return processStarter(args, repoDir)
} }
/** /**
* bwrap creates mountpoint directories for bind destinations inside the sandbox — * Loads the rootfs archive into the werkdock image store once per source.
* against the read-only rootfs bind that fails with "Can't mkdir parents ... * `werkdock images` answers existence through the CLI, like `docker image
* Read-only file system" for every destination that does not exist in the rootfs * inspect` does for the Docker runner.
* (the workspace under the repo, for example). The rootfs directory itself is a
* plain host directory, so we pre-create the mountpoints there; bwrap then finds
* them and has nothing left to mkdir.
*/ */
private fun ensureMountpoints( private fun ensureImage(
rootfsDir: Path, werkdock: String,
args: List<String>, image: String,
) {
var i = 0
while (i < args.size) {
val arg = args[i]
if (arg == "--bind" || arg == "--ro-bind") {
val dest = args[i + 2]
val mountpoint = rootfsDir.resolve(dest.substring(1))
// Skip anything that already exists in the rootfs (e.g. /etc/resolv.conf
// is a file the rootfs ships); only missing dirs are created.
if (dest.startsWith("/") && !Files.exists(mountpoint)) {
Files.createDirectories(mountpoint)
}
i += 3
} else if (arg == "--proc" || arg == "--dev" || arg == "--tmpfs") {
// The rootfs archive ships no /proc, /dev (excluded when packed), so
// these mountpoints must exist too.
val dest = args[i + 1]
if (dest.startsWith("/") && !Files.exists(rootfsDir.resolve(dest.substring(1)))) {
Files.createDirectories(rootfsDir.resolve(dest.substring(1)))
}
i += 2
} else {
i += 1
}
}
}
/**
* Unpacks the configured archive into [rootfsDir] once per environment version
* (identified by [envKey]). Missing means "not yet unpacked"; the environment is a
* cache like the Docker image and the Gradle volume, and stale ones are pruned with
* the rest of `.git/werkator`.
*/
private fun ensureRootfs(
bwrap: BwrapConfig, bwrap: BwrapConfig,
rootfsDir: Path,
repoDir: Path, repoDir: Path,
onAuxProcess: (Process) -> Unit, onAuxProcess: (Process) -> Unit,
) { ) {
if (Files.isDirectory(rootfsDir)) { val loaded = commandRunner.runOrThrow(listOf(werkdock, "images"), repoDir, onProcess = onAuxProcess).lines()
if (image in loaded) {
return return
} }
Files.createDirectories(rootfsDir) val envDir = repoDir.resolve(BUILDENV_DIR).resolve(sourceKey(bwrap.rootfs))
val archive = localArchive(bwrap.rootfs, rootfsDir.parent, repoDir, onAuxProcess) Files.createDirectories(envDir)
log.info("unpacking build environment {} into {}", bwrap.rootfs, rootfsDir) val archive = localArchive(bwrap.rootfs, envDir, repoDir, onAuxProcess)
log.info("loading build environment {} as werkdock image {}", bwrap.rootfs, image)
commandRunner.runOrThrow( commandRunner.runOrThrow(
listOf("tar", "--no-same-owner", "-xf", archive, "-C", rootfsDir.toString()), listOf(werkdock, "load", "-i", archive, "--name", image),
repoDir, repoDir,
onProcess = onAuxProcess, onProcess = onAuxProcess,
) )
@@ -123,9 +94,7 @@ class BwrapBuildRunner(
/** /**
* Resolves [BwrapConfig.rootfs] to a local archive path: a bare or `file:` path is * Resolves [BwrapConfig.rootfs] to a local archive path: a bare or `file:` path is
* used as-is; an `http(s)` URL is downloaded once into the buildenv root. GNU tar * used as-is; an `http(s)` URL is downloaded once into the buildenv cache.
* auto-detects the compression from the archive magic, so a `.tar.gz` or `.tar.zst`
* needs no extra flag.
*/ */
private fun localArchive( private fun localArchive(
rootfs: String, rootfs: String,
@@ -155,77 +124,48 @@ class BwrapBuildRunner(
environment: Map<String, String>, environment: Map<String, String>,
repoDir: Path, repoDir: Path,
bwrap: BwrapConfig, bwrap: BwrapConfig,
rootfsDir: Path, werkdock: String,
image: String,
homeDir: Path, homeDir: Path,
): List<String> { ): List<String> {
// bwrap creates mountpoints for bind destinations inside the sandbox; a // Mounts at absolute host paths — same contract as the Docker runner.
// relative workspace path would resolve there into the read-only rootfs // Relative paths come from the CLI relative to the repo, so resolve them
// ("Can't mkdir parents ...: Read-only file system"). Bind at absolute // against repoDir, not against the process working directory.
// host paths instead — same contract as the Docker runner. Relative
// paths come from the CLI relative to the repo, so resolve them against
// repoDir, not against the process working directory.
val repoDirAbs = repoDir.toAbsolutePath().normalize() val repoDirAbs = repoDir.toAbsolutePath().normalize()
val workspaceAbs = val workspaceAbs =
if (workspace.isAbsolute) workspace.normalize() else repoDirAbs.resolve(workspace).normalize() if (workspace.isAbsolute) workspace.normalize() else repoDirAbs.resolve(workspace).normalize()
val homeDirAbs = val homeDirAbs =
if (homeDir.isAbsolute) homeDir.normalize() else repoDirAbs.resolve(homeDir).normalize() if (homeDir.isAbsolute) homeDir.normalize() else repoDirAbs.resolve(homeDir).normalize()
val args = val args = mutableListOf(werkdock, "run", "--rm")
mutableListOf( // The repo read-write FIRST, as the base the later mountpoints (workspace,
"bwrap", // worktree admin dir) are created in; the git metadata mounts then layer
"--unshare-user", // the isolation on top, and the workspace bind last shadows the tmpfs mask
"--unshare-pid", // at exactly its own path (it lives under .git/werkator/worktrees).
"--die-with-parent", args += listOf("-v", "$repoDirAbs:$repoDirAbs")
"--uid",
"0",
"--gid",
"0",
"--ro-bind",
rootfsDir.toString(),
"/",
)
// Bind the repo read-write FIRST so bwrap can create the mountpoints of
// the later binds (workspace, worktree admin dir) inside it — creating
// them against the read-only rootfs fails with "Can't mkdir parents ...
// Read-only file system". The git metadata mounts below then layer the
// usual isolation on top: read-only .git, tmpfs mask over .git/werkator,
// read-write worktree admin dir.
args += listOf("--bind", "$repoDirAbs", "$repoDirAbs")
// Git metadata mounts BEFORE the workspace bind: the tmpfs mask over
// .git/werkator must not shadow the workspace, which lives under
// .git/werkator/worktrees — the later workspace bind shadows the mask
// at exactly its own path and nothing else.
args += gitMetadataMounts(workspaceAbs, repoDir) args += gitMetadataMounts(workspaceAbs, repoDir)
args += listOf("--bind", "$workspaceAbs", "$workspaceAbs") args += listOf("-v", "$workspaceAbs:$workspaceAbs")
args += listOf("--bind", "$homeDirAbs", "/root") args += listOf("-v", "$homeDirAbs:/root")
args += listOf("--ro-bind", "/etc/resolv.conf", "/etc/resolv.conf")
args += listOf("--proc", "/proc", "--dev", "/dev", "--tmpfs", "/tmp")
args += listOf("--setenv", "HOME", "/root")
// The sandbox /tmp is a fresh tmpfs, but bwrap inherits the server's
// environment — on hosts with pam_tmpdir that includes
// TMPDIR=/tmp/user/<uid>, which does not exist inside and breaks every
// tool honoring it (go: "creating work dir: stat ...: no such file or
// directory"; the JVM ignores TMPDIR, so Gradle never noticed). Set
// both back to /tmp; explicit env below can still override.
args += listOf("--setenv", "TMPDIR", "/tmp")
args += listOf("--setenv", "TMP", "/tmp")
for ((key, value) in environment) { for ((key, value) in environment) {
args += listOf("--setenv", key, value) args += listOf("-e", "$key=$value")
} }
for ((key, value) in bwrap.env) { for ((key, value) in bwrap.env) {
args += listOf("--setenv", key, value) args += listOf("-e", "$key=$value")
} }
args += listOf("--chdir", "$workspaceAbs", "/bin/sh", "-c", command) args += listOf("-w", "$workspaceAbs")
args += image
args += listOf("/bin/sh", "-c", command)
return args return args
} }
/** /**
* Makes git work inside the sandbox without exposing Werkator's secrets — the same * Makes git work inside the sandbox without exposing Werkator's secrets — the same
* three layered mounts as the Docker runner, expressed in `bwrap` flags (bwrap nests * three layered mounts as the Docker runner, expressed as werkdock flags (werkdock
* mounts by target path like Docker): the primary `.git` read-only, an empty tmpfs * preserves the -v/--tmpfs flag order, and bwrap nests mounts by target path): the
* masking `.git/werkator/` (machine config with `git.token`, control token, build * primary `.git` read-only, an empty tmpfs masking `.git/werkator/` (machine config
* state), and this worktree's admin directory read-write so index-refreshing commands * with `git.token`, control token, build state), and this worktree's admin directory
* keep working. Object and ref writes stay blocked by the read-only `.git` mount. * read-write so index-refreshing commands keep working. Object and ref writes stay
* No mounts are added when the workspace is not a worktree of [repoDir]. * blocked by the read-only `.git` mount. No mounts are added when the workspace is
* not a worktree of [repoDir].
*/ */
private fun gitMetadataMounts( private fun gitMetadataMounts(
workspace: Path, workspace: Path,
@@ -247,28 +187,27 @@ class BwrapBuildRunner(
if (!adminDir.startsWith(gitDir) || !Files.isDirectory(adminDir)) { if (!adminDir.startsWith(gitDir) || !Files.isDirectory(adminDir)) {
return emptyList() return emptyList()
} }
val args = mutableListOf("--ro-bind", "$gitDir", "$gitDir") val args = mutableListOf("-v", "$gitDir:$gitDir:ro")
val werkatorDir = gitDir.resolve("werkator") val werkatorDir = gitDir.resolve("werkator")
if (Files.isDirectory(werkatorDir)) { if (Files.isDirectory(werkatorDir)) {
args += listOf("--tmpfs", "$werkatorDir") args += listOf("--tmpfs", "$werkatorDir")
} }
args += listOf("--bind", "$adminDir", "$adminDir") args += listOf("-v", "$adminDir:$adminDir")
return args return args
} }
private fun buildEnvRoot(repoDir: Path): Path = repoDir.resolve(BUILDENV_DIR) /** A short hash of the archive source, so a changed source loads a fresh image. */
private fun sourceKey(rootfs: String): String =
/** A short hash of the archive source, so a changed source unpacks a fresh rootfs. */
private fun envKey(rootfs: String): String =
MessageDigest MessageDigest
.getInstance("SHA-256") .getInstance("SHA-256")
.digest(rootfs.toByteArray()) .digest(rootfs.toByteArray())
.joinToString("") { "%02x".format(it) } .joinToString("") { "%02x".format(it) }
.take(12) .take(12)
private fun imageName(rootfs: String): String = "werkator-buildenv-${sourceKey(rootfs)}"
companion object { companion object {
const val BUILDENV_DIR = ".git/werkator/buildenv" const val BUILDENV_DIR = ".git/werkator/buildenv"
const val ROOTFS_DIR = "rootfs"
const val HOME_DIR = "home" const val HOME_DIR = "home"
} }
} }
@@ -226,6 +226,7 @@ class InitCommand(
bwrap: bwrap:
enabled: false # run clean/build in a bwrap sandbox instead of natively (pinned) enabled: false # run clean/build in a bwrap sandbox instead of natively (pinned)
rootfs: "" # prepared rootfs archive (path or URL); required when enabled (pinned) rootfs: "" # prepared rootfs archive (path or URL); required when enabled (pinned)
werkdock: werkdock # the werkdock CLI executing the sandbox; default resolves via PATH (pinned)
env: {} # additional environment variables set inside the sandbox env: {} # additional environment variables set inside the sandbox
# Gitea check this build reports as; empty uses gitea.statusContext. # Gitea check this build reports as; empty uses gitea.statusContext.
# Two builds of one commit under the same context overwrite each other. # Two builds of one commit under the same context overwrite each other.
@@ -68,6 +68,7 @@ data class BuildDefinition(
branchConfig.bwrap.copy( branchConfig.bwrap.copy(
enabled = bwrap?.enabled ?: branchConfig.bwrap.enabled, enabled = bwrap?.enabled ?: branchConfig.bwrap.enabled,
rootfs = bwrap?.rootfs ?: branchConfig.bwrap.rootfs, rootfs = bwrap?.rootfs ?: branchConfig.bwrap.rootfs,
werkdock = bwrap?.werkdock ?: branchConfig.bwrap.werkdock,
env = bwrap?.env ?: branchConfig.bwrap.env, env = bwrap?.env ?: branchConfig.bwrap.env,
), ),
) )
@@ -174,5 +175,7 @@ data class BwrapOverrides(
val enabled: Boolean? = null, val enabled: Boolean? = null,
/** Rootfs archive source. Pinned — a branch must not substitute a foreign rootfs. */ /** Rootfs archive source. Pinned — a branch must not substitute a foreign rootfs. */
val rootfs: String? = null, val rootfs: String? = null,
/** The werkdock CLI executing the sandbox. Pinned — a branch must not substitute the executing binary. */
val werkdock: String? = null,
val env: Map<String, String>? = null, val env: Map<String, String>? = null,
) )
@@ -403,8 +403,8 @@ class ConfigLoader(
/** `docker` keys a branch must never override: the sandbox policy. */ /** `docker` keys a branch must never override: the sandbox policy. */
private val PINNED_DOCKER_KEYS = setOf("enabled", "network") private val PINNED_DOCKER_KEYS = setOf("enabled", "network")
/** `bwrap` keys a branch must never override: the sandbox policy (Step 17). */ /** `bwrap` keys a branch must never override: the sandbox policy (Step 17) and its executing binary. */
private val PINNED_BWRAP_KEYS = setOf("enabled", "rootfs") private val PINNED_BWRAP_KEYS = setOf("enabled", "rootfs", "werkdock")
/** /**
* The one key of a build definition that says *when* and *for which branches* it * The one key of a build definition that says *when* and *for which branches* it
@@ -197,6 +197,11 @@ data class BwrapConfig(
* Pinned — a branch must not substitute a foreign rootfs via its committed config. * Pinned — a branch must not substitute a foreign rootfs via its committed config.
*/ */
val rootfs: String = "", val rootfs: String = "",
/**
* The werkdock CLI executing the sandbox (step 21 session C); empty or the default
* resolves via PATH. Pinned — a branch must not substitute the executing binary.
*/
val werkdock: String = "werkdock",
/** Additional environment variables set inside the sandbox. */ /** Additional environment variables set inside the sandbox. */
val env: Map<String, String> = emptyMap(), val env: Map<String, String> = emptyMap(),
) )
@@ -35,11 +35,18 @@ class BwrapBuildRunnerTest : FunSpec() {
), ),
) )
private fun rootfsUnpacked(rootfs: String = "/srv/buildenv.tar.zst"): Path = private fun imageName(rootfs: String = "/srv/buildenv.tar.zst"): String = "werkator-buildenv-${rootfs.sha12()}"
repoDir
.resolve(BwrapBuildRunner.BUILDENV_DIR) /** The image is already loaded: `werkdock images` lists it, so no load runs. */
.resolve(rootfs.sha12()) private fun givenImageLoaded(rootfs: String = "/srv/buildenv.tar.zst") {
.resolve(BwrapBuildRunner.ROOTFS_DIR) every { commandRunner.runOrThrow(listOf("werkdock", "images"), repoDir, any(), any()) } returns
GitCommandResult(0, imageName(rootfs) + "\n", "")
}
private fun givenImageMissing() {
every { commandRunner.runOrThrow(listOf("werkdock", "images"), repoDir, any(), any()) } returns
GitCommandResult(0, "some-other-image\n", "")
}
init { init {
beforeEach { beforeEach {
@@ -54,110 +61,92 @@ class BwrapBuildRunnerTest : FunSpec() {
} }
} }
test("unpacks the rootfs on demand and assembles the exact bwrap command") { test("assembles the exact werkdock run command for a loaded image") {
every { givenImageLoaded()
commandRunner.runOrThrow(
listOf("tar", "--no-same-owner", "-xf", "/srv/buildenv.tar.zst", "-C", rootfsUnpacked().toString()),
repoDir,
any(),
any(),
)
} returns
GitCommandResult(0, "", "")
runner.start("./gradlew test", workspace, mapOf("branch" to "main"), repoDir, bwrapBranchConfig()) runner.start("./gradlew test", workspace, mapOf("branch" to "main"), repoDir, bwrapBranchConfig())
val args = captured.single() captured.single() shouldBe
val rootfsDir = args[args.indexOf("--ro-bind") + 1]
args shouldBe
listOf( listOf(
"bwrap", "werkdock",
"--unshare-user", "run",
"--unshare-pid", "--rm",
"--die-with-parent", "-v",
"--uid", "$repoDir:$repoDir",
"0", "-v",
"--gid", "$workspace:$workspace",
"0", "-v",
"--ro-bind", "${repoDir.resolve(".git/werkator/buildenv/home")}:/root",
rootfsUnpacked().toString(), "-e",
"/", "branch=main",
"--bind", "-w",
repoDir.toString(),
repoDir.toString(),
"--bind",
workspace.toString(),
workspace.toString(),
"--bind",
repoDir.resolve(".git/werkator/buildenv/home").toString(),
"/root",
"--ro-bind",
"/etc/resolv.conf",
"/etc/resolv.conf",
"--proc",
"/proc",
"--dev",
"/dev",
"--tmpfs",
"/tmp",
"--setenv",
"HOME",
"/root",
"--setenv",
"TMPDIR",
"/tmp",
"--setenv",
"TMP",
"/tmp",
"--setenv",
"branch",
"main",
"--chdir",
workspace.toString(), workspace.toString(),
imageName(),
"/bin/sh", "/bin/sh",
"-c", "-c",
"./gradlew test", "./gradlew test",
) )
Files.isDirectory(rootfsUnpacked()) shouldBe true
} }
test("binds a relative workspace path at its absolute location") { test("loads the image once when werkdock does not know it yet") {
// bwrap creates mountpoints for bind destinations inside the sandbox; givenImageMissing()
// a relative path would land in the read-only rootfs and fail with
// "Can't mkdir parents ...: Read-only file system" (seen on the webspace).
every { every {
commandRunner.runOrThrow( commandRunner.runOrThrow(
listOf("tar", "--no-same-owner", "-xf", "/srv/buildenv.tar.zst", "-C", rootfsUnpacked().toString()), listOf("werkdock", "load", "-i", "/srv/buildenv.tar.zst", "--name", imageName()),
repoDir, repoDir,
any(), any(),
any(), any(),
) )
} returns } returns GitCommandResult(0, "", "")
GitCommandResult(0, "", "")
runner.start("./gradlew test", workspace, mapOf("branch" to "main"), repoDir, bwrapBranchConfig())
verify {
commandRunner.runOrThrow(
listOf("werkdock", "load", "-i", "/srv/buildenv.tar.zst", "--name", imageName()),
repoDir,
any(),
any(),
)
}
}
test("does not load an image werkdock already has") {
givenImageLoaded()
runner.start("./gradlew test", workspace, mapOf("branch" to "main"), repoDir, bwrapBranchConfig())
verify(exactly = 0) { commandRunner.runOrThrow(match { "load" in it }, any(), any(), any()) }
}
test("uses the configured werkdock binary path") {
every { commandRunner.runOrThrow(listOf("/opt/bin/werkdock", "images"), repoDir, any(), any()) } returns
GitCommandResult(0, imageName() + "\n", "")
val branchConfig =
BranchConfig(
bwrap = BwrapConfig(enabled = true, rootfs = "/srv/buildenv.tar.zst", werkdock = "/opt/bin/werkdock"),
)
runner.start("./gradlew test", workspace, emptyMap(), repoDir, branchConfig)
captured.single().first() shouldBe "/opt/bin/werkdock"
}
test("mounts a relative workspace path at its absolute location") {
givenImageLoaded()
val relativeWorkspace = repoDir.relativize(workspace) val relativeWorkspace = repoDir.relativize(workspace)
runner.start("./gradlew test", relativeWorkspace, mapOf("branch" to "main"), repoDir, bwrapBranchConfig()) runner.start("./gradlew test", relativeWorkspace, mapOf("branch" to "main"), repoDir, bwrapBranchConfig())
val args = captured.single() val args = captured.single()
val absolute = workspace.toAbsolutePath().normalize().toString() val absolute = workspace.toAbsolutePath().normalize().toString()
val bindIdx = args.withIndex().filter { it.value == "--bind" }.map { it.index } args shouldContainElement "-v"
// first bind is the repo dir (mountpoint base), second is the workspace args[args.indexOf("-w") + 1] shouldBe absolute
args[bindIdx[1] + 1] shouldBe absolute args.count { it == "$absolute:$absolute" } shouldBe 1
args[bindIdx[1] + 2] shouldBe absolute
args[args.indexOf("--chdir") + 1] shouldBe absolute
} }
test("does not re-unpack an already prepared rootfs") { test("adds bwrap env after the branch environment") {
Files.createDirectories(rootfsUnpacked()) givenImageLoaded()
runner.start("./gradlew test", workspace, mapOf("branch" to "main"), repoDir, bwrapBranchConfig())
verify(exactly = 0) { commandRunner.runOrThrow(match { it.first() == "tar" }, any(), any(), any()) }
}
test("adds bwrap env and passes the branch environment through") {
Files.createDirectories(rootfsUnpacked())
runner.start( runner.start(
"./gradlew test", "./gradlew test",
@@ -168,39 +157,43 @@ class BwrapBuildRunnerTest : FunSpec() {
) )
val args = captured.single() val args = captured.single()
args[args.indexOf("branch") - 1] shouldBe "--setenv" args[args.indexOf("branch=main") - 1] shouldBe "-e"
args[args.indexOf("branch") + 1] shouldBe "main" args[args.indexOf("FOO=bar") - 1] shouldBe "-e"
args[args.indexOf("FOO") - 1] shouldBe "--setenv" args.indexOf("branch=main") shouldBe args.indexOf("FOO=bar") - 2
args[args.indexOf("FOO") + 1] shouldBe "bar"
} }
test("exposes git metadata read-only with the werkator dir masked for a worktree workspace") { test("exposes git metadata read-only with the werkator dir masked, in mount order") {
val gitDir = repoDir.resolve(".git") val gitDir = repoDir.resolve(".git")
val adminDir = gitDir.resolve("worktrees/workspace") val adminDir = gitDir.resolve("worktrees/workspace")
Files.createDirectories(adminDir) Files.createDirectories(adminDir)
Files.createDirectories(gitDir.resolve("werkator")) Files.createDirectories(gitDir.resolve("werkator"))
Files.createDirectories(workspace) Files.createDirectories(workspace)
Files.writeString(workspace.resolve(".git"), "gitdir: $adminDir\n") Files.writeString(workspace.resolve(".git"), "gitdir: $adminDir\n")
Files.createDirectories(rootfsUnpacked()) givenImageLoaded()
runner.start("./gradlew test", workspace, mapOf("branch" to "main"), repoDir, bwrapBranchConfig()) runner.start("./gradlew test", workspace, mapOf("branch" to "main"), repoDir, bwrapBranchConfig())
val args = captured.single() val args = captured.single()
args[args.indexOf(gitDir.toString()) - 1] shouldBe "--ro-bind" args[args.indexOf("$gitDir:$gitDir:ro") - 1] shouldBe "-v"
args[args.indexOf("$gitDir/werkator") - 1] shouldBe "--tmpfs" args[args.indexOf("$gitDir/werkator") - 1] shouldBe "--tmpfs"
args[args.indexOf(adminDir.toString()) - 1] shouldBe "--bind" args[args.indexOf("$adminDir:$adminDir") - 1] shouldBe "-v"
// order: ro .git, tmpfs mask, admin dir, then the workspace bind that
// shadows the mask at its own path
val roGit = args.indexOf("$gitDir:$gitDir:ro")
val mask = args.indexOf("$gitDir/werkator")
val admin = args.indexOf("$adminDir:$adminDir")
val workspaceBind = args.indexOf("$workspace:$workspace")
(roGit < mask && mask < admin && admin < workspaceBind) shouldBe true
} }
test("mounts no git metadata when the workspace is not a worktree") { test("mounts no git metadata when the workspace is not a worktree") {
Files.createDirectories(rootfsUnpacked()) givenImageLoaded()
Files.createDirectories(workspace) Files.createDirectories(workspace)
runner.start("./gradlew test", workspace, mapOf("branch" to "main"), repoDir, bwrapBranchConfig()) runner.start("./gradlew test", workspace, mapOf("branch" to "main"), repoDir, bwrapBranchConfig())
val args = captured.single() val args = captured.single()
val gitDir = repoDir.resolve(".git") val gitDir = repoDir.resolve(".git")
// the sandbox's own /tmp tmpfs is always present; the point is that no tmpfs
// masks .git/werkator and no worktree admin dir is bound
args.none { it == "$gitDir/werkator" } shouldBe true args.none { it == "$gitDir/werkator" } shouldBe true
args.none { it.contains("worktrees/") } shouldBe true args.none { it.contains("worktrees/") } shouldBe true
} }
@@ -216,16 +209,17 @@ class BwrapBuildRunnerTest : FunSpec() {
exception.message shouldContain "bwrap.rootfs" exception.message shouldContain "bwrap.rootfs"
} }
test("downloads a URL rootfs once before unpacking") { test("downloads a URL rootfs once before loading it") {
val url = "https://example.test/buildenv.tar.zst" val url = "https://example.test/buildenv.tar.zst"
val downloadTarget = val downloadTarget =
repoDir repoDir
.resolve(BwrapBuildRunner.BUILDENV_DIR) .resolve(BwrapBuildRunner.BUILDENV_DIR)
.resolve(url.sha12()) .resolve(url.sha12())
.resolve("buildenv.tar.zst") .resolve("buildenv.tar.zst")
givenImageMissing()
every { commandRunner.runOrThrow(listOf("curl", "-fsSL", "-o", downloadTarget.toString(), url), repoDir, any(), any()) } returns every { commandRunner.runOrThrow(listOf("curl", "-fsSL", "-o", downloadTarget.toString(), url), repoDir, any(), any()) } returns
GitCommandResult(0, "", "") GitCommandResult(0, "", "")
every { commandRunner.runOrThrow(match { it.first() == "tar" }, any(), any(), any()) } returns every { commandRunner.runOrThrow(match { "load" in it }, any(), any(), any()) } returns
GitCommandResult(0, "", "") GitCommandResult(0, "", "")
runner.start("./gradlew test", workspace, mapOf("branch" to "main"), repoDir, bwrapBranchConfig(rootfs = url)) runner.start("./gradlew test", workspace, mapOf("branch" to "main"), repoDir, bwrapBranchConfig(rootfs = url))
@@ -233,10 +227,19 @@ class BwrapBuildRunnerTest : FunSpec() {
verify { verify {
commandRunner.runOrThrow(listOf("curl", "-fsSL", "-o", downloadTarget.toString(), url), repoDir, any(), any()) commandRunner.runOrThrow(listOf("curl", "-fsSL", "-o", downloadTarget.toString(), url), repoDir, any(), any())
} }
verify { commandRunner.runOrThrow(match { it.first() == "tar" }, repoDir, any(), any()) } verify {
commandRunner.runOrThrow(
listOf("werkdock", "load", "-i", downloadTarget.toString(), "--name", "werkator-buildenv-${url.sha12()}"),
repoDir,
any(),
any(),
)
}
} }
} }
private infix fun List<String>.shouldContainElement(element: String) = (element in this) shouldBe true
private fun String.sha12(): String = private fun String.sha12(): String =
java.security.MessageDigest java.security.MessageDigest
.getInstance("SHA-256") .getInstance("SHA-256")