Bwrap build runtime und Installation in Hostsharing Managed Webspace (#4)

* Add the bubblewrap build runtime (step 17, ADR 0007)

BwrapBuildRunner: third runtime behind BuildRunner for hosts without root
and without Docker (e.g. Hostsharing managed webspaces). Shells out to the
bwrap CLI, unpacks a prepared rootfs on demand into
.git/werkator/buildenv/<envKey>/rootfs, reuses the Docker runner's git
metadata mounts, and returns the attached bwrap process for streaming and
cancellation.

Config: bwrap.enabled/rootfs/env on BranchConfig and BwrapOverrides on
BuildDefinition; enabled/rootfs are pinned like the docker sandbox policy.
Docker and bwrap are mutually exclusive per build, rejected in
buildSettings instead of picked silently. DispatchingBuildRunner routes
bwrap; InitCommand template, docs/configuration.md and AGENTS.md in sync.

* bwrap rollout tooling: remote script, prerequisites disk/quota check, absolute workspace binds

- tools/remote: central remote control script with check-prerequisites, install and build commands
- tools/werkator-build-prerequisites.sh: compact PASS/FAIL output, target-dir parameter, free-space and
  group-quota headroom checks against the ~5 GiB build footprint, home-filesystem reference
- BwrapBuildRunner: bind workspace and home at absolute paths resolved against repoDir — a relative
  path made bwrap create mountpoints inside the read-only rootfs (seen on the webspace); regression test
- TestcontainersSmokeTest: gated with enabledIf docker available (skip, never fail, without a daemon)
- docs: configuration reference, step-17 plan notes, PR-doc

* bwrap: bind the repo read-write before the workspace so mountpoints are creatable

bwrap creates mountpoints for bind destinations inside the sandbox; with only a
read-only rootfs bound at /, creating them for the workspace under .git/werkator/
worktrees failed with 'Read-only file system' (seen on the webspace). Binding the
repo dir read-write first provides the base; the git metadata mounts then layer
the usual isolation on top (read-only .git, tmpfs mask over .git/werkator,
read-write worktree admin dir).

* bwrap: pre-create bind mountpoints inside the unpacked rootfs

bwrap mkdirs mountpoints for bind destinations against the sandbox view; with the
rootfs ro-bound at / every destination missing from the rootfs (the repo dir under
/home/storage/... on the webspace) fails with 'Read-only file system'. The rootfs
directory is a plain host dir, so create the mountpoints there before launching
bwrap; it then finds them and has nothing left to create.

* bwrap: skip existing rootfs files when pre-creating bind mountpoints

/etc/resolv.conf is a file the rootfs already ships; createDirectories threw on it.
Only missing directories are created now.

* bwrap: pre-create proc/dev/tmpfs mountpoints in the rootfs too

The rootfs archive ships no /proc or /dev (excluded when packed), so bwrap failed
mkdir'ing their mountpoints against the read-only root.

* bwrap: bind the workspace after the git metadata mounts

The tmpfs mask over .git/werkator shadowed the earlier workspace bind, because the
worktree lives under .git/werkator/worktrees — chdir then failed with ENOENT. The
workspace bind now comes last and shadows the mask at exactly its own path.

* systemd resource limits and webspace start command (step 17, web access)

- server.systemd.memoryMax/tasksMax (empty = directive omitted): on platforms where
  the service runs in a shared memory slice (Hostsharing Managed Webspaces) a
  runaway Gradle build must not starve the whole package; init --systemd reads the
  effective config and bakes the values into the generated unit
- tools/remote werkator start: writes server settings (assigned port, loopback
  bind, publicBaseUrl, nginx off) plus the Apache reverse-proxy .htaccess into
  ~/doms/<domain>/subs/www, runs init --systemd and enables the user unit
- docs/configuration.md documents the new keys

* tools/remote: env-based configuration and background port-forward

All connection and deployment values come from .env in the repository root
(WERKATOR_REMOTE, WERKATOR_PATH, WERKATOR_PORT, WERKATOR_DOMAIN,
WERKATOR_LOCAL_PORT, optional WERKATOR_BRANCH/MEMORY_MAX/TASKS_MAX/ROOTFS);
missing values fail with a pointing error instead of positional parameters.

- port-forward is now 'tools/remote port-forward start|stop' with a detached
  ssh tunnel, pid file under /tmp, and idempotent start
- start restarts the systemd unit after updating the machine config
- control-token generates the token in place when the server has not yet
- the rootfs archive default moves to build/ (already gitignored)
This commit is contained in:
Michael Hönnig
2026-08-31 19:56:04 +02:00
committed by GitHub
parent 516765a717
commit 71f1fc62c6
21 changed files with 1564 additions and 37 deletions
@@ -44,13 +44,16 @@ class ProcessBuildRunner : BuildRunner {
/**
* Selects the runtime per branch: Docker when `branches.<name>.docker.enabled`,
* native shell execution otherwise (the unchanged default).
* bubblewrap when `branches.<name>.bwrap.enabled`, native shell execution otherwise
* (the unchanged default). Docker and bwrap are mutually exclusive per branch and are
* rejected together at config load, so the branch order here never has to "pick".
*/
@Primary
@Component
class DispatchingBuildRunner(
private val processBuildRunner: ProcessBuildRunner,
private val dockerBuildRunner: DockerBuildRunner,
private val bwrapBuildRunner: BwrapBuildRunner,
) : BuildRunner {
override fun start(
command: String,
@@ -60,7 +63,12 @@ class DispatchingBuildRunner(
branchConfig: BranchConfig,
onAuxProcess: (Process) -> Unit,
): Process {
val runner = if (branchConfig.docker.enabled) dockerBuildRunner else processBuildRunner
val runner =
when {
branchConfig.docker.enabled -> dockerBuildRunner
branchConfig.bwrap.enabled -> bwrapBuildRunner
else -> processBuildRunner
}
return runner.start(command, workingDir, environment, repoDir, branchConfig, onAuxProcess)
}
}
@@ -0,0 +1,266 @@
package de.hoennig.werkator.build
import de.hoennig.werkator.config.BranchConfig
import de.hoennig.werkator.config.BwrapConfig
import de.hoennig.werkator.git.GitCommandRunner
import org.slf4j.LoggerFactory
import org.springframework.stereotype.Component
import java.nio.file.Files
import java.nio.file.Path
import java.security.MessageDigest
/**
* Runs build commands inside a bubblewrap user-namespace sandbox (Step 17 / ADR 0007),
* for hosts without root and without a Docker daemon (e.g. Hostsharing managed
* webspaces). Shells out to the `bwrap` CLI via the generic [GitCommandRunner] process
* wrapper — no library, consistent with git and docker.
*
* The prepared rootfs (a Debian-base archive built elsewhere, since `debootstrap` is not
* available on the target) is unpacked on demand into `.git/werkator/buildenv/<envKey>/rootfs`,
* shared across all branch worktrees like the Docker gradle cache volume; `<envKey>` derives
* from a hash of the archive source, so a changed source unpacks a fresh rootfs and stale
* ones can be pruned. The returned [Process] is the attached `bwrap` process, so log
* streaming and cancellation work exactly like native builds (`--die-with-parent` plus
* `--unshare-pid` tear down the whole tree on cancel). Git works inside the sandbox with
* the same layered mounts as the Docker runner: the primary `.git` read-only with
* `.git/werkator/` masked, see [gitMetadataMounts].
*/
@Component
class BwrapBuildRunner(
private val commandRunner: GitCommandRunner,
) : BuildRunner {
private val log = LoggerFactory.getLogger(BwrapBuildRunner::class.java)
/** Replaceable process launcher so unit tests can capture the assembled `bwrap` argv. */
internal var processStarter: (List<String>, Path) -> Process = { command, dir ->
ProcessBuilder(command).directory(dir.toFile()).start()
}
override fun start(
command: String,
workingDir: Path,
environment: Map<String, String>,
repoDir: Path,
branchConfig: BranchConfig,
onAuxProcess: (Process) -> Unit,
): Process {
val bwrap = branchConfig.bwrap
require(bwrap.rootfs.isNotBlank()) { "branches.<name>.bwrap.rootfs must be set when bwrap.enabled is true" }
val buildEnvRoot = buildEnvRoot(repoDir)
val envKey = envKey(bwrap.rootfs)
val rootfsDir = buildEnvRoot.resolve(envKey).resolve(ROOTFS_DIR)
ensureRootfs(bwrap, rootfsDir, repoDir, onAuxProcess)
val homeDir = buildEnvRoot.resolve(HOME_DIR)
Files.createDirectories(homeDir)
val args =
invocation(command, workingDir, environment, repoDir, bwrap, rootfsDir, homeDir)
ensureMountpoints(rootfsDir, args)
return processStarter(args, repoDir)
}
/**
* bwrap creates mountpoint directories for bind destinations inside the sandbox —
* against the read-only rootfs bind that fails with "Can't mkdir parents ...
* Read-only file system" for every destination that does not exist in the rootfs
* (the workspace under the repo, for example). The rootfs directory itself is a
* plain host directory, so we pre-create the mountpoints there; bwrap then finds
* them and has nothing left to mkdir.
*/
private fun ensureMountpoints(
rootfsDir: Path,
args: List<String>,
) {
var i = 0
while (i < args.size) {
val arg = args[i]
if (arg == "--bind" || arg == "--ro-bind") {
val dest = args[i + 2]
val mountpoint = rootfsDir.resolve(dest.substring(1))
// Skip anything that already exists in the rootfs (e.g. /etc/resolv.conf
// is a file the rootfs ships); only missing dirs are created.
if (dest.startsWith("/") && !Files.exists(mountpoint)) {
Files.createDirectories(mountpoint)
}
i += 3
} else if (arg == "--proc" || arg == "--dev" || arg == "--tmpfs") {
// The rootfs archive ships no /proc, /dev (excluded when packed), so
// these mountpoints must exist too.
val dest = args[i + 1]
if (dest.startsWith("/") && !Files.exists(rootfsDir.resolve(dest.substring(1)))) {
Files.createDirectories(rootfsDir.resolve(dest.substring(1)))
}
i += 2
} else {
i += 1
}
}
}
/**
* Unpacks the configured archive into [rootfsDir] once per environment version
* (identified by [envKey]). Missing means "not yet unpacked"; the environment is a
* cache like the Docker image and the Gradle volume, and stale ones are pruned with
* the rest of `.git/werkator`.
*/
private fun ensureRootfs(
bwrap: BwrapConfig,
rootfsDir: Path,
repoDir: Path,
onAuxProcess: (Process) -> Unit,
) {
if (Files.isDirectory(rootfsDir)) {
return
}
Files.createDirectories(rootfsDir)
val archive = localArchive(bwrap.rootfs, rootfsDir.parent, repoDir, onAuxProcess)
log.info("unpacking build environment {} into {}", bwrap.rootfs, rootfsDir)
commandRunner.runOrThrow(
listOf("tar", "--no-same-owner", "-xf", archive, "-C", rootfsDir.toString()),
repoDir,
onProcess = onAuxProcess,
)
}
/**
* Resolves [BwrapConfig.rootfs] to a local archive path: a bare or `file:` path is
* used as-is; an `http(s)` URL is downloaded once into the buildenv root. GNU tar
* auto-detects the compression from the archive magic, so a `.tar.gz` or `.tar.zst`
* needs no extra flag.
*/
private fun localArchive(
rootfs: String,
envDir: Path,
repoDir: Path,
onAuxProcess: (Process) -> Unit,
): String {
if (!rootfs.startsWith("http://") && !rootfs.startsWith("https://")) {
return rootfs.removePrefix("file://")
}
val fileName = rootfs.substringAfterLast('/').ifBlank { "buildenv" }
val target = envDir.resolve(fileName)
if (!Files.exists(target)) {
log.info("downloading build environment {} from {}", fileName, rootfs)
commandRunner.runOrThrow(
listOf("curl", "-fsSL", "-o", target.toString(), rootfs),
repoDir,
onProcess = onAuxProcess,
)
}
return target.toString()
}
private fun invocation(
command: String,
workspace: Path,
environment: Map<String, String>,
repoDir: Path,
bwrap: BwrapConfig,
rootfsDir: Path,
homeDir: Path,
): List<String> {
// bwrap creates mountpoints for bind destinations inside the sandbox; a
// relative workspace path would resolve there into the read-only rootfs
// ("Can't mkdir parents ...: Read-only file system"). Bind at absolute
// host paths instead — same contract as the Docker runner. Relative
// paths come from the CLI relative to the repo, so resolve them against
// repoDir, not against the process working directory.
val repoDirAbs = repoDir.toAbsolutePath().normalize()
val workspaceAbs =
if (workspace.isAbsolute) workspace.normalize() else repoDirAbs.resolve(workspace).normalize()
val homeDirAbs =
if (homeDir.isAbsolute) homeDir.normalize() else repoDirAbs.resolve(homeDir).normalize()
val args =
mutableListOf(
"bwrap",
"--unshare-user",
"--unshare-pid",
"--die-with-parent",
"--uid",
"0",
"--gid",
"0",
"--ro-bind",
rootfsDir.toString(),
"/",
)
// Bind the repo read-write FIRST so bwrap can create the mountpoints of
// the later binds (workspace, worktree admin dir) inside it — creating
// them against the read-only rootfs fails with "Can't mkdir parents ...
// Read-only file system". The git metadata mounts below then layer the
// usual isolation on top: read-only .git, tmpfs mask over .git/werkator,
// read-write worktree admin dir.
args += listOf("--bind", "$repoDirAbs", "$repoDirAbs")
// Git metadata mounts BEFORE the workspace bind: the tmpfs mask over
// .git/werkator must not shadow the workspace, which lives under
// .git/werkator/worktrees — the later workspace bind shadows the mask
// at exactly its own path and nothing else.
args += gitMetadataMounts(workspaceAbs, repoDir)
args += listOf("--bind", "$workspaceAbs", "$workspaceAbs")
args += listOf("--bind", "$homeDirAbs", "/root")
args += listOf("--ro-bind", "/etc/resolv.conf", "/etc/resolv.conf")
args += listOf("--proc", "/proc", "--dev", "/dev", "--tmpfs", "/tmp")
args += listOf("--setenv", "HOME", "/root")
for ((key, value) in environment) {
args += listOf("--setenv", key, value)
}
for ((key, value) in bwrap.env) {
args += listOf("--setenv", key, value)
}
args += listOf("--chdir", "$workspaceAbs", "/bin/sh", "-c", command)
return args
}
/**
* Makes git work inside the sandbox without exposing Werkator's secrets — the same
* three layered mounts as the Docker runner, expressed in `bwrap` flags (bwrap nests
* mounts by target path like Docker): the primary `.git` read-only, an empty tmpfs
* masking `.git/werkator/` (machine config with `git.token`, control token, build
* state), and this worktree's admin directory read-write so index-refreshing commands
* keep working. Object and ref writes stay blocked by the read-only `.git` mount.
* No mounts are added when the workspace is not a worktree of [repoDir].
*/
private fun gitMetadataMounts(
workspace: Path,
repoDir: Path,
): List<String> {
val gitDir = repoDir.toAbsolutePath().normalize().resolve(".git")
val workspaceGitFile = workspace.resolve(".git")
if (!Files.isDirectory(gitDir) || !Files.isRegularFile(workspaceGitFile)) {
return emptyList()
}
val adminDir =
Files
.readString(workspaceGitFile)
.substringAfter("gitdir:", "")
.trim()
.takeIf { it.isNotEmpty() }
?.let { workspace.resolve(it).normalize() }
?: return emptyList()
if (!adminDir.startsWith(gitDir) || !Files.isDirectory(adminDir)) {
return emptyList()
}
val args = mutableListOf("--ro-bind", "$gitDir", "$gitDir")
val werkatorDir = gitDir.resolve("werkator")
if (Files.isDirectory(werkatorDir)) {
args += listOf("--tmpfs", "$werkatorDir")
}
args += listOf("--bind", "$adminDir", "$adminDir")
return args
}
private fun buildEnvRoot(repoDir: Path): Path = repoDir.resolve(BUILDENV_DIR)
/** A short hash of the archive source, so a changed source unpacks a fresh rootfs. */
private fun envKey(rootfs: String): String =
MessageDigest
.getInstance("SHA-256")
.digest(rootfs.toByteArray())
.joinToString("") { "%02x".format(it) }
.take(12)
companion object {
const val BUILDENV_DIR = ".git/werkator/buildenv"
const val ROOTFS_DIR = "rootfs"
const val HOME_DIR = "home"
}
}
@@ -19,6 +19,7 @@ import java.nio.file.Paths
)
class InitCommand(
private val gitService: GitService,
private val configLoader: de.hoennig.werkator.config.ConfigLoader,
/** The version written into the generated config as `werkator.version.since`. */
private val buildProperties: ObjectProvider<BuildProperties>? = null,
) : Runnable {
@@ -157,6 +158,11 @@ class InitCommand(
bindAddress: 127.0.0.1
# optional Impressum (legal disclosure) link in the web UI footer; empty hides the link
impressumUrl: ""
# Resource limits of the systemd user unit (init --systemd); empty = directive omitted.
# Needed where the service shares a memory slice, e.g. Hostsharing Managed Webspaces.
systemd:
memoryMax: "" # e.g. 1G — a runaway Gradle build must not starve the package
tasksMax: "" # e.g. 512
# Opt-in managed nginx+certbot Docker container for HTTPS, for hosts without
# a usable reverse proxy (see docs/deployment.md). Off by default.
nginx:
@@ -215,6 +221,12 @@ class InitCommand(
context: "." # Docker build context used with dockerfile
network: "" # Docker network mode for the build container; empty = Docker default (pinned)
env: {} # additional environment variables set inside the build container
# bubblewrap user-namespace sandbox — for hosts without root and without a
# Docker daemon (e.g. Hostsharing managed webspaces). Mutually exclusive with docker.
bwrap:
enabled: false # run clean/build in a bwrap sandbox instead of natively (pinned)
rootfs: "" # prepared rootfs archive (path or URL); required when enabled (pinned)
env: {} # additional environment variables set inside the sandbox
# Gitea check this build reports as; empty uses gitea.statusContext.
# Two builds of one commit under the same context overwrite each other.
statusContext: ""
@@ -256,6 +268,19 @@ class InitCommand(
println("created ${file.toFile().relativeTo(normalizedWorkingDir.toFile())}")
}
/**
* Resource limits for the unit come from the effective configuration when one is
* already loadable (re-running `init --systemd` on an installed instance); during
* the very first bootstrap they stay unset and the defaults (no directives) apply.
*/
private fun loadedSystemdConfig(): de.hoennig.werkator.config.SystemdConfig =
try {
configLoader.load(Paths.get(".")).server.systemd
} catch (_: Exception) {
de.hoennig.werkator.config
.SystemdConfig()
}
private fun createSystemdFiles(
root: Path,
normalizedWorkingDir: Path,
@@ -277,6 +302,8 @@ class InitCommand(
javaExecutable = javaExecutableResolver(),
jarPath = jarPath,
envFile = envFile,
memoryMax = loadedSystemdConfig().memoryMax,
tasksMax = loadedSystemdConfig().tasksMax,
),
)
println("created ${unitFile.toFile().relativeTo(normalizedWorkingDir.toFile())}")
@@ -22,24 +22,33 @@ object SystemdServiceFiles {
javaExecutable: Path,
jarPath: Path,
envFile: Path,
): String =
"""
[Unit]
Description=Werkator CI for ${repoRoot.fileName}
Wants=network-online.target
After=network-online.target docker.service
memoryMax: String = "",
tasksMax: String = "",
): String {
val limits =
listOfNotNull(
"MemoryMax=$memoryMax".takeIf { memoryMax.isNotBlank() },
"TasksMax=$tasksMax".takeIf { tasksMax.isNotBlank() },
).joinToString("\n")
val limitsLine = if (limits.isEmpty()) "" else "\n$limits"
return """
[Unit]
Description=Werkator CI for ${repoRoot.fileName}
Wants=network-online.target
After=network-online.target docker.service
[Service]
Type=simple
WorkingDirectory=${systemdPath("$repoRoot")}
EnvironmentFile=-${systemdPath("$envFile")}
ExecStart=${systemdQuote("$javaExecutable")} ${'$'}JAVA_OPTS -jar ${systemdQuote("$jarPath")} server
Restart=always
RestartSec=30
[Service]
Type=simple
WorkingDirectory=${systemdPath("$repoRoot")}
EnvironmentFile=-${systemdPath("$envFile")}
ExecStart=${systemdQuote("$javaExecutable")} ${'$'}JAVA_OPTS -jar ${systemdQuote("$jarPath")} server
Restart=always
RestartSec=30
[Install]
WantedBy=default.target
""".trimIndent() + "\n"
[Install]
WantedBy=default.target
""".trimIndent().replace("\n\n[Install]", "$limitsLine\n\n[Install]") + "\n"
}
/**
* Nightly Docker cleanup like the legacy `docker-prune.service`, but without `--volumes`:
@@ -42,6 +42,8 @@ data class BuildDefinition(
val statusContext: String? = null,
/** Overrides of the docker settings; null inherits them. */
val docker: DockerOverrides? = null,
/** Overrides of the bwrap settings; null inherits them. */
val bwrap: BwrapOverrides? = null,
) {
/** The settings this build runs with: [branchConfig] with this definition applied; unset values fall through. */
fun applyTo(branchConfig: BranchConfig): BranchConfig =
@@ -62,6 +64,12 @@ data class BuildDefinition(
network = docker?.network ?: branchConfig.docker.network,
env = docker?.env ?: branchConfig.docker.env,
),
bwrap =
branchConfig.bwrap.copy(
enabled = bwrap?.enabled ?: branchConfig.bwrap.enabled,
rootfs = bwrap?.rootfs ?: branchConfig.bwrap.rootfs,
env = bwrap?.env ?: branchConfig.bwrap.env,
),
)
companion object {
@@ -159,3 +167,12 @@ data class DockerOverrides(
val context: String? = null,
val env: Map<String, String>? = null,
)
/** Nullable bubblewrap overrides of a [BuildDefinition]; null values inherit the branch's setting. */
data class BwrapOverrides(
/** Run the build in the bwrap sandbox instead of natively. Pinned — a branch must not escape its sandbox. */
val enabled: Boolean? = null,
/** Rootfs archive source. Pinned — a branch must not substitute a foreign rootfs. */
val rootfs: String? = null,
val env: Map<String, String>? = null,
)
@@ -159,6 +159,11 @@ class ConfigLoader(
val strippedDocker = docker.toMutableMap().apply { PINNED_DOCKER_KEYS.forEach { remove(it) } }
if (strippedDocker.isEmpty()) result.remove("docker") else result["docker"] = strippedDocker
}
val bwrap = entry["bwrap"] as? Map<String, Any?>
if (bwrap != null) {
val strippedBwrap = bwrap.toMutableMap().apply { PINNED_BWRAP_KEYS.forEach { remove(it) } }
if (strippedBwrap.isEmpty()) result.remove("bwrap") else result["bwrap"] = strippedBwrap
}
return result
}
@@ -398,6 +403,9 @@ class ConfigLoader(
/** `docker` keys a branch must never override: the sandbox policy. */
private val PINNED_DOCKER_KEYS = setOf("enabled", "network")
/** `bwrap` keys a branch must never override: the sandbox policy (Step 17). */
private val PINNED_BWRAP_KEYS = setOf("enabled", "rootfs")
/**
* The one key of a build definition that says *when* and *for which branches* it
* runs; never inherited from `builds.default`. A single key on purpose: a selector
@@ -37,7 +37,14 @@ data class WerkatorConfig(
build: String,
): BranchConfig {
val branchConfig = branches[branch] ?: branches["default"] ?: BranchConfig()
return effectiveBuildDefinitions()[build]?.applyTo(branchConfig) ?: branchConfig
val settings = effectiveBuildDefinitions()[build]?.applyTo(branchConfig) ?: branchConfig
if (settings.docker.enabled && settings.bwrap.enabled) {
throw IllegalArgumentException(
"builds.$build on '$branch' enables both docker and bwrap; a build runs in exactly one sandbox. " +
"Disable one of them.",
)
}
return settings
}
}
@@ -57,9 +64,24 @@ data class ServerConfig(
val bindAddress: String = "127.0.0.1",
/** Optional Impressum (legal disclosure) link shown in the web UI footer; empty hides the link. */
val impressumUrl: String = "",
/**
* Resource limits for the generated systemd user unit (`init --systemd`); empty
* means the directive is not written. Needed on platforms where the service runs
* inside a shared memory slice, e.g. Hostsharing Managed Webspaces, where a
* runaway Gradle build would starve everything else in the package.
*/
val systemd: SystemdConfig = SystemdConfig(),
val nginx: NginxConfig = NginxConfig(),
)
/** Resource-limit directives of the systemd user unit (`server.systemd`, see [ServerConfig.systemd]). */
data class SystemdConfig(
/** `MemoryMax=` of the unit, e.g. `1G`; empty omits the directive. */
val memoryMax: String = "",
/** `TasksMax=` of the unit, e.g. `512`; empty omits the directive. */
val tasksMax: String = "",
)
/**
* Opt-in managed nginx+certbot Docker container serving Werkator over HTTPS,
* for hosts without a usable reverse proxy (ADR 0005). Off by default; the
@@ -157,6 +179,26 @@ data class BranchConfig(
val statusContext: String = "",
val autoBuild: AutoBuildConfig = AutoBuildConfig(),
val docker: DockerConfig = DockerConfig(),
/** bubblewrap user-namespace sandbox; mutually exclusive with [docker]. */
val bwrap: BwrapConfig = BwrapConfig(),
)
/**
* bubblewrap build sandbox (Step 17): runs the build in an unprivileged user namespace
* with a prepared Debian root filesystem. For hosts without root and without a Docker
* daemon (e.g. Hostsharing managed webspaces); see `docs/plan/17-bwrap-build-runtime.md`.
*/
data class BwrapConfig(
/** Run the clean and build commands in a bwrap sandbox instead of natively. */
val enabled: Boolean = false,
/**
* Path or URL of the prepared rootfs archive (e.g. `werkator-buildenv-trixie-java21.tar.zst`),
* unpacked on demand into `.git/werkator/buildenv/<envKey>/rootfs`; required when [enabled].
* Pinned — a branch must not substitute a foreign rootfs via its committed config.
*/
val rootfs: String = "",
/** Additional environment variables set inside the sandbox. */
val env: Map<String, String> = emptyMap(),
)
data class DockerConfig(
@@ -0,0 +1,240 @@
package de.hoennig.werkator.build
import de.hoennig.werkator.config.BranchConfig
import de.hoennig.werkator.config.BwrapConfig
import de.hoennig.werkator.git.GitCommandResult
import de.hoennig.werkator.git.GitCommandRunner
import io.kotest.assertions.throwables.shouldThrow
import io.kotest.core.spec.style.FunSpec
import io.kotest.matchers.shouldBe
import io.kotest.matchers.string.shouldContain
import io.mockk.clearMocks
import io.mockk.every
import io.mockk.mockk
import io.mockk.verify
import java.nio.file.Files
import java.nio.file.Path
class BwrapBuildRunnerTest : FunSpec() {
private val commandRunner = mockk<GitCommandRunner>()
private lateinit var runner: BwrapBuildRunner
private lateinit var repoDir: Path
private lateinit var workspace: Path
private val captured = mutableListOf<List<String>>()
private fun bwrapBranchConfig(
rootfs: String = "/srv/buildenv.tar.zst",
env: Map<String, String> = emptyMap(),
): BranchConfig =
BranchConfig(
bwrap =
BwrapConfig(
enabled = true,
rootfs = rootfs,
env = env,
),
)
private fun rootfsUnpacked(rootfs: String = "/srv/buildenv.tar.zst"): Path =
repoDir
.resolve(BwrapBuildRunner.BUILDENV_DIR)
.resolve(rootfs.sha12())
.resolve(BwrapBuildRunner.ROOTFS_DIR)
init {
beforeEach {
clearMocks(commandRunner)
captured.clear()
repoDir = Files.createTempDirectory("werkator-bwrap-runner")
workspace = repoDir.resolve("workspace")
runner = BwrapBuildRunner(commandRunner)
runner.processStarter = { command, _ ->
captured += command
ProcessBuilder("true").start()
}
}
test("unpacks the rootfs on demand and assembles the exact bwrap command") {
every {
commandRunner.runOrThrow(
listOf("tar", "--no-same-owner", "-xf", "/srv/buildenv.tar.zst", "-C", rootfsUnpacked().toString()),
repoDir,
any(),
any(),
)
} returns
GitCommandResult(0, "", "")
runner.start("./gradlew test", workspace, mapOf("branch" to "main"), repoDir, bwrapBranchConfig())
val args = captured.single()
val rootfsDir = args[args.indexOf("--ro-bind") + 1]
args shouldBe
listOf(
"bwrap",
"--unshare-user",
"--unshare-pid",
"--die-with-parent",
"--uid",
"0",
"--gid",
"0",
"--ro-bind",
rootfsUnpacked().toString(),
"/",
"--bind",
repoDir.toString(),
repoDir.toString(),
"--bind",
workspace.toString(),
workspace.toString(),
"--bind",
repoDir.resolve(".git/werkator/buildenv/home").toString(),
"/root",
"--ro-bind",
"/etc/resolv.conf",
"/etc/resolv.conf",
"--proc",
"/proc",
"--dev",
"/dev",
"--tmpfs",
"/tmp",
"--setenv",
"HOME",
"/root",
"--setenv",
"branch",
"main",
"--chdir",
workspace.toString(),
"/bin/sh",
"-c",
"./gradlew test",
)
Files.isDirectory(rootfsUnpacked()) shouldBe true
}
test("binds a relative workspace path at its absolute location") {
// bwrap creates mountpoints for bind destinations inside the sandbox;
// a relative path would land in the read-only rootfs and fail with
// "Can't mkdir parents ...: Read-only file system" (seen on the webspace).
every {
commandRunner.runOrThrow(
listOf("tar", "--no-same-owner", "-xf", "/srv/buildenv.tar.zst", "-C", rootfsUnpacked().toString()),
repoDir,
any(),
any(),
)
} returns
GitCommandResult(0, "", "")
val relativeWorkspace = repoDir.relativize(workspace)
runner.start("./gradlew test", relativeWorkspace, mapOf("branch" to "main"), repoDir, bwrapBranchConfig())
val args = captured.single()
val absolute = workspace.toAbsolutePath().normalize().toString()
val bindIdx = args.withIndex().filter { it.value == "--bind" }.map { it.index }
// first bind is the repo dir (mountpoint base), second is the workspace
args[bindIdx[1] + 1] shouldBe absolute
args[bindIdx[1] + 2] shouldBe absolute
args[args.indexOf("--chdir") + 1] shouldBe absolute
}
test("does not re-unpack an already prepared rootfs") {
Files.createDirectories(rootfsUnpacked())
runner.start("./gradlew test", workspace, mapOf("branch" to "main"), repoDir, bwrapBranchConfig())
verify(exactly = 0) { commandRunner.runOrThrow(match { it.first() == "tar" }, any(), any(), any()) }
}
test("adds bwrap env and passes the branch environment through") {
Files.createDirectories(rootfsUnpacked())
runner.start(
"./gradlew test",
workspace,
mapOf("branch" to "main"),
repoDir,
bwrapBranchConfig(env = mapOf("FOO" to "bar")),
)
val args = captured.single()
args[args.indexOf("branch") - 1] shouldBe "--setenv"
args[args.indexOf("branch") + 1] shouldBe "main"
args[args.indexOf("FOO") - 1] shouldBe "--setenv"
args[args.indexOf("FOO") + 1] shouldBe "bar"
}
test("exposes git metadata read-only with the werkator dir masked for a worktree workspace") {
val gitDir = repoDir.resolve(".git")
val adminDir = gitDir.resolve("worktrees/workspace")
Files.createDirectories(adminDir)
Files.createDirectories(gitDir.resolve("werkator"))
Files.createDirectories(workspace)
Files.writeString(workspace.resolve(".git"), "gitdir: $adminDir\n")
Files.createDirectories(rootfsUnpacked())
runner.start("./gradlew test", workspace, mapOf("branch" to "main"), repoDir, bwrapBranchConfig())
val args = captured.single()
args[args.indexOf(gitDir.toString()) - 1] shouldBe "--ro-bind"
args[args.indexOf("$gitDir/werkator") - 1] shouldBe "--tmpfs"
args[args.indexOf(adminDir.toString()) - 1] shouldBe "--bind"
}
test("mounts no git metadata when the workspace is not a worktree") {
Files.createDirectories(rootfsUnpacked())
Files.createDirectories(workspace)
runner.start("./gradlew test", workspace, mapOf("branch" to "main"), repoDir, bwrapBranchConfig())
val args = captured.single()
val gitDir = repoDir.resolve(".git")
// the sandbox's own /tmp tmpfs is always present; the point is that no tmpfs
// masks .git/werkator and no worktree admin dir is bound
args.none { it == "$gitDir/werkator" } shouldBe true
args.none { it.contains("worktrees/") } shouldBe true
}
test("fails without a configured rootfs") {
val branchConfig = BranchConfig(bwrap = BwrapConfig(enabled = true))
val exception =
shouldThrow<IllegalArgumentException> {
runner.start("./gradlew test", workspace, mapOf("branch" to "main"), repoDir, branchConfig)
}
exception.message shouldContain "bwrap.rootfs"
}
test("downloads a URL rootfs once before unpacking") {
val url = "https://example.test/buildenv.tar.zst"
val downloadTarget =
repoDir
.resolve(BwrapBuildRunner.BUILDENV_DIR)
.resolve(url.sha12())
.resolve("buildenv.tar.zst")
every { commandRunner.runOrThrow(listOf("curl", "-fsSL", "-o", downloadTarget.toString(), url), repoDir, any(), any()) } returns
GitCommandResult(0, "", "")
every { commandRunner.runOrThrow(match { it.first() == "tar" }, any(), any(), any()) } returns
GitCommandResult(0, "", "")
runner.start("./gradlew test", workspace, mapOf("branch" to "main"), repoDir, bwrapBranchConfig(rootfs = url))
verify {
commandRunner.runOrThrow(listOf("curl", "-fsSL", "-o", downloadTarget.toString(), url), repoDir, any(), any())
}
verify { commandRunner.runOrThrow(match { it.first() == "tar" }, repoDir, any(), any()) }
}
}
private fun String.sha12(): String =
java.security.MessageDigest
.getInstance("SHA-256")
.digest(toByteArray())
.joinToString("") { "%02x".format(it) }
.take(12)
}
@@ -1,6 +1,7 @@
package de.hoennig.werkator.build
import de.hoennig.werkator.config.BranchConfig
import de.hoennig.werkator.config.BwrapConfig
import de.hoennig.werkator.config.DockerConfig
import io.kotest.core.spec.style.FunSpec
import io.kotest.matchers.shouldBe
@@ -14,12 +15,13 @@ import java.nio.file.Paths
class DispatchingBuildRunnerTest : FunSpec() {
private val processBuildRunner = mockk<ProcessBuildRunner>()
private val dockerBuildRunner = mockk<DockerBuildRunner>()
private val dispatcher = DispatchingBuildRunner(processBuildRunner, dockerBuildRunner)
private val bwrapBuildRunner = mockk<BwrapBuildRunner>()
private val dispatcher = DispatchingBuildRunner(processBuildRunner, dockerBuildRunner, bwrapBuildRunner)
private val process = mockk<Process>()
private val dir = Paths.get(".")
init {
beforeEach { clearMocks(processBuildRunner, dockerBuildRunner) }
beforeEach { clearMocks(processBuildRunner, dockerBuildRunner, bwrapBuildRunner) }
test("runs natively by default") {
val branchConfig = BranchConfig()
@@ -28,6 +30,7 @@ class DispatchingBuildRunnerTest : FunSpec() {
dispatcher.start("cmd", dir, emptyMap(), dir, branchConfig) shouldBe process
verify { dockerBuildRunner wasNot Called }
verify { bwrapBuildRunner wasNot Called }
}
test("runs in Docker when the branch enables it") {
@@ -37,6 +40,17 @@ class DispatchingBuildRunnerTest : FunSpec() {
dispatcher.start("cmd", dir, emptyMap(), dir, branchConfig) shouldBe process
verify { processBuildRunner wasNot Called }
verify { bwrapBuildRunner wasNot Called }
}
test("runs in bwrap when the branch enables it (and not Docker)") {
val branchConfig = BranchConfig(bwrap = BwrapConfig(enabled = true, rootfs = "/srv/buildenv.tar.zst"))
every { bwrapBuildRunner.start("cmd", dir, emptyMap(), dir, branchConfig) } returns process
dispatcher.start("cmd", dir, emptyMap(), dir, branchConfig) shouldBe process
verify { processBuildRunner wasNot Called }
verify { dockerBuildRunner wasNot Called }
}
}
}
@@ -14,7 +14,12 @@ import java.nio.file.attribute.PosixFilePermissions
class InitCommandTest : FunSpec() {
private val gitService = mockk<GitService>()
private val initCommand = InitCommand(gitService)
private val initCommand =
InitCommand(
gitService,
de.hoennig.werkator.config
.ConfigLoader(mockk(relaxed = true)),
)
init {
test("creates config files with auto-detected values") {
@@ -33,6 +33,26 @@ class SystemdServiceFilesTest : FunSpec() {
content shouldContain "WantedBy=default.target"
}
test("resource limits are written when configured and omitted when unset") {
fun unit(
memoryMax: String,
tasksMax: String,
) = SystemdServiceFiles.unitFileContent(
repoRoot = Paths.get("/srv/repos/my-repo"),
javaExecutable = Paths.get("/usr/bin/java"),
jarPath = Paths.get("/srv/repos/my-repo/werkator.jar"),
envFile = Paths.get("/srv/repos/my-repo/werkator.env"),
memoryMax = memoryMax,
tasksMax = tasksMax,
)
val with = unit(memoryMax = "1G", tasksMax = "512")
with shouldContain "MemoryMax=1G"
with shouldContain "TasksMax=512"
val without = unit(memoryMax = "", tasksMax = "")
without shouldNotContain "MemoryMax"
without shouldNotContain "TasksMax"
}
test("percent signs in paths are escaped for systemd") {
val content =
SystemdServiceFiles.unitFileContent(
@@ -350,6 +350,39 @@ class ConfigLoaderTest : FunSpec() {
settings.docker.image shouldBe "attacker-image"
}
test("a branch cannot disable its bwrap sandbox or substitute a foreign rootfs through a build definition") {
val dir = Files.createTempDirectory("werkator-test")
dir.resolve(".werkator.yml").toFile().writeText(
"""
builds:
default:
bwrap:
enabled: true
rootfs: /host/rootfs.tar.zst
""".trimIndent(),
)
val worktree = Files.createTempDirectory("werkator-test-worktree")
worktree.resolve(".werkator.yml").toFile().writeText(
"""
builds:
default:
bwrap:
enabled: false
rootfs: /attacker/rootfs.tar.zst
env:
FOO: from-branch
""".trimIndent(),
)
val settings = loader.loadForWorktree(dir, worktree).buildSettings("any-branch", "default")
// pinned: the sandbox can neither be switched off nor pointed at a foreign rootfs
settings.bwrap.enabled shouldBe true
settings.bwrap.rootfs shouldBe "/host/rootfs.tar.zst"
// everything that describes the build itself stays the branch's own business
settings.bwrap.env shouldBe mapOf("FOO" to "from-branch")
}
test("a build the branch invents inherits the host's sandbox policy") {
val dir = Files.createTempDirectory("werkator-test")
dir.resolve(".werkator.yml").toFile().writeText(
@@ -386,6 +419,31 @@ class ConfigLoaderTest : FunSpec() {
settings.requirePullRequest shouldBe true
}
test("enabling both docker and bwrap on a build is rejected, not picked silently") {
val dir = Files.createTempDirectory("werkator-test")
dir.resolve(".werkator.yml").toFile().writeText(
"""
builds:
default:
docker:
enabled: true
image: build-env
bwrap:
enabled: true
rootfs: /srv/rootfs.tar.zst
""".trimIndent(),
)
val config = loader.load(dir)
val exception =
shouldThrow<IllegalArgumentException> {
config.buildSettings("any-branch", "default")
}
exception.message shouldContain "both docker and bwrap"
exception.message shouldContain "builds.default"
}
test("an exclusion pattern takes a branch out of a build that would otherwise select it") {
val dir = Files.createTempDirectory("werkator-test")
dir.resolve(".werkator.yml").toFile().writeText(
@@ -2,18 +2,35 @@ package de.hoennig.werkator.framework
import io.kotest.core.spec.style.FunSpec
import io.kotest.matchers.shouldBe
import org.testcontainers.DockerClientFactory
import org.testcontainers.containers.GenericContainer
import org.testcontainers.utility.DockerImageName
/**
* Probes that Testcontainers can actually start a container on this host.
* Skipped, not failed, when no Docker is present — that is what lets Werkator
* build itself on a Docker-less host (e.g. a Hostsharing webspace, where its own
* build runs in the bubblewrap sandbox); see `tools/werkator-build-prerequisites.sh`.
*/
class TestcontainersSmokeTest :
FunSpec({
test("Testcontainers starts a container") {
val container =
GenericContainer(DockerImageName.parse("alpine:3"))
.withCommand("sh", "-c", "sleep 30")
container.start()
container.isRunning shouldBe true
container.stop()
}
})
test("Testcontainers starts a container")
.config(enabledIf = { dockerAvailable() }) {
val container =
GenericContainer(DockerImageName.parse("alpine:3"))
.withCommand("sh", "-c", "sleep 30")
container.start()
container.isRunning shouldBe true
container.stop()
}
}) {
companion object {
private fun dockerAvailable(): Boolean =
try {
DockerClientFactory.instance().isDockerAvailable()
} catch (_: Throwable) {
false
}
}
}