From 516765a717f3781b442089fc85da60e107aad19d Mon Sep 17 00:00:00 2001 From: =?UTF-8?q?Michael=20H=C3=B6nnig?= Date: Mon, 31 Aug 2026 13:58:34 +0200 Subject: [PATCH 1/2] build duration tracking Feature notiert (#2) MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit * renaming from gitTally to Werkator * Rename GitTally to Werkator `gitTally` is the name of another product in the git space, so the rename is a precaution; nothing about what the build system does changes. The name follows one rule: `Werkator` where it is prose, capitalized where it is a Kotlin type and its file, lowercase everywhere a machine reads it — the command, packages, paths, configuration keys and values, the Gitea check context. Environment variables keep their convention and are uppercase throughout. Every configuration file is still found under its pre-rename name (`ConfigFiles`): `.gittally.yml` at the repository root, in a build worktree and as committed on a branch, `.git/gittally/.gittally.yml` for the machine layer. The current name wins where both exist, and the old file is then ignored rather than merged — two files side by side are a half-done rename, not a layering. Without the fallback an installation that updated without renaming would not fail: a configuration that is not found leaves every setting at its default, so it would come up looking healthy while having forgotten its credentials and its builds. `docs/werkator-migrationsplan.md` lists what the fallback does not cover and has to be moved by hand — above all the state directory `.git/werkator/`, which holds the build history, the control token and the worktrees, and has no fallback of its own. `docs/migration-from-legacy.md` is deleted with this: it mapped the legacy script's environment variables, and every host it addressed has long since moved to the YAML configuration. Co-Authored-By: Claude Opus 5 * Move the pre-rename state directory at the first start The configuration is found under either name, the state is not: build history, control token, auto-build slots and worktrees live at one fixed path. An installation that updates without moving `.git/gittally` would not fail — it would come up with an empty history and a fresh control token, quietly. So the first start moves it instead of the release notes asking for it. Only when the old directory exists and the new one does not. Where both exist nothing is touched and a warning names the leftover: which of the two is the live state is not something to guess. A failed move is an error in the log, never an abort — a CI must not hang on it. The worktrees are dropped rather than moved, since they point at their old path in both directions; `GitWorktreeWorkspaces` prunes the stale admin entry and recreates each on its branch's next build. A generated systemd unit moves with the directory and leaves its symlink dangling, which is warned about — the running service is unaffected, the next start is not. Runs from `CliRunner`, before any command resolves a path under the directory, and so before the second context of `server` exists. Co-Authored-By: Claude Opus 5 * Document PR#1: the rename to Werkator Co-Authored-By: Claude Opus 5 * Drop the legacy env-to-YAML conversion from the setup tool The old bash script configured itself through `GITTALLY_*` environment variables. The blanket rename rewrote those literals, so the converter was looking for `WERKATOR_*` — a spelling no host has ever written. Fed a real legacy file it would have found nothing and written an almost empty configuration, without an error, which is the same silent failure this rename is otherwise careful to avoid. The conversion has served its purpose with the vm2176 to vm4006 migration, so it goes instead of being repaired. What remains is the setup of a new instance: the preconditions, the credential prompt, and the machine configuration written mode 600 — now carrying the host's public URL as well, since that is host-specific too. Everything the repository builds comes from `init` and its templates. It also stops emitting a legacy `branches:` section, which step 18 is about to reject outright. `docs/plan/00-legacy-analysis.md` and `13-nginx-tls.md` get the real `GITTALLY_*` spelling back: they record what the old script read. Co-Authored-By: Claude Opus 5 * Describe this repository's build with a build definition Its own `.werkator.yml` still used the deprecated `branches` section with an `autoBuild` schedule that was switched off. That section is read only while nothing defines a build at all, and step 18 rejects it by name — so this repository would have blocked the precondition of that step, which asks that no configuration still in play carries it. Nothing about the build changes: `builds.default` with `trigger.onPush` is a build of every new commit on every branch, which is what the branch section said. `config:print --full` resolves the definition completely and logs no deprecation warning any more. Co-Authored-By: Claude Opus 5 * Stop documenting the pre-rename fallback for users Exactly one repository is configured with the old names, and it is migrated by hand in the same move as this release. The fallback is therefore a transition of days, not a feature anyone reading the release notes or the configuration reference has to plan around. Removed from `releases.html` and `docs/configuration.md`. The mechanism itself is unchanged and stays described where it is worked on: in `ConfigFiles`, in `StateDirMigration`, and in the migration plan. Co-Authored-By: Claude Opus 5 * Bring the PR-doc to its final state The `statusContext` question is answered and marked as decided rather than left standing: it is the one value a human reads as a label, and it stays lowercase because Gitea matches it and the client reads it back, which makes it a value. Also records that the pre-rename fallback is deliberately absent from the release notes and the configuration reference. Co-Authored-By: Claude Opus 5 * Release v1.0.0: Werkator The release after 0.9.21 is 1.0.0, because a product that changes its name is better off counting from one under it. The release note says as much, so the jump is not read as a claim about maturity — plan steps 14, 17 and 18 are still open. Co-Authored-By: Claude Opus 5 * Correct the PR-doc about the version It claimed the PR carries no version bump, which the release commit made untrue, and records why the number is 1.0.0 instead of 0.9.22. Co-Authored-By: Claude Opus 5 * Point the legacy references at the history `legacy/gitTally` was removed from the tree with the rename, but the README still described it as a reference kept in the repository, and the plan told an executing session to read parts of it — including step 14, which is open. The README section is gone; `docs/plan/README.md`, step 14 and the legacy analysis now say where the script actually is (`git show 7f55068^:legacy/gitTally`). Executed steps and ADR 0004 keep their wording: they record what was true when they ran. Co-Authored-By: Claude Opus 5 * Retarget the links in the historic PR-docs The package rename moved every file the older PR-docs link to, leaving 60 dead links. Only the link targets are rewritten, never the visible text and never a statement: those documents record what was true when they were written, GitTally in the prose included. A snapshot may be outdated; it should still be navigable. Co-Authored-By: Claude Opus 5 * Add the v1.0.0 deployment procedure for vm4006 Measured, not estimated: the state directory is 878 MB, of which 878 MB are the nine build worktrees. What cannot be recreated is 84 KB, so the snapshot before an in-place switch is instant and the rollback is one sequence of moves. Records the three expected non-failures — a cold Gradle volume, one image rebuild, containers left under the old label — and that `gitea.statusContext` needs no attention because it comes from the watched repository's committed configuration. Co-Authored-By: Claude Opus 5 * Rename the machine configuration along with its directory Found by the deployment to vm4006: the move renames the directory and leaves the file inside it alone, so the machine configuration ended up at `.git/werkator/.gittally.yml` — a pair of names the lookup did not expect, because it pairs directory and file name. The instance resolved empty credentials, no public URL and none of the host's build definitions, and said nothing about it. That is the exact failure this change exists to prevent, produced by the change itself. `StateDirMigration` now renames the configuration with the directory, unless one under the current name is already there. `ConfigFiles` carries `.git/werkator/.gittally.yml` as a third candidate as well, for a directory somebody moved by hand, where the migration never runs and so can rename nothing. Co-Authored-By: Claude Opus 5 * Let init see a configuration under its previous name `init --systemd` runs `init`, and its "already exists" check knew only the current name. On the one repository still carrying `.gittally.yml` it therefore wrote a fresh template `.werkator.yml` beside it — and since the current name wins, that repository would have built the template's `./gradlew test` instead of what its own configuration says. Found on vm4006, where the file was created in the watched working tree and removed again by hand. Both checks now ask `ConfigFiles`, so init decides existence by the same rule the loader uses to read. Co-Authored-By: Claude Opus 5 * Record the v1.0.0 deployment to vm4006 Deployed from the branch as the final test of PR#1, and it did what a final test is for: it found two silent-failure defects before the service was started, both fixed and redeployed in the same window. Co-Authored-By: Claude Opus 5 * Drop the control token left under the old localStorage key The key is named after the product, so the rename left every browser with a token under `gittally.controlToken`, which nothing reads any more and which "forget token" can no longer reach. It is a write-scope token in a browser store, not a password, but a secret nobody owns is worth one line to remove. Removed on load. The token on the server is unchanged, so re-entering it once per browser is all the rename costs. Co-Authored-By: Claude Opus 5 * Plan tracking the build duration over time (step 20) --------- Co-authored-by: Claude Opus 5 --- docs/plan/20-build-duration-tracking.md | 98 +++++++++++++++++++++++++ docs/plan/README.md | 5 ++ 2 files changed, 103 insertions(+) create mode 100644 docs/plan/20-build-duration-tracking.md diff --git a/docs/plan/20-build-duration-tracking.md b/docs/plan/20-build-duration-tracking.md new file mode 100644 index 0000000..cdb6232 --- /dev/null +++ b/docs/plan/20-build-duration-tracking.md @@ -0,0 +1,98 @@ +# Step 20: Track build duration over time + +Prerequisites: the single-build duration already exists — `BuildResult.duration` (pure execution time from `runningSince`, without the queue wait) is shown in the History view, the JSON API (`durationSeconds`) and the CLI status. +Read `README.md` first. + +## Why + +Step 14's `Out of Scope` deliberately left out *historical* phase series and alerting; it only gave a per-build breakdown. +A per-build number answers "how long was the last run", not "is the build getting slower". +On a CI serving one repository the first sign of trouble is usually a build that used to take 6 minutes now taking 9 — before a failure, at a point where the per-build figure still looks fine. + +This step turns the already-recorded durations into a trend, so a regression in build time is visible as it happens instead of after the fact. + +## What Already Exists + +- `BuildResult.duration` — set once on the final transition in `BuildExecutor` (`Duration.between(runningSince, now)`). +- `repository.history()` returns every recorded build, newest first, each carrying `duration`. +- The History view (`/history`, `/api/builds/history`) already renders `durationSeconds` per row via `BuildResultDto`. +- No aggregation, no trend, no comparison against history exists anywhere. + +So the raw data is complete; nothing about build recording needs to change. + +## Goal + +A per-job (build-definition name) duration trend derived from the existing history, shown where the History view already lives. +The trend answers, for a chosen job and a bounded time window (default: last 30 days): + +- how long each build took, chronologically; +- the running/window average, the min and the max; +- the slowest recent build and how far it sits above the window average — a visible outlier instead of a silent one; +- how the latest finished build compares to the window average (slower/faster by how much). + +Derived on read from `repository.history()`, not stored: it is the same data, and deriving keeps the repository schema untouched and the numbers always fresh as the retention window prunes old runs. + +## Scope Decision 1: whole build, not phases + +This step tracks the *whole execution* duration — the number that tells a user "the build is getting slower". +Step 14 tracks the *phase* breakdown (where orchestration time goes) and already owns the per-build budget warning. +Keep the two apart: phase timing is Step 14's, whole-duration trends are this step's. +Only compute a trend where `duration` is not null (queued/cancelled builds have none and carry no signal for this purpose). + +## Scope Decision 2: which builds go into the trend — jobs, not branches + +Branch builds matter here as much as the primary branch, often more: on our projects the feature-branch build is the performance gate, not `main`. +But the history is already grouped by `name` (`branch`, or `@`), and a branch may sit behind the machine config by up to a week, yet still build several times a day. +So the trend must **not** merge every branch into one per-job line — that would average a stale branch together with the current one and hide exactly the regression the feature exists to show. + +Rule: group the trend by the history's own grouping — `name` — which is already the natural comparison unit (every "latest" and every retention pool is keyed by it). +A job defined on the host (`main@nightly`) gets its own line; each branch build (`mihoe/feature-x`) gets its own line. +Where `build == DEFAULT` the line is just the branch, and still distinct from every other branch and from every named job. +The series stays comparable because each line is a single, self-consistent grouping; the window average of `main@nightly` is never polluted by a branch that lags the config. + +What this deliberately does *not* do is aggregate branches together for a project-level view. +"A branch build got slower" is a per-name question and is answered per line; conflating branches has no one right answer while they lag the config by different amounts. +The default UI shows the primary branch or the default job, and the other lines are a lookup, not a summary. + +This assumes the per-branch duration itself is a fair comparison: every build of one `name` runs the same build steps with the same config (pinning strips only a fixed set). +A branch lagging the machine config by under a week still runs its committed `.werkator.yml` — see the branch-layer invariant — so its durations come from the same definition it has always run, and the trend is honest within that line. + +## Design + +1. `build/BuildDurationTrend.kt` — pure function over `List`, grouped by `name` (the history's grouping — one line per name, see Scope Decision 2), optionally filtered by a `since: Instant` window: + per name return the chronological `(startedAt, duration)` series plus window average, min, max, and the latest finished duration's deviation from the window average. + Pure and dependency-free so it is unit-testable without Spring. + Durations with a `null` `duration` are excluded up front. +2. `server/BuildsApiController.kt` — a `GET /api/builds/duration` returning the trend for all names over the default window. The History view polls it on the same cycle as the rest. +3. Web UI — a compact trend block on the History view (`/history`): for the primary branch (the default job), the series of recent durations with the window average and the latest build's delta, with an explicit marker when the latest build is slower than the window average by a margin (the exact ink and wording are free; the marker must call out *slowness* in the user's words, not just plot a line). The other names are a lookup, not a summary — see Scope Decision 2. Follow the existing discipline: a fetch timeout, and a failure of this request must not break the view's own refresh. + `UiFormats` and `werkator.js` keep producing identical formats (invariant in `AGENTS.md`). +4. CLI — `status --history` already prints per-build duration; add nothing to the CLI. The trend is a server-side view; the CLI has no need to chart it. + +## Out of Scope + +- Phase timing and the overhead budget warning — Step 14. +- Storing a time series separate from the build results; the retention window already prunes old runs, and the trend lives where the history lives. +- Alerting or notifications (Webhook, Gitea status) on a slow build — this step surfaces the trend in the UI; wiring it to push is a later concern and belongs nowhere here. + +## Tests + +- `build/BuildDurationTrendTest.kt` — grouping by name (a branch build and a named job stay separate, never merged); the window filter; a `duration == null` build is excluded; average/min/max correct for an asymmetric series; the latest build's deviation (slower and faster) against the window average. +- `server/BuildsApiControllerTest.kt` — a history with several durations across two names resolves through `/api/builds/duration` to the expected per-name trends. +- The JavaScript has no test harness; the trend block is verified manually below. + +## Documentation + +- `docs/configuration.md` — nothing configurable is added (the window is a constant for now); no config key changes. +- `docs/adrs/` — none: this repeats no architectural decision, it derives read-only from existing data. +- `AGENTS.md` web-UI invariant is already covered; no new invariant. + +## Verification + +- In a scratch install, run several builds of different durations (e.g. by building on commits with different workloads), then open `/history` and confirm the trend shows the series, the window average, and a slower-latest marker on the last run. +- Delete a run via the API and confirm the trend reflects the pruned history, since it is derived on read. +- On a narrow viewport the trend block must not push the table off screen. + +## Production + +Nothing to deploy beyond the next release: the feature is read-only over existing history and needs no config or migration on vm4006. +Deploy as usual. diff --git a/docs/plan/README.md b/docs/plan/README.md index a9ff329..41a1c6c 100644 --- a/docs/plan/README.md +++ b/docs/plan/README.md @@ -91,6 +91,10 @@ Added for running Werkator on Hostsharing Managed Webspaces (2026-08-10): - [ ] `17-bwrap-build-runtime.md` — Werkator on a Managed Webspace: bubblewrap user-namespace build sandbox with a prepared rootfs (precondition check first — see the step file), plus web access under a domain via the platform's Apache proxy and Let's Encrypt +Added for surfacing build time as a trend (2026-08-31): + +- [ ] `20-build-duration-tracking.md` — a per-name duration trend over the existing history, derived on read in the History view: series, window average/min/max, and a visible marker when the latest build is slower than its window average (grouped by the history's own `name`, so branch builds and named jobs stay separate — complements Step 14, which owns phase timing) + Steps 01–03 are independent of each other. Steps 04–06 depend on 01–03. Steps 07–09 depend on 04–06. @@ -100,3 +104,4 @@ Step 15 depends on 12 and 13 and revises the containerized-runtime sketch in `do Step 17 depends on 11, 15, and 16, and starts with a hard precondition check on the target webspace (ADR 0007 is written as part of the step). Step 18 depends on nothing in code but on the watched repository having migrated — its precondition check is a hard gate, not a formality. Step 19 depends on nothing; `WatcherState` and `/api/watcher` already carry everything it needs to render. +Step 20 depends on nothing; the duration is already recorded, and the trend is derived read-only from `repository.history()`. From 71f1fc62c6ea634fb3a2db71238d4e8cffecd589 Mon Sep 17 00:00:00 2001 From: =?UTF-8?q?Michael=20H=C3=B6nnig?= Date: Mon, 31 Aug 2026 19:56:04 +0200 Subject: [PATCH 2/2] Bwrap build runtime und Installation in Hostsharing Managed Webspace (#4) MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit * Add the bubblewrap build runtime (step 17, ADR 0007) BwrapBuildRunner: third runtime behind BuildRunner for hosts without root and without Docker (e.g. Hostsharing managed webspaces). Shells out to the bwrap CLI, unpacks a prepared rootfs on demand into .git/werkator/buildenv//rootfs, reuses the Docker runner's git metadata mounts, and returns the attached bwrap process for streaming and cancellation. Config: bwrap.enabled/rootfs/env on BranchConfig and BwrapOverrides on BuildDefinition; enabled/rootfs are pinned like the docker sandbox policy. Docker and bwrap are mutually exclusive per build, rejected in buildSettings instead of picked silently. DispatchingBuildRunner routes bwrap; InitCommand template, docs/configuration.md and AGENTS.md in sync. * bwrap rollout tooling: remote script, prerequisites disk/quota check, absolute workspace binds - tools/remote: central remote control script with check-prerequisites, install and build commands - tools/werkator-build-prerequisites.sh: compact PASS/FAIL output, target-dir parameter, free-space and group-quota headroom checks against the ~5 GiB build footprint, home-filesystem reference - BwrapBuildRunner: bind workspace and home at absolute paths resolved against repoDir — a relative path made bwrap create mountpoints inside the read-only rootfs (seen on the webspace); regression test - TestcontainersSmokeTest: gated with enabledIf docker available (skip, never fail, without a daemon) - docs: configuration reference, step-17 plan notes, PR-doc * bwrap: bind the repo read-write before the workspace so mountpoints are creatable bwrap creates mountpoints for bind destinations inside the sandbox; with only a read-only rootfs bound at /, creating them for the workspace under .git/werkator/ worktrees failed with 'Read-only file system' (seen on the webspace). Binding the repo dir read-write first provides the base; the git metadata mounts then layer the usual isolation on top (read-only .git, tmpfs mask over .git/werkator, read-write worktree admin dir). * bwrap: pre-create bind mountpoints inside the unpacked rootfs bwrap mkdirs mountpoints for bind destinations against the sandbox view; with the rootfs ro-bound at / every destination missing from the rootfs (the repo dir under /home/storage/... on the webspace) fails with 'Read-only file system'. The rootfs directory is a plain host dir, so create the mountpoints there before launching bwrap; it then finds them and has nothing left to create. * bwrap: skip existing rootfs files when pre-creating bind mountpoints /etc/resolv.conf is a file the rootfs already ships; createDirectories threw on it. Only missing directories are created now. * bwrap: pre-create proc/dev/tmpfs mountpoints in the rootfs too The rootfs archive ships no /proc or /dev (excluded when packed), so bwrap failed mkdir'ing their mountpoints against the read-only root. * bwrap: bind the workspace after the git metadata mounts The tmpfs mask over .git/werkator shadowed the earlier workspace bind, because the worktree lives under .git/werkator/worktrees — chdir then failed with ENOENT. The workspace bind now comes last and shadows the mask at exactly its own path. * systemd resource limits and webspace start command (step 17, web access) - server.systemd.memoryMax/tasksMax (empty = directive omitted): on platforms where the service runs in a shared memory slice (Hostsharing Managed Webspaces) a runaway Gradle build must not starve the whole package; init --systemd reads the effective config and bakes the values into the generated unit - tools/remote werkator start: writes server settings (assigned port, loopback bind, publicBaseUrl, nginx off) plus the Apache reverse-proxy .htaccess into ~/doms//subs/www, runs init --systemd and enables the user unit - docs/configuration.md documents the new keys * tools/remote: env-based configuration and background port-forward All connection and deployment values come from .env in the repository root (WERKATOR_REMOTE, WERKATOR_PATH, WERKATOR_PORT, WERKATOR_DOMAIN, WERKATOR_LOCAL_PORT, optional WERKATOR_BRANCH/MEMORY_MAX/TASKS_MAX/ROOTFS); missing values fail with a pointing error instead of positional parameters. - port-forward is now 'tools/remote port-forward start|stop' with a detached ssh tunnel, pid file under /tmp, and idempotent start - start restarts the systemd unit after updating the machine config - control-token generates the token in place when the server has not yet - the rootfs archive default moves to build/ (already gitignored) --- .gitignore | 2 + AGENTS.md | 4 +- docs/configuration.md | 32 +- docs/plan/17-bwrap-build-runtime.md | 11 + .../2026-08-31-PR#000-bwrap-build-runtime.md | 135 +++++++ .../de/hoennig/werkator/build/BuildRunner.kt | 12 +- .../werkator/build/BwrapBuildRunner.kt | 266 ++++++++++++++ .../hoennig/werkator/commands/InitCommand.kt | 27 ++ .../werkator/commands/SystemdServiceFiles.kt | 41 ++- .../werkator/config/BuildDefinition.kt | 17 + .../hoennig/werkator/config/ConfigLoader.kt | 8 + .../hoennig/werkator/config/WerkatorConfig.kt | 44 ++- .../werkator/build/BwrapBuildRunnerTest.kt | 240 ++++++++++++ .../build/DispatchingBuildRunnerTest.kt | 18 +- .../werkator/commands/InitCommandTest.kt | 7 +- .../commands/SystemdServiceFilesTest.kt | 20 + .../werkator/config/ConfigLoaderTest.kt | 58 +++ .../framework/TestcontainersSmokeTest.kt | 35 +- tools/build-bwrap-rootfs.sh | 116 ++++++ tools/remote | 344 ++++++++++++++++++ tools/werkator-build-prerequisites.sh | 164 +++++++++ 21 files changed, 1564 insertions(+), 37 deletions(-) create mode 100644 docs/prs/2026-08-31-PR#000-bwrap-build-runtime.md create mode 100644 src/main/kotlin/de/hoennig/werkator/build/BwrapBuildRunner.kt create mode 100644 src/test/kotlin/de/hoennig/werkator/build/BwrapBuildRunnerTest.kt create mode 100755 tools/build-bwrap-rootfs.sh create mode 100755 tools/remote create mode 100755 tools/werkator-build-prerequisites.sh diff --git a/.gitignore b/.gitignore index a081a8f..3750b5c 100644 --- a/.gitignore +++ b/.gitignore @@ -34,3 +34,5 @@ replay_pid* # Other /.local/ +/.env + diff --git a/AGENTS.md b/AGENTS.md index d5849ba..254d830 100644 --- a/AGENTS.md +++ b/AGENTS.md @@ -39,9 +39,9 @@ All production code lives under `de.hoennig.werkator`, with sub-packages `comman - Builds run detached in worktrees under `.git/werkator/worktrees/`; the primary checkout is never used for builds; never assume a single running build. - When config keys change, three places must stay in sync: the `WerkatorConfig` data classes, the `InitCommand` templates, and `docs/configuration.md`. - Every config file may declare `werkator.version.since`/`below` (the Werkator it is written for, never a format version — no API is involved). `since` is enforced in both directions, using `ConfigVersions.FORMAT_BROKE_IN` for "file predates a breaking change"; `below` only warns. A violation aborts the start for the machine and project config, but fails only that branch's builds for a branch config. -- A branch describes its own CI: its committed `.werkator.yml` is the branch layer (`ConfigLoader.loadWithBranchLayer`, used by the watcher per origin branch and by `loadForWorktree` at build time) and takes precedence over `.git`/project — including the whole `builds` section, so a new configuration can be tried out on a branch without affecting other branches. Only the pinned set is stripped from that layer: secrets (`git`), host/repository sections (`server`, `gitea`, `executor`, `watcher`), the docker sandbox policy (`docker.enabled`, `docker.network`), and the trust gate (`requirePullRequest`). A branch must never reach credentials, disable its container, change its network, raise global concurrency, or bypass its own pull-request gate; a branch's definitions apply to that branch alone. +- A branch describes its own CI: its committed `.werkator.yml` is the branch layer (`ConfigLoader.loadWithBranchLayer`, used by the watcher per origin branch and by `loadForWorktree` at build time) and takes precedence over `.git`/project — including the whole `builds` section, so a new configuration can be tried out on a branch without affecting other branches. Only the pinned set is stripped from that layer: secrets (`git`), host/repository sections (`server`, `gitea`, `executor`, `watcher`), the docker (`docker.enabled`, `docker.network`) and bubblewrap (`bwrap.enabled`, `bwrap.rootfs`) sandbox policies, and the trust gate (`requirePullRequest`). A branch must never reach credentials, disable its container or sandbox, change its network, substitute a foreign rootfs, raise global concurrency, or bypass its own pull-request gate; a branch's definitions apply to that branch alone. - A build definition carries the complete description of its build, split in two: the `trigger` block (`onPush`, `atTimes`, `branches`, `activeWithin`) says when and for which branches it runs, everything else what it does. `builds.default` is the base every other definition inherits its settings — never its `trigger` — from. The split is structural so that a selector added to `TriggerConfig` later is non-inheritable by construction; writing a trigger key flat is refused, never ignored, because ignoring it leaves a build that silently stops running. A `!` prefix in `trigger.branches` excludes and always wins. -- The inheritance is applied after all layers are merged: that order is what makes a build a branch invents inherit the host's sandbox policy instead of the data-class default, so the pinning also holds for a build the host has never heard of. Pinned are `requirePullRequest`, `statusContext`, `docker.enabled`, and `docker.network`. +- The inheritance is applied after all layers are merged: that order is what makes a build a branch invents inherit the host's sandbox policy instead of the data-class default, so the pinning also holds for a build the host has never heard of. Pinned are `requirePullRequest`, `statusContext`, `docker.enabled`, `docker.network`, `bwrap.enabled`, and `bwrap.rootfs`. Docker and bwrap are mutually exclusive per branch — enabling both is rejected at start. - `builds` or the legacy `branches`, never both: `branches` is read only while the merged config defines no build at all (`builds.maxConcurrent` is not one), and ignored with a warning as soon as one exists. The section is deprecated and goes away once the repositories have migrated; then `ConfigVersions.FORMAT_BROKE_IN` gets set and a leftover `branches:` key must be rejected by name — the version check alone cannot catch a file that declares no version. - Web UI: server-rendered Thymeleaf plus one hand-written `static/werkator.js` — no SPA framework, no frontend build pipeline; every fetch has a timeout and an explicit error badge; `UiFormats` and `werkator.js` must produce identical display formats. - Git and Docker access shells out to the CLIs (`GitCommandRunner`, `docker`) — no JGit, no Docker SDK. diff --git a/docs/configuration.md b/docs/configuration.md index 6ad2150..7cb34b0 100644 --- a/docs/configuration.md +++ b/docs/configuration.md @@ -63,7 +63,7 @@ above, giving the precedence **branch > repo install > project**. It takes prece everything that describes how this branch is built: the whole `builds` section — its own definitions and its overrides of the definitions from the project config, with `buildCommand`, `cleanCommand`, `artifactDirs`, log file names, and -`docker.image`/`dockerfile`/`context`/`env` inside them. That is how a new configuration is tried out: change it on a branch, and +`docker.image`/`dockerfile`/`context`/`env` and `bwrap.env` inside them. That is how a new configuration is tried out: change it on a branch, and no other branch's builds are affected. The branch layer is used in both places where it matters: the watcher reads the committed @@ -90,7 +90,8 @@ single branch may decide it: - the repository-side settings: the whole `gitea`, `executor`, and `watcher` sections; - the trust gate: `requirePullRequest`, and the Gitea status context: `statusContext`; -- the container sandbox policy: `docker.enabled` and `docker.network` — host-pinned as +- the container sandbox policy: `docker.enabled`/`docker.network` and + `bwrap.enabled`/`bwrap.rootfs` — host-pinned as long as only the host's configuration sets them, master-pinned once the committed configuration does. @@ -138,6 +139,12 @@ server: bindAddress: 127.0.0.1 # optional Impressum (legal disclosure) link in the web UI footer; empty hides the link impressumUrl: "" + # Resource limits of the systemd user unit generated by `init --systemd`; empty = directive omitted. + # Needed where the service shares a memory slice, e.g. Hostsharing Managed Webspaces, where a + # runaway Gradle build would starve everything else in the package. + systemd: + memoryMax: "" # e.g. 1G — written as `MemoryMax=` into the unit + tasksMax: "" # e.g. 512 — written as `TasksMax=` into the unit # Opt-in managed nginx+certbot Docker container for HTTPS, for hosts without # a usable reverse proxy (ADR 0005; see notes below and deployment.md). nginx: @@ -342,9 +349,9 @@ That is how a branch gets a build of its own without being built by the default `activeWithin` (e.g. `24h`) additionally keeps only branches whose origin head commit is younger than the duration — useful to run a nightly deep check over all recently active branches. Both parts combine as an intersection. -Settings: `buildCommand`, `cleanCommand`, `artifactDirs`, `stdoutLog`/`stderrLog`, `requirePullRequest`, `statusContext`, and `docker` with all its keys. +Settings: `buildCommand`, `cleanCommand`, `artifactDirs`, `stdoutLog`/`stderrLog`, `requirePullRequest`, `statusContext`, and `docker` and `bwrap` with all their keys. A definition carries the complete description of its build; unset keys fall back to `builds.default` and then to Werkator's own defaults. -`requirePullRequest`, `statusContext`, `docker.enabled`, and `docker.network` are pinned (master-pinned, see [the branch layer](#the-branch-layer-a-branch-describes-its-own-ci)): they are read from the repo install/project config even when a branch sets them in its own committed config. +`requirePullRequest`, `statusContext`, `docker.enabled`, `docker.network`, `bwrap.enabled`, and `bwrap.rootfs` are pinned (master-pinned, see [the branch layer](#the-branch-layer-a-branch-describes-its-own-ci)): they are read from the repo install/project config even when a branch sets them in its own committed config. Inheritance from `builds.default` covers the settings only — the `trigger` block says when and where *this* build runs and is never inherited. Definitions are part of the branch layer: a branch may add its own and override those from the project config, for its own builds only. Because the inheritance is applied after all layers are merged, a build a branch invents still inherits the host's `builds.default` — its sandbox policy included, which is what keeps the pinning effective for a build the host has never heard of. @@ -400,6 +407,23 @@ Note that the rest of `.git` — including `.git/config` — is visible to build The Docker socket is mounted into the container and `DOCKER_HOST`/`TESTCONTAINERS_*` variables are set, so Testcontainers-based builds work inside the container. All Werkator containers carry `org.hoennig.werkator` labels; stale build containers of the repository are removed before the first Docker build after a restart. +### Notes on `builds..bwrap` + +With `bwrap.enabled`, Werkator shells out to the `bwrap` CLI (bubblewrap) instead of native execution. +This is the third runtime, for hosts without root and without a Docker daemon (e.g. Hostsharing managed webspaces); see `docs/plan/17-bwrap-build-runtime.md` and ADR 0007. +`bwrap` must be on the `PATH`. + +`bwrap.rootfs` names the prepared root filesystem archive — a Debian-base rootfs with the build tools (JDK, git, locales, project-specific tooling) built elsewhere, since `debootstrap` is unavailable on the target. +It is a local path or an `http(s)` URL; a URL is downloaded once. +Build the archive with `tools/build-bwrap-rootfs.sh` on any machine with Docker; verify the host's user-namespace capability first with `tools/werkator-build-prerequisites.sh`. +The archive is unpacked on demand (`tar --no-same-owner`) into `.git/werkator/buildenv//rootfs`, shared across all branch worktrees like the Docker Gradle cache volume; `` derives from a hash of the source, so a changed `rootfs` unpacks a fresh environment and stale ones can be pruned. +Per-branch Gradle caches persist in `.git/werkator/buildenv/home`, bound as `/root`. +`bwrap.env` adds environment variables inside the sandbox. +Files created inside the sandbox are owned by the host user, because uid 0 maps back to the unprivileged webspace user. + +`docker` and `bwrap` are mutually exclusive per branch: enabling both is rejected at start, not silently picked. +Git works inside the sandbox exactly as inside the Docker container: the primary `.git` is mounted read-only with `.git/werkator/` masked, so builds can run read-only git commands but never reach the machine config or the control token. + ## `.git/werkator/.werkator.yml` (not committed) ```yaml diff --git a/docs/plan/17-bwrap-build-runtime.md b/docs/plan/17-bwrap-build-runtime.md index 0cd22fe..42a2960 100644 --- a/docs/plan/17-bwrap-build-runtime.md +++ b/docs/plan/17-bwrap-build-runtime.md @@ -95,6 +95,17 @@ bwrap --unshare-user --unshare-pid --die-with-parent --uid 0 --gid 0 \ - No Docker inside the sandbox, so no Testcontainers-based tests; build commands must select a Docker-free test subset. For Werkator's own build this means `TestcontainersSmokeTest` must become conditional (`enabledIf` docker present) — that change is part of this step. +**Done (branch `bwrap-build-runtime`):** `TestcontainersSmokeTest` is now gated with `enabledIf docker available` — it is reported as skipped (never failed) when no Docker daemon is reachable, and runs as before when one is. +The two manual steps of the workflow have scripts in `tools/`: +`tools/build-bwrap-rootfs.sh` builds the rootfs archive (debootstrap-minbase Debian + JDK 21 + git + locales) on any Docker machine — the rootfs is *not* built on the target; `tools/werkator-build-prerequisites.sh` re-runs the precondition command line above on the target webspace and checks all three signals. +It also checks the disk/quota situation: free space via `df` plus group-quota headroom (`quota -g` limit minus usage) against the ~4 GiB build footprint (unpacked rootfs + Gradle cache + artifacts) — an undersized quota fails the check, since a build would otherwise be blocked mid-flight (experienced on h68: 1 GiB group quota). + +**Planned: a central `tools/remote` control script** (same pattern as the user's other repos; first argument is the repo, here `werkator`, then a command): + +- `tools/remote werkator check-prerequisites @` — runs `werkator-build-prerequisites.sh` on the target host (uploaded if missing). +- `tools/remote werkator install @` — full setup, idempotent: ensure SSH access (`ssh-copy-id`, first login asks for the password), run the prerequisites check, upload runtime bundle + rootfs archive to `~/.werkator/`, unpack, clone the repo, run `werkator init`, write the machine-local bwrap config. + The one manual dependency remains: the host's public SSH key must be added to GitHub once; the script prints the key and waits. + ## Web Access under a Domain (no Docker, no managed nginx) The managed nginx/TLS container from ADR 0005 is for container hosts without a reverse proxy. diff --git a/docs/prs/2026-08-31-PR#000-bwrap-build-runtime.md b/docs/prs/2026-08-31-PR#000-bwrap-build-runtime.md new file mode 100644 index 0000000..a13cd8e --- /dev/null +++ b/docs/prs/2026-08-31-PR#000-bwrap-build-runtime.md @@ -0,0 +1,135 @@ +> **WARNING:** This document describes only the change applied in this PR. +> It may already be outdated once the next PR is merged. +> Historic PR-documentation is not maintained along with new PRs — treat it as a snapshot, not as current documentation. + +## The Problem + +Werkator runs its builds on the host, either natively or inside a Docker container. +A Hostsharing **Managed Webspace** has neither root nor a Docker daemon, so neither runtime works there — yet that is exactly where some users want to run a Werkator that builds Werkator itself. + +The only sandboxing primitive available there is `bwrap` (bubblewrap): unprivileged user namespaces with a uid-0 mapping and read-only root binds. +Step 17 (docs/plan/17-bwrap-build-runtime.md) defines a third build runtime behind the `BuildRunner` interface that is based on it. + +## Non-Goals + +- No change to the native and Docker runtimes; bwrap is added alongside them and selected per branch via config. +- No Docker inside the sandbox; Testcontainers-based tests cannot run there and are excluded by a branch's own build command. +- No overlayfs: the webspace's bubblewrap 0.8.0 predates `--overlay`, so a throwaway writable rootfs per build is out of scope. +- No web-access deployment; that half of step 17 (Apache `.htaccess` proxy, systemd user unit, Let's Encrypt) needs a real webspace and is written up separately. +- No per-repo build automation on the webspace; this PR makes it possible, and a follow-up runs it on a real host. + +## The Scenarios + +### Feature: bubblewrap as the third build runtime + +#### Background + +- A branch selects exactly one runtime: nothing (native), `docker.enabled`, or `bwrap.enabled`. +- `bwrap.enabled` and `bwrap.rootfs` are pinned (host-set), so a branch's committed config cannot switch its own sandbox off or substitute a foreign rootfs. +- `docker.enabled` and `bwrap.enabled` are mutually exclusive per branch; enabling both is rejected, not silently picked. + +#### Scenario#000.01: A bwrap build runs the command inside the sandbox as root + +So that the build is isolated from the host exactly as the native and Docker runtimes intend. + +- **Given** a branch with `bwrap.enabled` and a `bwrap.rootfs` archive +- **When** a build for that branch starts +- **Then** Werkator unpacks the rootfs on demand into `.git/werkator/buildenv//rootfs` + - **and** invokes `bwrap` with a uid-0 mapping, a read-only root bind of the rootfs, the workspace bound at its host path, and `--chdir` into it + - **and** the returned process is the attached `bwrap` process, so streaming and cancellation behave like native builds +- **and** the environment plus `bwrap.env` are passed via `--setenv` + +##### Verified by + +- [BwrapBuildRunnerTest](../../src/test/kotlin/de/hoennig/werkator/build/BwrapBuildRunnerTest.kt) + +#### Scenario#000.02: Git metadata mounts keep secrets out of the sandbox + +So that builds can run read-only git commands but never reach the machine config or the control token. + +- **Given** a workspace that is a worktree of the repository +- **When** the sandbox is assembled +- **Then** the primary `.git` is bound read-only + - **and** `.git/werkator/` is masked by an empty tmpfs + - **and** the worktree admin directory is bound read-write + +##### Verified by + +- [BwrapBuildRunnerTest](../../src/test/kotlin/de/hoennig/werkator/build/BwrapBuildRunnerTest.kt) + +#### Scenario#000.03: A branch cannot turn its sandbox off or swap its rootfs + +So that the pinned sandbox policy holds for builds a branch invents as well as for ones the host already knows. + +- **Given** a branch whose committed config sets `bwrap.enabled` or `bwrap.rootfs` +- **When** that config is resolved into a build +- **Then** the pinned keys are stripped from the worktree layer + - **and** enabling both `docker` and `bwrap` on a build is rejected, not picked silently + +##### Verified by + +- [ConfigLoaderTest](../../src/test/kotlin/de/hoennig/werkator/config/ConfigLoaderTest.kt) + +#### Scenario#000.04: The dispatcher routes builds to the bwrap runtime + +So that a bwrap-explicit branch builds inside the sandbox rather than natively. + +- **Given** a branch with `bwrap.enabled` +- **When** its build is dispatched +- **Then** `BwrapBuildRunner` is selected + +##### Verified by + +- [DispatchingBuildRunnerTest](../../src/test/kotlin/de/hoennig/werkator/build/DispatchingBuildRunnerTest.kt) + +### Feature: Werkator builds itself without Docker + +#### Background + +- Werkator's own build runs the full test suite, which includes `TestcontainersSmokeTest`. +- On a Docker-less host (the webspace, and the bwrap sandbox that builds there), that test must not fail the self-build. + +#### Scenario#000.05: The Testcontainers smoke test is skipped, not failed, without Docker + +So that a Docker-less build of Werkator itself stays green. + +- **Given** no reachable Docker daemon +- **When** the test suite runs +- **Then** `TestcontainersSmokeTest` is reported as skipped + - **and** the build is not failed by it +- **and** with a Docker daemon present the test still runs and verifies a container + +##### Verified by + +- [TestcontainersSmokeTest](../../src/test/kotlin/de/hoennig/werkator/framework/TestcontainersSmokeTest.kt) + +## The Solution + +**A third `BuildRunner` by the same shell-out pattern as git and Docker.** +`BwrapBuildRunner` shells out to the `bwrap` CLI (no library), unpacks a prepared Debian rootfs on demand into `.git/werkator/buildenv//rootfs`, reuses the step-16 git-metadata mounts verbatim, binds a persistent `.git/werkator/buildenv/home` as `/root`, and returns the attached `bwrap` process so streaming and cancellation match the other runtimes. +`DispatchingBuildRunner` routes by `bwrap.enabled`. + +**Pinning and mutual exclusion hold at the choke point.** +`bwrap.enabled` and `bwrap.rootfs` join the pinned sandbox-policy set, stripped from the worktree layer so a branch cannot disable its sandbox or substitute a foreign rootfs. +`docker` and `bwrap` are mutually exclusive per build, rejected in `buildSettings` — the single point every build passes through — instead of picked silently. + +**Tooling makes the two manual steps reproducible.** +`tools/build-bwrap-rootfs.sh` builds the rootfs archive (debootstrap-minbase Debian + JDK 21 + git + locales) on any machine with Docker, since `debootstrap` is not available on the target. +`tools/werkator-build-prerequisites.sh` re-runs the exact precondition command line from the plan on the target webspace and checks all three signals. +`TestcontainersSmokeTest` is gated with `enabledIf docker available`, so a Docker-less self-build skips it. + +## Open Questions + +- Whether the rootfs archive built by `tools/build-bwrap-rootfs.sh` is complete for `./gradlew build` has not been exercised on a real webspace; the JDK 21, git and locales package set is a good baseline but project-specific tooling must be added. +- The `systemd` user unit for a webspace should gain `MemoryMax`/`TasksMax` (configurable per the plan); not implemented here. + +## Additional Changes + +- Config template (`InitCommand`), `docs/configuration.md` and `AGENTS.md` updated so all three config places stay in sync and the pinned set is documented. +- The plan file 17 records the precondition result and the new tooling. + +## Follow-up PRs + +- ADR 0007 recording the bubblewrap runtime decision (options: bwrap vs proot/fakechroot vs plain native). +- Web-access deployment on a real webspace and the `docs/deployment.md` third variant, written once verified there. +- `MemoryMax`/`TasksMax` on the webspace systemd unit. diff --git a/src/main/kotlin/de/hoennig/werkator/build/BuildRunner.kt b/src/main/kotlin/de/hoennig/werkator/build/BuildRunner.kt index d016b0d..9d1dcb8 100644 --- a/src/main/kotlin/de/hoennig/werkator/build/BuildRunner.kt +++ b/src/main/kotlin/de/hoennig/werkator/build/BuildRunner.kt @@ -44,13 +44,16 @@ class ProcessBuildRunner : BuildRunner { /** * Selects the runtime per branch: Docker when `branches..docker.enabled`, - * native shell execution otherwise (the unchanged default). + * bubblewrap when `branches..bwrap.enabled`, native shell execution otherwise + * (the unchanged default). Docker and bwrap are mutually exclusive per branch and are + * rejected together at config load, so the branch order here never has to "pick". */ @Primary @Component class DispatchingBuildRunner( private val processBuildRunner: ProcessBuildRunner, private val dockerBuildRunner: DockerBuildRunner, + private val bwrapBuildRunner: BwrapBuildRunner, ) : BuildRunner { override fun start( command: String, @@ -60,7 +63,12 @@ class DispatchingBuildRunner( branchConfig: BranchConfig, onAuxProcess: (Process) -> Unit, ): Process { - val runner = if (branchConfig.docker.enabled) dockerBuildRunner else processBuildRunner + val runner = + when { + branchConfig.docker.enabled -> dockerBuildRunner + branchConfig.bwrap.enabled -> bwrapBuildRunner + else -> processBuildRunner + } return runner.start(command, workingDir, environment, repoDir, branchConfig, onAuxProcess) } } diff --git a/src/main/kotlin/de/hoennig/werkator/build/BwrapBuildRunner.kt b/src/main/kotlin/de/hoennig/werkator/build/BwrapBuildRunner.kt new file mode 100644 index 0000000..f41cb69 --- /dev/null +++ b/src/main/kotlin/de/hoennig/werkator/build/BwrapBuildRunner.kt @@ -0,0 +1,266 @@ +package de.hoennig.werkator.build + +import de.hoennig.werkator.config.BranchConfig +import de.hoennig.werkator.config.BwrapConfig +import de.hoennig.werkator.git.GitCommandRunner +import org.slf4j.LoggerFactory +import org.springframework.stereotype.Component +import java.nio.file.Files +import java.nio.file.Path +import java.security.MessageDigest + +/** + * Runs build commands inside a bubblewrap user-namespace sandbox (Step 17 / ADR 0007), + * for hosts without root and without a Docker daemon (e.g. Hostsharing managed + * webspaces). Shells out to the `bwrap` CLI via the generic [GitCommandRunner] process + * wrapper — no library, consistent with git and docker. + * + * The prepared rootfs (a Debian-base archive built elsewhere, since `debootstrap` is not + * available on the target) is unpacked on demand into `.git/werkator/buildenv//rootfs`, + * shared across all branch worktrees like the Docker gradle cache volume; `` derives + * from a hash of the archive source, so a changed source unpacks a fresh rootfs and stale + * ones can be pruned. The returned [Process] is the attached `bwrap` process, so log + * streaming and cancellation work exactly like native builds (`--die-with-parent` plus + * `--unshare-pid` tear down the whole tree on cancel). Git works inside the sandbox with + * the same layered mounts as the Docker runner: the primary `.git` read-only with + * `.git/werkator/` masked, see [gitMetadataMounts]. + */ +@Component +class BwrapBuildRunner( + private val commandRunner: GitCommandRunner, +) : BuildRunner { + private val log = LoggerFactory.getLogger(BwrapBuildRunner::class.java) + + /** Replaceable process launcher so unit tests can capture the assembled `bwrap` argv. */ + internal var processStarter: (List, Path) -> Process = { command, dir -> + ProcessBuilder(command).directory(dir.toFile()).start() + } + + override fun start( + command: String, + workingDir: Path, + environment: Map, + repoDir: Path, + branchConfig: BranchConfig, + onAuxProcess: (Process) -> Unit, + ): Process { + val bwrap = branchConfig.bwrap + require(bwrap.rootfs.isNotBlank()) { "branches..bwrap.rootfs must be set when bwrap.enabled is true" } + val buildEnvRoot = buildEnvRoot(repoDir) + val envKey = envKey(bwrap.rootfs) + val rootfsDir = buildEnvRoot.resolve(envKey).resolve(ROOTFS_DIR) + ensureRootfs(bwrap, rootfsDir, repoDir, onAuxProcess) + val homeDir = buildEnvRoot.resolve(HOME_DIR) + Files.createDirectories(homeDir) + val args = + invocation(command, workingDir, environment, repoDir, bwrap, rootfsDir, homeDir) + ensureMountpoints(rootfsDir, args) + return processStarter(args, repoDir) + } + + /** + * bwrap creates mountpoint directories for bind destinations inside the sandbox — + * against the read-only rootfs bind that fails with "Can't mkdir parents ... + * Read-only file system" for every destination that does not exist in the rootfs + * (the workspace under the repo, for example). The rootfs directory itself is a + * plain host directory, so we pre-create the mountpoints there; bwrap then finds + * them and has nothing left to mkdir. + */ + private fun ensureMountpoints( + rootfsDir: Path, + args: List, + ) { + var i = 0 + while (i < args.size) { + val arg = args[i] + if (arg == "--bind" || arg == "--ro-bind") { + val dest = args[i + 2] + val mountpoint = rootfsDir.resolve(dest.substring(1)) + // Skip anything that already exists in the rootfs (e.g. /etc/resolv.conf + // is a file the rootfs ships); only missing dirs are created. + if (dest.startsWith("/") && !Files.exists(mountpoint)) { + Files.createDirectories(mountpoint) + } + i += 3 + } else if (arg == "--proc" || arg == "--dev" || arg == "--tmpfs") { + // The rootfs archive ships no /proc, /dev (excluded when packed), so + // these mountpoints must exist too. + val dest = args[i + 1] + if (dest.startsWith("/") && !Files.exists(rootfsDir.resolve(dest.substring(1)))) { + Files.createDirectories(rootfsDir.resolve(dest.substring(1))) + } + i += 2 + } else { + i += 1 + } + } + } + + /** + * Unpacks the configured archive into [rootfsDir] once per environment version + * (identified by [envKey]). Missing means "not yet unpacked"; the environment is a + * cache like the Docker image and the Gradle volume, and stale ones are pruned with + * the rest of `.git/werkator`. + */ + private fun ensureRootfs( + bwrap: BwrapConfig, + rootfsDir: Path, + repoDir: Path, + onAuxProcess: (Process) -> Unit, + ) { + if (Files.isDirectory(rootfsDir)) { + return + } + Files.createDirectories(rootfsDir) + val archive = localArchive(bwrap.rootfs, rootfsDir.parent, repoDir, onAuxProcess) + log.info("unpacking build environment {} into {}", bwrap.rootfs, rootfsDir) + commandRunner.runOrThrow( + listOf("tar", "--no-same-owner", "-xf", archive, "-C", rootfsDir.toString()), + repoDir, + onProcess = onAuxProcess, + ) + } + + /** + * Resolves [BwrapConfig.rootfs] to a local archive path: a bare or `file:` path is + * used as-is; an `http(s)` URL is downloaded once into the buildenv root. GNU tar + * auto-detects the compression from the archive magic, so a `.tar.gz` or `.tar.zst` + * needs no extra flag. + */ + private fun localArchive( + rootfs: String, + envDir: Path, + repoDir: Path, + onAuxProcess: (Process) -> Unit, + ): String { + if (!rootfs.startsWith("http://") && !rootfs.startsWith("https://")) { + return rootfs.removePrefix("file://") + } + val fileName = rootfs.substringAfterLast('/').ifBlank { "buildenv" } + val target = envDir.resolve(fileName) + if (!Files.exists(target)) { + log.info("downloading build environment {} from {}", fileName, rootfs) + commandRunner.runOrThrow( + listOf("curl", "-fsSL", "-o", target.toString(), rootfs), + repoDir, + onProcess = onAuxProcess, + ) + } + return target.toString() + } + + private fun invocation( + command: String, + workspace: Path, + environment: Map, + repoDir: Path, + bwrap: BwrapConfig, + rootfsDir: Path, + homeDir: Path, + ): List { + // bwrap creates mountpoints for bind destinations inside the sandbox; a + // relative workspace path would resolve there into the read-only rootfs + // ("Can't mkdir parents ...: Read-only file system"). Bind at absolute + // host paths instead — same contract as the Docker runner. Relative + // paths come from the CLI relative to the repo, so resolve them against + // repoDir, not against the process working directory. + val repoDirAbs = repoDir.toAbsolutePath().normalize() + val workspaceAbs = + if (workspace.isAbsolute) workspace.normalize() else repoDirAbs.resolve(workspace).normalize() + val homeDirAbs = + if (homeDir.isAbsolute) homeDir.normalize() else repoDirAbs.resolve(homeDir).normalize() + val args = + mutableListOf( + "bwrap", + "--unshare-user", + "--unshare-pid", + "--die-with-parent", + "--uid", + "0", + "--gid", + "0", + "--ro-bind", + rootfsDir.toString(), + "/", + ) + // Bind the repo read-write FIRST so bwrap can create the mountpoints of + // the later binds (workspace, worktree admin dir) inside it — creating + // them against the read-only rootfs fails with "Can't mkdir parents ... + // Read-only file system". The git metadata mounts below then layer the + // usual isolation on top: read-only .git, tmpfs mask over .git/werkator, + // read-write worktree admin dir. + args += listOf("--bind", "$repoDirAbs", "$repoDirAbs") + // Git metadata mounts BEFORE the workspace bind: the tmpfs mask over + // .git/werkator must not shadow the workspace, which lives under + // .git/werkator/worktrees — the later workspace bind shadows the mask + // at exactly its own path and nothing else. + args += gitMetadataMounts(workspaceAbs, repoDir) + args += listOf("--bind", "$workspaceAbs", "$workspaceAbs") + args += listOf("--bind", "$homeDirAbs", "/root") + args += listOf("--ro-bind", "/etc/resolv.conf", "/etc/resolv.conf") + args += listOf("--proc", "/proc", "--dev", "/dev", "--tmpfs", "/tmp") + args += listOf("--setenv", "HOME", "/root") + for ((key, value) in environment) { + args += listOf("--setenv", key, value) + } + for ((key, value) in bwrap.env) { + args += listOf("--setenv", key, value) + } + args += listOf("--chdir", "$workspaceAbs", "/bin/sh", "-c", command) + return args + } + + /** + * Makes git work inside the sandbox without exposing Werkator's secrets — the same + * three layered mounts as the Docker runner, expressed in `bwrap` flags (bwrap nests + * mounts by target path like Docker): the primary `.git` read-only, an empty tmpfs + * masking `.git/werkator/` (machine config with `git.token`, control token, build + * state), and this worktree's admin directory read-write so index-refreshing commands + * keep working. Object and ref writes stay blocked by the read-only `.git` mount. + * No mounts are added when the workspace is not a worktree of [repoDir]. + */ + private fun gitMetadataMounts( + workspace: Path, + repoDir: Path, + ): List { + val gitDir = repoDir.toAbsolutePath().normalize().resolve(".git") + val workspaceGitFile = workspace.resolve(".git") + if (!Files.isDirectory(gitDir) || !Files.isRegularFile(workspaceGitFile)) { + return emptyList() + } + val adminDir = + Files + .readString(workspaceGitFile) + .substringAfter("gitdir:", "") + .trim() + .takeIf { it.isNotEmpty() } + ?.let { workspace.resolve(it).normalize() } + ?: return emptyList() + if (!adminDir.startsWith(gitDir) || !Files.isDirectory(adminDir)) { + return emptyList() + } + val args = mutableListOf("--ro-bind", "$gitDir", "$gitDir") + val werkatorDir = gitDir.resolve("werkator") + if (Files.isDirectory(werkatorDir)) { + args += listOf("--tmpfs", "$werkatorDir") + } + args += listOf("--bind", "$adminDir", "$adminDir") + return args + } + + private fun buildEnvRoot(repoDir: Path): Path = repoDir.resolve(BUILDENV_DIR) + + /** A short hash of the archive source, so a changed source unpacks a fresh rootfs. */ + private fun envKey(rootfs: String): String = + MessageDigest + .getInstance("SHA-256") + .digest(rootfs.toByteArray()) + .joinToString("") { "%02x".format(it) } + .take(12) + + companion object { + const val BUILDENV_DIR = ".git/werkator/buildenv" + const val ROOTFS_DIR = "rootfs" + const val HOME_DIR = "home" + } +} diff --git a/src/main/kotlin/de/hoennig/werkator/commands/InitCommand.kt b/src/main/kotlin/de/hoennig/werkator/commands/InitCommand.kt index d4a1a3c..ac78d34 100644 --- a/src/main/kotlin/de/hoennig/werkator/commands/InitCommand.kt +++ b/src/main/kotlin/de/hoennig/werkator/commands/InitCommand.kt @@ -19,6 +19,7 @@ import java.nio.file.Paths ) class InitCommand( private val gitService: GitService, + private val configLoader: de.hoennig.werkator.config.ConfigLoader, /** The version written into the generated config as `werkator.version.since`. */ private val buildProperties: ObjectProvider? = null, ) : Runnable { @@ -157,6 +158,11 @@ class InitCommand( bindAddress: 127.0.0.1 # optional Impressum (legal disclosure) link in the web UI footer; empty hides the link impressumUrl: "" + # Resource limits of the systemd user unit (init --systemd); empty = directive omitted. + # Needed where the service shares a memory slice, e.g. Hostsharing Managed Webspaces. + systemd: + memoryMax: "" # e.g. 1G — a runaway Gradle build must not starve the package + tasksMax: "" # e.g. 512 # Opt-in managed nginx+certbot Docker container for HTTPS, for hosts without # a usable reverse proxy (see docs/deployment.md). Off by default. nginx: @@ -215,6 +221,12 @@ class InitCommand( context: "." # Docker build context used with dockerfile network: "" # Docker network mode for the build container; empty = Docker default (pinned) env: {} # additional environment variables set inside the build container + # bubblewrap user-namespace sandbox — for hosts without root and without a + # Docker daemon (e.g. Hostsharing managed webspaces). Mutually exclusive with docker. + bwrap: + enabled: false # run clean/build in a bwrap sandbox instead of natively (pinned) + rootfs: "" # prepared rootfs archive (path or URL); required when enabled (pinned) + env: {} # additional environment variables set inside the sandbox # Gitea check this build reports as; empty uses gitea.statusContext. # Two builds of one commit under the same context overwrite each other. statusContext: "" @@ -256,6 +268,19 @@ class InitCommand( println("created ${file.toFile().relativeTo(normalizedWorkingDir.toFile())}") } + /** + * Resource limits for the unit come from the effective configuration when one is + * already loadable (re-running `init --systemd` on an installed instance); during + * the very first bootstrap they stay unset and the defaults (no directives) apply. + */ + private fun loadedSystemdConfig(): de.hoennig.werkator.config.SystemdConfig = + try { + configLoader.load(Paths.get(".")).server.systemd + } catch (_: Exception) { + de.hoennig.werkator.config + .SystemdConfig() + } + private fun createSystemdFiles( root: Path, normalizedWorkingDir: Path, @@ -277,6 +302,8 @@ class InitCommand( javaExecutable = javaExecutableResolver(), jarPath = jarPath, envFile = envFile, + memoryMax = loadedSystemdConfig().memoryMax, + tasksMax = loadedSystemdConfig().tasksMax, ), ) println("created ${unitFile.toFile().relativeTo(normalizedWorkingDir.toFile())}") diff --git a/src/main/kotlin/de/hoennig/werkator/commands/SystemdServiceFiles.kt b/src/main/kotlin/de/hoennig/werkator/commands/SystemdServiceFiles.kt index 4dbcd12..566532c 100644 --- a/src/main/kotlin/de/hoennig/werkator/commands/SystemdServiceFiles.kt +++ b/src/main/kotlin/de/hoennig/werkator/commands/SystemdServiceFiles.kt @@ -22,24 +22,33 @@ object SystemdServiceFiles { javaExecutable: Path, jarPath: Path, envFile: Path, - ): String = - """ - [Unit] - Description=Werkator CI for ${repoRoot.fileName} - Wants=network-online.target - After=network-online.target docker.service + memoryMax: String = "", + tasksMax: String = "", + ): String { + val limits = + listOfNotNull( + "MemoryMax=$memoryMax".takeIf { memoryMax.isNotBlank() }, + "TasksMax=$tasksMax".takeIf { tasksMax.isNotBlank() }, + ).joinToString("\n") + val limitsLine = if (limits.isEmpty()) "" else "\n$limits" + return """ + [Unit] + Description=Werkator CI for ${repoRoot.fileName} + Wants=network-online.target + After=network-online.target docker.service - [Service] - Type=simple - WorkingDirectory=${systemdPath("$repoRoot")} - EnvironmentFile=-${systemdPath("$envFile")} - ExecStart=${systemdQuote("$javaExecutable")} ${'$'}JAVA_OPTS -jar ${systemdQuote("$jarPath")} server - Restart=always - RestartSec=30 + [Service] + Type=simple + WorkingDirectory=${systemdPath("$repoRoot")} + EnvironmentFile=-${systemdPath("$envFile")} + ExecStart=${systemdQuote("$javaExecutable")} ${'$'}JAVA_OPTS -jar ${systemdQuote("$jarPath")} server + Restart=always + RestartSec=30 - [Install] - WantedBy=default.target - """.trimIndent() + "\n" + [Install] + WantedBy=default.target + """.trimIndent().replace("\n\n[Install]", "$limitsLine\n\n[Install]") + "\n" + } /** * Nightly Docker cleanup like the legacy `docker-prune.service`, but without `--volumes`: diff --git a/src/main/kotlin/de/hoennig/werkator/config/BuildDefinition.kt b/src/main/kotlin/de/hoennig/werkator/config/BuildDefinition.kt index 7be9dcc..0c64dad 100644 --- a/src/main/kotlin/de/hoennig/werkator/config/BuildDefinition.kt +++ b/src/main/kotlin/de/hoennig/werkator/config/BuildDefinition.kt @@ -42,6 +42,8 @@ data class BuildDefinition( val statusContext: String? = null, /** Overrides of the docker settings; null inherits them. */ val docker: DockerOverrides? = null, + /** Overrides of the bwrap settings; null inherits them. */ + val bwrap: BwrapOverrides? = null, ) { /** The settings this build runs with: [branchConfig] with this definition applied; unset values fall through. */ fun applyTo(branchConfig: BranchConfig): BranchConfig = @@ -62,6 +64,12 @@ data class BuildDefinition( network = docker?.network ?: branchConfig.docker.network, env = docker?.env ?: branchConfig.docker.env, ), + bwrap = + branchConfig.bwrap.copy( + enabled = bwrap?.enabled ?: branchConfig.bwrap.enabled, + rootfs = bwrap?.rootfs ?: branchConfig.bwrap.rootfs, + env = bwrap?.env ?: branchConfig.bwrap.env, + ), ) companion object { @@ -159,3 +167,12 @@ data class DockerOverrides( val context: String? = null, val env: Map? = null, ) + +/** Nullable bubblewrap overrides of a [BuildDefinition]; null values inherit the branch's setting. */ +data class BwrapOverrides( + /** Run the build in the bwrap sandbox instead of natively. Pinned — a branch must not escape its sandbox. */ + val enabled: Boolean? = null, + /** Rootfs archive source. Pinned — a branch must not substitute a foreign rootfs. */ + val rootfs: String? = null, + val env: Map? = null, +) diff --git a/src/main/kotlin/de/hoennig/werkator/config/ConfigLoader.kt b/src/main/kotlin/de/hoennig/werkator/config/ConfigLoader.kt index f208927..ca59590 100644 --- a/src/main/kotlin/de/hoennig/werkator/config/ConfigLoader.kt +++ b/src/main/kotlin/de/hoennig/werkator/config/ConfigLoader.kt @@ -159,6 +159,11 @@ class ConfigLoader( val strippedDocker = docker.toMutableMap().apply { PINNED_DOCKER_KEYS.forEach { remove(it) } } if (strippedDocker.isEmpty()) result.remove("docker") else result["docker"] = strippedDocker } + val bwrap = entry["bwrap"] as? Map + if (bwrap != null) { + val strippedBwrap = bwrap.toMutableMap().apply { PINNED_BWRAP_KEYS.forEach { remove(it) } } + if (strippedBwrap.isEmpty()) result.remove("bwrap") else result["bwrap"] = strippedBwrap + } return result } @@ -398,6 +403,9 @@ class ConfigLoader( /** `docker` keys a branch must never override: the sandbox policy. */ private val PINNED_DOCKER_KEYS = setOf("enabled", "network") + /** `bwrap` keys a branch must never override: the sandbox policy (Step 17). */ + private val PINNED_BWRAP_KEYS = setOf("enabled", "rootfs") + /** * The one key of a build definition that says *when* and *for which branches* it * runs; never inherited from `builds.default`. A single key on purpose: a selector diff --git a/src/main/kotlin/de/hoennig/werkator/config/WerkatorConfig.kt b/src/main/kotlin/de/hoennig/werkator/config/WerkatorConfig.kt index 8555d01..38eb09c 100644 --- a/src/main/kotlin/de/hoennig/werkator/config/WerkatorConfig.kt +++ b/src/main/kotlin/de/hoennig/werkator/config/WerkatorConfig.kt @@ -37,7 +37,14 @@ data class WerkatorConfig( build: String, ): BranchConfig { val branchConfig = branches[branch] ?: branches["default"] ?: BranchConfig() - return effectiveBuildDefinitions()[build]?.applyTo(branchConfig) ?: branchConfig + val settings = effectiveBuildDefinitions()[build]?.applyTo(branchConfig) ?: branchConfig + if (settings.docker.enabled && settings.bwrap.enabled) { + throw IllegalArgumentException( + "builds.$build on '$branch' enables both docker and bwrap; a build runs in exactly one sandbox. " + + "Disable one of them.", + ) + } + return settings } } @@ -57,9 +64,24 @@ data class ServerConfig( val bindAddress: String = "127.0.0.1", /** Optional Impressum (legal disclosure) link shown in the web UI footer; empty hides the link. */ val impressumUrl: String = "", + /** + * Resource limits for the generated systemd user unit (`init --systemd`); empty + * means the directive is not written. Needed on platforms where the service runs + * inside a shared memory slice, e.g. Hostsharing Managed Webspaces, where a + * runaway Gradle build would starve everything else in the package. + */ + val systemd: SystemdConfig = SystemdConfig(), val nginx: NginxConfig = NginxConfig(), ) +/** Resource-limit directives of the systemd user unit (`server.systemd`, see [ServerConfig.systemd]). */ +data class SystemdConfig( + /** `MemoryMax=` of the unit, e.g. `1G`; empty omits the directive. */ + val memoryMax: String = "", + /** `TasksMax=` of the unit, e.g. `512`; empty omits the directive. */ + val tasksMax: String = "", +) + /** * Opt-in managed nginx+certbot Docker container serving Werkator over HTTPS, * for hosts without a usable reverse proxy (ADR 0005). Off by default; the @@ -157,6 +179,26 @@ data class BranchConfig( val statusContext: String = "", val autoBuild: AutoBuildConfig = AutoBuildConfig(), val docker: DockerConfig = DockerConfig(), + /** bubblewrap user-namespace sandbox; mutually exclusive with [docker]. */ + val bwrap: BwrapConfig = BwrapConfig(), +) + +/** + * bubblewrap build sandbox (Step 17): runs the build in an unprivileged user namespace + * with a prepared Debian root filesystem. For hosts without root and without a Docker + * daemon (e.g. Hostsharing managed webspaces); see `docs/plan/17-bwrap-build-runtime.md`. + */ +data class BwrapConfig( + /** Run the clean and build commands in a bwrap sandbox instead of natively. */ + val enabled: Boolean = false, + /** + * Path or URL of the prepared rootfs archive (e.g. `werkator-buildenv-trixie-java21.tar.zst`), + * unpacked on demand into `.git/werkator/buildenv//rootfs`; required when [enabled]. + * Pinned — a branch must not substitute a foreign rootfs via its committed config. + */ + val rootfs: String = "", + /** Additional environment variables set inside the sandbox. */ + val env: Map = emptyMap(), ) data class DockerConfig( diff --git a/src/test/kotlin/de/hoennig/werkator/build/BwrapBuildRunnerTest.kt b/src/test/kotlin/de/hoennig/werkator/build/BwrapBuildRunnerTest.kt new file mode 100644 index 0000000..65c8932 --- /dev/null +++ b/src/test/kotlin/de/hoennig/werkator/build/BwrapBuildRunnerTest.kt @@ -0,0 +1,240 @@ +package de.hoennig.werkator.build + +import de.hoennig.werkator.config.BranchConfig +import de.hoennig.werkator.config.BwrapConfig +import de.hoennig.werkator.git.GitCommandResult +import de.hoennig.werkator.git.GitCommandRunner +import io.kotest.assertions.throwables.shouldThrow +import io.kotest.core.spec.style.FunSpec +import io.kotest.matchers.shouldBe +import io.kotest.matchers.string.shouldContain +import io.mockk.clearMocks +import io.mockk.every +import io.mockk.mockk +import io.mockk.verify +import java.nio.file.Files +import java.nio.file.Path + +class BwrapBuildRunnerTest : FunSpec() { + private val commandRunner = mockk() + private lateinit var runner: BwrapBuildRunner + private lateinit var repoDir: Path + private lateinit var workspace: Path + private val captured = mutableListOf>() + + private fun bwrapBranchConfig( + rootfs: String = "/srv/buildenv.tar.zst", + env: Map = emptyMap(), + ): BranchConfig = + BranchConfig( + bwrap = + BwrapConfig( + enabled = true, + rootfs = rootfs, + env = env, + ), + ) + + private fun rootfsUnpacked(rootfs: String = "/srv/buildenv.tar.zst"): Path = + repoDir + .resolve(BwrapBuildRunner.BUILDENV_DIR) + .resolve(rootfs.sha12()) + .resolve(BwrapBuildRunner.ROOTFS_DIR) + + init { + beforeEach { + clearMocks(commandRunner) + captured.clear() + repoDir = Files.createTempDirectory("werkator-bwrap-runner") + workspace = repoDir.resolve("workspace") + runner = BwrapBuildRunner(commandRunner) + runner.processStarter = { command, _ -> + captured += command + ProcessBuilder("true").start() + } + } + + test("unpacks the rootfs on demand and assembles the exact bwrap command") { + every { + commandRunner.runOrThrow( + listOf("tar", "--no-same-owner", "-xf", "/srv/buildenv.tar.zst", "-C", rootfsUnpacked().toString()), + repoDir, + any(), + any(), + ) + } returns + GitCommandResult(0, "", "") + + runner.start("./gradlew test", workspace, mapOf("branch" to "main"), repoDir, bwrapBranchConfig()) + + val args = captured.single() + val rootfsDir = args[args.indexOf("--ro-bind") + 1] + args shouldBe + listOf( + "bwrap", + "--unshare-user", + "--unshare-pid", + "--die-with-parent", + "--uid", + "0", + "--gid", + "0", + "--ro-bind", + rootfsUnpacked().toString(), + "/", + "--bind", + repoDir.toString(), + repoDir.toString(), + "--bind", + workspace.toString(), + workspace.toString(), + "--bind", + repoDir.resolve(".git/werkator/buildenv/home").toString(), + "/root", + "--ro-bind", + "/etc/resolv.conf", + "/etc/resolv.conf", + "--proc", + "/proc", + "--dev", + "/dev", + "--tmpfs", + "/tmp", + "--setenv", + "HOME", + "/root", + "--setenv", + "branch", + "main", + "--chdir", + workspace.toString(), + "/bin/sh", + "-c", + "./gradlew test", + ) + Files.isDirectory(rootfsUnpacked()) shouldBe true + } + + test("binds a relative workspace path at its absolute location") { + // bwrap creates mountpoints for bind destinations inside the sandbox; + // a relative path would land in the read-only rootfs and fail with + // "Can't mkdir parents ...: Read-only file system" (seen on the webspace). + every { + commandRunner.runOrThrow( + listOf("tar", "--no-same-owner", "-xf", "/srv/buildenv.tar.zst", "-C", rootfsUnpacked().toString()), + repoDir, + any(), + any(), + ) + } returns + GitCommandResult(0, "", "") + + val relativeWorkspace = repoDir.relativize(workspace) + + runner.start("./gradlew test", relativeWorkspace, mapOf("branch" to "main"), repoDir, bwrapBranchConfig()) + + val args = captured.single() + val absolute = workspace.toAbsolutePath().normalize().toString() + val bindIdx = args.withIndex().filter { it.value == "--bind" }.map { it.index } + // first bind is the repo dir (mountpoint base), second is the workspace + args[bindIdx[1] + 1] shouldBe absolute + args[bindIdx[1] + 2] shouldBe absolute + args[args.indexOf("--chdir") + 1] shouldBe absolute + } + + test("does not re-unpack an already prepared rootfs") { + Files.createDirectories(rootfsUnpacked()) + + runner.start("./gradlew test", workspace, mapOf("branch" to "main"), repoDir, bwrapBranchConfig()) + + verify(exactly = 0) { commandRunner.runOrThrow(match { it.first() == "tar" }, any(), any(), any()) } + } + + test("adds bwrap env and passes the branch environment through") { + Files.createDirectories(rootfsUnpacked()) + + runner.start( + "./gradlew test", + workspace, + mapOf("branch" to "main"), + repoDir, + bwrapBranchConfig(env = mapOf("FOO" to "bar")), + ) + + val args = captured.single() + args[args.indexOf("branch") - 1] shouldBe "--setenv" + args[args.indexOf("branch") + 1] shouldBe "main" + args[args.indexOf("FOO") - 1] shouldBe "--setenv" + args[args.indexOf("FOO") + 1] shouldBe "bar" + } + + test("exposes git metadata read-only with the werkator dir masked for a worktree workspace") { + val gitDir = repoDir.resolve(".git") + val adminDir = gitDir.resolve("worktrees/workspace") + Files.createDirectories(adminDir) + Files.createDirectories(gitDir.resolve("werkator")) + Files.createDirectories(workspace) + Files.writeString(workspace.resolve(".git"), "gitdir: $adminDir\n") + Files.createDirectories(rootfsUnpacked()) + + runner.start("./gradlew test", workspace, mapOf("branch" to "main"), repoDir, bwrapBranchConfig()) + + val args = captured.single() + args[args.indexOf(gitDir.toString()) - 1] shouldBe "--ro-bind" + args[args.indexOf("$gitDir/werkator") - 1] shouldBe "--tmpfs" + args[args.indexOf(adminDir.toString()) - 1] shouldBe "--bind" + } + + test("mounts no git metadata when the workspace is not a worktree") { + Files.createDirectories(rootfsUnpacked()) + Files.createDirectories(workspace) + + runner.start("./gradlew test", workspace, mapOf("branch" to "main"), repoDir, bwrapBranchConfig()) + + val args = captured.single() + val gitDir = repoDir.resolve(".git") + // the sandbox's own /tmp tmpfs is always present; the point is that no tmpfs + // masks .git/werkator and no worktree admin dir is bound + args.none { it == "$gitDir/werkator" } shouldBe true + args.none { it.contains("worktrees/") } shouldBe true + } + + test("fails without a configured rootfs") { + val branchConfig = BranchConfig(bwrap = BwrapConfig(enabled = true)) + + val exception = + shouldThrow { + runner.start("./gradlew test", workspace, mapOf("branch" to "main"), repoDir, branchConfig) + } + + exception.message shouldContain "bwrap.rootfs" + } + + test("downloads a URL rootfs once before unpacking") { + val url = "https://example.test/buildenv.tar.zst" + val downloadTarget = + repoDir + .resolve(BwrapBuildRunner.BUILDENV_DIR) + .resolve(url.sha12()) + .resolve("buildenv.tar.zst") + every { commandRunner.runOrThrow(listOf("curl", "-fsSL", "-o", downloadTarget.toString(), url), repoDir, any(), any()) } returns + GitCommandResult(0, "", "") + every { commandRunner.runOrThrow(match { it.first() == "tar" }, any(), any(), any()) } returns + GitCommandResult(0, "", "") + + runner.start("./gradlew test", workspace, mapOf("branch" to "main"), repoDir, bwrapBranchConfig(rootfs = url)) + + verify { + commandRunner.runOrThrow(listOf("curl", "-fsSL", "-o", downloadTarget.toString(), url), repoDir, any(), any()) + } + verify { commandRunner.runOrThrow(match { it.first() == "tar" }, repoDir, any(), any()) } + } + } + + private fun String.sha12(): String = + java.security.MessageDigest + .getInstance("SHA-256") + .digest(toByteArray()) + .joinToString("") { "%02x".format(it) } + .take(12) +} diff --git a/src/test/kotlin/de/hoennig/werkator/build/DispatchingBuildRunnerTest.kt b/src/test/kotlin/de/hoennig/werkator/build/DispatchingBuildRunnerTest.kt index d39da45..906d451 100644 --- a/src/test/kotlin/de/hoennig/werkator/build/DispatchingBuildRunnerTest.kt +++ b/src/test/kotlin/de/hoennig/werkator/build/DispatchingBuildRunnerTest.kt @@ -1,6 +1,7 @@ package de.hoennig.werkator.build import de.hoennig.werkator.config.BranchConfig +import de.hoennig.werkator.config.BwrapConfig import de.hoennig.werkator.config.DockerConfig import io.kotest.core.spec.style.FunSpec import io.kotest.matchers.shouldBe @@ -14,12 +15,13 @@ import java.nio.file.Paths class DispatchingBuildRunnerTest : FunSpec() { private val processBuildRunner = mockk() private val dockerBuildRunner = mockk() - private val dispatcher = DispatchingBuildRunner(processBuildRunner, dockerBuildRunner) + private val bwrapBuildRunner = mockk() + private val dispatcher = DispatchingBuildRunner(processBuildRunner, dockerBuildRunner, bwrapBuildRunner) private val process = mockk() private val dir = Paths.get(".") init { - beforeEach { clearMocks(processBuildRunner, dockerBuildRunner) } + beforeEach { clearMocks(processBuildRunner, dockerBuildRunner, bwrapBuildRunner) } test("runs natively by default") { val branchConfig = BranchConfig() @@ -28,6 +30,7 @@ class DispatchingBuildRunnerTest : FunSpec() { dispatcher.start("cmd", dir, emptyMap(), dir, branchConfig) shouldBe process verify { dockerBuildRunner wasNot Called } + verify { bwrapBuildRunner wasNot Called } } test("runs in Docker when the branch enables it") { @@ -37,6 +40,17 @@ class DispatchingBuildRunnerTest : FunSpec() { dispatcher.start("cmd", dir, emptyMap(), dir, branchConfig) shouldBe process verify { processBuildRunner wasNot Called } + verify { bwrapBuildRunner wasNot Called } + } + + test("runs in bwrap when the branch enables it (and not Docker)") { + val branchConfig = BranchConfig(bwrap = BwrapConfig(enabled = true, rootfs = "/srv/buildenv.tar.zst")) + every { bwrapBuildRunner.start("cmd", dir, emptyMap(), dir, branchConfig) } returns process + + dispatcher.start("cmd", dir, emptyMap(), dir, branchConfig) shouldBe process + + verify { processBuildRunner wasNot Called } + verify { dockerBuildRunner wasNot Called } } } } diff --git a/src/test/kotlin/de/hoennig/werkator/commands/InitCommandTest.kt b/src/test/kotlin/de/hoennig/werkator/commands/InitCommandTest.kt index 57dd631..554d6d0 100644 --- a/src/test/kotlin/de/hoennig/werkator/commands/InitCommandTest.kt +++ b/src/test/kotlin/de/hoennig/werkator/commands/InitCommandTest.kt @@ -14,7 +14,12 @@ import java.nio.file.attribute.PosixFilePermissions class InitCommandTest : FunSpec() { private val gitService = mockk() - private val initCommand = InitCommand(gitService) + private val initCommand = + InitCommand( + gitService, + de.hoennig.werkator.config + .ConfigLoader(mockk(relaxed = true)), + ) init { test("creates config files with auto-detected values") { diff --git a/src/test/kotlin/de/hoennig/werkator/commands/SystemdServiceFilesTest.kt b/src/test/kotlin/de/hoennig/werkator/commands/SystemdServiceFilesTest.kt index 8b9c0e0..1417ac3 100644 --- a/src/test/kotlin/de/hoennig/werkator/commands/SystemdServiceFilesTest.kt +++ b/src/test/kotlin/de/hoennig/werkator/commands/SystemdServiceFilesTest.kt @@ -33,6 +33,26 @@ class SystemdServiceFilesTest : FunSpec() { content shouldContain "WantedBy=default.target" } + test("resource limits are written when configured and omitted when unset") { + fun unit( + memoryMax: String, + tasksMax: String, + ) = SystemdServiceFiles.unitFileContent( + repoRoot = Paths.get("/srv/repos/my-repo"), + javaExecutable = Paths.get("/usr/bin/java"), + jarPath = Paths.get("/srv/repos/my-repo/werkator.jar"), + envFile = Paths.get("/srv/repos/my-repo/werkator.env"), + memoryMax = memoryMax, + tasksMax = tasksMax, + ) + val with = unit(memoryMax = "1G", tasksMax = "512") + with shouldContain "MemoryMax=1G" + with shouldContain "TasksMax=512" + val without = unit(memoryMax = "", tasksMax = "") + without shouldNotContain "MemoryMax" + without shouldNotContain "TasksMax" + } + test("percent signs in paths are escaped for systemd") { val content = SystemdServiceFiles.unitFileContent( diff --git a/src/test/kotlin/de/hoennig/werkator/config/ConfigLoaderTest.kt b/src/test/kotlin/de/hoennig/werkator/config/ConfigLoaderTest.kt index cd34a4f..018f689 100644 --- a/src/test/kotlin/de/hoennig/werkator/config/ConfigLoaderTest.kt +++ b/src/test/kotlin/de/hoennig/werkator/config/ConfigLoaderTest.kt @@ -350,6 +350,39 @@ class ConfigLoaderTest : FunSpec() { settings.docker.image shouldBe "attacker-image" } + test("a branch cannot disable its bwrap sandbox or substitute a foreign rootfs through a build definition") { + val dir = Files.createTempDirectory("werkator-test") + dir.resolve(".werkator.yml").toFile().writeText( + """ + builds: + default: + bwrap: + enabled: true + rootfs: /host/rootfs.tar.zst + """.trimIndent(), + ) + val worktree = Files.createTempDirectory("werkator-test-worktree") + worktree.resolve(".werkator.yml").toFile().writeText( + """ + builds: + default: + bwrap: + enabled: false + rootfs: /attacker/rootfs.tar.zst + env: + FOO: from-branch + """.trimIndent(), + ) + + val settings = loader.loadForWorktree(dir, worktree).buildSettings("any-branch", "default") + + // pinned: the sandbox can neither be switched off nor pointed at a foreign rootfs + settings.bwrap.enabled shouldBe true + settings.bwrap.rootfs shouldBe "/host/rootfs.tar.zst" + // everything that describes the build itself stays the branch's own business + settings.bwrap.env shouldBe mapOf("FOO" to "from-branch") + } + test("a build the branch invents inherits the host's sandbox policy") { val dir = Files.createTempDirectory("werkator-test") dir.resolve(".werkator.yml").toFile().writeText( @@ -386,6 +419,31 @@ class ConfigLoaderTest : FunSpec() { settings.requirePullRequest shouldBe true } + test("enabling both docker and bwrap on a build is rejected, not picked silently") { + val dir = Files.createTempDirectory("werkator-test") + dir.resolve(".werkator.yml").toFile().writeText( + """ + builds: + default: + docker: + enabled: true + image: build-env + bwrap: + enabled: true + rootfs: /srv/rootfs.tar.zst + """.trimIndent(), + ) + + val config = loader.load(dir) + val exception = + shouldThrow { + config.buildSettings("any-branch", "default") + } + + exception.message shouldContain "both docker and bwrap" + exception.message shouldContain "builds.default" + } + test("an exclusion pattern takes a branch out of a build that would otherwise select it") { val dir = Files.createTempDirectory("werkator-test") dir.resolve(".werkator.yml").toFile().writeText( diff --git a/src/test/kotlin/de/hoennig/werkator/framework/TestcontainersSmokeTest.kt b/src/test/kotlin/de/hoennig/werkator/framework/TestcontainersSmokeTest.kt index fd8b3da..ece0d97 100644 --- a/src/test/kotlin/de/hoennig/werkator/framework/TestcontainersSmokeTest.kt +++ b/src/test/kotlin/de/hoennig/werkator/framework/TestcontainersSmokeTest.kt @@ -2,18 +2,35 @@ package de.hoennig.werkator.framework import io.kotest.core.spec.style.FunSpec import io.kotest.matchers.shouldBe +import org.testcontainers.DockerClientFactory import org.testcontainers.containers.GenericContainer import org.testcontainers.utility.DockerImageName +/** + * Probes that Testcontainers can actually start a container on this host. + * Skipped, not failed, when no Docker is present — that is what lets Werkator + * build itself on a Docker-less host (e.g. a Hostsharing webspace, where its own + * build runs in the bubblewrap sandbox); see `tools/werkator-build-prerequisites.sh`. + */ class TestcontainersSmokeTest : FunSpec({ - test("Testcontainers starts a container") { - val container = - GenericContainer(DockerImageName.parse("alpine:3")) - .withCommand("sh", "-c", "sleep 30") - container.start() - container.isRunning shouldBe true - container.stop() - } - }) + test("Testcontainers starts a container") + .config(enabledIf = { dockerAvailable() }) { + val container = + GenericContainer(DockerImageName.parse("alpine:3")) + .withCommand("sh", "-c", "sleep 30") + container.start() + container.isRunning shouldBe true + container.stop() + } + }) { + companion object { + private fun dockerAvailable(): Boolean = + try { + DockerClientFactory.instance().isDockerAvailable() + } catch (_: Throwable) { + false + } + } +} diff --git a/tools/build-bwrap-rootfs.sh b/tools/build-bwrap-rootfs.sh new file mode 100755 index 0000000..5b1a5fe --- /dev/null +++ b/tools/build-bwrap-rootfs.sh @@ -0,0 +1,116 @@ +#!/usr/bin/env bash +# +# Build the Werkator bwrap build environment (rootfs) archive. +# +# The bwrap build runtime (step 17 / ADR 0007) runs each build inside a +# bubblewrap user namespace, chrooted into a *prepared* Debian root filesystem. +# That rootfs is NOT built on the target (the webspace has no root and no +# debootstrap), it is built once on any machine that can — most comfortably a +# machine with Docker — and distributed as an archive, e.g. +# `werkator-buildenv-trixie-java21.tar.zst`. +# +# This script builds exactly that archive: a debootstrap-minbase Debian release +# plus the packages Werkator itself needs to run `./gradlew build` inside the +# sandbox (JDK 21, git, ca-certificates, locales, curl/unzip for the wrapper). +# The whole build runs inside a throwaway Docker container, so no root is +# needed on the machine running this script. +# +# Why it works this way (each quirk learned the hard way): +# - The whole job runs in ONE container whose stdin carries a base64-encoded +# script (no file is bind-mounted for the script — a bind-mounted script +# hit noexec/tmpfs trouble and vanished inside the container). +# - The rootfs is built inside the container's own writable layer, NOT on a +# bind-mounted host directory — debootstrap "Tried to extract package, but +# tar failed" when its target sat on some bind-mounted/special filesystems. +# - Only the final `tar --zstd` writes to stdout; every build step is +# redirected to stderr, so the archive coming out of `docker run` is pure. +# +# Usage: build-bwrap-rootfs.sh [--release trixie] [--mirror URL] [--out path] +# --release Debian release/architecture tail, default "trixie" +# --mirror apt mirror for debootstrap, default http://deb.debian.org/debian +# --out output archive path, default ./werkator-buildenv-.tar.zst +# + +set -euo pipefail + +die() { echo "ERROR: $*" >&2; exit 1; } +warn() { echo "WARNING: $*" >&2; } + +usage() { + echo "usage: build-bwrap-rootfs.sh [--release trixie] [--mirror URL] [--out path]" >&2 + exit 1 +} + +# --------------------------------------------------------------- arguments -- + +release="trixie" +out="" +while [ $# -gt 0 ]; do + case "$1" in + --release) release="${2:?missing value for --release}"; shift 2 ;; + --mirror) mirror="${2:?missing value for --mirror}"; shift 2 ;; + --out) out="${2:?missing value for --out}"; shift 2 ;; + -*) die "unknown option: $1" ;; + *) usage ;; + esac +done +mirror="${mirror:-http://deb.debian.org/debian}" +[ -n "$out" ] || out="$(pwd)/werkator-buildenv-${release}.tar.zst" + +command -v docker >/dev/null 2>&1 || die "docker is required to build the rootfs" + +# Rootfs content: Werkator's own build needs a JDK 21 toolchain (Gradle +# toolchain resolution), git, ca-certificates for HTTPS, locales for git, and +# curl/unzip/xz-utils/zstd for the Gradle wrapper and general build hygiene. +# Keep this list additive — project-specific tooling goes on top of this base. +PKGS="openjdk-21-jdk git ca-certificates locales procps file curl unzip xz-utils zstd" + +# The chroot step runs inside the freshly debootstrapped rootfs; passed into +# the container as base64 so no nested heredoc corrupts the piped script. +inner="$(printf '%s' '#!/bin/bash +set -euxo pipefail +mount -t proc none /proc +apt-get update -qq +apt-get install -y --no-install-recommends '"${PKGS}"' +apt-get clean +rm -f /etc/localtime +locale-gen en_US.UTF-8 de_DE.UTF-8 >/dev/null 2>&1 || true +update-locale LANG=en_US.UTF-8 >/dev/null 2>&1 || true +' | base64 -w0)" + +# The outer script runs inside the Debian container as root. Build noise goes +# to stderr (fd 1 is saved on fd 3 and restored only for the final tar), so +# docker stdout is exactly the archive. +outer="$(printf '%s' '#!/bin/bash +set -euo pipefail +export DEBIAN_FRONTEND=noninteractive +exec 3>&1 +exec 1>&2 +apt-get update -qq +apt-get install -y --no-install-recommends debootstrap zstd ca-certificates +mkdir -p /b/rootfs +debootstrap --variant=minbase --components=main,contrib --include=apt,ca-certificates '"${release}"' /b/rootfs '"${mirror}"' +mount --bind /proc /b/rootfs/proc +mount --bind /sys /b/rootfs/sys +mount --bind /dev /b/rootfs/dev +echo '"${inner}"' | base64 -d > /b/rootfs/inner.sh +chmod +x /b/rootfs/inner.sh +chroot /b/rootfs /bin/bash /inner.sh +umount /b/rootfs/proc; umount /b/rootfs/sys; umount /b/rootfs/dev +exec 1>&3 +tar --zstd --exclude=proc --exclude=sys --exclude=dev -C /b/rootfs -cf - . +' | base64 -w0)" + +echo "building ${release} rootfs (downloads packages, takes a while; log below)..." +echo "archive → $out" + +# Stream the base64-encoded outer script into the container over stdin; the +# archive lands on stdout (redirected to $out), the build log on stderr. +docker run --rm -i --privileged debian:"${release}-slim" \ + bash -c 'base64 -d | bash' \ + <<<"$outer" >"$out" + +echo +echo "OK: build environment written to $out" +echo " Configure it as branches..bwrap.rootfs (a bare path or a URL)" +echo " on the target Werkator instance to build in this environment." diff --git a/tools/remote b/tools/remote new file mode 100755 index 0000000..2fc45ea --- /dev/null +++ b/tools/remote @@ -0,0 +1,344 @@ +#!/usr/bin/env bash +# +# Central control script for remote Werkator operations (same pattern as the +# `remote` scripts in the other repos): the first argument is the repo selector, +# the second the command. All connection and deployment values come from the +# `.env` file in the repository root — never as command line parameters. +# +# Usage: +# tools/remote werkator check-prerequisites +# tools/remote werkator install +# tools/remote werkator build # WERKATOR_BRANCH to override, default main +# tools/remote werkator start +# tools/remote port-forward start # background tunnel to the Werkator UI +# tools/remote port-forward stop +# tools/remote werkator control-token +# +# Required in .env: +# WERKATOR_REMOTE user@host to operate on, e.g. mih34-werkator@mih34.hostsharing.net +# WERKATOR_PATH target directory on that host, e.g. /home/storage/mih34/users/werkator +# +# Required for `start`: +# WERKATOR_PORT the localhost port assigned by Hostsharing (eigener Serverdienst) +# WERKATOR_DOMAIN the domain served by the managed Apache, e.g. ci.example.de +# +# Required for `port-forward`: +# WERKATOR_LOCAL_PORT the local port the browser uses +# Optional in .env: +# WERKATOR_BRANCH branch for `build` (default: main) +# WERKATOR_MEMORY_MAX systemd MemoryMax for the unit, e.g. 1G (start) +# WERKATOR_TASKS_MAX systemd TasksMax for the unit, e.g. 512 (start) +# WERKATOR_ROOTFS rootfs archive path +# (default: /build/werkator-buildenv-trixie.tar.zst) +# +# Install layout on the host: +# $WERKATOR_PATH/werkator/ the repository clone +# $WERKATOR_PATH/.werkator/ runtime bundle + rootfs archive +# +# `install` performs, in order: +# 1. check-prerequisites (bwrap capability + disk/quota, aborts on FAIL) +# 2. ensure SSH access (ssh-copy-id on first use; asks for the password) +# 3. upload artifacts (runtime bundle, built locally if missing, + rootfs) +# 4. clone the repository (needs the host SSH key registered at GitHub once — +# the script prints the key and waits) +# 5. `werkator init` + machine-local bwrap configuration +# + +set -euo pipefail + +REPO="${1:-}" +COMMAND="${2:-}" + +REPO_ROOT="$(cd "$(dirname "${BASH_SOURCE[0]}")/.." && pwd)" +PREREQ_SCRIPT="$REPO_ROOT/tools/werkator-build-prerequisites.sh" +RUNTIME_BUNDLE="$REPO_ROOT/build/distributions/werkator-runtime-linux-x64.tar.gz" +PID_FILE="/tmp/werkator-port-forward-$(id -u).pid" +LOG_FILE="/tmp/werkator-port-forward-$(id -u).log" + +usage() { + sed -n '3,32p' "${BASH_SOURCE[0]}" | sed 's/^# \{0,1\}//' + exit 2 +} + +require_env() { + local missing=0 + for name in "$@"; do + if [ -z "${!name:-}" ]; then + echo "ERROR: $name is not set — define it in $REPO_ROOT/.env" >&2 + missing=1 + fi + done + [ "$missing" -eq 0 ] || exit 1 +} + +[ -n "$REPO" ] && [ -n "$COMMAND" ] || usage + +# Load the connection and deployment values; explicit environment wins, the +# .env in the repository root fills the rest. +set -a +[ -f "$REPO_ROOT/.env" ] && source "$REPO_ROOT/.env" +set +a + +require_env WERKATOR_REMOTE WERKATOR_PATH +HOST="$WERKATOR_REMOTE" +TARGET_DIR="$WERKATOR_PATH" +ROOTFS="${WERKATOR_ROOTFS:-$REPO_ROOT/build/werkator-buildenv-trixie.tar.zst}" + +ssh_present() { + ssh -o BatchMode=yes -o ConnectTimeout=10 "$HOST" true 2>/dev/null +} + +ensure_ssh() { + if ssh_present; then + echo "==> SSH access to $HOST: ok" + else + echo "==> No key-based SSH access yet; running ssh-copy-id (password prompt expected)" + ssh-copy-id "$HOST" + ssh_present || { echo "ERROR: SSH access still not working after ssh-copy-id" >&2; exit 1; } + fi +} + +# Run the prerequisites script remotely by piping it over stdin; TARGET_DIR and +# ROOTFS_ARCHIVE are passed as arguments to `bash -s --`. +check_prerequisites() { + echo "==> Checking prerequisites on $HOST (target dir: $TARGET_DIR)" + local rootfs_remote="$TARGET_DIR/.werkator/$(basename "$ROOTFS")" + if ! ssh "$HOST" "WERKATOR_SSH_TARGET='$HOST' bash -s -- '$TARGET_DIR' '$rootfs_remote'" < "$PREREQ_SCRIPT"; then + echo "ERROR: prerequisites failed on $HOST — install aborted" >&2 + exit 1 + fi +} + +ensure_local_artifacts() { + if [ ! -f "$RUNTIME_BUNDLE" ]; then + echo "==> Runtime bundle not found; building it locally (./gradlew runtimeBundle)" + (cd "$REPO_ROOT" && ./gradlew runtimeBundle --console=plain -q) + fi + [ -f "$RUNTIME_BUNDLE" ] || { echo "ERROR: runtime bundle missing: $RUNTIME_BUNDLE" >&2; exit 1; } + [ -f "$ROOTFS" ] || { + echo "ERROR: rootfs archive missing: $ROOTFS" >&2 + echo " build it with tools/build-bwrap-rootfs.sh or set WERKATOR_ROOTFS" >&2 + exit 1 + } +} + +ensure_github_access() { + # `ssh -T git@github.com` exits 1 even on success ("does not provide shell + # access") — neutralize remotely, then match on the greeting text. + if ssh "$HOST" 'ssh -o BatchMode=yes -o ConnectTimeout=10 -T git@github.com 2>&1 || true' | grep -q "successfully authenticated"; then + echo "==> GitHub SSH access from $HOST: ok" + return 0 + fi + echo + echo "==> The host cannot reach GitHub via SSH yet." + echo " Add THIS public key to GitHub (Settings > SSH and GPG keys > New SSH key):" + ssh "$HOST" 'cat ~/.ssh/id_*.pub 2>/dev/null' || { + echo "ERROR: no public key on the host; create one with ssh-keygen -t ed25519" >&2 + exit 1 + } + read -r -p " Press Enter once the key is registered at GitHub... " + ssh "$HOST" 'ssh -o BatchMode=yes -T git@github.com 2>&1 || true' | grep -q "successfully authenticated" || { + echo "ERROR: GitHub authentication from $HOST still failing" >&2 + exit 1 + } + echo "==> GitHub SSH access from $HOST: ok" +} + +install() { + ensure_ssh + check_prerequisites + ensure_local_artifacts + + echo "==> Uploading runtime bundle and rootfs archive" + ssh "$HOST" "mkdir -p '$TARGET_DIR/.werkator'" + scp -q "$RUNTIME_BUNDLE" "$HOST:$TARGET_DIR/.werkator/" + scp -q "$ROOTFS" "$HOST:$TARGET_DIR/.werkator/" + + echo "==> Unpacking runtime bundle" + ssh "$HOST" "tar xzf '$TARGET_DIR/.werkator/$(basename "$RUNTIME_BUNDLE")' -C '$TARGET_DIR/.werkator'" + ssh "$HOST" "'$TARGET_DIR/.werkator/werkator/bin/werkator' --version" + + ensure_github_access + + echo "==> Cloning the repository" + if ssh "$HOST" "test -d '$TARGET_DIR/werkator/.git'"; then + echo " (already cloned, skipping)" + else + ssh "$HOST" "git clone git@github.com:mhoennig/werkator.git '$TARGET_DIR/werkator'" + fi + + echo "==> Running werkator init" + ssh "$HOST" "cd '$TARGET_DIR/werkator' && '$TARGET_DIR/.werkator/werkator/bin/werkator' init" + + echo "==> Writing machine-local bwrap configuration" + ssh "$HOST" "grep -q '^ bwrap:' '$TARGET_DIR/werkator/.git/werkator/.werkator.yml' 2>/dev/null" || ssh "$HOST" "cat >> '$TARGET_DIR/werkator/.git/werkator/.werkator.yml' <<'CFG' + +# Build in the bubblewrap sandbox instead of natively (Step 17 / ADR 0007). +# Both keys are pinned: read from this machine config even if a branch sets +# its own values in a committed .werkator.yml. +builds: + default: + bwrap: + enabled: true + rootfs: $TARGET_DIR/.werkator/$(basename "$ROOTFS") +CFG" + + echo "==> Verifying the effective configuration" + ssh "$HOST" "cd '$TARGET_DIR/werkator' && '$TARGET_DIR/.werkator/werkator/bin/werkator' config:print 2>/dev/null | grep -A3 'bwrap:' | head -4" + + echo + echo "==> Install complete." + echo " Repo: $TARGET_DIR/werkator" + echo " Runtime: $TARGET_DIR/.werkator/werkator/bin/werkator" + echo " Next: tools/remote werkator build" +} + +build() { + ensure_ssh + local branch="${WERKATOR_BRANCH:-main}" + echo "==> Running one initial build of branch '$branch' on $HOST (in the bwrap sandbox)" + ssh -t "$HOST" "cd '$TARGET_DIR/werkator' && '$TARGET_DIR/.werkator/werkator/bin/werkator' build '$branch'" +} + +# Start the server as a systemd user unit behind the managed Apache. +# WERKATOR_MEMORY_MAX / WERKATOR_TASKS_MAX (optional) are written into the +# machine config so `init --systemd` bakes them into the unit. +start() { + ensure_ssh + require_env WERKATOR_PORT WERKATOR_DOMAIN + local machine="$TARGET_DIR/werkator/.git/werkator/.werkator.yml" + local unit="werkator-$(basename "$TARGET_DIR/werkator").service" + local htaccess="$TARGET_DIR/doms/$WERKATOR_DOMAIN/subs/www/.htaccess" + + echo "==> Writing server settings to the machine config" + if ssh "$HOST" "grep -q '^server:' '$machine' 2>/dev/null"; then + # re-run: update port and publicBaseUrl in place (systemd limits stay as written) + ssh "$HOST" "sed -i 's/^ port: .*/ port: $WERKATOR_PORT/; s|^ publicBaseUrl: .*| publicBaseUrl: \"https://$WERKATOR_DOMAIN/\"|' '$machine'" + else + ssh "$HOST" "cat >> '$machine' <<'CFG' + +# Web access: the managed Apache terminates TLS and proxies to the localhost +# port assigned by Hostsharing (eigener Serverdienst); TLS is the domain's +# Let's Encrypt certificate, so Werkator itself stays on 127.0.0.1. +server: + port: $WERKATOR_PORT + bindAddress: 127.0.0.1 + publicBaseUrl: \"https://$WERKATOR_DOMAIN/\" + nginx: + enabled: false + systemd: + memoryMax: \"${WERKATOR_MEMORY_MAX:-}\" + tasksMax: \"${WERKATOR_TASKS_MAX:-}\" +CFG" + fi + + echo "==> Writing the Apache reverse proxy to $htaccess" + ssh "$HOST" "mkdir -p '$TARGET_DIR/doms/$WERKATOR_DOMAIN/subs/www' && cat > '$htaccess' <<'HT' +DirectoryIndex disabled +RewriteEngine On +RewriteBase / +RewriteRule .* http://127.0.0.1:$WERKATOR_PORT%{REQUEST_URI} [proxy] +HT" + + echo "==> Generating the systemd user unit (init --systemd)" + ssh "$HOST" "cd '$TARGET_DIR/werkator' && '$TARGET_DIR/.werkator/werkator/bin/werkator' init --systemd" + + echo "==> Linking the units into ~/.config/systemd/user and enabling the service" + ssh "$HOST" "mkdir -p ~/.config/systemd/user && \ + ln -sf '$TARGET_DIR/werkator/.git/werkator/$unit' ~/.config/systemd/user/ && \ + ln -sf '$TARGET_DIR/werkator/.git/werkator/werkator-docker-prune.service' ~/.config/systemd/user/ && \ + ln -sf '$TARGET_DIR/werkator/.git/werkator/werkator-docker-prune.timer' ~/.config/systemd/user/ && \ + systemctl --user daemon-reload && systemctl --user restart '$unit' && systemctl --user status '$unit' --no-pager -l | head -12" + + echo + echo "==> Server started. Verify: https://$WERKATOR_DOMAIN/" + echo " Logs: ssh $HOST -- systemctl --user status '$unit'" +} + +# Background SSH tunnel to the Werkator server, so the browser reaches the UI +# at http://localhost: without keeping a terminal busy. +# `start` runs ssh -N -L detached with a pid file; `stop` kills it. +port_forward() { + require_env WERKATOR_LOCAL_PORT + local remote_port + remote_port="$(ssh "$HOST" "awk '/^server:/{f=1;next} f && /^ port:/{print \$2; exit}' '$TARGET_DIR/werkator/.git/werkator/.werkator.yml'")" + [ -n "$remote_port" ] || { echo "ERROR: no server.port in the machine config — run 'tools/remote werkator start' first" >&2; exit 1; } + + case "$COMMAND" in + start) + if [ -f "$PID_FILE" ] && kill -0 "$(cat "$PID_FILE")" 2>/dev/null; then + echo "==> Port-forward already running (pid $(cat "$PID_FILE")) — http://localhost:$WERKATOR_LOCAL_PORT" + exit 0 + fi + nohup ssh -N -L "$WERKATOR_LOCAL_PORT:127.0.0.1:$remote_port" "$HOST" \ + >"$LOG_FILE" 2>&1 & + echo $! > "$PID_FILE" + sleep 1 + if kill -0 "$(cat "$PID_FILE")" 2>/dev/null; then + echo "==> Forwarding http://localhost:$WERKATOR_LOCAL_PORT -> $HOST:127.0.0.1:$remote_port (pid $(cat "$PID_FILE"))" + else + echo "ERROR: port-forward failed to start — see $LOG_FILE" >&2 + rm -f "$PID_FILE" + exit 1 + fi + ;; + stop) + if [ -f "$PID_FILE" ] && kill -0 "$(cat "$PID_FILE")" 2>/dev/null; then + kill "$(cat "$PID_FILE")" + rm -f "$PID_FILE" + echo "==> Port-forward stopped" + else + rm -f "$PID_FILE" + echo "==> Port-forward is not running" + fi + ;; + *) + echo "ERROR: unknown port-forward command: $COMMAND (use start or stop)" >&2 + exit 2 + ;; + esac +} + +# Print the control token guarding the mutating build endpoints. If the server +# has not created it yet (it does so on first use), generate one in place — the +# server reads the file lazily, so a pre-created token is equivalent. +control_token() { + ensure_ssh + local token_file="$TARGET_DIR/werkator/.git/werkator/control-token" + ssh "$HOST" "if [ -f '$token_file' ]; then cat '$token_file'; else \ + umask 077 && head -c 32 /dev/urandom | od -An -tx1 | tr -d ' \\n' > '$token_file' && cat '$token_file'; fi" +} + +case "$REPO" in + port-forward) + port_forward + ;; + werkator) + case "$COMMAND" in + check-prerequisites) + ensure_ssh + check_prerequisites + ;; + install) + install + ;; + build) + build + ;; + start) + start + ;; + control-token) + control_token + ;; + *) + echo "ERROR: unknown command: $COMMAND" >&2 + usage + ;; + esac + ;; + *) + echo "ERROR: unknown repo selector: $REPO" >&2 + usage + ;; +esac diff --git a/tools/werkator-build-prerequisites.sh b/tools/werkator-build-prerequisites.sh new file mode 100755 index 0000000..8534448 --- /dev/null +++ b/tools/werkator-build-prerequisites.sh @@ -0,0 +1,164 @@ +#!/usr/bin/env bash +# +# Verify the bwrap (bubblewrap) build precondition on a target host before +# running Werkator's bwrap build runtime there (step 17 / ADR 0007). +# +# The whole "build Werkator inside bubblewrap on a Managed Webspace" approach +# hinges on one hard precondition: unprivileged user namespaces with a uid-0 +# mapping and read-only root binds must work. This script runs the exact +# command line recorded in docs/plan/17-bwrap-build-runtime.md, checks the +# expected signals, and additionally verifies the disk/quota situation: +# a bwrap build unpacks the rootfs (a zstd archive expands to several GiB) +# plus a Gradle distribution and per-branch caches, so the host needs both +# raw free space and enough group-quota headroom. +# +# Run this ON the target host (the webspace), no root needed. +# +# Optional: the rootfs archive to size the disk/quota check against, e.g. +# werkator-build-prerequisites.sh /path/to/werkator-buildenv-trixie.tar.zst +# When omitted, the check runs against a conservative default footprint. +# +# Usage: werkator-build-prerequisites.sh [TARGET_DIR] [ROOTFS_ARCHIVE] +# +# TARGET_DIR is the directory the build workspace will live in (default: $HOME). +# The check verifies it sits on the home filesystem and has enough free space. +# ROOTFS_ARCHIVE, when given, is the rootfs archive that will be used there. +# +# Output is one PASS/FAIL line per check plus a final RESULT line, e.g.: +# PASS: bwrap version: bubblewrap 0.8.0 +# PASS: build runs as root inside the namespace (uid 0) +# PASS: uid_map maps root back to the unprivileged user (uid 120957) +# PASS: read-only root bind is enforced +# PASS: at least 5 GiB free space on the build working filesystem +# FAIL: group quota headroom below the 5 GiB build footprint ... +# RESULT: FAIL (4/5) — Werkator bubblewrap builds are not usable on this host. +# + +set -euo pipefail + +target_dir_arg="${1:-}" +rootfs_arg="${2:-}" + +die() { echo "ERROR: $*" >&2; exit 1; } + +# Disk footprint a bwrap build needs headroom for, in 1K blocks: unpacked +# rootfs (zstd expands roughly 3-4x), Gradle distribution + per-branch cache, +# build output and artifacts. ~5 GiB. +MIN_FREE_BLOCKS=$((5 * 1024 * 1024)) + +# Reference filesystem: the one the invoking user's home directory lives on. +# Builds (repo clone, buildenv, caches) must run there — other mounts, such as +# a slow mass-storage volume, are rejected. +HOME_FS="$(df -Pk "$HOME" 2>/dev/null | awk 'NR==2 {print $1}')" + +pass=0 +fail=0 +result() { # result PASS|FAIL "message" + echo "$1: $2" + if [ "$1" = "PASS" ]; then pass=$((pass+1)); else fail=$((fail+1)); fi +} + +command -v bwrap >/dev/null 2>&1 || die "bwrap is not installed on this host" + +output="$(bwrap --unshare-user --unshare-pid --die-with-parent --uid 0 --gid 0 \ + --ro-bind / / --dev /dev --proc /proc --tmpfs /tmp \ + sh -c 'id -u && cat /proc/self/uid_map && (touch /usr/ro-test 2>&1 || true)' 2>&1)" || + die "bwrap invocation failed (no user namespace support?): $output" + +# Signal 0: bwrap itself is usable (version as a visible marker). +result PASS "bwrap version: $(bwrap --version 2>&1)" + +# Signal 1: runs as root (uid 0) inside the namespace. +first="$(printf '%s\n' "$output" | sed -n '1p')" +if [ "$first" = "0" ]; then + result PASS "build runs as root inside the namespace (uid 0)" +else + result FAIL "expected uid 0 inside the namespace, got: $first" +fi + +# Signal 2: uid_map maps root to the invoking unprivileged user. +uid_line="$(printf '%s\n' "$output" | sed -n '2p')" +self_uid="$(id -u)" +if printf '%s\n' "$uid_line" | grep -E "^[[:space:]]*0[[:space:]]+${self_uid}[[:space:]]+1" >/dev/null; then + result PASS "uid_map maps root back to the unprivileged user (uid $self_uid)" +else + result FAIL "expected uid_map '0 $self_uid 1', got: $uid_line" +fi + +# Signal 3: the read-only root bind is enforced (a write to /usr fails). +if printf '%s\n' "$output" | grep -qi "read-only file system"; then + result PASS "read-only root bind is enforced" +else + result FAIL "the read-only root bind did not reject a write to /usr" +fi + +# --- Disk / quota checks ------------------------------------------------ + +target_dir="${target_dir_arg:-$HOME}" +target_dir="$(realpath -m "$target_dir")" +min_gib=$((MIN_FREE_BLOCKS / 1024 / 1024)) + +if [ -n "$rootfs_arg" ] && [ ! -f "$rootfs_arg" ]; then + echo "WARNING: rootfs archive not found: $rootfs_arg (continuing without it)" +fi + +target_fs="$(df -Pk "$target_dir" 2>/dev/null | awk 'NR==2 {print $1}')" +df_output="$(df -Pk "$target_dir" 2>/dev/null | awk 'NR==2 {print int($4) " " $6}')" +if [ -n "$df_output" ]; then + avail_k="${df_output%% *}" + mount="${df_output##* }" + if [ -n "$HOME_FS" ] && [ "$target_fs" != "$HOME_FS" ]; then + # An explicitly chosen foreign filesystem is allowed (e.g. for testing) + # but flagged: builds there will be slow. + echo "WARNING: target dir is on $target_fs (mounted at $mount), not the home filesystem ($HOME_FS) — builds will run on slower storage" + fi + if [ "${avail_k:-0}" -lt "$MIN_FREE_BLOCKS" ]; then + result FAIL "less than ${min_gib} GiB free space on the build working filesystem ($mount)" + else + result PASS "at least ${min_gib} GiB free space on the build working filesystem ($mount, device $target_fs)" + fi +else + echo "WARNING: could not measure free space on $target_dir — only the quota check below applies" +fi + +if quota_output="$(quota -g 2>/dev/null)" && [ -n "$quota_output" ]; then + quota_ok=1 + quota_seen=0 + detail="" + while read -r fs blocks quota_limit; do + quota_seen=1 + # Only the quota of the target filesystem counts — other volumes may + # legitimately be full or unquota'd without affecting the build. + if [ -n "$target_fs" ] && [ "$(basename "$fs")" != "$(basename "$target_fs")" ] && [ "$fs" != "$target_fs" ]; then + continue + fi + headroom=$((quota_limit - blocks)) + if [ "$headroom" -lt "$MIN_FREE_BLOCKS" ]; then + quota_ok=0 + detail+=" $(basename "$fs"): $(awk -v b="$headroom" 'BEGIN{printf "%.1f", b/1024/1024}') GiB free of quota;" + fi + done < <(printf '%s\n' "$quota_output" | awk ' + NF==1 && $1 ~ /^\// { pending_fs=$1; next } + $1 ~ /^\// && $2 ~ /^[0-9]+$/ { print $1, $2, $4; pending_fs=""; next } + $1 ~ /^[0-9]+[*]?/ && pending_fs != "" { gsub(/\*/, "", $1); print pending_fs, $1, $3; pending_fs="" }') + if [ "$quota_seen" -eq 0 ]; then + echo "WARNING: quota tooling present but no group quota lines could be parsed — only free space was checked" + elif [ "$quota_ok" -eq 1 ]; then + result PASS "group quota headroom covers the ${min_gib} GiB build footprint" + else + result FAIL "group quota headroom below the ${min_gib} GiB build footprint (rootfs + Gradle cache); raise the quota before building.$detail" + fi +else + echo "WARNING: no readable group quota tooling on this host — only free space was checked" +fi + +total=$((pass + fail)) +echo +if [ "$fail" -eq 0 ]; then + echo "RESULT: PASS ($pass/$total) — Werkator bubblewrap builds are usable on this host." + echo "Next: install the Werkator instance with: tools/remote werkator install ${WERKATOR_SSH_TARGET:-@} '$target_dir'" + exit 0 +else + echo "RESULT: FAIL ($pass/$total) — Werkator bubblewrap builds are not usable on this host." + exit 1 +fi