diff --git a/docs/configuration.md b/docs/configuration.md index 7a4add2..5d52e48 100644 --- a/docs/configuration.md +++ b/docs/configuration.md @@ -475,7 +475,9 @@ The section was called `bwrap` and its binary key `bwrap.werkdock` until v1.2.0; `werkdock.rootfs` names the prepared root filesystem archive — a Debian-base rootfs with the build tools (JDK, git, locales, project-specific tooling) built elsewhere, since `debootstrap` is unavailable on the target. It is a local path or an `http(s)` URL; a URL is downloaded once into `.git/werkator/buildenv/`. Build the archive with `tools/build-bwrap-rootfs.sh` on any machine with Docker. -The archive is loaded once per source as the werkdock image `werkator-buildenv-` into werkdock's store (`$WERKDOCK_HOME`, default `~/.werkdock`) — shared by every repository of this OS user; the hash derives from the source string, so a changed `rootfs` loads a fresh image and stale ones can be removed from the store. +The archive is imported once per source as the werkdock image `werkator-buildenv-` into werkdock's store (`$WERKDOCK_HOME`, default `~/.werkdock`) — shared by every repository of this OS user; the hash derives from the source string, so a changed `rootfs` imports a fresh image and stale ones can be removed from the store. +Werkator uses `werkdock import ARCHIVE IMAGE` (docker import semantics) and falls back to the older `werkdock load -i ARCHIVE --name IMAGE` when the installed werkdock does not know the verb yet, so Werkator and werkdock can be updated in either order. +The image keeps its untagged name (it equals `werkator-buildenv-:latest` in werkdock's docker-style naming), so an existing store needs no re-import. Per-repo Gradle caches persist in `.git/werkator/buildenv/home`, bound as `/root`. `werkdock.env` adds environment variables inside the sandbox; the environment is otherwise cleared (docker semantics) — the server's environment does not leak in. Files created inside the sandbox are owned by the host user, because uid 0 maps back to the unprivileged webspace user. diff --git a/docs/deployment.md b/docs/deployment.md index 48722d3..e846961 100644 --- a/docs/deployment.md +++ b/docs/deployment.md @@ -342,7 +342,7 @@ tools/remote --env-file .env.mih34 port-forward start # browser tunne Layout on the host: the watched repository at `$WERKATOR_PATH/werkator/`, the unpacked runtime at `$WERKATOR_PATH/.werkator/werkator/`, the werkdock binary at `$WERKATOR_PATH/.werkator/bin/werkdock`. That is the default, not a requirement: `WERKATOR_REPO_DIR`, `WERKATOR_INSTALL_DIR` and `WERKATOR_SANDBOX` bend it to an installation that predates the script, see [Updating an Existing Installation](#updating-an-existing-installation). -The rootfs archive is loaded once per source into werkdock's image store (`~/.werkdock`), shared by every repository of the user. +The rootfs archive is imported once per source into werkdock's image store (`~/.werkdock`), shared by every repository of the user. Fill `git.account`/`git.token` in the machine config when the origin is private, and make the user's services survive logout with `loginctl enable-linger`. Updates are one command, refused while a build runs (`FORCE=1` overrides): diff --git a/docs/prs/2026-09-05-PR#000-werkdock-import-verb.md b/docs/prs/2026-09-05-PR#000-werkdock-import-verb.md new file mode 100644 index 0000000..2fb0f3b --- /dev/null +++ b/docs/prs/2026-09-05-PR#000-werkdock-import-verb.md @@ -0,0 +1,102 @@ +> **WARNING:** This document describes only the change applied in this PR. +> It may already be outdated once the next PR is merged. +> Historic PR-documentation is not maintained along with new PRs — treat it as a snapshot, not as current documentation. + +## Related Links + +- Werkdock `CHANGELOG.md` on its `docker-compat` branch — the release notes this PR adapts to: `import` replaces `load -i --name` for rootfs archives, `load` is reserved for docker/OCI image archives, image names gained docker-style tags. +- ADR 0008 — the bwrap build runtime; `docs/configuration.md`, notes on `builds..werkdock`. + +## The Problem + +Werkdock moves its CLI closer to `docker run --rm`. +For the rootfs archives Werkator uses, the verb is now `werkdock import ARCHIVE IMAGE` (docker import semantics). +`werkdock load -i ARCHIVE --name IMAGE` still works but is a compatibility path with a note on stderr, and is announced to go away once Werkator has switched. +Werkator ships the werkdock binary with its deployment, but nothing forces the two to be updated together: an installation may run a new Werkator against an older werkdock for a while, or the other way round. + +## Non-Goals + +- No switch of the existence check from `werkdock images` to `werkdock inspect`: `images` prints the bare name for untagged images on every werkdock version, the exact-line match keeps working, and `inspect` would tie Werkator to the new werkdock. +- No tagged image name (`werkator-buildenv:`): it would re-import every build environment on the webspace and orphan the old image, for no functional gain today. +- None of the new `run` flags (`--mount`, `--entrypoint`, `--network host`): Werkator's invocation needs none of them. + +## The Scenarios + +### Feature: rootfs archives are imported with werkdock's docker-shaped verb + +#### Background + +- The build environment is a rootfs archive named by `builds..werkdock.rootfs`. +- Werkator creates the werkdock image `werkator-buildenv-` from it once per source. +- Werkdock answers an unknown verb with `werkdock: unknown command "import"` and exit code 125, its code for its own errors. + +#### Scenario#000.01: A missing image is imported with `werkdock import` + +So that Werkator uses the verb werkdock names for rootfs archives, and the deprecated path can be removed on werkdock's side. + +- **Given** `werkdock images` does not list the image +- **When** a build starts +- **Then** Werkator runs `werkdock import ARCHIVE werkator-buildenv-` + - **and** runs no `werkdock load`. + +##### Verified by + +- [WerkdockBuildRunnerTest "imports the image once when werkdock does not know it yet"](../../src/test/kotlin/de/hoennig/werkator/build/WerkdockBuildRunnerTest.kt) +- [WerkdockBuildRunnerTest "downloads a URL rootfs once before importing it"](../../src/test/kotlin/de/hoennig/werkator/build/WerkdockBuildRunnerTest.kt) + +#### Scenario#000.02: An older werkdock without the verb still works + +So that Werkator and werkdock can be updated in either order. + +- **Given** the installed werkdock answers `import` with `unknown command` and exit 125 +- **When** a build starts with a missing image +- **Then** Werkator falls back to `werkdock load -i ARCHIVE --name werkator-buildenv-` + - **and** logs the fallback. + +##### Verified by + +- [WerkdockBuildRunnerTest "falls back to load -i --name on a werkdock without the import verb"](../../src/test/kotlin/de/hoennig/werkator/build/WerkdockBuildRunnerTest.kt) + +#### Scenario#000.03: A real import failure is not masked by the fallback + +So that a broken archive fails the build with werkdock's message, as before. + +- **Given** `werkdock import` fails for any other reason (any other exit code, or exit 125 without `unknown command`) +- **When** a build starts with a missing image +- **Then** the build fails with that command's output + - **and** no `werkdock load` runs. + +##### Verified by + +- [WerkdockBuildRunnerTest "propagates an import failure that is not a missing verb"](../../src/test/kotlin/de/hoennig/werkator/build/WerkdockBuildRunnerTest.kt) + +#### Scenario#000.04: An existing image is neither imported nor loaded + +- **Given** `werkdock images` lists the image +- **When** a build starts +- **Then** neither `import` nor `load` runs. + +##### Verified by + +- [WerkdockBuildRunnerTest "does not import an image werkdock already has"](../../src/test/kotlin/de/hoennig/werkator/build/WerkdockBuildRunnerTest.kt) + +## The Solution + +`WerkdockBuildRunner.ensureImage` calls the new `importImage`: `werkdock import ARCHIVE IMAGE` through the non-throwing `run`, then either returns, falls back to `load -i --name` on exactly the unknown-verb signature (exit 125 plus `unknown command` on stderr), or rethrows the import's result as a `GitCommandException` — the same exception and message the old code produced. +The fallback is keyed on werkdock's own error signature rather than on a version string, because werkdock's version output does not yet distinguish the two builds. + +## Open Questions + +- When to drop the fallback: once every installation runs a werkdock with `import`, the `load -i --name` branch and its test go, and werkdock can remove the compatibility path. + +## Additional Changes + +- `docs/configuration.md` and `docs/deployment.md` say "imported" where they said "loaded", and name the fallback and the reason the image name stays untagged. + +## Prerequisite PRs + +- Werkdock's `docker-compat` branch (repository `mi/werkdock`), which introduces `import`; without it the fallback path runs. + +## Follow-up PRs + +- Remove the fallback (see Open Questions). diff --git a/src/main/kotlin/de/hoennig/werkator/build/WerkdockBuildRunner.kt b/src/main/kotlin/de/hoennig/werkator/build/WerkdockBuildRunner.kt index 060401a..6461158 100644 --- a/src/main/kotlin/de/hoennig/werkator/build/WerkdockBuildRunner.kt +++ b/src/main/kotlin/de/hoennig/werkator/build/WerkdockBuildRunner.kt @@ -2,6 +2,7 @@ package de.hoennig.werkator.build import de.hoennig.werkator.config.BranchConfig import de.hoennig.werkator.config.WerkdockConfig +import de.hoennig.werkator.git.GitCommandException import de.hoennig.werkator.git.GitCommandRunner import org.slf4j.LoggerFactory import org.springframework.stereotype.Component @@ -16,9 +17,9 @@ import java.security.MessageDigest * it shells out to the `werkdock` CLI (`werkdock.binary`, default via PATH) — the same * pattern as git and docker, CLI, no library. * - * The rootfs archive becomes a werkdock *image*, loaded once per source + * The rootfs archive becomes a werkdock *image*, imported once per source * (`werkator-buildenv-`, the hash over the source string, so a changed source - * loads a fresh image) into werkdock's own store (`$WERKDOCK_HOME`, default + * imports a fresh image) into werkdock's own store (`$WERKDOCK_HOME`, default * `~/.werkdock`) — shared by every repository of this OS user, unlike the old * per-repo unpack. Only the download cache for URL sources and the persistent * toolchain home (bound to `/root` for Gradle/Go caches) stay under @@ -66,9 +67,11 @@ class WerkdockBuildRunner( } /** - * Loads the rootfs archive into the werkdock image store once per source. + * Imports the rootfs archive into the werkdock image store once per source. * `werkdock images` answers existence through the CLI, like `docker image - * inspect` does for the Docker runner. + * inspect` does for the Docker runner; it prints the bare name for an untagged + * image on every werkdock version, so the exact-line match holds across the + * werkdock upgrade that introduced tags. */ private fun ensureImage( werkdock: String, @@ -84,7 +87,33 @@ class WerkdockBuildRunner( val envDir = repoDir.resolve(BUILDENV_DIR).resolve(sourceKey(sandbox.rootfs)) Files.createDirectories(envDir) val archive = localArchive(sandbox.rootfs, envDir, repoDir, onAuxProcess) - log.info("loading build environment {} as werkdock image {}", sandbox.rootfs, image) + log.info("importing build environment {} as werkdock image {}", sandbox.rootfs, image) + importImage(werkdock, archive, image, repoDir, onAuxProcess) + } + + /** + * `werkdock import ARCHIVE IMAGE` (docker import semantics) creates the image from a + * rootfs archive. A werkdock that predates the verb answers `unknown command` with + * exit 125; then the older `load -i ARCHIVE --name IMAGE` does the same job, so an + * installation can update Werkator and werkdock in either order. Any other failure + * is the import's own and propagates as it did before. + */ + private fun importImage( + werkdock: String, + archive: String, + image: String, + repoDir: Path, + onAuxProcess: (Process) -> Unit, + ) { + val importCommand = listOf(werkdock, "import", archive, image) + val imported = commandRunner.run(importCommand, repoDir, onProcess = onAuxProcess) + if (imported.isSuccess) { + return + } + if (imported.exitCode != WERKDOCK_CLI_ERROR || "unknown command" !in imported.stderr) { + throw GitCommandException(importCommand, imported) + } + log.info("this werkdock has no import verb yet, falling back to load -i --name") commandRunner.runOrThrow( listOf(werkdock, "load", "-i", archive, "--name", image), repoDir, @@ -209,5 +238,8 @@ class WerkdockBuildRunner( companion object { const val BUILDENV_DIR = ".git/werkator/buildenv" const val HOME_DIR = "home" + + /** werkdock's exit code for its own errors (docker's 125), as opposed to the sandboxed command's. */ + const val WERKDOCK_CLI_ERROR = 125 } } diff --git a/src/test/kotlin/de/hoennig/werkator/build/WerkdockBuildRunnerTest.kt b/src/test/kotlin/de/hoennig/werkator/build/WerkdockBuildRunnerTest.kt index 21e9031..8894fa7 100644 --- a/src/test/kotlin/de/hoennig/werkator/build/WerkdockBuildRunnerTest.kt +++ b/src/test/kotlin/de/hoennig/werkator/build/WerkdockBuildRunnerTest.kt @@ -2,6 +2,7 @@ package de.hoennig.werkator.build import de.hoennig.werkator.config.BranchConfig import de.hoennig.werkator.config.WerkdockConfig +import de.hoennig.werkator.git.GitCommandException import de.hoennig.werkator.git.GitCommandResult import de.hoennig.werkator.git.GitCommandRunner import io.kotest.assertions.throwables.shouldThrow @@ -37,7 +38,11 @@ class WerkdockBuildRunnerTest : FunSpec() { private fun imageName(rootfs: String = "/srv/buildenv.tar.zst"): String = "werkator-buildenv-${rootfs.sha12()}" - /** The image is already loaded: `werkdock images` lists it, so no load runs. */ + private fun importCommand(): List = listOf("werkdock", "import", "/srv/buildenv.tar.zst", imageName()) + + private fun loadCommand(): List = listOf("werkdock", "load", "-i", "/srv/buildenv.tar.zst", "--name", imageName()) + + /** The image is already there: `werkdock images` lists it, so no import runs. */ private fun givenImageLoaded(rootfs: String = "/srv/buildenv.tar.zst") { every { commandRunner.runOrThrow(listOf("werkdock", "images"), repoDir, any(), any()) } returns GitCommandResult(0, imageName(rootfs) + "\n", "") @@ -88,34 +93,47 @@ class WerkdockBuildRunnerTest : FunSpec() { ) } - test("loads the image once when werkdock does not know it yet") { + test("imports the image once when werkdock does not know it yet") { givenImageMissing() - every { - commandRunner.runOrThrow( - listOf("werkdock", "load", "-i", "/srv/buildenv.tar.zst", "--name", imageName()), - repoDir, - any(), - any(), - ) - } returns GitCommandResult(0, "", "") + every { commandRunner.run(importCommand(), repoDir, any(), any()) } returns GitCommandResult(0, "", "") runner.start("./gradlew test", workspace, mapOf("branch" to "main"), repoDir, werkdockBranchConfig()) - verify { - commandRunner.runOrThrow( - listOf("werkdock", "load", "-i", "/srv/buildenv.tar.zst", "--name", imageName()), - repoDir, - any(), - any(), - ) - } + verify { commandRunner.run(importCommand(), repoDir, any(), any()) } + verify(exactly = 0) { commandRunner.runOrThrow(match { "load" in it }, any(), any(), any()) } } - test("does not load an image werkdock already has") { + test("falls back to load -i --name on a werkdock without the import verb") { + givenImageMissing() + every { commandRunner.run(importCommand(), repoDir, any(), any()) } returns + GitCommandResult(125, "", "werkdock: unknown command \"import\"\n") + every { commandRunner.runOrThrow(loadCommand(), repoDir, any(), any()) } returns GitCommandResult(0, "", "") + + runner.start("./gradlew test", workspace, mapOf("branch" to "main"), repoDir, werkdockBranchConfig()) + + verify { commandRunner.runOrThrow(loadCommand(), repoDir, any(), any()) } + } + + test("propagates an import failure that is not a missing verb") { + givenImageMissing() + every { commandRunner.run(importCommand(), repoDir, any(), any()) } returns + GitCommandResult(125, "", "werkdock: unpacking /srv/buildenv.tar.zst failed\n") + + val exception = + shouldThrow { + runner.start("./gradlew test", workspace, mapOf("branch" to "main"), repoDir, werkdockBranchConfig()) + } + + exception.message shouldContain "unpacking" + verify(exactly = 0) { commandRunner.runOrThrow(match { "load" in it }, any(), any(), any()) } + } + + test("does not import an image werkdock already has") { givenImageLoaded() runner.start("./gradlew test", workspace, mapOf("branch" to "main"), repoDir, werkdockBranchConfig()) + verify(exactly = 0) { commandRunner.run(match { "import" in it }, any(), any(), any()) } verify(exactly = 0) { commandRunner.runOrThrow(match { "load" in it }, any(), any(), any()) } } @@ -209,7 +227,7 @@ class WerkdockBuildRunnerTest : FunSpec() { exception.message shouldContain "werkdock.rootfs" } - test("downloads a URL rootfs once before loading it") { + test("downloads a URL rootfs once before importing it") { val url = "https://example.test/buildenv.tar.zst" val downloadTarget = repoDir @@ -219,7 +237,7 @@ class WerkdockBuildRunnerTest : FunSpec() { givenImageMissing() every { commandRunner.runOrThrow(listOf("curl", "-fsSL", "-o", downloadTarget.toString(), url), repoDir, any(), any()) } returns GitCommandResult(0, "", "") - every { commandRunner.runOrThrow(match { "load" in it }, any(), any(), any()) } returns + every { commandRunner.run(match { "import" in it }, any(), any(), any()) } returns GitCommandResult(0, "", "") runner.start("./gradlew test", workspace, mapOf("branch" to "main"), repoDir, werkdockBranchConfig(rootfs = url)) @@ -228,8 +246,8 @@ class WerkdockBuildRunnerTest : FunSpec() { commandRunner.runOrThrow(listOf("curl", "-fsSL", "-o", downloadTarget.toString(), url), repoDir, any(), any()) } verify { - commandRunner.runOrThrow( - listOf("werkdock", "load", "-i", downloadTarget.toString(), "--name", "werkator-buildenv-${url.sha12()}"), + commandRunner.run( + listOf("werkdock", "import", downloadTarget.toString(), "werkator-buildenv-${url.sha12()}"), repoDir, any(), any(),