Author SHA1 Message Date
mhoennigandClaude Opus 5 3ffc1a8eed docs(prs): PR#19 — the sandbox config section is werkdock
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
2026-09-03 20:27:16 +02:00
mhoennigandClaude Opus 5 34af8ba1bf feat(config): the sandbox section is werkdock, not bwrap
`bwrap` named the mechanism one layer below the tool that actually runs it: since
v1.0.0 Werkator does not invoke bwrap at all, it shells out to the werkdock CLI —
which made `bwrap.werkdock` a key naming its own executor.

The section is `werkdock` now and that key is `werkdock.binary`; BwrapConfig,
BwrapOverrides and BwrapBuildRunner follow the name. A file still writing `bwrap`
is read as before and warned about once per file, in `renameLegacySandbox` on the
raw map of every layer before merging — so nothing downstream knows two names, and
the old name is not a way around the pinning either. Renaming rather than refusing,
because the section lives in the machine configuration of every webspace instance,
which no repository tracks; the hard refusal belongs to the release that sets
ConfigVersions.FORMAT_BROKE_IN, where a file declaring no version can be caught
by name at all.

WERKATOR_SANDBOX in tools/remote follows, and still accepts `bwrap`.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
2026-09-03 20:27:16 +02:00
mhoennigandClaude Opus 5 95564587d3 docs(prs): PR#18 — tools/remote drives any host layout
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
2026-09-03 20:27:11 +02:00
mhoennigandClaude Opus 5 618a2acb9f docs(deployment): tools/remote drives any host layout
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
2026-09-03 20:27:11 +02:00
mhoennigandClaude Opus 5 2ffc14a20c fix(remote): upload before stopping the service, and verify the transfer
instance-update stopped the unit and only then started the upload, so a transfer
that dies mid-way leaves the host with no running Werkator and nothing to start
again. That is not theoretical: deploying to vm4006 on 2026-09-03 failed with
"scp: Connection closed" with the service already stopped.

The upload now happens before the stop, and each artifact is transferred to a
.part file whose sha256 is compared with the local one before it is moved into
place, retrying twice. A truncated archive would otherwise unpack into a broken
runtime, which is worse than the failed transfer it came from.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
2026-09-03 20:27:11 +02:00
mhoennigandClaude Opus 5 b5df75f844 feat(remote): the host layout is configurable, not the mih convention
tools/remote assumed the layout instance-install creates: the watched repository
in $WERKATOR_PATH/werkator, the runtime in $WERKATOR_PATH/.werkator, a werkdock
binary and a rootfs beside it, and the unit hardcoded as werkator-werkator.service.
An installation that predates the script — vm4006, a docker host with the repository
in ~/hs.hsadmin.ng and the runtime in ~/opt — could not be deployed with it at all.

WERKATOR_REPO_DIR, WERKATOR_INSTALL_DIR and WERKATOR_SANDBOX name the three values
that actually differ; their defaults are what instance-install writes, so the
existing env files resolve to exactly the same paths as before. The unit name is
derived from the repository directory the way SystemdServiceFiles.unitName does it,
instead of being spelled out. With WERKATOR_SANDBOX=docker the werkdock binary and
the rootfs archive are neither built nor uploaded — a docker host has no sandbox to
install, and check-prerequisites asks the docker daemon instead of werkdock doctor.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
2026-09-03 20:27:11 +02:00
14 changed files with 61 additions and 625 deletions
-22
View File
@@ -8,28 +8,6 @@ Lightweight, declarative and highly opinionated software build system (CI/CD).
- [docs/bootstrapping.md](docs/bootstrapping.md) — initializing a repository with `init`
- [docs/deployment.md](docs/deployment.md) — running Werkator as a systemd service behind a reverse proxy
## Adding a Gitea Repository
One instance serves several repositories (`docs/deployment.md`, ADR 0009).
From the workstation, clone and initialise, then register:
```bash
tools/remote --env-file .env.<instance> werkator repo-add https://gitea.example.org/<owner>/<repo>.git [<name>]
```
It prints the registry entry: add it to `~/.werkator.yml` under `repositories:`, then restart the service.
The optional `[<name>]` overrides the directory basename: it becomes the route segment (`/repos/<name>/…`) and the UI switcher entry, so it must be unique.
Needed only when the clone directory name is wrong or collides — e.g. `michael.hoennig.de.git` checked out as `michael.hoennig.de`, or two forges serving a repo of the same name.
A **public** repository needs nothing else: the clone runs anonymously.
A **private** repository needs shared credentials once on the host, in `~/.werkator.yml` of the service user, before cloning:
```yaml
defaults:
git:
account: <gitea-user>
token: <token> # Gitea → Settings → Applications → Generate Token, scope read:repository
```
## Developer Setup
Source `.envrc` to add `tools/` to your `PATH`, or install [direnv](#direnv) to have this done automatically on `cd`:
+1 -3
View File
@@ -475,9 +475,7 @@ The section was called `bwrap` and its binary key `bwrap.werkdock` until v1.2.0;
`werkdock.rootfs` names the prepared root filesystem archive — a Debian-base rootfs with the build tools (JDK, git, locales, project-specific tooling) built elsewhere, since `debootstrap` is unavailable on the target.
It is a local path or an `http(s)` URL; a URL is downloaded once into `.git/werkator/buildenv/`.
Build the archive with `tools/build-bwrap-rootfs.sh` on any machine with Docker.
The archive is imported once per source as the werkdock image `werkator-buildenv-<hash>` into werkdock's store (`$WERKDOCK_HOME`, default `~/.werkdock`) — shared by every repository of this OS user; the hash derives from the source string, so a changed `rootfs` imports a fresh image and stale ones can be removed from the store.
Werkator uses `werkdock import ARCHIVE IMAGE` (docker import semantics) and falls back to the older `werkdock load -i ARCHIVE --name IMAGE` when the installed werkdock does not know the verb yet, so Werkator and werkdock can be updated in either order.
The image keeps its untagged name (it equals `werkator-buildenv-<hash>:latest` in werkdock's docker-style naming), so an existing store needs no re-import.
The archive is loaded once per source as the werkdock image `werkator-buildenv-<hash>` into werkdock's store (`$WERKDOCK_HOME`, default `~/.werkdock`) — shared by every repository of this OS user; the hash derives from the source string, so a changed `rootfs` loads a fresh image and stale ones can be removed from the store.
Per-repo Gradle caches persist in `.git/werkator/buildenv/home`, bound as `/root`.
`werkdock.env` adds environment variables inside the sandbox; the environment is otherwise cleared (docker semantics) — the server's environment does not leak in.
Files created inside the sandbox are owned by the host user, because uid 0 maps back to the unprivileged webspace user.
+1 -2
View File
@@ -123,7 +123,6 @@ Adding a repository is editing a registry entry — never a data migration, beca
It clones the repository next to the ones already served, runs `init` in it, and **prints** the registry entry.
It does not write `~/.werkator.yml`: that file is the instance's own — port, global concurrency, possibly shared credentials — and a script editing it in place would rewrite the operator's configuration behind their back.
Cloning and initialising is mechanical; registering is a decision.
A private `https` origin authenticates with the shared `defaults.git.account`/`defaults.git.token` of `~/.werkator.yml` (the token travels via a one-shot `GIT_ASKPASS` on the host, never in a URL or process list); enter those once before cloning a private repository — without them only public origins clone.
4. **Restart** the service; startup recovery re-enqueues what was in flight:
@@ -342,7 +341,7 @@ tools/remote --env-file .env.mih34 port-forward start # browser tunne
Layout on the host: the watched repository at `$WERKATOR_PATH/werkator/`, the unpacked runtime at `$WERKATOR_PATH/.werkator/werkator/`, the werkdock binary at `$WERKATOR_PATH/.werkator/bin/werkdock`.
That is the default, not a requirement: `WERKATOR_REPO_DIR`, `WERKATOR_INSTALL_DIR` and `WERKATOR_SANDBOX` bend it to an installation that predates the script, see [Updating an Existing Installation](#updating-an-existing-installation).
The rootfs archive is imported once per source into werkdock's image store (`~/.werkdock`), shared by every repository of the user.
The rootfs archive is loaded once per source into werkdock's image store (`~/.werkdock`), shared by every repository of the user.
Fill `git.account`/`git.token` in the machine config when the origin is private, and make the user's services survive logout with `loginctl enable-linger`.
Updates are one command, refused while a build runs (`FORCE=1` overrides):
@@ -1,42 +0,0 @@
> **WARNING:** This document describes only the change applied in this PR.
> It may already be outdated once the next PR is merged.
> Historic PR-documentation is not maintained along with new PRs — treat it as a snapshot, not as current documentation.
## The Problem
The build for commit `7028ca8` on `main` failed on the mih09 production instance with
`BuildExecutorTest > with maxConcurrent 1 a second branch stays PENDING until the first finished`,
while a retry of the very same commit passed, and the test passes locally.
The test was timing-dependent.
It started a build for `branch-a` whose build command was `sleep 1`, immediately started a second build for `branch-b`,
and then asserted — without any synchronization at all — that `branch-b` was still `PENDING`.
That assertion only held as long as the test thread reached it within the one second `branch-a` slept.
On a shared host under CPU contention the executor can get through `branch-a` entirely (queued, running, slept, succeeded) first,
and `branch-b` is then already `RUNNING` or `SUCCESS` when the assertion runs.
The failure therefore says nothing about the executor; it is pure scheduling noise that costs a build and a retry every time it hits.
## Non-Goals
- No change to production code — `BuildExecutor` is not touched, its queueing behaviour is unchanged.
- No sweep of the other timing-sensitive tests in the suite; only the one that actually flaked is fixed.
## The Solution
`branch-a` no longer sleeps for a fixed time, it blocks until the test says so:
its build command is `until [ -f gate ]; do sleep 0.05; done`, and the build workspace is the test's working directory.
The test now
1. waits (via `eventually`) until `branch-a` is `RUNNING`, so the single executor slot is provably occupied,
2. asserts that `branch-b` is `PENDING` — which cannot race anything, because `branch-a` cannot finish before the gate file exists,
3. creates the gate file, and only then awaits both builds' `SUCCESS`.
The assertion on the event transitions (`branch-b` goes `RUNNING` only after `branch-a` reached `SUCCESS`) is unchanged.
A blocking gate was chosen over a mocked `BuildRunner` because it keeps the test on the real `ProcessBuildRunner`,
so it still covers the actual process handling rather than only the executor's bookkeeping.
Verified by running `BuildExecutorTest` five times on an idle machine and three more times with twice `nproc` busy-loops saturating the CPU,
which is the condition that produced the original failure.
- [BuildExecutorTest](../../src/test/kotlin/de/hoennig/werkator/build/BuildExecutorTest.kt)
@@ -1,30 +0,0 @@
> **WARNING:** This document describes only the change applied in this PR.
> It may already be outdated once the next PR is merged.
> Historic PR-documentation is not maintained along with new PRs — treat it as a snapshot, not as current documentation.
## The Problem
The repository's home is `https://git.javagil.de/mi/werkator.git` since the move to the own Gitea instance,
but `tools/remote` still defaulted `WERKATOR_REPO_URL` to the GitHub mirror.
The mih09 production instance was cloned from that mirror and consequently watched a `main` that nobody pushes to any more:
it kept reporting the last GitHub state as green while three merged pull requests sat unbuilt on the real `main`.
The failure that started this — a flaky test already fixed on Gitea's `main` — could not be re-verified live,
because the instance had no way to see the fix.
## Non-Goals
- The existing clone on mih09 is not touched by this change; its remote was repointed by hand (`git remote set-url`), and `repo-init` skips an existing clone.
- The generic placeholder URLs in `docs/deployment.md` stay as they are — they describe cloning *any* watched repository, not werkator's own.
- No decision about mirroring to GitHub; the mirror simply stops being the source an instance builds from.
## The Solution
`REPO_URL` in [tools/remote](../../tools/remote) defaults to the Gitea URL, and the usage comment says so.
The value stays overridable via `WERKATOR_REPO_URL` in the instance's env file,
so an installation that deliberately watches a different remote is unaffected.
Anonymous HTTPS works against Gitea exactly as it did against GitHub, so no deploy key or token is involved.
Note that pull requests opened via AGit-Flow create no branch in Gitea,
so an instance watching this remote sees `main` only — branch builds require pushing real branches.
@@ -1,90 +0,0 @@
> **WARNING:** This document describes only the change applied in this PR.
> It may already be outdated once the next PR is merged.
> Historic PR-documentation is not maintained along with new PRs — treat it as a snapshot, not as current documentation.
## The Problem
`init --systemd` generates the host integration — the systemd unit's resource limits, the Apache `.htaccess` and the maintenance page — from the effective configuration.
It read that configuration through a helper with two independent defects, both of which fail silently.
**It swallowed every error.**
The load sat in `try { … } catch (_: Exception) { ServerConfig() }`.
Any configuration error at all — a missing required field, a malformed layer, a version floor violation — produced a default `ServerConfig` with a blank `publicBaseUrl`.
A repository with a broken `.werkator.yml` then looked exactly like one that simply has no public base URL configured:
the `.htaccess` and the maintenance page were skipped without a word.
This surfaced while verifying [PR#17](2026-09-03-PR%2317-maintenance-page.md) on mih09, where the missing files looked like an unconfigured `publicBaseUrl` and were in fact an unrelated validation error.
**It read from the wrong directory.**
The helper called `configLoader.load(Paths.get("."))` — the process's current directory — while everything else in the command works off the git top level resolved by `GitService.getTopLevel`.
`ConfigLoader.loadRaw` resolves the layers directly under the directory it is given and does not walk up to the repository root, so the two agree only when `init` happens to be invoked from the root itself.
From a subdirectory the command read another repository's configuration, or none.
That also broke `--apply`: the fragment is installed into the repository root deliberately before the systemd files are written, so that its port and limits reach the generated unit, and a current-directory read does not see it.
`init --systemd` runs during initial deployment setup, which is exactly when a silent wrong answer is most expensive.
## Non-Goals
- The fallback itself is kept: a configuration that cannot be loaded is not fatal for `init`, the units are still generated with the defaults.
During the very first bootstrap there is legitimately nothing to load yet.
- No change to `ConfigLoader`, to the configuration schema, or to any other command.
- No sweep for catch-all exception handlers elsewhere in the code base;
the two other `catch` blocks in `InitCommand` already print an `Error:` and abort, so they were only checked, not changed.
## The Scenarios
### Feature: init reports what it read and where it read it from
#### Background
- The *repository root* is the git top level as resolved by `GitService.getTopLevel`, the directory holding `.werkator.yml`, `.git/werkator/.werkator.yml` and an applied fragment.
- The *current directory* is the process working directory, which is the repository root only when `init` is invoked there.
#### Scenario#22.01: A broken configuration is named, not defaulted over
So that a validation error during deployment setup is not mistaken for an unconfigured installation.
- **Given** a repository whose effective configuration cannot be loaded
- **When** `init --systemd` runs
- **Then** the exception message is printed as a warning
- **and** the unit files are still generated with the default settings
- **and** the warning appears exactly once, although three settings are read from the configuration
##### Verified by
- [InitCommandTest: `--systemd warns once when the effective configuration cannot be loaded`](../../src/test/kotlin/de/hoennig/werkator/commands/InitCommandTest.kt)
#### Scenario#22.02: The configuration is read from the repository root
So that the generated host integration reflects the repository being initialized, whatever directory `init` was invoked from.
- **Given** a repository whose root configuration sets `server.publicBaseUrl` and `server.port`
- **and** a current directory that is not that repository root
- **When** `init --systemd` runs
- **Then** the `.htaccess` and the maintenance page are generated
- **and** the `.htaccess` proxies to the port from the root configuration
##### Verified by
- [InitCommandTest: `--systemd reads the configuration from the repository root, not the current directory`](../../src/test/kotlin/de/hoennig/werkator/commands/InitCommandTest.kt)
## The Solution
The catch-all now prints the exception message before falling back:
```
Warning: the effective configuration could not be loaded (<message>)
continuing with default server settings — check the generated unit and host files
```
The configuration is read three times while the systemd files are written (`memoryMax`, `tasksMax`, `publicBaseUrl`), which would repeat the warning three times.
It is therefore loaded once per run and cached in the command, and the cache is reset at the top of `run()` so a reused instance — the command is a Spring singleton — re-reads.
The repository root is passed down into the two accessors instead of `Paths.get(".")`.
This matches every other caller of `ConfigLoader.load` in the code base, all of which pass an explicit working directory;
`InitCommand` was the only one relying on the process's current directory.
Both fixes are the same failure in two forms — the command answered from a configuration it never actually read — which is why they are in one PR.
## Additional Changes
- None.
@@ -1,92 +0,0 @@
> **WARNING:** This document describes only the change applied in this PR.
> It may already be outdated once the next PR is merged.
> Historic PR-documentation is not maintained along with new PRs — treat it as a snapshot, not as current documentation.
## Related Links
- ADR 0009 — multi-repo instance: the `defaults` block carries shared repository-level keys, read by `ConfigLoader`, never directly by consumers.
- `docs/deployment.md` — registry setup: `repo-add` clones, initialises, and prints the registry entry.
## The Problem
`tools/remote werkator repo-add <private-https-url>` fails on the host with `could not read Username`.
The clone runs anonymously, but the credentials exist only in the instance file's `defaults.git` block — which the script never consults.
Registering a private repository therefore needs a manual SSH session today.
## Non-Goals
The script still does not write `~/.werkator.yml`: registering stays the operator's decision.
No SSH-URL support: the forge is reached over `https` from the host.
No new config keys: `defaults.git.account`/`defaults.git.token` already exist.
## The Scenarios
### Feature: authenticated clone for private origins
#### Background
- The shared credentials live in `~/.werkator.yml` under `defaults.git` (ADR 0009).
- Public origins and instances without shared credentials must keep cloning anonymously.
#### Scenario#000.01: Private https origin clones with shared credentials
- **Given** `defaults.git.account`/`defaults.git.token` in `~/.werkator.yml` on the host
- **When** `tools/remote werkator repo-add <private-https-url>` runs
- **Then** the clone authenticates with those credentials and succeeds.
##### Verified by
- Manual stub-`git` test: URL carries `account@`, askpass answers the token, `GIT_TERMINAL_PROMPT=0`.
#### Scenario#000.02: Public origin clones anonymously
- **Given** no shared credentials (or a public repository)
- **When** `repo-add` or `repo-init` runs
- **Then** the clone runs exactly as before, without authentication.
##### Verified by
- Local helper test against a `file://` origin with and without an instance file.
#### Scenario#000.03: Token never leaks locally
- **Given** an authenticated clone
- **When** the command runs from the workstation
- **Then** the token appears in neither the local process list nor a repository config.
##### Verified by
- Code inspection: the token is read on the host and passed via a one-shot `GIT_ASKPASS` script.
## The Solution
`tools/remote` gained a `clone_repo` helper used by both `repo-init` and `repo-add`.
For `https://` URLs it ships a small Python helper (base64-encoded, so no `$` is expanded locally) to the host.
The helper reads `defaults.git.account`/`defaults.git.token` from `~/.werkator.yml`, puts the account into the URL, and hands the token to git via a one-shot `0700` `GIT_ASKPASS` script deleted in `finally`.
Without credentials it falls back to the plain anonymous clone; non-`https` URLs clone unchanged.
`docs/deployment.md` documents that the shared credentials must exist before cloning a private repository.
## Open Questions
- None.
## Attachments
### Adding a Gitea repository — example
From the workstation, clone and initialise, then register:
```bash
tools/remote --env-file .env.<instance> werkator repo-add https://gitea.example.org/<owner>/<repo>.git [<name>]
```
Add the printed entry to `~/.werkator.yml` under `repositories:`, then restart the service.
A public repository needs nothing else: the clone runs anonymously.
A private repository needs shared credentials once on the host, in `~/.werkator.yml` of the service user, before cloning (token: Gitea → Settings → Applications → Generate Token, scope `read:repository`):
```yaml
defaults:
git:
account: <gitea-user>
token: <token>
```
@@ -1,102 +0,0 @@
> **WARNING:** This document describes only the change applied in this PR.
> It may already be outdated once the next PR is merged.
> Historic PR-documentation is not maintained along with new PRs — treat it as a snapshot, not as current documentation.
## Related Links
- Werkdock `CHANGELOG.md` (repository `mi/werkdock`, Unreleased section) — the release notes this PR adapts to: `import` replaces `load -i --name` for rootfs archives, `load` is reserved for docker/OCI image archives, image names gained docker-style tags.
- ADR 0008 — the bwrap build runtime; `docs/configuration.md`, notes on `builds.<name>.werkdock`.
## The Problem
Werkdock moves its CLI closer to `docker run --rm`.
For the rootfs archives Werkator uses, the verb is now `werkdock import ARCHIVE IMAGE` (docker import semantics).
`werkdock load -i ARCHIVE --name IMAGE` still works but is a compatibility path with a note on stderr, and is announced to go away once Werkator has switched.
Werkator ships the werkdock binary with its deployment, but nothing forces the two to be updated together: an installation may run a new Werkator against an older werkdock for a while, or the other way round.
## Non-Goals
- No switch of the existence check from `werkdock images` to `werkdock inspect`: `images` prints the bare name for untagged images on every werkdock version, the exact-line match keeps working, and `inspect` would tie Werkator to the new werkdock.
- No tagged image name (`werkator-buildenv:<hash>`): it would re-import every build environment on the webspace and orphan the old image, for no functional gain today.
- None of the new `run` flags (`--mount`, `--entrypoint`, `--network host`): Werkator's invocation needs none of them.
## The Scenarios
### Feature: rootfs archives are imported with werkdock's docker-shaped verb
#### Background
- The build environment is a rootfs archive named by `builds.<name>.werkdock.rootfs`.
- Werkator creates the werkdock image `werkator-buildenv-<hash>` from it once per source.
- Werkdock answers an unknown verb with `werkdock: unknown command "import"` and exit code 125, its code for its own errors.
#### Scenario#000.01: A missing image is imported with `werkdock import`
So that Werkator uses the verb werkdock names for rootfs archives, and the deprecated path can be removed on werkdock's side.
- **Given** `werkdock images` does not list the image
- **When** a build starts
- **Then** Werkator runs `werkdock import ARCHIVE werkator-buildenv-<hash>`
- **and** runs no `werkdock load`.
##### Verified by
- [WerkdockBuildRunnerTest "imports the image once when werkdock does not know it yet"](../../src/test/kotlin/de/hoennig/werkator/build/WerkdockBuildRunnerTest.kt)
- [WerkdockBuildRunnerTest "downloads a URL rootfs once before importing it"](../../src/test/kotlin/de/hoennig/werkator/build/WerkdockBuildRunnerTest.kt)
#### Scenario#000.02: An older werkdock without the verb still works
So that Werkator and werkdock can be updated in either order.
- **Given** the installed werkdock answers `import` with `unknown command` and exit 125
- **When** a build starts with a missing image
- **Then** Werkator falls back to `werkdock load -i ARCHIVE --name werkator-buildenv-<hash>`
- **and** logs the fallback.
##### Verified by
- [WerkdockBuildRunnerTest "falls back to load -i --name on a werkdock without the import verb"](../../src/test/kotlin/de/hoennig/werkator/build/WerkdockBuildRunnerTest.kt)
#### Scenario#000.03: A real import failure is not masked by the fallback
So that a broken archive fails the build with werkdock's message, as before.
- **Given** `werkdock import` fails for any other reason (any other exit code, or exit 125 without `unknown command`)
- **When** a build starts with a missing image
- **Then** the build fails with that command's output
- **and** no `werkdock load` runs.
##### Verified by
- [WerkdockBuildRunnerTest "propagates an import failure that is not a missing verb"](../../src/test/kotlin/de/hoennig/werkator/build/WerkdockBuildRunnerTest.kt)
#### Scenario#000.04: An existing image is neither imported nor loaded
- **Given** `werkdock images` lists the image
- **When** a build starts
- **Then** neither `import` nor `load` runs.
##### Verified by
- [WerkdockBuildRunnerTest "does not import an image werkdock already has"](../../src/test/kotlin/de/hoennig/werkator/build/WerkdockBuildRunnerTest.kt)
## The Solution
`WerkdockBuildRunner.ensureImage` calls the new `importImage`: `werkdock import ARCHIVE IMAGE` through the non-throwing `run`, then either returns, falls back to `load -i --name` on exactly the unknown-verb signature (exit 125 plus `unknown command` on stderr), or rethrows the import's result as a `GitCommandException` — the same exception and message the old code produced.
The fallback is keyed on werkdock's own error signature rather than on a version string, because werkdock's version output does not yet distinguish the two builds.
## Open Questions
- When to drop the fallback: once every installation runs a werkdock with `import`, the `load -i --name` branch and its test go, and werkdock can remove the compatibility path.
## Additional Changes
- `docs/configuration.md` and `docs/deployment.md` say "imported" where they said "loaded", and name the fallback and the reason the image name stays untagged.
## Prerequisite PRs
- The werkdock change that introduces `import` (repository `mi/werkdock`, CHANGELOG Unreleased); against an older werkdock the fallback path runs.
## Follow-up PRs
- Remove the fallback (see Open Questions).
@@ -2,7 +2,6 @@ package de.hoennig.werkator.build
import de.hoennig.werkator.config.BranchConfig
import de.hoennig.werkator.config.WerkdockConfig
import de.hoennig.werkator.git.GitCommandException
import de.hoennig.werkator.git.GitCommandRunner
import org.slf4j.LoggerFactory
import org.springframework.stereotype.Component
@@ -17,9 +16,9 @@ import java.security.MessageDigest
* it shells out to the `werkdock` CLI (`werkdock.binary`, default via PATH) the same
* pattern as git and docker, CLI, no library.
*
* The rootfs archive becomes a werkdock *image*, imported once per source
* The rootfs archive becomes a werkdock *image*, loaded once per source
* (`werkator-buildenv-<hash>`, the hash over the source string, so a changed source
* imports a fresh image) into werkdock's own store (`$WERKDOCK_HOME`, default
* loads a fresh image) into werkdock's own store (`$WERKDOCK_HOME`, default
* `~/.werkdock`) shared by every repository of this OS user, unlike the old
* per-repo unpack. Only the download cache for URL sources and the persistent
* toolchain home (bound to `/root` for Gradle/Go caches) stay under
@@ -67,11 +66,9 @@ class WerkdockBuildRunner(
}
/**
* Imports the rootfs archive into the werkdock image store once per source.
* Loads the rootfs archive into the werkdock image store once per source.
* `werkdock images` answers existence through the CLI, like `docker image
* inspect` does for the Docker runner; it prints the bare name for an untagged
* image on every werkdock version, so the exact-line match holds across the
* werkdock upgrade that introduced tags.
* inspect` does for the Docker runner.
*/
private fun ensureImage(
werkdock: String,
@@ -87,33 +84,7 @@ class WerkdockBuildRunner(
val envDir = repoDir.resolve(BUILDENV_DIR).resolve(sourceKey(sandbox.rootfs))
Files.createDirectories(envDir)
val archive = localArchive(sandbox.rootfs, envDir, repoDir, onAuxProcess)
log.info("importing build environment {} as werkdock image {}", sandbox.rootfs, image)
importImage(werkdock, archive, image, repoDir, onAuxProcess)
}
/**
* `werkdock import ARCHIVE IMAGE` (docker import semantics) creates the image from a
* rootfs archive. A werkdock that predates the verb answers `unknown command` with
* exit 125; then the older `load -i ARCHIVE --name IMAGE` does the same job, so an
* installation can update Werkator and werkdock in either order. Any other failure
* is the import's own and propagates as it did before.
*/
private fun importImage(
werkdock: String,
archive: String,
image: String,
repoDir: Path,
onAuxProcess: (Process) -> Unit,
) {
val importCommand = listOf(werkdock, "import", archive, image)
val imported = commandRunner.run(importCommand, repoDir, onProcess = onAuxProcess)
if (imported.isSuccess) {
return
}
if (imported.exitCode != WERKDOCK_CLI_ERROR || "unknown command" !in imported.stderr) {
throw GitCommandException(importCommand, imported)
}
log.info("this werkdock has no import verb yet, falling back to load -i --name")
log.info("loading build environment {} as werkdock image {}", sandbox.rootfs, image)
commandRunner.runOrThrow(
listOf(werkdock, "load", "-i", archive, "--name", image),
repoDir,
@@ -238,8 +209,5 @@ class WerkdockBuildRunner(
companion object {
const val BUILDENV_DIR = ".git/werkator/buildenv"
const val HOME_DIR = "home"
/** werkdock's exit code for its own errors (docker's 125), as opposed to the sandboxed command's. */
const val WERKDOCK_CLI_ERROR = 125
}
}
@@ -48,7 +48,6 @@ class InitCommand(
internal var javaExecutableResolver: () -> Path = { Paths.get(System.getProperty("java.home"), "bin", "java") }
override fun run() {
cachedServerConfig = null
val normalizedWorkingDir = workingDir.toAbsolutePath().normalize()
val root =
try {
@@ -89,7 +88,7 @@ class InitCommand(
if (url == null) return DetectedValues()
if (url.startsWith("http")) {
val regex = Regex("""https?://(?:([^@]+)@)?([^/]+)/([^/]+)/(.+?)(?:\.git)?$""")
val regex = Regex("""https?://(?:([^@]+)@)?([^/]+)/([^/]+)/([^/.]+)(?:\.git)?""")
val match = regex.find(url)
if (match != null) {
val (user, host, owner, repo) = match.destructured
@@ -102,7 +101,7 @@ class InitCommand(
}
} else if (url.contains("@") && url.contains(":")) {
// Assume SSH: git@host:owner/repo.git
val regex = Regex("""([^@]+)@([^:]+):([^/]+)/(.+?)(?:\.git)?$""")
val regex = Regex("""([^@]+)@([^:]+):([^/]+)/([^/.]+)(?:\.git)?""")
val match = regex.find(url)
if (match != null) {
val (_, host, owner, repo) = match.destructured
@@ -297,31 +296,14 @@ class InitCommand(
* already loadable (re-running `init --systemd` on an installed instance); during
* the very first bootstrap they stay unset and the defaults (no directives) apply.
*/
private fun loadedSystemdConfig(root: Path): de.hoennig.werkator.config.SystemdConfig = loadedServerConfig(root).systemd
private fun loadedSystemdConfig(): de.hoennig.werkator.config.SystemdConfig = loadedServerConfig().systemd
/** Loaded once per run, so a broken configuration is reported once and not per caller. */
private var cachedServerConfig: de.hoennig.werkator.config.ServerConfig? = null
/**
* Read from the repository root like every other file this command touches the
* layers sit there, not in whatever directory the process happens to run in, and
* an applied fragment must reach the generated unit even when `init` is invoked
* from a subdirectory.
*
* A configuration error here is not fatal the units are still generated with defaults
* but it must not pass for "nothing configured": without the warning a broken `.werkator.yml`
* looks exactly like an unset `publicBaseUrl` and the host integration is skipped silently.
*/
private fun loadedServerConfig(root: Path): de.hoennig.werkator.config.ServerConfig =
cachedServerConfig ?: run {
private fun loadedServerConfig(): de.hoennig.werkator.config.ServerConfig =
try {
configLoader.load(root).server
} catch (e: Exception) {
println("Warning: the effective configuration could not be loaded (${e.message})")
println(" continuing with default server settings — check the generated unit and host files")
configLoader.load(Paths.get(".")).server
} catch (_: Exception) {
de.hoennig.werkator.config
.ServerConfig()
}.also { cachedServerConfig = it }
}
private fun createSystemdFiles(
@@ -345,8 +327,8 @@ class InitCommand(
javaExecutable = javaExecutableResolver(),
jarPath = jarPath,
envFile = envFile,
memoryMax = loadedSystemdConfig(root).memoryMax,
tasksMax = loadedSystemdConfig(root).tasksMax,
memoryMax = loadedSystemdConfig().memoryMax,
tasksMax = loadedSystemdConfig().tasksMax,
),
)
println("created ${unitFile.toFile().relativeTo(normalizedWorkingDir.toFile())}")
@@ -369,7 +351,7 @@ class InitCommand(
// generated host integration like the units: only meaningful behind a web
// frontend, so it needs a public base URL; unused elsewhere and harmless
val server = loadedServerConfig(root)
val server = loadedServerConfig()
if (server.publicBaseUrl.isNotBlank()) {
val htaccessFile = werkatorDir.resolve(SystemdServiceFiles.HTACCESS_NAME)
htaccessFile.toFile().writeText(SystemdServiceFiles.htaccessContent(server.port))
@@ -505,8 +505,6 @@ class BuildExecutorTest : FunSpec() {
}
test("with maxConcurrent 1 a second branch stays PENDING until the first finished") {
// branch-a blocks on a gate file the test creates, so the PENDING assertion
// below cannot race the first build finishing on a loaded machine
val h =
Harness(
"""
@@ -514,7 +512,7 @@ class BuildExecutorTest : FunSpec() {
maxConcurrent: 1
branches:
branch-a:
buildCommand: "until [ -f gate ]; do sleep 0.05; done"
buildCommand: "sleep 1"
cleanCommand: ""
branch-b:
buildCommand: "echo ok"
@@ -525,13 +523,8 @@ class BuildExecutorTest : FunSpec() {
h.executor.startBuild(h.repo, "branch-a", "sha-a")
h.executor.startBuild(h.repo, "branch-b", "sha-b")
eventually(30.seconds) {
h.repository.latestFor("branch-a")?.status shouldBe BuildStatus.RUNNING
}
h.repository.latestFor("branch-b")?.status shouldBe BuildStatus.PENDING
Files.createFile(h.workingDir.resolve("gate"))
awaitStatus(h, "branch-b", BuildStatus.SUCCESS)
awaitStatus(h, "branch-a", BuildStatus.SUCCESS)
val transitions = h.events.map { it.result.branch to it.result.status }
@@ -2,7 +2,6 @@ package de.hoennig.werkator.build
import de.hoennig.werkator.config.BranchConfig
import de.hoennig.werkator.config.WerkdockConfig
import de.hoennig.werkator.git.GitCommandException
import de.hoennig.werkator.git.GitCommandResult
import de.hoennig.werkator.git.GitCommandRunner
import io.kotest.assertions.throwables.shouldThrow
@@ -38,11 +37,7 @@ class WerkdockBuildRunnerTest : FunSpec() {
private fun imageName(rootfs: String = "/srv/buildenv.tar.zst"): String = "werkator-buildenv-${rootfs.sha12()}"
private fun importCommand(): List<String> = listOf("werkdock", "import", "/srv/buildenv.tar.zst", imageName())
private fun loadCommand(): List<String> = listOf("werkdock", "load", "-i", "/srv/buildenv.tar.zst", "--name", imageName())
/** The image is already there: `werkdock images` lists it, so no import runs. */
/** The image is already loaded: `werkdock images` lists it, so no load runs. */
private fun givenImageLoaded(rootfs: String = "/srv/buildenv.tar.zst") {
every { commandRunner.runOrThrow(listOf("werkdock", "images"), repoDir, any(), any()) } returns
GitCommandResult(0, imageName(rootfs) + "\n", "")
@@ -93,47 +88,34 @@ class WerkdockBuildRunnerTest : FunSpec() {
)
}
test("imports the image once when werkdock does not know it yet") {
test("loads the image once when werkdock does not know it yet") {
givenImageMissing()
every { commandRunner.run(importCommand(), repoDir, any(), any()) } returns GitCommandResult(0, "", "")
every {
commandRunner.runOrThrow(
listOf("werkdock", "load", "-i", "/srv/buildenv.tar.zst", "--name", imageName()),
repoDir,
any(),
any(),
)
} returns GitCommandResult(0, "", "")
runner.start("./gradlew test", workspace, mapOf("branch" to "main"), repoDir, werkdockBranchConfig())
verify { commandRunner.run(importCommand(), repoDir, any(), any()) }
verify(exactly = 0) { commandRunner.runOrThrow(match { "load" in it }, any(), any(), any()) }
verify {
commandRunner.runOrThrow(
listOf("werkdock", "load", "-i", "/srv/buildenv.tar.zst", "--name", imageName()),
repoDir,
any(),
any(),
)
}
}
test("falls back to load -i --name on a werkdock without the import verb") {
givenImageMissing()
every { commandRunner.run(importCommand(), repoDir, any(), any()) } returns
GitCommandResult(125, "", "werkdock: unknown command \"import\"\n")
every { commandRunner.runOrThrow(loadCommand(), repoDir, any(), any()) } returns GitCommandResult(0, "", "")
runner.start("./gradlew test", workspace, mapOf("branch" to "main"), repoDir, werkdockBranchConfig())
verify { commandRunner.runOrThrow(loadCommand(), repoDir, any(), any()) }
}
test("propagates an import failure that is not a missing verb") {
givenImageMissing()
every { commandRunner.run(importCommand(), repoDir, any(), any()) } returns
GitCommandResult(125, "", "werkdock: unpacking /srv/buildenv.tar.zst failed\n")
val exception =
shouldThrow<GitCommandException> {
runner.start("./gradlew test", workspace, mapOf("branch" to "main"), repoDir, werkdockBranchConfig())
}
exception.message shouldContain "unpacking"
verify(exactly = 0) { commandRunner.runOrThrow(match { "load" in it }, any(), any(), any()) }
}
test("does not import an image werkdock already has") {
test("does not load an image werkdock already has") {
givenImageLoaded()
runner.start("./gradlew test", workspace, mapOf("branch" to "main"), repoDir, werkdockBranchConfig())
verify(exactly = 0) { commandRunner.run(match { "import" in it }, any(), any(), any()) }
verify(exactly = 0) { commandRunner.runOrThrow(match { "load" in it }, any(), any(), any()) }
}
@@ -227,7 +209,7 @@ class WerkdockBuildRunnerTest : FunSpec() {
exception.message shouldContain "werkdock.rootfs"
}
test("downloads a URL rootfs once before importing it") {
test("downloads a URL rootfs once before loading it") {
val url = "https://example.test/buildenv.tar.zst"
val downloadTarget =
repoDir
@@ -237,7 +219,7 @@ class WerkdockBuildRunnerTest : FunSpec() {
givenImageMissing()
every { commandRunner.runOrThrow(listOf("curl", "-fsSL", "-o", downloadTarget.toString(), url), repoDir, any(), any()) } returns
GitCommandResult(0, "", "")
every { commandRunner.run(match { "import" in it }, any(), any(), any()) } returns
every { commandRunner.runOrThrow(match { "load" in it }, any(), any(), any()) } returns
GitCommandResult(0, "", "")
runner.start("./gradlew test", workspace, mapOf("branch" to "main"), repoDir, werkdockBranchConfig(rootfs = url))
@@ -246,8 +228,8 @@ class WerkdockBuildRunnerTest : FunSpec() {
commandRunner.runOrThrow(listOf("curl", "-fsSL", "-o", downloadTarget.toString(), url), repoDir, any(), any())
}
verify {
commandRunner.run(
listOf("werkdock", "import", downloadTarget.toString(), "werkator-buildenv-${url.sha12()}"),
commandRunner.runOrThrow(
listOf("werkdock", "load", "-i", downloadTarget.toString(), "--name", "werkator-buildenv-${url.sha12()}"),
repoDir,
any(),
any(),
@@ -109,21 +109,6 @@ class InitCommandTest : FunSpec() {
projectContent shouldContain "repo: my-repo"
}
test("keeps dots in repository names") {
val tempDir = Files.createTempDirectory("werkator-init-test")
initCommand.workingDir = tempDir
every { gitService.getTopLevel(tempDir) } returns tempDir
every { gitService.getOriginUrl(tempDir) } returns "https://git.example.org/mi/michael.hoennig.de.git"
initCommand.run()
val projectConfig = tempDir.resolve(".werkator.yml")
val projectContent = projectConfig.toFile().readText()
projectContent shouldContain "owner: mi"
projectContent shouldContain "repo: michael.hoennig.de"
}
test("does not overwrite existing files") {
val tempDir = Files.createTempDirectory("werkator-init-test")
initCommand.workingDir = tempDir
@@ -267,50 +252,5 @@ class InitCommandTest : FunSpec() {
// This should not throw IllegalArgumentException
initCommand.run()
}
test("--systemd reads the configuration from the repository root, not the current directory") {
val tempDir = Files.createTempDirectory("werkator-init-test")
// written before the run, so `init` keeps it instead of creating a template
tempDir.resolve(".werkator.yml").toFile().writeText(
"server:\n publicBaseUrl: \"https://werkator.example.org/\"\n port: 18099\n",
)
initCommand.workingDir = tempDir
initCommand.systemd = true
initCommand.jarPathResolver = { Paths.get("/home/ci/bin/werkator.jar") }
initCommand.javaExecutableResolver = { Paths.get("/usr/bin/java") }
every { gitService.getTopLevel(tempDir) } returns tempDir
every { gitService.getOriginUrl(tempDir) } returns "https://git.example.org/my-org/my-repo.git"
initCommand.run()
// the host integration is generated only when the root's config has a public base URL
val htaccess = tempDir.resolve(".git/werkator/${SystemdServiceFiles.HTACCESS_NAME}")
htaccess.toFile().shouldExist()
htaccess.toFile().readText() shouldContain "18099"
tempDir.resolve(".git/werkator/${SystemdServiceFiles.MAINTENANCE_PAGE_NAME}").toFile().shouldExist()
}
test("--systemd warns once when the effective configuration cannot be loaded") {
val tempDir = Files.createTempDirectory("werkator-init-test")
val brokenLoader = mockk<de.hoennig.werkator.config.ConfigLoader>()
every { brokenLoader.load(any()) } throws IllegalStateException("gitea.owner is required")
val command = InitCommand(gitService, brokenLoader)
command.workingDir = tempDir
command.systemd = true
command.jarPathResolver = { Paths.get("/home/ci/bin/werkator.jar") }
command.javaExecutableResolver = { Paths.get("/usr/bin/java") }
every { gitService.getTopLevel(tempDir) } returns tempDir
every { gitService.getOriginUrl(tempDir) } returns "https://git.example.org/my-org/my-repo.git"
val console = captureConsole { command.run() }
console.stdout shouldContain "gitea.owner is required"
// the three readers of the configuration must not repeat the warning
console.stdout.windowed("Warning:".length).count { it == "Warning:" } shouldBe 1
// the units are still written with the defaults
tempDir.resolve(".git/werkator/${SystemdServiceFiles.unitName(tempDir)}").toFile().shouldExist()
}
}
}
+15 -63
View File
@@ -44,7 +44,7 @@
# Optional in the env file:
# WERKATOR_INIT_CONFIG the init fragment to apply (repo-init, instance-start)
# WERKATOR_REPO_URL https clone URL of the watched repository
# (default: https://git.javagil.de/mi/werkator.git)
# (default: https://github.com/mhoennig/werkator.git)
# WERKATOR_REPO_DIR directory of the watched repository, absolute or relative to
# WERKATOR_PATH (default: werkator); it also names the systemd
# unit, exactly as `init --systemd` derives it
@@ -122,7 +122,7 @@ require_env WERKATOR_REMOTE WERKATOR_PATH
HOST="$WERKATOR_REMOTE"
TARGET_DIR="$WERKATOR_PATH"
ROOTFS="${WERKATOR_ROOTFS:-$REPO_ROOT/build/werkator-buildenv-trixie-java-go-node.tar.zst}"
REPO_URL="${WERKATOR_REPO_URL:-https://git.javagil.de/mi/werkator.git}"
REPO_URL="${WERKATOR_REPO_URL:-https://github.com/mhoennig/werkator.git}"
# The host layout is three values, not one convention: an installation that grew
# before this script existed puts them elsewhere, and the defaults are exactly what
@@ -332,65 +332,9 @@ instance_update() {
echo "==> Instance updated."
}
# Clones one URL into one directory on the host.
# A private https origin authenticates with the shared `defaults.git.account` /
# `defaults.git.token` of `~/.werkator.yml` (ADR 0009). The whole authenticated
# clone runs in one remote python script: the token is read from the instance
# file on the host and passed to git via a one-shot GIT_ASKPASS script, so it
# appears in neither the local process list nor a repository config.
# Public origins (or an instance without shared credentials) clone anonymously.
clone_repo() {
local url="$1" dest="$2"
if ssh "$HOST" "test -d '$dest/.git'"; then
echo " (already cloned, skipping)"
return 0
fi
case "$url" in
https://*)
# The python helper travels base64-encoded: the clone command itself
# stays a plain `ssh` line, so no `$` inside the script is ever
# expanded by the local shell, and the token never leaves the host.
local helper_b64
helper_b64="$(python3 -c 'import base64,sys; print(base64.b64encode(sys.stdin.read().encode()).decode())' <<'PYEOF_CLONE'
import os, stat, subprocess, sys, tempfile, urllib.parse
url, dest = sys.argv[1], sys.argv[2]
try:
import yaml
cfg = yaml.safe_load(open(os.path.expanduser("~/.werkator.yml"))) or {}
except (FileNotFoundError, ImportError):
cfg = {}
d = (cfg.get("defaults") or {}).get("git") or {}
account, token = d.get("account"), d.get("token")
env = dict(os.environ, GIT_TERMINAL_PROMPT="0")
ask = None
if account and token:
parts = urllib.parse.urlsplit(url)
host = parts.netloc.rsplit("@", 1)[-1]
url = urllib.parse.urlunsplit(parts._replace(netloc=account + "@" + host))
ask = tempfile.NamedTemporaryFile(mode="w", prefix="werkator-clone-askpass-",
suffix=".sh", delete=False)
ask.write("#!/bin/sh\nexec echo \"$WERKATOR_CLONE_TOKEN\"\n")
ask.close()
os.chmod(ask.name, stat.S_IRWXU)
env.update(GIT_ASKPASS=ask.name, WERKATOR_CLONE_TOKEN=token)
try:
subprocess.run(["git", "clone", url, dest], env=env, check=True)
finally:
if ask is not None:
os.unlink(ask.name)
PYEOF_CLONE
)"
ssh "$HOST" "echo '$helper_b64' | base64 -d | python3 - '$url' '$dest'"
;;
*)
ssh "$HOST" "git clone '$url' '$dest'"
;;
esac
}
# Sets up the WATCHED repository: an https clone (a private origin
# authenticates with the shared `defaults.git.*` credentials of
# `~/.werkator.yml`; see `clone_repo`), the werkator init with the instance
# Sets up the WATCHED repository: an anonymous https clone (a private origin
# gets its credentials via git.account/git.token in the machine config that
# `werkator init` creates), the werkator init with the instance fragment
# applied, and the rootfs archive for the sandbox builds. All configuration
# writing is init's — this script transports and invokes (step 23).
repo_init() {
@@ -400,7 +344,11 @@ repo_init() {
ssh "$HOST" "test -x '$WERKATOR_BIN'" || die "no instance on $HOST — run instance-install first"
echo "==> Cloning the watched repository"
clone_repo "$REPO_URL" "$REPO_DIR"
if ssh "$HOST" "test -d '$REPO_DIR/.git'"; then
echo " (already cloned, skipping)"
else
ssh "$HOST" "git clone '$REPO_URL' '$REPO_DIR'"
fi
if [ "$SANDBOX" = "docker" ]; then
echo "==> No rootfs needed (WERKATOR_SANDBOX=docker) — the build image is the repository's own Dockerfile"
@@ -453,7 +401,11 @@ repo_add() {
ssh "$HOST" "test -x '$WERKATOR_BIN'" || die "no instance on $HOST — run instance-install first"
echo "==> Cloning $url as '$name'"
clone_repo "$url" "$SIBLING_DIR/$name"
if ssh "$HOST" "test -d '$SIBLING_DIR/$name/.git'"; then
echo " (already cloned, skipping)"
else
ssh "$HOST" "git clone '$url' '$SIBLING_DIR/$name'"
fi
# The instance fragment carries the sandbox policy (bwrap rootfs and werkdock
# binary). Without it a watched repository builds on the bare host, where the