Compare commits
11
Commits
| Author | SHA1 | Date | |
|---|---|---|---|
|
|
ec07269cca | ||
|
|
b8600bb556 | ||
|
|
0f9f119687 | ||
|
|
5c64391f5f | ||
|
|
cd6f915231 | ||
|
|
092183ca30 | ||
|
|
0761a274db | ||
|
|
5c73c4cc21 | ||
|
|
16543f038b | ||
|
|
3ccc901d1b | ||
|
|
9221550a2a |
@@ -8,6 +8,28 @@ Lightweight, declarative and highly opinionated software build system (CI/CD).
|
||||
- [docs/bootstrapping.md](docs/bootstrapping.md) — initializing a repository with `init`
|
||||
- [docs/deployment.md](docs/deployment.md) — running Werkator as a systemd service behind a reverse proxy
|
||||
|
||||
## Adding a Gitea Repository
|
||||
|
||||
One instance serves several repositories (`docs/deployment.md`, ADR 0009).
|
||||
From the workstation, clone and initialise, then register:
|
||||
|
||||
```bash
|
||||
tools/remote --env-file .env.<instance> werkator repo-add https://gitea.example.org/<owner>/<repo>.git [<name>]
|
||||
```
|
||||
|
||||
It prints the registry entry: add it to `~/.werkator.yml` under `repositories:`, then restart the service.
|
||||
The optional `[<name>]` overrides the directory basename: it becomes the route segment (`/repos/<name>/…`) and the UI switcher entry, so it must be unique.
|
||||
Needed only when the clone directory name is wrong or collides — e.g. `michael.hoennig.de.git` checked out as `michael.hoennig.de`, or two forges serving a repo of the same name.
|
||||
A **public** repository needs nothing else: the clone runs anonymously.
|
||||
A **private** repository needs shared credentials once on the host, in `~/.werkator.yml` of the service user, before cloning:
|
||||
|
||||
```yaml
|
||||
defaults:
|
||||
git:
|
||||
account: <gitea-user>
|
||||
token: <token> # Gitea → Settings → Applications → Generate Token, scope read:repository
|
||||
```
|
||||
|
||||
## Developer Setup
|
||||
|
||||
Source `.envrc` to add `tools/` to your `PATH`, or install [direnv](#direnv) to have this done automatically on `cd`:
|
||||
|
||||
@@ -475,7 +475,9 @@ The section was called `bwrap` and its binary key `bwrap.werkdock` until v1.2.0;
|
||||
`werkdock.rootfs` names the prepared root filesystem archive — a Debian-base rootfs with the build tools (JDK, git, locales, project-specific tooling) built elsewhere, since `debootstrap` is unavailable on the target.
|
||||
It is a local path or an `http(s)` URL; a URL is downloaded once into `.git/werkator/buildenv/`.
|
||||
Build the archive with `tools/build-bwrap-rootfs.sh` on any machine with Docker.
|
||||
The archive is loaded once per source as the werkdock image `werkator-buildenv-<hash>` into werkdock's store (`$WERKDOCK_HOME`, default `~/.werkdock`) — shared by every repository of this OS user; the hash derives from the source string, so a changed `rootfs` loads a fresh image and stale ones can be removed from the store.
|
||||
The archive is imported once per source as the werkdock image `werkator-buildenv-<hash>` into werkdock's store (`$WERKDOCK_HOME`, default `~/.werkdock`) — shared by every repository of this OS user; the hash derives from the source string, so a changed `rootfs` imports a fresh image and stale ones can be removed from the store.
|
||||
Werkator uses `werkdock import ARCHIVE IMAGE` (docker import semantics) and falls back to the older `werkdock load -i ARCHIVE --name IMAGE` when the installed werkdock does not know the verb yet, so Werkator and werkdock can be updated in either order.
|
||||
The image keeps its untagged name (it equals `werkator-buildenv-<hash>:latest` in werkdock's docker-style naming), so an existing store needs no re-import.
|
||||
Per-repo Gradle caches persist in `.git/werkator/buildenv/home`, bound as `/root`.
|
||||
`werkdock.env` adds environment variables inside the sandbox; the environment is otherwise cleared (docker semantics) — the server's environment does not leak in.
|
||||
Files created inside the sandbox are owned by the host user, because uid 0 maps back to the unprivileged webspace user.
|
||||
|
||||
+2
-1
@@ -123,6 +123,7 @@ Adding a repository is editing a registry entry — never a data migration, beca
|
||||
It clones the repository next to the ones already served, runs `init` in it, and **prints** the registry entry.
|
||||
It does not write `~/.werkator.yml`: that file is the instance's own — port, global concurrency, possibly shared credentials — and a script editing it in place would rewrite the operator's configuration behind their back.
|
||||
Cloning and initialising is mechanical; registering is a decision.
|
||||
A private `https` origin authenticates with the shared `defaults.git.account`/`defaults.git.token` of `~/.werkator.yml` (the token travels via a one-shot `GIT_ASKPASS` on the host, never in a URL or process list); enter those once before cloning a private repository — without them only public origins clone.
|
||||
|
||||
4. **Restart** the service; startup recovery re-enqueues what was in flight:
|
||||
|
||||
@@ -341,7 +342,7 @@ tools/remote --env-file .env.mih34 port-forward start # browser tunne
|
||||
|
||||
Layout on the host: the watched repository at `$WERKATOR_PATH/werkator/`, the unpacked runtime at `$WERKATOR_PATH/.werkator/werkator/`, the werkdock binary at `$WERKATOR_PATH/.werkator/bin/werkdock`.
|
||||
That is the default, not a requirement: `WERKATOR_REPO_DIR`, `WERKATOR_INSTALL_DIR` and `WERKATOR_SANDBOX` bend it to an installation that predates the script, see [Updating an Existing Installation](#updating-an-existing-installation).
|
||||
The rootfs archive is loaded once per source into werkdock's image store (`~/.werkdock`), shared by every repository of the user.
|
||||
The rootfs archive is imported once per source into werkdock's image store (`~/.werkdock`), shared by every repository of the user.
|
||||
Fill `git.account`/`git.token` in the machine config when the origin is private, and make the user's services survive logout with `loginctl enable-linger`.
|
||||
|
||||
Updates are one command, refused while a build runs (`FORCE=1` overrides):
|
||||
|
||||
@@ -0,0 +1,42 @@
|
||||
> **WARNING:** This document describes only the change applied in this PR.
|
||||
> It may already be outdated once the next PR is merged.
|
||||
> Historic PR-documentation is not maintained along with new PRs — treat it as a snapshot, not as current documentation.
|
||||
|
||||
## The Problem
|
||||
|
||||
The build for commit `7028ca8` on `main` failed on the mih09 production instance with
|
||||
`BuildExecutorTest > with maxConcurrent 1 a second branch stays PENDING until the first finished`,
|
||||
while a retry of the very same commit passed, and the test passes locally.
|
||||
|
||||
The test was timing-dependent.
|
||||
It started a build for `branch-a` whose build command was `sleep 1`, immediately started a second build for `branch-b`,
|
||||
and then asserted — without any synchronization at all — that `branch-b` was still `PENDING`.
|
||||
|
||||
That assertion only held as long as the test thread reached it within the one second `branch-a` slept.
|
||||
On a shared host under CPU contention the executor can get through `branch-a` entirely (queued, running, slept, succeeded) first,
|
||||
and `branch-b` is then already `RUNNING` or `SUCCESS` when the assertion runs.
|
||||
The failure therefore says nothing about the executor; it is pure scheduling noise that costs a build and a retry every time it hits.
|
||||
|
||||
## Non-Goals
|
||||
|
||||
- No change to production code — `BuildExecutor` is not touched, its queueing behaviour is unchanged.
|
||||
- No sweep of the other timing-sensitive tests in the suite; only the one that actually flaked is fixed.
|
||||
|
||||
## The Solution
|
||||
|
||||
`branch-a` no longer sleeps for a fixed time, it blocks until the test says so:
|
||||
its build command is `until [ -f gate ]; do sleep 0.05; done`, and the build workspace is the test's working directory.
|
||||
|
||||
The test now
|
||||
1. waits (via `eventually`) until `branch-a` is `RUNNING`, so the single executor slot is provably occupied,
|
||||
2. asserts that `branch-b` is `PENDING` — which cannot race anything, because `branch-a` cannot finish before the gate file exists,
|
||||
3. creates the gate file, and only then awaits both builds' `SUCCESS`.
|
||||
|
||||
The assertion on the event transitions (`branch-b` goes `RUNNING` only after `branch-a` reached `SUCCESS`) is unchanged.
|
||||
A blocking gate was chosen over a mocked `BuildRunner` because it keeps the test on the real `ProcessBuildRunner`,
|
||||
so it still covers the actual process handling rather than only the executor's bookkeeping.
|
||||
|
||||
Verified by running `BuildExecutorTest` five times on an idle machine and three more times with twice `nproc` busy-loops saturating the CPU,
|
||||
which is the condition that produced the original failure.
|
||||
|
||||
- [BuildExecutorTest](../../src/test/kotlin/de/hoennig/werkator/build/BuildExecutorTest.kt)
|
||||
@@ -0,0 +1,30 @@
|
||||
> **WARNING:** This document describes only the change applied in this PR.
|
||||
> It may already be outdated once the next PR is merged.
|
||||
> Historic PR-documentation is not maintained along with new PRs — treat it as a snapshot, not as current documentation.
|
||||
|
||||
## The Problem
|
||||
|
||||
The repository's home is `https://git.javagil.de/mi/werkator.git` since the move to the own Gitea instance,
|
||||
but `tools/remote` still defaulted `WERKATOR_REPO_URL` to the GitHub mirror.
|
||||
|
||||
The mih09 production instance was cloned from that mirror and consequently watched a `main` that nobody pushes to any more:
|
||||
it kept reporting the last GitHub state as green while three merged pull requests sat unbuilt on the real `main`.
|
||||
The failure that started this — a flaky test already fixed on Gitea's `main` — could not be re-verified live,
|
||||
because the instance had no way to see the fix.
|
||||
|
||||
## Non-Goals
|
||||
|
||||
- The existing clone on mih09 is not touched by this change; its remote was repointed by hand (`git remote set-url`), and `repo-init` skips an existing clone.
|
||||
- The generic placeholder URLs in `docs/deployment.md` stay as they are — they describe cloning *any* watched repository, not werkator's own.
|
||||
- No decision about mirroring to GitHub; the mirror simply stops being the source an instance builds from.
|
||||
|
||||
## The Solution
|
||||
|
||||
`REPO_URL` in [tools/remote](../../tools/remote) defaults to the Gitea URL, and the usage comment says so.
|
||||
|
||||
The value stays overridable via `WERKATOR_REPO_URL` in the instance's env file,
|
||||
so an installation that deliberately watches a different remote is unaffected.
|
||||
Anonymous HTTPS works against Gitea exactly as it did against GitHub, so no deploy key or token is involved.
|
||||
|
||||
Note that pull requests opened via AGit-Flow create no branch in Gitea,
|
||||
so an instance watching this remote sees `main` only — branch builds require pushing real branches.
|
||||
@@ -0,0 +1,90 @@
|
||||
> **WARNING:** This document describes only the change applied in this PR.
|
||||
> It may already be outdated once the next PR is merged.
|
||||
> Historic PR-documentation is not maintained along with new PRs — treat it as a snapshot, not as current documentation.
|
||||
|
||||
## The Problem
|
||||
|
||||
`init --systemd` generates the host integration — the systemd unit's resource limits, the Apache `.htaccess` and the maintenance page — from the effective configuration.
|
||||
It read that configuration through a helper with two independent defects, both of which fail silently.
|
||||
|
||||
**It swallowed every error.**
|
||||
The load sat in `try { … } catch (_: Exception) { ServerConfig() }`.
|
||||
Any configuration error at all — a missing required field, a malformed layer, a version floor violation — produced a default `ServerConfig` with a blank `publicBaseUrl`.
|
||||
A repository with a broken `.werkator.yml` then looked exactly like one that simply has no public base URL configured:
|
||||
the `.htaccess` and the maintenance page were skipped without a word.
|
||||
This surfaced while verifying [PR#17](2026-09-03-PR%2317-maintenance-page.md) on mih09, where the missing files looked like an unconfigured `publicBaseUrl` and were in fact an unrelated validation error.
|
||||
|
||||
**It read from the wrong directory.**
|
||||
The helper called `configLoader.load(Paths.get("."))` — the process's current directory — while everything else in the command works off the git top level resolved by `GitService.getTopLevel`.
|
||||
`ConfigLoader.loadRaw` resolves the layers directly under the directory it is given and does not walk up to the repository root, so the two agree only when `init` happens to be invoked from the root itself.
|
||||
From a subdirectory the command read another repository's configuration, or none.
|
||||
That also broke `--apply`: the fragment is installed into the repository root deliberately before the systemd files are written, so that its port and limits reach the generated unit, and a current-directory read does not see it.
|
||||
|
||||
`init --systemd` runs during initial deployment setup, which is exactly when a silent wrong answer is most expensive.
|
||||
|
||||
## Non-Goals
|
||||
|
||||
- The fallback itself is kept: a configuration that cannot be loaded is not fatal for `init`, the units are still generated with the defaults.
|
||||
During the very first bootstrap there is legitimately nothing to load yet.
|
||||
- No change to `ConfigLoader`, to the configuration schema, or to any other command.
|
||||
- No sweep for catch-all exception handlers elsewhere in the code base;
|
||||
the two other `catch` blocks in `InitCommand` already print an `Error:` and abort, so they were only checked, not changed.
|
||||
|
||||
## The Scenarios
|
||||
|
||||
### Feature: init reports what it read and where it read it from
|
||||
|
||||
#### Background
|
||||
|
||||
- The *repository root* is the git top level as resolved by `GitService.getTopLevel`, the directory holding `.werkator.yml`, `.git/werkator/.werkator.yml` and an applied fragment.
|
||||
- The *current directory* is the process working directory, which is the repository root only when `init` is invoked there.
|
||||
|
||||
#### Scenario#22.01: A broken configuration is named, not defaulted over
|
||||
|
||||
So that a validation error during deployment setup is not mistaken for an unconfigured installation.
|
||||
|
||||
- **Given** a repository whose effective configuration cannot be loaded
|
||||
- **When** `init --systemd` runs
|
||||
- **Then** the exception message is printed as a warning
|
||||
- **and** the unit files are still generated with the default settings
|
||||
- **and** the warning appears exactly once, although three settings are read from the configuration
|
||||
|
||||
##### Verified by
|
||||
|
||||
- [InitCommandTest: `--systemd warns once when the effective configuration cannot be loaded`](../../src/test/kotlin/de/hoennig/werkator/commands/InitCommandTest.kt)
|
||||
|
||||
#### Scenario#22.02: The configuration is read from the repository root
|
||||
|
||||
So that the generated host integration reflects the repository being initialized, whatever directory `init` was invoked from.
|
||||
|
||||
- **Given** a repository whose root configuration sets `server.publicBaseUrl` and `server.port`
|
||||
- **and** a current directory that is not that repository root
|
||||
- **When** `init --systemd` runs
|
||||
- **Then** the `.htaccess` and the maintenance page are generated
|
||||
- **and** the `.htaccess` proxies to the port from the root configuration
|
||||
|
||||
##### Verified by
|
||||
|
||||
- [InitCommandTest: `--systemd reads the configuration from the repository root, not the current directory`](../../src/test/kotlin/de/hoennig/werkator/commands/InitCommandTest.kt)
|
||||
|
||||
## The Solution
|
||||
|
||||
The catch-all now prints the exception message before falling back:
|
||||
|
||||
```
|
||||
Warning: the effective configuration could not be loaded (<message>)
|
||||
continuing with default server settings — check the generated unit and host files
|
||||
```
|
||||
|
||||
The configuration is read three times while the systemd files are written (`memoryMax`, `tasksMax`, `publicBaseUrl`), which would repeat the warning three times.
|
||||
It is therefore loaded once per run and cached in the command, and the cache is reset at the top of `run()` so a reused instance — the command is a Spring singleton — re-reads.
|
||||
|
||||
The repository root is passed down into the two accessors instead of `Paths.get(".")`.
|
||||
This matches every other caller of `ConfigLoader.load` in the code base, all of which pass an explicit working directory;
|
||||
`InitCommand` was the only one relying on the process's current directory.
|
||||
|
||||
Both fixes are the same failure in two forms — the command answered from a configuration it never actually read — which is why they are in one PR.
|
||||
|
||||
## Additional Changes
|
||||
|
||||
- None.
|
||||
@@ -0,0 +1,92 @@
|
||||
> **WARNING:** This document describes only the change applied in this PR.
|
||||
> It may already be outdated once the next PR is merged.
|
||||
> Historic PR-documentation is not maintained along with new PRs — treat it as a snapshot, not as current documentation.
|
||||
|
||||
## Related Links
|
||||
|
||||
- ADR 0009 — multi-repo instance: the `defaults` block carries shared repository-level keys, read by `ConfigLoader`, never directly by consumers.
|
||||
- `docs/deployment.md` — registry setup: `repo-add` clones, initialises, and prints the registry entry.
|
||||
|
||||
## The Problem
|
||||
|
||||
`tools/remote werkator repo-add <private-https-url>` fails on the host with `could not read Username`.
|
||||
The clone runs anonymously, but the credentials exist only in the instance file's `defaults.git` block — which the script never consults.
|
||||
Registering a private repository therefore needs a manual SSH session today.
|
||||
|
||||
## Non-Goals
|
||||
|
||||
The script still does not write `~/.werkator.yml`: registering stays the operator's decision.
|
||||
No SSH-URL support: the forge is reached over `https` from the host.
|
||||
No new config keys: `defaults.git.account`/`defaults.git.token` already exist.
|
||||
|
||||
## The Scenarios
|
||||
|
||||
### Feature: authenticated clone for private origins
|
||||
|
||||
#### Background
|
||||
|
||||
- The shared credentials live in `~/.werkator.yml` under `defaults.git` (ADR 0009).
|
||||
- Public origins and instances without shared credentials must keep cloning anonymously.
|
||||
|
||||
#### Scenario#000.01: Private https origin clones with shared credentials
|
||||
|
||||
- **Given** `defaults.git.account`/`defaults.git.token` in `~/.werkator.yml` on the host
|
||||
- **When** `tools/remote werkator repo-add <private-https-url>` runs
|
||||
- **Then** the clone authenticates with those credentials and succeeds.
|
||||
|
||||
##### Verified by
|
||||
|
||||
- Manual stub-`git` test: URL carries `account@`, askpass answers the token, `GIT_TERMINAL_PROMPT=0`.
|
||||
|
||||
#### Scenario#000.02: Public origin clones anonymously
|
||||
|
||||
- **Given** no shared credentials (or a public repository)
|
||||
- **When** `repo-add` or `repo-init` runs
|
||||
- **Then** the clone runs exactly as before, without authentication.
|
||||
|
||||
##### Verified by
|
||||
|
||||
- Local helper test against a `file://` origin with and without an instance file.
|
||||
|
||||
#### Scenario#000.03: Token never leaks locally
|
||||
|
||||
- **Given** an authenticated clone
|
||||
- **When** the command runs from the workstation
|
||||
- **Then** the token appears in neither the local process list nor a repository config.
|
||||
|
||||
##### Verified by
|
||||
|
||||
- Code inspection: the token is read on the host and passed via a one-shot `GIT_ASKPASS` script.
|
||||
|
||||
## The Solution
|
||||
|
||||
`tools/remote` gained a `clone_repo` helper used by both `repo-init` and `repo-add`.
|
||||
For `https://` URLs it ships a small Python helper (base64-encoded, so no `$` is expanded locally) to the host.
|
||||
The helper reads `defaults.git.account`/`defaults.git.token` from `~/.werkator.yml`, puts the account into the URL, and hands the token to git via a one-shot `0700` `GIT_ASKPASS` script deleted in `finally`.
|
||||
Without credentials it falls back to the plain anonymous clone; non-`https` URLs clone unchanged.
|
||||
`docs/deployment.md` documents that the shared credentials must exist before cloning a private repository.
|
||||
|
||||
## Open Questions
|
||||
|
||||
- None.
|
||||
|
||||
## Attachments
|
||||
|
||||
### Adding a Gitea repository — example
|
||||
|
||||
From the workstation, clone and initialise, then register:
|
||||
|
||||
```bash
|
||||
tools/remote --env-file .env.<instance> werkator repo-add https://gitea.example.org/<owner>/<repo>.git [<name>]
|
||||
```
|
||||
|
||||
Add the printed entry to `~/.werkator.yml` under `repositories:`, then restart the service.
|
||||
A public repository needs nothing else: the clone runs anonymously.
|
||||
A private repository needs shared credentials once on the host, in `~/.werkator.yml` of the service user, before cloning (token: Gitea → Settings → Applications → Generate Token, scope `read:repository`):
|
||||
|
||||
```yaml
|
||||
defaults:
|
||||
git:
|
||||
account: <gitea-user>
|
||||
token: <token>
|
||||
```
|
||||
@@ -0,0 +1,102 @@
|
||||
> **WARNING:** This document describes only the change applied in this PR.
|
||||
> It may already be outdated once the next PR is merged.
|
||||
> Historic PR-documentation is not maintained along with new PRs — treat it as a snapshot, not as current documentation.
|
||||
|
||||
## Related Links
|
||||
|
||||
- Werkdock `CHANGELOG.md` (repository `mi/werkdock`, Unreleased section) — the release notes this PR adapts to: `import` replaces `load -i --name` for rootfs archives, `load` is reserved for docker/OCI image archives, image names gained docker-style tags.
|
||||
- ADR 0008 — the bwrap build runtime; `docs/configuration.md`, notes on `builds.<name>.werkdock`.
|
||||
|
||||
## The Problem
|
||||
|
||||
Werkdock moves its CLI closer to `docker run --rm`.
|
||||
For the rootfs archives Werkator uses, the verb is now `werkdock import ARCHIVE IMAGE` (docker import semantics).
|
||||
`werkdock load -i ARCHIVE --name IMAGE` still works but is a compatibility path with a note on stderr, and is announced to go away once Werkator has switched.
|
||||
Werkator ships the werkdock binary with its deployment, but nothing forces the two to be updated together: an installation may run a new Werkator against an older werkdock for a while, or the other way round.
|
||||
|
||||
## Non-Goals
|
||||
|
||||
- No switch of the existence check from `werkdock images` to `werkdock inspect`: `images` prints the bare name for untagged images on every werkdock version, the exact-line match keeps working, and `inspect` would tie Werkator to the new werkdock.
|
||||
- No tagged image name (`werkator-buildenv:<hash>`): it would re-import every build environment on the webspace and orphan the old image, for no functional gain today.
|
||||
- None of the new `run` flags (`--mount`, `--entrypoint`, `--network host`): Werkator's invocation needs none of them.
|
||||
|
||||
## The Scenarios
|
||||
|
||||
### Feature: rootfs archives are imported with werkdock's docker-shaped verb
|
||||
|
||||
#### Background
|
||||
|
||||
- The build environment is a rootfs archive named by `builds.<name>.werkdock.rootfs`.
|
||||
- Werkator creates the werkdock image `werkator-buildenv-<hash>` from it once per source.
|
||||
- Werkdock answers an unknown verb with `werkdock: unknown command "import"` and exit code 125, its code for its own errors.
|
||||
|
||||
#### Scenario#000.01: A missing image is imported with `werkdock import`
|
||||
|
||||
So that Werkator uses the verb werkdock names for rootfs archives, and the deprecated path can be removed on werkdock's side.
|
||||
|
||||
- **Given** `werkdock images` does not list the image
|
||||
- **When** a build starts
|
||||
- **Then** Werkator runs `werkdock import ARCHIVE werkator-buildenv-<hash>`
|
||||
- **and** runs no `werkdock load`.
|
||||
|
||||
##### Verified by
|
||||
|
||||
- [WerkdockBuildRunnerTest "imports the image once when werkdock does not know it yet"](../../src/test/kotlin/de/hoennig/werkator/build/WerkdockBuildRunnerTest.kt)
|
||||
- [WerkdockBuildRunnerTest "downloads a URL rootfs once before importing it"](../../src/test/kotlin/de/hoennig/werkator/build/WerkdockBuildRunnerTest.kt)
|
||||
|
||||
#### Scenario#000.02: An older werkdock without the verb still works
|
||||
|
||||
So that Werkator and werkdock can be updated in either order.
|
||||
|
||||
- **Given** the installed werkdock answers `import` with `unknown command` and exit 125
|
||||
- **When** a build starts with a missing image
|
||||
- **Then** Werkator falls back to `werkdock load -i ARCHIVE --name werkator-buildenv-<hash>`
|
||||
- **and** logs the fallback.
|
||||
|
||||
##### Verified by
|
||||
|
||||
- [WerkdockBuildRunnerTest "falls back to load -i --name on a werkdock without the import verb"](../../src/test/kotlin/de/hoennig/werkator/build/WerkdockBuildRunnerTest.kt)
|
||||
|
||||
#### Scenario#000.03: A real import failure is not masked by the fallback
|
||||
|
||||
So that a broken archive fails the build with werkdock's message, as before.
|
||||
|
||||
- **Given** `werkdock import` fails for any other reason (any other exit code, or exit 125 without `unknown command`)
|
||||
- **When** a build starts with a missing image
|
||||
- **Then** the build fails with that command's output
|
||||
- **and** no `werkdock load` runs.
|
||||
|
||||
##### Verified by
|
||||
|
||||
- [WerkdockBuildRunnerTest "propagates an import failure that is not a missing verb"](../../src/test/kotlin/de/hoennig/werkator/build/WerkdockBuildRunnerTest.kt)
|
||||
|
||||
#### Scenario#000.04: An existing image is neither imported nor loaded
|
||||
|
||||
- **Given** `werkdock images` lists the image
|
||||
- **When** a build starts
|
||||
- **Then** neither `import` nor `load` runs.
|
||||
|
||||
##### Verified by
|
||||
|
||||
- [WerkdockBuildRunnerTest "does not import an image werkdock already has"](../../src/test/kotlin/de/hoennig/werkator/build/WerkdockBuildRunnerTest.kt)
|
||||
|
||||
## The Solution
|
||||
|
||||
`WerkdockBuildRunner.ensureImage` calls the new `importImage`: `werkdock import ARCHIVE IMAGE` through the non-throwing `run`, then either returns, falls back to `load -i --name` on exactly the unknown-verb signature (exit 125 plus `unknown command` on stderr), or rethrows the import's result as a `GitCommandException` — the same exception and message the old code produced.
|
||||
The fallback is keyed on werkdock's own error signature rather than on a version string, because werkdock's version output does not yet distinguish the two builds.
|
||||
|
||||
## Open Questions
|
||||
|
||||
- When to drop the fallback: once every installation runs a werkdock with `import`, the `load -i --name` branch and its test go, and werkdock can remove the compatibility path.
|
||||
|
||||
## Additional Changes
|
||||
|
||||
- `docs/configuration.md` and `docs/deployment.md` say "imported" where they said "loaded", and name the fallback and the reason the image name stays untagged.
|
||||
|
||||
## Prerequisite PRs
|
||||
|
||||
- The werkdock change that introduces `import` (repository `mi/werkdock`, CHANGELOG Unreleased); against an older werkdock the fallback path runs.
|
||||
|
||||
## Follow-up PRs
|
||||
|
||||
- Remove the fallback (see Open Questions).
|
||||
@@ -2,6 +2,7 @@ package de.hoennig.werkator.build
|
||||
|
||||
import de.hoennig.werkator.config.BranchConfig
|
||||
import de.hoennig.werkator.config.WerkdockConfig
|
||||
import de.hoennig.werkator.git.GitCommandException
|
||||
import de.hoennig.werkator.git.GitCommandRunner
|
||||
import org.slf4j.LoggerFactory
|
||||
import org.springframework.stereotype.Component
|
||||
@@ -16,9 +17,9 @@ import java.security.MessageDigest
|
||||
* it shells out to the `werkdock` CLI (`werkdock.binary`, default via PATH) — the same
|
||||
* pattern as git and docker, CLI, no library.
|
||||
*
|
||||
* The rootfs archive becomes a werkdock *image*, loaded once per source
|
||||
* The rootfs archive becomes a werkdock *image*, imported once per source
|
||||
* (`werkator-buildenv-<hash>`, the hash over the source string, so a changed source
|
||||
* loads a fresh image) into werkdock's own store (`$WERKDOCK_HOME`, default
|
||||
* imports a fresh image) into werkdock's own store (`$WERKDOCK_HOME`, default
|
||||
* `~/.werkdock`) — shared by every repository of this OS user, unlike the old
|
||||
* per-repo unpack. Only the download cache for URL sources and the persistent
|
||||
* toolchain home (bound to `/root` for Gradle/Go caches) stay under
|
||||
@@ -66,9 +67,11 @@ class WerkdockBuildRunner(
|
||||
}
|
||||
|
||||
/**
|
||||
* Loads the rootfs archive into the werkdock image store once per source.
|
||||
* Imports the rootfs archive into the werkdock image store once per source.
|
||||
* `werkdock images` answers existence through the CLI, like `docker image
|
||||
* inspect` does for the Docker runner.
|
||||
* inspect` does for the Docker runner; it prints the bare name for an untagged
|
||||
* image on every werkdock version, so the exact-line match holds across the
|
||||
* werkdock upgrade that introduced tags.
|
||||
*/
|
||||
private fun ensureImage(
|
||||
werkdock: String,
|
||||
@@ -84,7 +87,33 @@ class WerkdockBuildRunner(
|
||||
val envDir = repoDir.resolve(BUILDENV_DIR).resolve(sourceKey(sandbox.rootfs))
|
||||
Files.createDirectories(envDir)
|
||||
val archive = localArchive(sandbox.rootfs, envDir, repoDir, onAuxProcess)
|
||||
log.info("loading build environment {} as werkdock image {}", sandbox.rootfs, image)
|
||||
log.info("importing build environment {} as werkdock image {}", sandbox.rootfs, image)
|
||||
importImage(werkdock, archive, image, repoDir, onAuxProcess)
|
||||
}
|
||||
|
||||
/**
|
||||
* `werkdock import ARCHIVE IMAGE` (docker import semantics) creates the image from a
|
||||
* rootfs archive. A werkdock that predates the verb answers `unknown command` with
|
||||
* exit 125; then the older `load -i ARCHIVE --name IMAGE` does the same job, so an
|
||||
* installation can update Werkator and werkdock in either order. Any other failure
|
||||
* is the import's own and propagates as it did before.
|
||||
*/
|
||||
private fun importImage(
|
||||
werkdock: String,
|
||||
archive: String,
|
||||
image: String,
|
||||
repoDir: Path,
|
||||
onAuxProcess: (Process) -> Unit,
|
||||
) {
|
||||
val importCommand = listOf(werkdock, "import", archive, image)
|
||||
val imported = commandRunner.run(importCommand, repoDir, onProcess = onAuxProcess)
|
||||
if (imported.isSuccess) {
|
||||
return
|
||||
}
|
||||
if (imported.exitCode != WERKDOCK_CLI_ERROR || "unknown command" !in imported.stderr) {
|
||||
throw GitCommandException(importCommand, imported)
|
||||
}
|
||||
log.info("this werkdock has no import verb yet, falling back to load -i --name")
|
||||
commandRunner.runOrThrow(
|
||||
listOf(werkdock, "load", "-i", archive, "--name", image),
|
||||
repoDir,
|
||||
@@ -209,5 +238,8 @@ class WerkdockBuildRunner(
|
||||
companion object {
|
||||
const val BUILDENV_DIR = ".git/werkator/buildenv"
|
||||
const val HOME_DIR = "home"
|
||||
|
||||
/** werkdock's exit code for its own errors (docker's 125), as opposed to the sandboxed command's. */
|
||||
const val WERKDOCK_CLI_ERROR = 125
|
||||
}
|
||||
}
|
||||
|
||||
@@ -48,6 +48,7 @@ class InitCommand(
|
||||
internal var javaExecutableResolver: () -> Path = { Paths.get(System.getProperty("java.home"), "bin", "java") }
|
||||
|
||||
override fun run() {
|
||||
cachedServerConfig = null
|
||||
val normalizedWorkingDir = workingDir.toAbsolutePath().normalize()
|
||||
val root =
|
||||
try {
|
||||
@@ -88,7 +89,7 @@ class InitCommand(
|
||||
if (url == null) return DetectedValues()
|
||||
|
||||
if (url.startsWith("http")) {
|
||||
val regex = Regex("""https?://(?:([^@]+)@)?([^/]+)/([^/]+)/([^/.]+)(?:\.git)?""")
|
||||
val regex = Regex("""https?://(?:([^@]+)@)?([^/]+)/([^/]+)/(.+?)(?:\.git)?$""")
|
||||
val match = regex.find(url)
|
||||
if (match != null) {
|
||||
val (user, host, owner, repo) = match.destructured
|
||||
@@ -101,7 +102,7 @@ class InitCommand(
|
||||
}
|
||||
} else if (url.contains("@") && url.contains(":")) {
|
||||
// Assume SSH: git@host:owner/repo.git
|
||||
val regex = Regex("""([^@]+)@([^:]+):([^/]+)/([^/.]+)(?:\.git)?""")
|
||||
val regex = Regex("""([^@]+)@([^:]+):([^/]+)/(.+?)(?:\.git)?$""")
|
||||
val match = regex.find(url)
|
||||
if (match != null) {
|
||||
val (_, host, owner, repo) = match.destructured
|
||||
@@ -296,14 +297,31 @@ class InitCommand(
|
||||
* already loadable (re-running `init --systemd` on an installed instance); during
|
||||
* the very first bootstrap they stay unset and the defaults (no directives) apply.
|
||||
*/
|
||||
private fun loadedSystemdConfig(): de.hoennig.werkator.config.SystemdConfig = loadedServerConfig().systemd
|
||||
private fun loadedSystemdConfig(root: Path): de.hoennig.werkator.config.SystemdConfig = loadedServerConfig(root).systemd
|
||||
|
||||
private fun loadedServerConfig(): de.hoennig.werkator.config.ServerConfig =
|
||||
/** Loaded once per run, so a broken configuration is reported once and not per caller. */
|
||||
private var cachedServerConfig: de.hoennig.werkator.config.ServerConfig? = null
|
||||
|
||||
/**
|
||||
* Read from the repository root like every other file this command touches — the
|
||||
* layers sit there, not in whatever directory the process happens to run in, and
|
||||
* an applied fragment must reach the generated unit even when `init` is invoked
|
||||
* from a subdirectory.
|
||||
*
|
||||
* A configuration error here is not fatal — the units are still generated with defaults —
|
||||
* but it must not pass for "nothing configured": without the warning a broken `.werkator.yml`
|
||||
* looks exactly like an unset `publicBaseUrl` and the host integration is skipped silently.
|
||||
*/
|
||||
private fun loadedServerConfig(root: Path): de.hoennig.werkator.config.ServerConfig =
|
||||
cachedServerConfig ?: run {
|
||||
try {
|
||||
configLoader.load(Paths.get(".")).server
|
||||
} catch (_: Exception) {
|
||||
configLoader.load(root).server
|
||||
} catch (e: Exception) {
|
||||
println("Warning: the effective configuration could not be loaded (${e.message})")
|
||||
println(" continuing with default server settings — check the generated unit and host files")
|
||||
de.hoennig.werkator.config
|
||||
.ServerConfig()
|
||||
}.also { cachedServerConfig = it }
|
||||
}
|
||||
|
||||
private fun createSystemdFiles(
|
||||
@@ -327,8 +345,8 @@ class InitCommand(
|
||||
javaExecutable = javaExecutableResolver(),
|
||||
jarPath = jarPath,
|
||||
envFile = envFile,
|
||||
memoryMax = loadedSystemdConfig().memoryMax,
|
||||
tasksMax = loadedSystemdConfig().tasksMax,
|
||||
memoryMax = loadedSystemdConfig(root).memoryMax,
|
||||
tasksMax = loadedSystemdConfig(root).tasksMax,
|
||||
),
|
||||
)
|
||||
println("created ${unitFile.toFile().relativeTo(normalizedWorkingDir.toFile())}")
|
||||
@@ -351,7 +369,7 @@ class InitCommand(
|
||||
|
||||
// generated host integration like the units: only meaningful behind a web
|
||||
// frontend, so it needs a public base URL; unused elsewhere and harmless
|
||||
val server = loadedServerConfig()
|
||||
val server = loadedServerConfig(root)
|
||||
if (server.publicBaseUrl.isNotBlank()) {
|
||||
val htaccessFile = werkatorDir.resolve(SystemdServiceFiles.HTACCESS_NAME)
|
||||
htaccessFile.toFile().writeText(SystemdServiceFiles.htaccessContent(server.port))
|
||||
|
||||
@@ -505,6 +505,8 @@ class BuildExecutorTest : FunSpec() {
|
||||
}
|
||||
|
||||
test("with maxConcurrent 1 a second branch stays PENDING until the first finished") {
|
||||
// branch-a blocks on a gate file the test creates, so the PENDING assertion
|
||||
// below cannot race the first build finishing on a loaded machine
|
||||
val h =
|
||||
Harness(
|
||||
"""
|
||||
@@ -512,7 +514,7 @@ class BuildExecutorTest : FunSpec() {
|
||||
maxConcurrent: 1
|
||||
branches:
|
||||
branch-a:
|
||||
buildCommand: "sleep 1"
|
||||
buildCommand: "until [ -f gate ]; do sleep 0.05; done"
|
||||
cleanCommand: ""
|
||||
branch-b:
|
||||
buildCommand: "echo ok"
|
||||
@@ -523,8 +525,13 @@ class BuildExecutorTest : FunSpec() {
|
||||
h.executor.startBuild(h.repo, "branch-a", "sha-a")
|
||||
h.executor.startBuild(h.repo, "branch-b", "sha-b")
|
||||
|
||||
eventually(30.seconds) {
|
||||
h.repository.latestFor("branch-a")?.status shouldBe BuildStatus.RUNNING
|
||||
}
|
||||
h.repository.latestFor("branch-b")?.status shouldBe BuildStatus.PENDING
|
||||
|
||||
Files.createFile(h.workingDir.resolve("gate"))
|
||||
|
||||
awaitStatus(h, "branch-b", BuildStatus.SUCCESS)
|
||||
awaitStatus(h, "branch-a", BuildStatus.SUCCESS)
|
||||
val transitions = h.events.map { it.result.branch to it.result.status }
|
||||
|
||||
@@ -2,6 +2,7 @@ package de.hoennig.werkator.build
|
||||
|
||||
import de.hoennig.werkator.config.BranchConfig
|
||||
import de.hoennig.werkator.config.WerkdockConfig
|
||||
import de.hoennig.werkator.git.GitCommandException
|
||||
import de.hoennig.werkator.git.GitCommandResult
|
||||
import de.hoennig.werkator.git.GitCommandRunner
|
||||
import io.kotest.assertions.throwables.shouldThrow
|
||||
@@ -37,7 +38,11 @@ class WerkdockBuildRunnerTest : FunSpec() {
|
||||
|
||||
private fun imageName(rootfs: String = "/srv/buildenv.tar.zst"): String = "werkator-buildenv-${rootfs.sha12()}"
|
||||
|
||||
/** The image is already loaded: `werkdock images` lists it, so no load runs. */
|
||||
private fun importCommand(): List<String> = listOf("werkdock", "import", "/srv/buildenv.tar.zst", imageName())
|
||||
|
||||
private fun loadCommand(): List<String> = listOf("werkdock", "load", "-i", "/srv/buildenv.tar.zst", "--name", imageName())
|
||||
|
||||
/** The image is already there: `werkdock images` lists it, so no import runs. */
|
||||
private fun givenImageLoaded(rootfs: String = "/srv/buildenv.tar.zst") {
|
||||
every { commandRunner.runOrThrow(listOf("werkdock", "images"), repoDir, any(), any()) } returns
|
||||
GitCommandResult(0, imageName(rootfs) + "\n", "")
|
||||
@@ -88,34 +93,47 @@ class WerkdockBuildRunnerTest : FunSpec() {
|
||||
)
|
||||
}
|
||||
|
||||
test("loads the image once when werkdock does not know it yet") {
|
||||
test("imports the image once when werkdock does not know it yet") {
|
||||
givenImageMissing()
|
||||
every {
|
||||
commandRunner.runOrThrow(
|
||||
listOf("werkdock", "load", "-i", "/srv/buildenv.tar.zst", "--name", imageName()),
|
||||
repoDir,
|
||||
any(),
|
||||
any(),
|
||||
)
|
||||
} returns GitCommandResult(0, "", "")
|
||||
every { commandRunner.run(importCommand(), repoDir, any(), any()) } returns GitCommandResult(0, "", "")
|
||||
|
||||
runner.start("./gradlew test", workspace, mapOf("branch" to "main"), repoDir, werkdockBranchConfig())
|
||||
|
||||
verify {
|
||||
commandRunner.runOrThrow(
|
||||
listOf("werkdock", "load", "-i", "/srv/buildenv.tar.zst", "--name", imageName()),
|
||||
repoDir,
|
||||
any(),
|
||||
any(),
|
||||
)
|
||||
}
|
||||
verify { commandRunner.run(importCommand(), repoDir, any(), any()) }
|
||||
verify(exactly = 0) { commandRunner.runOrThrow(match { "load" in it }, any(), any(), any()) }
|
||||
}
|
||||
|
||||
test("does not load an image werkdock already has") {
|
||||
test("falls back to load -i --name on a werkdock without the import verb") {
|
||||
givenImageMissing()
|
||||
every { commandRunner.run(importCommand(), repoDir, any(), any()) } returns
|
||||
GitCommandResult(125, "", "werkdock: unknown command \"import\"\n")
|
||||
every { commandRunner.runOrThrow(loadCommand(), repoDir, any(), any()) } returns GitCommandResult(0, "", "")
|
||||
|
||||
runner.start("./gradlew test", workspace, mapOf("branch" to "main"), repoDir, werkdockBranchConfig())
|
||||
|
||||
verify { commandRunner.runOrThrow(loadCommand(), repoDir, any(), any()) }
|
||||
}
|
||||
|
||||
test("propagates an import failure that is not a missing verb") {
|
||||
givenImageMissing()
|
||||
every { commandRunner.run(importCommand(), repoDir, any(), any()) } returns
|
||||
GitCommandResult(125, "", "werkdock: unpacking /srv/buildenv.tar.zst failed\n")
|
||||
|
||||
val exception =
|
||||
shouldThrow<GitCommandException> {
|
||||
runner.start("./gradlew test", workspace, mapOf("branch" to "main"), repoDir, werkdockBranchConfig())
|
||||
}
|
||||
|
||||
exception.message shouldContain "unpacking"
|
||||
verify(exactly = 0) { commandRunner.runOrThrow(match { "load" in it }, any(), any(), any()) }
|
||||
}
|
||||
|
||||
test("does not import an image werkdock already has") {
|
||||
givenImageLoaded()
|
||||
|
||||
runner.start("./gradlew test", workspace, mapOf("branch" to "main"), repoDir, werkdockBranchConfig())
|
||||
|
||||
verify(exactly = 0) { commandRunner.run(match { "import" in it }, any(), any(), any()) }
|
||||
verify(exactly = 0) { commandRunner.runOrThrow(match { "load" in it }, any(), any(), any()) }
|
||||
}
|
||||
|
||||
@@ -209,7 +227,7 @@ class WerkdockBuildRunnerTest : FunSpec() {
|
||||
exception.message shouldContain "werkdock.rootfs"
|
||||
}
|
||||
|
||||
test("downloads a URL rootfs once before loading it") {
|
||||
test("downloads a URL rootfs once before importing it") {
|
||||
val url = "https://example.test/buildenv.tar.zst"
|
||||
val downloadTarget =
|
||||
repoDir
|
||||
@@ -219,7 +237,7 @@ class WerkdockBuildRunnerTest : FunSpec() {
|
||||
givenImageMissing()
|
||||
every { commandRunner.runOrThrow(listOf("curl", "-fsSL", "-o", downloadTarget.toString(), url), repoDir, any(), any()) } returns
|
||||
GitCommandResult(0, "", "")
|
||||
every { commandRunner.runOrThrow(match { "load" in it }, any(), any(), any()) } returns
|
||||
every { commandRunner.run(match { "import" in it }, any(), any(), any()) } returns
|
||||
GitCommandResult(0, "", "")
|
||||
|
||||
runner.start("./gradlew test", workspace, mapOf("branch" to "main"), repoDir, werkdockBranchConfig(rootfs = url))
|
||||
@@ -228,8 +246,8 @@ class WerkdockBuildRunnerTest : FunSpec() {
|
||||
commandRunner.runOrThrow(listOf("curl", "-fsSL", "-o", downloadTarget.toString(), url), repoDir, any(), any())
|
||||
}
|
||||
verify {
|
||||
commandRunner.runOrThrow(
|
||||
listOf("werkdock", "load", "-i", downloadTarget.toString(), "--name", "werkator-buildenv-${url.sha12()}"),
|
||||
commandRunner.run(
|
||||
listOf("werkdock", "import", downloadTarget.toString(), "werkator-buildenv-${url.sha12()}"),
|
||||
repoDir,
|
||||
any(),
|
||||
any(),
|
||||
|
||||
@@ -109,6 +109,21 @@ class InitCommandTest : FunSpec() {
|
||||
projectContent shouldContain "repo: my-repo"
|
||||
}
|
||||
|
||||
test("keeps dots in repository names") {
|
||||
val tempDir = Files.createTempDirectory("werkator-init-test")
|
||||
initCommand.workingDir = tempDir
|
||||
|
||||
every { gitService.getTopLevel(tempDir) } returns tempDir
|
||||
every { gitService.getOriginUrl(tempDir) } returns "https://git.example.org/mi/michael.hoennig.de.git"
|
||||
|
||||
initCommand.run()
|
||||
|
||||
val projectConfig = tempDir.resolve(".werkator.yml")
|
||||
val projectContent = projectConfig.toFile().readText()
|
||||
projectContent shouldContain "owner: mi"
|
||||
projectContent shouldContain "repo: michael.hoennig.de"
|
||||
}
|
||||
|
||||
test("does not overwrite existing files") {
|
||||
val tempDir = Files.createTempDirectory("werkator-init-test")
|
||||
initCommand.workingDir = tempDir
|
||||
@@ -252,5 +267,50 @@ class InitCommandTest : FunSpec() {
|
||||
// This should not throw IllegalArgumentException
|
||||
initCommand.run()
|
||||
}
|
||||
|
||||
test("--systemd reads the configuration from the repository root, not the current directory") {
|
||||
val tempDir = Files.createTempDirectory("werkator-init-test")
|
||||
// written before the run, so `init` keeps it instead of creating a template
|
||||
tempDir.resolve(".werkator.yml").toFile().writeText(
|
||||
"server:\n publicBaseUrl: \"https://werkator.example.org/\"\n port: 18099\n",
|
||||
)
|
||||
initCommand.workingDir = tempDir
|
||||
initCommand.systemd = true
|
||||
initCommand.jarPathResolver = { Paths.get("/home/ci/bin/werkator.jar") }
|
||||
initCommand.javaExecutableResolver = { Paths.get("/usr/bin/java") }
|
||||
|
||||
every { gitService.getTopLevel(tempDir) } returns tempDir
|
||||
every { gitService.getOriginUrl(tempDir) } returns "https://git.example.org/my-org/my-repo.git"
|
||||
|
||||
initCommand.run()
|
||||
|
||||
// the host integration is generated only when the root's config has a public base URL
|
||||
val htaccess = tempDir.resolve(".git/werkator/${SystemdServiceFiles.HTACCESS_NAME}")
|
||||
htaccess.toFile().shouldExist()
|
||||
htaccess.toFile().readText() shouldContain "18099"
|
||||
tempDir.resolve(".git/werkator/${SystemdServiceFiles.MAINTENANCE_PAGE_NAME}").toFile().shouldExist()
|
||||
}
|
||||
|
||||
test("--systemd warns once when the effective configuration cannot be loaded") {
|
||||
val tempDir = Files.createTempDirectory("werkator-init-test")
|
||||
val brokenLoader = mockk<de.hoennig.werkator.config.ConfigLoader>()
|
||||
every { brokenLoader.load(any()) } throws IllegalStateException("gitea.owner is required")
|
||||
val command = InitCommand(gitService, brokenLoader)
|
||||
command.workingDir = tempDir
|
||||
command.systemd = true
|
||||
command.jarPathResolver = { Paths.get("/home/ci/bin/werkator.jar") }
|
||||
command.javaExecutableResolver = { Paths.get("/usr/bin/java") }
|
||||
|
||||
every { gitService.getTopLevel(tempDir) } returns tempDir
|
||||
every { gitService.getOriginUrl(tempDir) } returns "https://git.example.org/my-org/my-repo.git"
|
||||
|
||||
val console = captureConsole { command.run() }
|
||||
|
||||
console.stdout shouldContain "gitea.owner is required"
|
||||
// the three readers of the configuration must not repeat the warning
|
||||
console.stdout.windowed("Warning:".length).count { it == "Warning:" } shouldBe 1
|
||||
// the units are still written with the defaults
|
||||
tempDir.resolve(".git/werkator/${SystemdServiceFiles.unitName(tempDir)}").toFile().shouldExist()
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
+63
-15
@@ -44,7 +44,7 @@
|
||||
# Optional in the env file:
|
||||
# WERKATOR_INIT_CONFIG the init fragment to apply (repo-init, instance-start)
|
||||
# WERKATOR_REPO_URL https clone URL of the watched repository
|
||||
# (default: https://github.com/mhoennig/werkator.git)
|
||||
# (default: https://git.javagil.de/mi/werkator.git)
|
||||
# WERKATOR_REPO_DIR directory of the watched repository, absolute or relative to
|
||||
# WERKATOR_PATH (default: werkator); it also names the systemd
|
||||
# unit, exactly as `init --systemd` derives it
|
||||
@@ -122,7 +122,7 @@ require_env WERKATOR_REMOTE WERKATOR_PATH
|
||||
HOST="$WERKATOR_REMOTE"
|
||||
TARGET_DIR="$WERKATOR_PATH"
|
||||
ROOTFS="${WERKATOR_ROOTFS:-$REPO_ROOT/build/werkator-buildenv-trixie-java-go-node.tar.zst}"
|
||||
REPO_URL="${WERKATOR_REPO_URL:-https://github.com/mhoennig/werkator.git}"
|
||||
REPO_URL="${WERKATOR_REPO_URL:-https://git.javagil.de/mi/werkator.git}"
|
||||
|
||||
# The host layout is three values, not one convention: an installation that grew
|
||||
# before this script existed puts them elsewhere, and the defaults are exactly what
|
||||
@@ -332,9 +332,65 @@ instance_update() {
|
||||
echo "==> Instance updated."
|
||||
}
|
||||
|
||||
# Sets up the WATCHED repository: an anonymous https clone (a private origin
|
||||
# gets its credentials via git.account/git.token in the machine config that
|
||||
# `werkator init` creates), the werkator init with the instance fragment
|
||||
# Clones one URL into one directory on the host.
|
||||
# A private https origin authenticates with the shared `defaults.git.account` /
|
||||
# `defaults.git.token` of `~/.werkator.yml` (ADR 0009). The whole authenticated
|
||||
# clone runs in one remote python script: the token is read from the instance
|
||||
# file on the host and passed to git via a one-shot GIT_ASKPASS script, so it
|
||||
# appears in neither the local process list nor a repository config.
|
||||
# Public origins (or an instance without shared credentials) clone anonymously.
|
||||
clone_repo() {
|
||||
local url="$1" dest="$2"
|
||||
if ssh "$HOST" "test -d '$dest/.git'"; then
|
||||
echo " (already cloned, skipping)"
|
||||
return 0
|
||||
fi
|
||||
case "$url" in
|
||||
https://*)
|
||||
# The python helper travels base64-encoded: the clone command itself
|
||||
# stays a plain `ssh` line, so no `$` inside the script is ever
|
||||
# expanded by the local shell, and the token never leaves the host.
|
||||
local helper_b64
|
||||
helper_b64="$(python3 -c 'import base64,sys; print(base64.b64encode(sys.stdin.read().encode()).decode())' <<'PYEOF_CLONE'
|
||||
import os, stat, subprocess, sys, tempfile, urllib.parse
|
||||
url, dest = sys.argv[1], sys.argv[2]
|
||||
try:
|
||||
import yaml
|
||||
cfg = yaml.safe_load(open(os.path.expanduser("~/.werkator.yml"))) or {}
|
||||
except (FileNotFoundError, ImportError):
|
||||
cfg = {}
|
||||
d = (cfg.get("defaults") or {}).get("git") or {}
|
||||
account, token = d.get("account"), d.get("token")
|
||||
env = dict(os.environ, GIT_TERMINAL_PROMPT="0")
|
||||
ask = None
|
||||
if account and token:
|
||||
parts = urllib.parse.urlsplit(url)
|
||||
host = parts.netloc.rsplit("@", 1)[-1]
|
||||
url = urllib.parse.urlunsplit(parts._replace(netloc=account + "@" + host))
|
||||
ask = tempfile.NamedTemporaryFile(mode="w", prefix="werkator-clone-askpass-",
|
||||
suffix=".sh", delete=False)
|
||||
ask.write("#!/bin/sh\nexec echo \"$WERKATOR_CLONE_TOKEN\"\n")
|
||||
ask.close()
|
||||
os.chmod(ask.name, stat.S_IRWXU)
|
||||
env.update(GIT_ASKPASS=ask.name, WERKATOR_CLONE_TOKEN=token)
|
||||
try:
|
||||
subprocess.run(["git", "clone", url, dest], env=env, check=True)
|
||||
finally:
|
||||
if ask is not None:
|
||||
os.unlink(ask.name)
|
||||
PYEOF_CLONE
|
||||
)"
|
||||
ssh "$HOST" "echo '$helper_b64' | base64 -d | python3 - '$url' '$dest'"
|
||||
;;
|
||||
*)
|
||||
ssh "$HOST" "git clone '$url' '$dest'"
|
||||
;;
|
||||
esac
|
||||
}
|
||||
|
||||
# Sets up the WATCHED repository: an https clone (a private origin
|
||||
# authenticates with the shared `defaults.git.*` credentials of
|
||||
# `~/.werkator.yml`; see `clone_repo`), the werkator init with the instance
|
||||
# applied, and the rootfs archive for the sandbox builds. All configuration
|
||||
# writing is init's — this script transports and invokes (step 23).
|
||||
repo_init() {
|
||||
@@ -344,11 +400,7 @@ repo_init() {
|
||||
ssh "$HOST" "test -x '$WERKATOR_BIN'" || die "no instance on $HOST — run instance-install first"
|
||||
|
||||
echo "==> Cloning the watched repository"
|
||||
if ssh "$HOST" "test -d '$REPO_DIR/.git'"; then
|
||||
echo " (already cloned, skipping)"
|
||||
else
|
||||
ssh "$HOST" "git clone '$REPO_URL' '$REPO_DIR'"
|
||||
fi
|
||||
clone_repo "$REPO_URL" "$REPO_DIR"
|
||||
|
||||
if [ "$SANDBOX" = "docker" ]; then
|
||||
echo "==> No rootfs needed (WERKATOR_SANDBOX=docker) — the build image is the repository's own Dockerfile"
|
||||
@@ -401,11 +453,7 @@ repo_add() {
|
||||
ssh "$HOST" "test -x '$WERKATOR_BIN'" || die "no instance on $HOST — run instance-install first"
|
||||
|
||||
echo "==> Cloning $url as '$name'"
|
||||
if ssh "$HOST" "test -d '$SIBLING_DIR/$name/.git'"; then
|
||||
echo " (already cloned, skipping)"
|
||||
else
|
||||
ssh "$HOST" "git clone '$url' '$SIBLING_DIR/$name'"
|
||||
fi
|
||||
clone_repo "$url" "$SIBLING_DIR/$name"
|
||||
|
||||
# The instance fragment carries the sandbox policy (bwrap rootfs and werkdock
|
||||
# binary). Without it a watched repository builds on the bare host, where the
|
||||
|
||||
Reference in New Issue
Block a user