diff --git a/.claude/skills/architecture/SKILL.md b/.claude/skills/architecture/SKILL.md index 0ed169d..0660ec9 100644 --- a/.claude/skills/architecture/SKILL.md +++ b/.claude/skills/architecture/SKILL.md @@ -19,12 +19,13 @@ WerkatorApplication ← @SpringBootApplication CliRunner ← CommandLineRunner + ExitCodeGenerator WerkatorCommand ← root @Command, delegates to subcommands commands/ - InitCommand ← "init [--systemd]" + InitCommand ← "init [--systemd] [--apply FILE]" ServerCommand ← "server" StatusCommand ← "status [--history]" BuildCommand ← "build []" RetryCommand ← "retry" ConfigPrintCommand ← "config:print [--full]" + ControlTokenCommand ← "control-token" ``` `status`, `build`, and `retry` implement `Callable` for their exit codes (0 success, 1 build failure, 2 usage/config errors). @@ -40,14 +41,15 @@ Two independent staleness signals, never merged: the `live-indicator` badge says ## Configuration System -Werkator is configured by two YAML files, deep-merged by `ConfigLoader` (later wins): +Werkator is configured by three YAML files, deep-merged by `ConfigLoader` (later wins): 1. `.werkator.yml` at the repo root — committed, shared team settings. -2. `.git/werkator/.werkator.yml` — not committed; machine-specific overrides and secrets (`git.account`, `git.token`). +2. `.git/werkator/.werkator.applied.yml` — not committed; the instance fragment installed verbatim by `init --apply` (step 23), validated strictly (unknown keys refused) and replaced wholesale on re-apply. +3. `.git/werkator/.werkator.yml` — not committed; machine-specific overrides and secrets (`git.account`, `git.token`); hand-edited, always wins. Every lookup falls back to the pre-rename name (`ConfigFiles`): `.gittally.yml`, and `.git/gittally/.gittally.yml` for the machine layer. Current name first, and where both exist the old one is ignored rather than merged — a missing config is not an error, so an un-renamed installation would otherwise start on defaults without a single failure. -On top of those comes the **branch layer**: the `.werkator.yml` committed on a branch, applied by `loadWithBranchLayer` (the watcher passes the content read via `git show`, `loadForWorktree` the file in the build worktree). A branch describes its own CI and wins over both layers — the whole `builds` section — so a configuration can be tried out on a branch without touching other branches' builds. `stripPinned` removes what is not a description of this branch's build: `git`, `server`, `gitea`, `executor`, `watcher`, and — inside every `builds` definition as well as every legacy `branches` entry — `requirePullRequest`, `statusContext`, and `docker.enabled`/`docker.network`. +On top of those comes the **branch layer**: the `.werkator.yml` committed on a branch, applied by `loadWithBranchLayer` (the watcher passes the content read via `git show`, `loadForWorktree` the file in the build worktree). A branch describes its own CI and wins over both layers — the whole `builds` section — so a configuration can be tried out on a branch without touching other branches' builds. `stripPinned` removes what is not a description of this branch's build: `git`, `server`, `gitea`, `executor`, `watcher`, and — inside every `builds` definition as well as every legacy `branches` entry — `requirePullRequest`, `statusContext`, `docker.enabled`/`docker.network`, and `bwrap.enabled`/`bwrap.rootfs`/`bwrap.werkdock`. Each file is version-checked before merging (`werkator.version.since`/`below`, `ConfigVersions.verdict`), so the message can name the file to fix: `since` is hard in both directions — too old a Werkator, or a file written before `ConfigVersions.FORMAT_BROKE_IN` and read after it — while `below` only warns. There is no format version (`apiVersion`) on purpose: only one configuration generation is supported, and the declared version exists to make the incompatibility nameable. diff --git a/.gitignore b/.gitignore index 3750b5c..c96b12b 100644 --- a/.gitignore +++ b/.gitignore @@ -35,4 +35,5 @@ replay_pid* # Other /.local/ /.env +/.env.* diff --git a/docs/adrs/0008-2026-09-01.bwrap-build-runtime.md b/docs/adrs/0008-2026-09-01.bwrap-build-runtime.md index 8f55dde..3915193 100644 --- a/docs/adrs/0008-2026-09-01.bwrap-build-runtime.md +++ b/docs/adrs/0008-2026-09-01.bwrap-build-runtime.md @@ -42,7 +42,7 @@ Good: Bad: -- The rootfs archive is project infrastructure that must be built and uploaded (~1.5 GB unpacked, disk/quota checked by `tools/werkator-build-prerequisites.sh`). +- The rootfs archive is project infrastructure that must be built and uploaded (~1.5 GB unpacked, disk/quota checked by `werkdock doctor`, which ported the original prerequisites script). - Filesystem-only isolation: network and process view are the host's — acceptable here, and pinned so a branch cannot widen it, but weaker than Docker. - Mountpoint pre-creation and mount ordering are subtle (hardened on the real webspace; see the fix messages preserved in commit `71f1fc6`). diff --git a/docs/bootstrapping.md b/docs/bootstrapping.md index 14aa526..a6f1c59 100644 --- a/docs/bootstrapping.md +++ b/docs/bootstrapping.md @@ -86,6 +86,12 @@ gitea: Then, you have to configure *Werkator* by amending this config file according to [configuration.md](configuration.md). +### 5. Optionally Install an Instance Fragment (`--apply`) + +`init --apply FILE` installs a YAML fragment in the configuration schema as the applied instance layer — see [configuration.md](configuration.md#the-applied-instance-fragment-init---apply). +Deployment tooling hands its parameters over this way instead of patching config files; the fragment is validated strictly and replaced wholesale on re-apply. +It runs before `--systemd`, so an applied `server.port` reaches the generated unit and the Apache `.htaccess` (written beside the units when a `publicBaseUrl` is configured). + ## Output `init` prints one line per action taken: diff --git a/docs/configuration.md b/docs/configuration.md index 97a0bfc..b84bcbb 100644 --- a/docs/configuration.md +++ b/docs/configuration.md @@ -7,10 +7,17 @@ Werkator is configured via YAML files. Settings are merged from several sources | Layer | Path | Committed to Git | Purpose | |--------------------------|----------------------------|------------------|----------------------------------------------| | Project config | `.werkator.yml` | Yes | Shared team settings | +| Applied instance fragment | `.git/werkator/.werkator.applied.yml` | No | Instance parameters installed by `init --apply` | | Repo installation config | `.git/werkator/.werkator.yml` | No | Machine- or user-specific overrides, secrets | | Branch config | `.werkator.yml` committed on a branch | Yes | That branch's build settings and build definitions | -The repo install config (`.git/werkator/.werkator.yml`) wins on any key present in both files. Typically used to set `git.token` and `git.account` without committing them. +The repo install config (`.git/werkator/.werkator.yml`) wins on any key present in several files; the applied fragment wins over the project config. Typically the repo install config sets `git.token` and `git.account` without committing them. + +### The applied instance fragment (`init --apply`) + +`werkator init --apply FILE` installs a YAML fragment in this very schema as its own layer (step 23) — the file a deployment wrapper hands over instead of patching configs. +The fragment is validated strictly before installing: an unknown key is refused loudly, never ignored, because a typo would otherwise install a value that silently does nothing. +It is then copied verbatim (comments included) to `.git/werkator/.werkator.applied.yml`; re-applying replaces the file, so nothing accumulates or duplicates, and the hand-edited repo install config — which always wins — is never rewritten. ### Which Werkator a file is written for @@ -414,7 +421,7 @@ All Werkator containers carry `org.hoennig.werkator` labels; stale build contain With `bwrap.enabled`, Werkator runs the build in a bubblewrap sandbox instead of native execution. This is the third runtime, for hosts without root and without a Docker daemon (e.g. Hostsharing managed webspaces); see `docs/plan/17-bwrap-build-runtime.md` and ADR 0008. Since step 21 session C the sandbox is executed by the `werkdock` CLI (`bwrap.werkdock`, default: resolved via `PATH`) — Werkator no longer invokes `bwrap` itself; `bwrap` must be installed for werkdock. -`werkdock doctor` checks the host's capability, superseding `tools/werkator-build-prerequisites.sh`. +`werkdock doctor` checks the host's capability (it replaced the retired `tools/werkator-build-prerequisites.sh` in step 23). `bwrap.rootfs` names the prepared root filesystem archive — a Debian-base rootfs with the build tools (JDK, git, locales, project-specific tooling) built elsewhere, since `debootstrap` is unavailable on the target. It is a local path or an `http(s)` URL; a URL is downloaded once into `.git/werkator/buildenv/`. diff --git a/docs/deployment.md b/docs/deployment.md index 0e824f9..f4f5809 100644 --- a/docs/deployment.md +++ b/docs/deployment.md @@ -268,14 +268,16 @@ The third deployment variant (plan step 21, verified live on a real webspace): n Werkator runs as a systemd *user* service on the assigned localhost port ("eigener Serverdienst"), the platform's managed Apache terminates TLS and proxies via `.htaccess`, and builds run in the bubblewrap sandbox executed by the [werkdock](../werkdock/README.md) CLI (ADR 0008, step 21 session C). Werkator is never built on the webspace: the runtime bundle and the werkdock binary are built locally and uploaded (ADR 0006). -All steps are driven by `tools/remote`, configured through the `.env` file in the repository root; commands name their role — `instance-*` manages the installed Werkator, `repo-*` the repository it watches. +All steps are driven by `tools/remote`; commands name their role — `instance-*` manages the installed Werkator, `repo-*` the repository it watches. +Each instance is a pair of files (step 23): a transport env file selected with `--env-file` (default `.env`), and a YAML fragment in the configuration schema, named by its `WERKATOR_INIT_CONFIG` key and installed remotely via `werkator init --apply` — e.g. `.env.mih34` + `.env.mih34.yml`, both gitignored. +The fragment carries the Werkator configuration (`server.port`, `publicBaseUrl`, systemd limits, `builds.default.bwrap.*`); the env file only says where and how to reach the host. ```bash -tools/remote werkator check-prerequisites # bwrap capability, disk and quota headroom -tools/remote werkator instance-install # upload + unpack the runtime bundle and werkdock -tools/remote werkator repo-init # clone the watched repo, init, rootfs archive, bwrap config -tools/remote werkator instance-start # server config, Apache proxy, systemd user unit -tools/remote port-forward start # browser tunnel while no public domain is set up +tools/remote --env-file .env.mih34 werkator check-prerequisites # uploads werkdock, runs its doctor +tools/remote --env-file .env.mih34 werkator instance-install # upload + unpack the runtime bundle and werkdock +tools/remote --env-file .env.mih34 werkator repo-init # clone the watched repo, rootfs archive, init --apply +tools/remote --env-file .env.mih34 werkator instance-start # init --apply --systemd, place .htaccess, enable the unit +tools/remote --env-file .env.mih34 port-forward start # browser tunnel while no public domain is set up ``` Layout on the host: the watched repository at `$WERKATOR_PATH/werkator/`, the unpacked runtime at `$WERKATOR_PATH/.werkator/werkator/`, the werkdock binary at `$WERKATOR_PATH/.werkator/bin/werkdock`. @@ -285,7 +287,7 @@ Fill `git.account`/`git.token` in the machine config when the origin is private, Updates are one command, refused while a build runs (`FORCE=1` overrides): ```bash -tools/remote werkator instance-update +tools/remote --env-file .env.mih34 werkator instance-update ``` The previous runtime stays as `.werkator/werkator.prev` for one deployment as the rollback asset. diff --git a/docs/plan/23-init-owns-the-files.md b/docs/plan/23-init-owns-the-files.md index 2a0fc17..f33e621 100644 --- a/docs/plan/23-init-owns-the-files.md +++ b/docs/plan/23-init-owns-the-files.md @@ -15,7 +15,7 @@ Werkator becomes the executing app wherever possible; `tools/remote` shrinks to Parameters travel as **files**, not as many CLI options — and each side gets the format that is native to it (refined 2026-09-01): -- The **wrapper** keeps a small, bash-sourceable env file with the transport values only: `tools/remote --env .env.mih34 werkator repo-init` selects the target (default: `.env`), so several instances (`.env.mih34`, `.env.vm4006`, later a Werkbaum instance) are files, not edits. +- The **wrapper** keeps a small, bash-sourceable env file with the transport values only: `tools/remote --env-file .env.mih34 werkator repo-init` selects the target (default: `.env`; named `--env-file` like docker's flag for the same thing, since `--env` means a single variable there), so several instances (`.env.mih34`, `.env.vm4006`, later a Werkbaum instance) are files, not edits. - **Werkator** takes a **YAML fragment in its own config schema**: `werkator init --apply mih34.yml` deep-merges the fragment into the machine config, idempotently — creating sections that are missing, updating the given values, never duplicating. No mapping table exists: the fragment says `server: {port: …}` and `builds: {default: {bwrap: …}}` directly, is validated by the existing schema binding, and is documented by the existing `docs/configuration.md`. - The wrapper uploads the fragment alongside the artifacts and calls `werkator init --apply …` remotely — the heredocs and `sed` calls in `tools/remote` disappear. @@ -31,25 +31,28 @@ The removed legacy env-to-YAML conversion stays removed — there is no conversi ## The Sessions -### A — Werkator side +### A — Werkator side (implemented 2026-09-01 on branch `init-apply-config`) - `init --apply FILE`: deep-merge the YAML fragment into the machine config — reusing the loader's merge, creating missing sections, updating given values, never duplicating (the duplication class dies here); a fragment that fails the schema binding or carries unknown keys is refused loudly. - `init --systemd` keeps generating the units; decide in the session whether the Apache `.htaccess` becomes part of the host-integration output when the applied config carries `server.port` and a public domain (proposal: yes, under `init --systemd`, since it is generated host integration exactly like the units). - New subcommand `werkator control-token`: print the token, creating it exactly like `ControlTokenService` does — the bash duplication in the wrapper dies. - Tests per the writing-tests conventions; `docs/bootstrapping.md` documents `--apply` (the fragment keys need no new reference — they are ordinary `docs/configuration.md` keys). -### B — Wrapper side +### B — Wrapper side (implemented 2026-09-01 on branch `init-apply-config`) -- `tools/remote --env FILE` (default `.env`); the init fragment named by `WERKATOR_INIT_CONFIG` is uploaded, and the remote init runs with `--apply`. +- `tools/remote --env-file FILE` (default `.env`); the init fragment named by `WERKATOR_INIT_CONFIG` is uploaded, and the remote init runs with `--apply`. - `repo-init` and `instance-start` lose their heredoc/`sed` config writing; `control-token` delegates to the new subcommand. - `check-prerequisites` uploads the werkdock binary first and runs `werkdock doctor`; `tools/werkator-build-prerequisites.sh` retires (its werkdock port is the survivor). -### C — Live verification and docs +### C — Live verification and docs (done 2026-09-01) - Run the full wrapper flow against mih34 (`instance-update`, `repo-init`, `instance-start` as no-op re-runs); `docs/deployment.md`'s webspace section switches to the `--env` invocations. ## Acceptance Criteria -- Session A: `werkator init --apply …` merges and re-merges a fragment idempotently; `werkator control-token` exists; full suite green. -- Session B: `tools/remote` contains no YAML heredocs and no `sed` into the machine config; the prerequisites bash script is gone. -- Session C: the mih34 re-runs change nothing on a configured host and the deployment docs show only `--env`-style calls. +- Session A: done 2026-09-01 — `werkator init --apply …` installs and replaces a fragment idempotently; `werkator control-token` exists; full suite green. + Deviation from the sketch above: the fragment is NOT merged into the machine config — it is installed verbatim as its own layer (`.git/werkator/.werkator.applied.yml`, above project, below machine config), because an in-place merge would re-serialize the machine config, destroying its comments and rewriting the file that holds the secrets; a verbatim copy also makes re-apply a plain file replacement. + The `.htaccess` decision fell as proposed: generated beside the units by `init --systemd` whenever a `publicBaseUrl` is configured; the wrapper copies it into the domain docroot. +- Session B: done 2026-09-01 — `tools/remote` contains no YAML heredocs and no `sed` into the machine config (the port lookups for idle check and port-forward read the *effective* config via `config:print`, so a port living in the applied fragment is found too); the prerequisites bash script is gone. +- Session C: done 2026-09-01 — verified live on mih34 with the `.env.mih34` + `.env.mih34.yml` pair: instance-update, doctor-based check-prerequisites (PASS 6/6), repo-init applying the fragment, instance-start placing the generated `.htaccess` and restarting the unit, control-token via the CLI; `docs/deployment.md` shows only `--env-file`-style calls. + Known niggle: validating a fragment that carries a `builds.default` without triggers logs the loader's "no build defines onPush" warning, although a fragment is judged out of context — cosmetic, fix when it annoys. diff --git a/docs/prs/2026-09-02-PR#9-init-apply-config.md b/docs/prs/2026-09-02-PR#9-init-apply-config.md new file mode 100644 index 0000000..3c7ed42 --- /dev/null +++ b/docs/prs/2026-09-02-PR#9-init-apply-config.md @@ -0,0 +1,144 @@ +> **WARNING:** This document describes only the change applied in this PR. +> It may already be outdated once the next PR is merged. +> Historic PR-documentation is not maintained along with new PRs — treat it as a snapshot, not as current documentation. + +## The Problem + +`tools/remote` wrote the machine config by appending heredoc blocks and patching values with `sed`, duplicating configuration knowledge (YAML shape, indentation, key names) that `WerkatorConfig` already owns — an indentation mismatch in one append guard produced nine duplicate `bwrap` blocks on mih34 before it was found (step 21 session D). +The script also re-implemented control-token generation in bash and still piped a prerequisites bash script whose generic half now exists as `werkdock doctor`. +PR #8 laid out the fix; this PR implements it. + +## Non-Goals + +- Multi-repository support for one Werkator instance (step 22, PR #10). +- A mapping table between env keys and config keys — deliberately absent, see The Solution. + +## The Scenarios + +### Feature: `werkator init --apply` installs an instance config fragment + +#### Background + +- The applied fragment is a fourth config layer: project `.werkator.yml` → applied fragment → repo-install machine config (secrets, always wins) → branch layer. +- It is strictly validated (unknown keys rejected) and installed verbatim, never merged in place — an in-place merge would re-serialize and destroy the machine config's comments and secrets. + +#### Scenario#9.01: A valid fragment becomes its own layer, above the project config and below the machine config + +So that an instance-specific setting (e.g. `server.port`) takes effect without touching the committed project config or the machine config's secrets. + +- **Given** a project `.werkator.yml` and a machine config with a secret +- **When** `init --apply FILE` installs a fragment that also sets a key the machine config sets +- **Then** the effective config shows the fragment's value where the machine config is silent, and the machine config's value where both set the same key. + +##### Verified by + +- [the applied instance fragment layers above the project config and below the machine config](../../src/test/kotlin/de/hoennig/werkator/config/ConfigLoaderTest.kt) +- [--apply installs the fragment as the applied layer and the effective config sees it](../../src/test/kotlin/de/hoennig/werkator/commands/InitCommandTest.kt) + +#### Scenario#9.02: Re-applying a fragment replaces it, never duplicates it + +So that repeated `instance-update` runs stay idempotent — the duplication class PR #8 was written against dies here structurally. + +- **Given** a fragment already installed as the applied layer +- **When** `init --apply` runs again with a changed fragment +- **Then** the applied layer file is atomically replaced, not appended to. + +##### Verified by + +- [applyInstanceFragment installs a valid fragment verbatim, and re-applying replaces it](../../src/test/kotlin/de/hoennig/werkator/config/ConfigLoaderTest.kt) + +#### Scenario#9.03: An invalid fragment is refused loudly and installs nothing + +So that a typo in a fragment never becomes a silent no-op or a half-applied layer. + +- **Given** a fragment with an unknown key, or a missing/empty file +- **When** `init --apply FILE` runs +- **Then** it fails loudly and the applied layer is left exactly as it was before the attempt. + +##### Verified by + +- [applyInstanceFragment refuses an unknown key loudly instead of installing a silent no-op](../../src/test/kotlin/de/hoennig/werkator/config/ConfigLoaderTest.kt) +- [applyInstanceFragment refuses a missing or empty fragment](../../src/test/kotlin/de/hoennig/werkator/config/ConfigLoaderTest.kt) +- [--apply with an invalid fragment installs nothing](../../src/test/kotlin/de/hoennig/werkator/commands/InitCommandTest.kt) + +#### Scenario#9.04: `init --systemd` generates the Apache reverse-proxy file alongside the units + +So that host integration for a public domain is generated exactly like the systemd units, not hand-written. + +- **Given** an effective config with `server.publicBaseUrl` set +- **When** `init --systemd` runs +- **Then** `werkator.htaccess` is generated proxying to the configured localhost port, for the wrapper to place in the domain docroot. + +##### Verified by + +- [the htaccess proxies everything to the configured localhost port](../../src/test/kotlin/de/hoennig/werkator/commands/SystemdServiceFilesTest.kt) + +#### Scenario#9.05: `werkator control-token` prints the same token the server would create + +So that the wrapper's bash re-implementation of token generation is no longer needed. + +- **Given** a repository with or without an existing control token +- **When** `werkator control-token` runs +- **Then** it creates the token exactly like `ControlTokenService` would and prints the same value on a re-run + - **and** it fails with exit code 2 outside a repository. + +##### Verified by + +- [creates the token like the server would and prints the same one on a re-run](../../src/test/kotlin/de/hoennig/werkator/commands/ControlTokenCommandTest.kt) +- [fails with exit code 2 outside a repository](../../src/test/kotlin/de/hoennig/werkator/commands/ControlTokenCommandTest.kt) + +### Feature: `tools/remote` becomes a thin wrapper + +#### Scenario#9.06: The wrapper selects its instance by file, not by many options + +So that several instances (mih34, vm4006, a future Werkbaum instance) are files, not edits to one script. + +- **Given** a transport env file (default `.env`) naming a config fragment via `WERKATOR_INIT_CONFIG` +- **When** `tools/remote --env-file .env.mih34 werkator repo-init` runs +- **Then** the fragment is uploaded and `werkator init --apply` is invoked remotely with it — no heredoc or `sed` writes the machine config. + +##### Verified by + +- live run on mih34 with `.env.mih34` + `.env.mih34.yml`: `instance-update`, `repo-init`, `instance-start` as idempotent re-runs (shell script; no automated test harness for `tools/remote`) + +#### Scenario#9.07: `check-prerequisites` delegates to `werkdock doctor` + +So that the generic host-readiness checks are not duplicated between a bash script and Werkdock's own porting of it. + +- **Given** a target host +- **When** `tools/remote werkator check-prerequisites` runs +- **Then** it uploads the `werkdock` binary and runs `werkdock doctor`, and `tools/werkator-build-prerequisites.sh` is deleted. + +##### Verified by + +- live run on mih34: doctor-based check-prerequisites, PASS 6/6 + +## The Solution + +Session A (Werkator): `ConfigLoader` reads a new `ConfigFiles.APPLIED` path (`.git/werkator/.werkator.applied.yml`) as a layer between project and repo-install config; `applyInstanceFragment(workingDir, fragment)` validates the fragment against a strict Jackson mapper (`FAIL_ON_UNKNOWN_PROPERTIES=true`, fragment validation only — the regular mapper stays lenient for forward-compatibility) and then copies it verbatim via an atomic move, never merging in place. +`InitCommand` gained `--apply FILE`, applied before `--systemd` handling so generated units/htaccess see the fragment; `SystemdServiceFiles.htaccessContent(port)` is the new generated file, written when `server.publicBaseUrl` is non-blank. +`ControlTokenCommand` is a new subcommand delegating to the existing `ControlTokenService`. +Session B (wrapper): `tools/remote --env-file FILE` (default `.env`) replaced positional/flag-heavy invocation; `repo-init`/`instance-start` lost their heredoc/`sed` config writing in favor of uploading the named fragment and calling `werkator init --apply`; port lookups (`require_idle`, `port_forward`) now parse `werkator config:print` output instead of grepping the machine config file directly, so a port living in the applied fragment is found too; `tools/werkator-build-prerequisites.sh` is deleted. +Session C: verified live end-to-end on mih34 with a `.env.mih34` + `.env.mih34.yml` pair. + +Deviation from the PR #8 plan: the fragment is not deep-merged into the machine config as first sketched — it is installed as its own verbatim layer, because an in-place merge would re-serialize the machine config and destroy its comments and secrets; a verbatim copy also makes re-apply a plain file replacement instead of a merge algorithm. + +## Open Questions + +- Applying a fragment that carries `builds.default` without triggers logs the loader's "no build defines onPush" warning, even though a fragment is judged out of context — cosmetic, fix when it annoys. + +## Additional Changes + +- `docs/configuration.md`: the layer table now shows four layers; a new subsection documents the applied instance fragment. +- `docs/bootstrapping.md`: a new section documents `--apply`. +- `docs/deployment.md`: the webspace section shows only `--env-file`-style invocations. +- `.gitignore`: `/.env.*` added for per-instance env files. + +## Prerequisite PRs + +- PR #7 (webspace install path) — the role-separated `tools/remote` this PR simplifies. +- PR #8 (step 23 plan) — the decision this PR implements. + +## Follow-up PRs + +- PR #10: multi-repository support for one Werkator instance (step 22). diff --git a/src/main/kotlin/de/hoennig/werkator/WerkatorCommand.kt b/src/main/kotlin/de/hoennig/werkator/WerkatorCommand.kt index e9409c6..ee7dc4f 100644 --- a/src/main/kotlin/de/hoennig/werkator/WerkatorCommand.kt +++ b/src/main/kotlin/de/hoennig/werkator/WerkatorCommand.kt @@ -2,6 +2,7 @@ package de.hoennig.werkator import de.hoennig.werkator.commands.BuildCommand import de.hoennig.werkator.commands.ConfigPrintCommand +import de.hoennig.werkator.commands.ControlTokenCommand import de.hoennig.werkator.commands.InitCommand import de.hoennig.werkator.commands.RetryCommand import de.hoennig.werkator.commands.ServerCommand @@ -22,6 +23,7 @@ import picocli.CommandLine.Command BuildCommand::class, RetryCommand::class, ConfigPrintCommand::class, + ControlTokenCommand::class, ], mixinStandardHelpOptions = true, versionProvider = BuildPropertiesVersionProvider::class, diff --git a/src/main/kotlin/de/hoennig/werkator/commands/ControlTokenCommand.kt b/src/main/kotlin/de/hoennig/werkator/commands/ControlTokenCommand.kt new file mode 100644 index 0000000..4d64c8a --- /dev/null +++ b/src/main/kotlin/de/hoennig/werkator/commands/ControlTokenCommand.kt @@ -0,0 +1,38 @@ +package de.hoennig.werkator.commands + +import de.hoennig.werkator.git.GitService +import de.hoennig.werkator.server.ControlTokenService +import org.springframework.stereotype.Component +import picocli.CommandLine.Command +import java.nio.file.Path +import java.nio.file.Paths +import java.util.concurrent.Callable + +/** + * Prints the control token guarding the mutating build endpoints, creating it + * exactly like the server does ([ControlTokenService] owns the format) — so no + * wrapper script ever needs its own token generator (step 23). + */ +@Component +@Command( + name = "control-token", + description = ["Print the control token for the mutating build endpoints, creating it if missing"], + mixinStandardHelpOptions = true, +) +class ControlTokenCommand( + private val gitService: GitService, +) : Callable { + var workingDir: Path = Paths.get(".") + + override fun call(): Int { + val root = + try { + gitService.getTopLevel(workingDir.toAbsolutePath().normalize()) + } catch (e: Exception) { + println("Error: ${e.message}") + return 2 + } + println(ControlTokenService(root.resolve(".git/werkator/control-token")).token()) + return 0 + } +} diff --git a/src/main/kotlin/de/hoennig/werkator/commands/InitCommand.kt b/src/main/kotlin/de/hoennig/werkator/commands/InitCommand.kt index 139490b..1871685 100644 --- a/src/main/kotlin/de/hoennig/werkator/commands/InitCommand.kt +++ b/src/main/kotlin/de/hoennig/werkator/commands/InitCommand.kt @@ -31,6 +31,16 @@ class InitCommand( ) var systemd: Boolean = false + @Option( + names = ["--apply"], + description = [ + "install a config-schema YAML fragment as the applied instance layer " + + "(validated strictly; re-applying replaces the previous fragment)", + ], + paramLabel = "FILE", + ) + var apply: Path? = null + /** Replaceable for tests: the jar this JVM was started from, or null when not run via `java -jar`. */ internal var jarPathResolver: () -> Path? = { runningJarPath() } @@ -52,6 +62,17 @@ class InitCommand( createRepoInstallConfig(root, detected, normalizedWorkingDir) createProjectConfig(root, detected, normalizedWorkingDir) + // before the systemd files, which read the effective configuration — + // an applied fragment's port and limits must reach the generated unit + apply?.let { fragment -> + try { + val target = configLoader.applyInstanceFragment(root, fragment) + println("applied $fragment as ${target.toFile().relativeTo(normalizedWorkingDir.toFile())}") + } catch (e: Exception) { + println("Error: ${e.message}") + return + } + } if (systemd) { createSystemdFiles(root, normalizedWorkingDir) } @@ -274,12 +295,14 @@ class InitCommand( * already loadable (re-running `init --systemd` on an installed instance); during * the very first bootstrap they stay unset and the defaults (no directives) apply. */ - private fun loadedSystemdConfig(): de.hoennig.werkator.config.SystemdConfig = + private fun loadedSystemdConfig(): de.hoennig.werkator.config.SystemdConfig = loadedServerConfig().systemd + + private fun loadedServerConfig(): de.hoennig.werkator.config.ServerConfig = try { - configLoader.load(Paths.get(".")).server.systemd + configLoader.load(Paths.get(".")).server } catch (_: Exception) { de.hoennig.werkator.config - .SystemdConfig() + .ServerConfig() } private fun createSystemdFiles( @@ -325,6 +348,19 @@ class InitCommand( pruneTimerFile.toFile().writeText(SystemdServiceFiles.pruneTimerContent()) println("created ${pruneTimerFile.toFile().relativeTo(normalizedWorkingDir.toFile())}") + // generated host integration like the units: only meaningful behind a web + // frontend, so it needs a public base URL; unused elsewhere and harmless + val server = loadedServerConfig() + if (server.publicBaseUrl.isNotBlank()) { + val htaccessFile = werkatorDir.resolve(SystemdServiceFiles.HTACCESS_NAME) + htaccessFile.toFile().writeText(SystemdServiceFiles.htaccessContent(server.port)) + println( + "created ${htaccessFile.toFile().relativeTo( + normalizedWorkingDir.toFile(), + )} (Apache reverse proxy; copy it into the domain docroot on a managed webspace)", + ) + } + println("install and start the service and the nightly Docker cleanup with:") println(" ln -sf $unitFile ~/.config/systemd/user/$unitName") println(" ln -sf $pruneServiceFile ~/.config/systemd/user/${SystemdServiceFiles.PRUNE_SERVICE_NAME}") diff --git a/src/main/kotlin/de/hoennig/werkator/commands/SystemdServiceFiles.kt b/src/main/kotlin/de/hoennig/werkator/commands/SystemdServiceFiles.kt index 566532c..2b7e2ba 100644 --- a/src/main/kotlin/de/hoennig/werkator/commands/SystemdServiceFiles.kt +++ b/src/main/kotlin/de/hoennig/werkator/commands/SystemdServiceFiles.kt @@ -11,6 +11,8 @@ object SystemdServiceFiles { const val ENV_FILE_NAME = "werkator.env" /** Host-global unit names of the nightly Docker cleanup — shared by all Werkator repositories on the host. */ + const val HTACCESS_NAME = "werkator.htaccess" + const val PRUNE_SERVICE_NAME = "werkator-docker-prune.service" const val PRUNE_TIMER_NAME = "werkator-docker-prune.timer" @@ -89,6 +91,20 @@ object SystemdServiceFiles { #JAVA_OPTS=-Xmx256m """.trimIndent() + "\n" + /** + * Apache reverse proxy for a Hostsharing managed webspace: the platform's + * Apache terminates TLS for the domain and forwards everything to the + * localhost port of the "eigener Serverdienst". Generated host integration + * like the units — the wrapper copies it into the domain's docroot. + */ + fun htaccessContent(port: Int): String = + """ + DirectoryIndex disabled + RewriteEngine On + RewriteBase / + RewriteRule .* http://127.0.0.1:$port%{REQUEST_URI} [proxy] + """.trimIndent() + "\n" + private fun sanitize(name: String): String = name.replace(Regex("[^A-Za-z0-9_.-]"), "-") /** Escape `%` specifiers in systemd unit values (legacy `systemd_path`). */ diff --git a/src/main/kotlin/de/hoennig/werkator/config/ConfigFiles.kt b/src/main/kotlin/de/hoennig/werkator/config/ConfigFiles.kt index ad95d74..c9a8bb8 100644 --- a/src/main/kotlin/de/hoennig/werkator/config/ConfigFiles.kt +++ b/src/main/kotlin/de/hoennig/werkator/config/ConfigFiles.kt @@ -19,6 +19,15 @@ object ConfigFiles { /** The machine-specific configuration inside `.git`; secrets live here. */ const val REPO_INSTALL = ".git/werkator/$COMMITTED" + /** + * The applied instance fragment (`init --apply`, step 23): a config-schema YAML + * fragment installed verbatim as its own layer — above the committed project + * config, below the hand-edited machine config. Kept separate so applying never + * rewrites the machine config (its comments and secrets stay untouched) and + * re-applying is a plain file replacement, never a merge that can duplicate. + */ + const val APPLIED = ".git/werkator/.werkator.applied.yml" + /** The name the committed configuration had before the rename. */ const val LEGACY_COMMITTED = ".gittally.yml" diff --git a/src/main/kotlin/de/hoennig/werkator/config/ConfigLoader.kt b/src/main/kotlin/de/hoennig/werkator/config/ConfigLoader.kt index 88c909f..49b5a04 100644 --- a/src/main/kotlin/de/hoennig/werkator/config/ConfigLoader.kt +++ b/src/main/kotlin/de/hoennig/werkator/config/ConfigLoader.kt @@ -11,8 +11,10 @@ import org.springframework.beans.factory.ObjectProvider import org.springframework.boot.info.BuildProperties import org.springframework.stereotype.Service import java.io.File +import java.nio.file.Files import java.nio.file.Path import java.nio.file.Paths +import java.nio.file.StandardCopyOption import java.util.concurrent.ConcurrentHashMap @Service @@ -28,6 +30,17 @@ class ConfigLoader( .configure(DeserializationFeature.FAIL_ON_UNKNOWN_PROPERTIES, false) .configure(SerializationFeature.WRITE_DATES_AS_TIMESTAMPS, false) + /** + * For validating instance fragments ([applyInstanceFragment]) only: unknown keys + * fail there instead of being ignored — the regular layers stay lenient so an old + * Werkator can read a newer file. + */ + private val strictYaml = + ObjectMapper(YAMLFactory().disable(YAMLGenerator.Feature.WRITE_DOC_START_MARKER)) + .registerKotlinModule() + .configure(DeserializationFeature.FAIL_ON_UNKNOWN_PROPERTIES, true) + .configure(SerializationFeature.WRITE_DATES_AS_TIMESTAMPS, false) + /** Keys already reported by [dropNonDefinitionBuilds]; the config is loaded on every poll cycle. */ private val warnedBuildKeys = ConcurrentHashMap.newKeySet() @@ -285,13 +298,50 @@ class ConfigLoader( val repoInstallName = ConfigFiles.firstExisting(workingDir, ConfigFiles.repoInstall) val projectName = ConfigFiles.firstExisting(workingDir) val repoInstall = loadFile(workingDir.resolve(repoInstallName).toFile()) + val applied = loadFile(workingDir.resolve(ConfigFiles.APPLIED).toFile()) val project = loadFile(workingDir.resolve(projectName).toFile()) // per file, so the message names the file to fix — the merged map has no provenance checkVersion(project, projectName, ROLLBACK_HINT) + checkVersion(applied, ConfigFiles.APPLIED, ROLLBACK_HINT) checkVersion(repoInstall, repoInstallName, ROLLBACK_HINT) checkTriggerBlocks(project, projectName, ROLLBACK_HINT) + checkTriggerBlocks(applied, ConfigFiles.APPLIED, ROLLBACK_HINT) checkTriggerBlocks(repoInstall, repoInstallName, ROLLBACK_HINT) - return deepMerge(project, repoInstall) + // the applied instance fragment sits above the committed project config and + // below the hand-edited machine config, which always has the last word + return deepMerge(deepMerge(project, applied), repoInstall) + } + + /** + * Validates and installs an instance fragment (`init --apply`, step 23): the file + * must be non-empty, pass the version and trigger checks, and bind *strictly* + * against the schema — an unknown key is refused loudly, never ignored, because a + * typo in a fragment would otherwise install a value that silently does nothing. + * The fragment is then copied verbatim (comments included) to [ConfigFiles.APPLIED]; + * re-applying replaces the file, so nothing can accumulate or duplicate. + */ + fun applyInstanceFragment( + workingDir: Path, + fragment: Path, + ): Path { + val raw = loadFile(fragment.toFile()) + require(raw.isNotEmpty()) { "instance fragment $fragment is missing, empty, or not a YAML mapping" } + checkVersion(raw, fragment.toString(), ROLLBACK_HINT) + checkTriggerBlocks(raw, fragment.toString(), ROLLBACK_HINT) + try { + strictYaml.convertValue(resolveBuildSections(dropNonDefinitionBuilds(raw)), WerkatorConfig::class.java) + } catch (e: IllegalArgumentException) { + throw IllegalArgumentException( + "instance fragment $fragment does not match the configuration schema: ${e.message}", + e, + ) + } + val target = workingDir.resolve(ConfigFiles.APPLIED) + Files.createDirectories(target.parent) + val temp = Files.createTempFile(target.parent, ".werkator.applied", ".tmp") + Files.copy(fragment, temp, StandardCopyOption.REPLACE_EXISTING) + Files.move(temp, target, StandardCopyOption.REPLACE_EXISTING, StandardCopyOption.ATOMIC_MOVE) + return target } /** diff --git a/src/test/kotlin/de/hoennig/werkator/commands/ControlTokenCommandTest.kt b/src/test/kotlin/de/hoennig/werkator/commands/ControlTokenCommandTest.kt new file mode 100644 index 0000000..abec3da --- /dev/null +++ b/src/test/kotlin/de/hoennig/werkator/commands/ControlTokenCommandTest.kt @@ -0,0 +1,37 @@ +package de.hoennig.werkator.commands + +import de.hoennig.werkator.git.GitService +import io.kotest.core.spec.style.FunSpec +import io.kotest.matchers.shouldBe +import io.kotest.matchers.string.shouldMatch +import io.mockk.every +import io.mockk.mockk +import java.nio.file.Files + +class ControlTokenCommandTest : FunSpec() { + private val gitService = mockk() + private val command = ControlTokenCommand(gitService) + + init { + test("creates the token like the server would and prints the same one on a re-run") { + val tempDir = Files.createTempDirectory("werkator-token-test") + command.workingDir = tempDir + every { gitService.getTopLevel(any()) } returns tempDir + + command.call() shouldBe 0 + + val tokenFile = tempDir.resolve(".git/werkator/control-token") + val token = tokenFile.toFile().readText().trim() + token shouldMatch Regex("[0-9a-f]{48}") + + command.call() shouldBe 0 + tokenFile.toFile().readText().trim() shouldBe token + } + + test("fails with exit code 2 outside a repository") { + every { gitService.getTopLevel(any()) } throws IllegalStateException("not a git repository") + + command.call() shouldBe 2 + } + } +} diff --git a/src/test/kotlin/de/hoennig/werkator/commands/InitCommandTest.kt b/src/test/kotlin/de/hoennig/werkator/commands/InitCommandTest.kt index 554d6d0..14ae12d 100644 --- a/src/test/kotlin/de/hoennig/werkator/commands/InitCommandTest.kt +++ b/src/test/kotlin/de/hoennig/werkator/commands/InitCommandTest.kt @@ -17,8 +17,10 @@ class InitCommandTest : FunSpec() { private val initCommand = InitCommand( gitService, + // default (null) BuildProperties provider: a relaxed ObjectProvider mock + // returns a raw Object under type erasure and breaks the version check de.hoennig.werkator.config - .ConfigLoader(mockk(relaxed = true)), + .ConfigLoader(), ) init { @@ -122,6 +124,47 @@ class InitCommandTest : FunSpec() { projectConfig.toFile().readText() shouldBe "existing: content" } + test("--apply installs the fragment as the applied layer and the effective config sees it") { + val tempDir = Files.createTempDirectory("werkator-init-test") + initCommand.workingDir = tempDir + val fragment = tempDir.resolve("mih34.yml") + fragment.toFile().writeText("server:\n port: 18088\n") + initCommand.apply = fragment + + every { gitService.getTopLevel(tempDir) } returns tempDir + every { gitService.getOriginUrl(tempDir) } returns "https://git.example.org/my-org/my-repo.git" + + initCommand.run() + + tempDir + .resolve(de.hoennig.werkator.config.ConfigFiles.APPLIED) + .toFile() + .shouldExist() + de.hoennig.werkator.config + .ConfigLoader() + .load(tempDir) + .server.port shouldBe 18088 + initCommand.apply = null + } + + test("--apply with an invalid fragment installs nothing") { + val tempDir = Files.createTempDirectory("werkator-init-test") + initCommand.workingDir = tempDir + val fragment = tempDir.resolve("typo.yml") + fragment.toFile().writeText("server:\n prot: 18088\n") + initCommand.apply = fragment + + every { gitService.getTopLevel(tempDir) } returns tempDir + every { gitService.getOriginUrl(tempDir) } returns "https://git.example.org/my-org/my-repo.git" + + initCommand.run() + + Files + .exists(tempDir.resolve(de.hoennig.werkator.config.ConfigFiles.APPLIED)) + .shouldBeFalse() + initCommand.apply = null + } + test("--systemd generates unit and environment file with install instructions") { val tempDir = Files.createTempDirectory("werkator-init-test") initCommand.workingDir = tempDir diff --git a/src/test/kotlin/de/hoennig/werkator/commands/SystemdServiceFilesTest.kt b/src/test/kotlin/de/hoennig/werkator/commands/SystemdServiceFilesTest.kt index 1417ac3..feb2053 100644 --- a/src/test/kotlin/de/hoennig/werkator/commands/SystemdServiceFilesTest.kt +++ b/src/test/kotlin/de/hoennig/werkator/commands/SystemdServiceFilesTest.kt @@ -91,5 +91,11 @@ class SystemdServiceFilesTest : FunSpec() { content shouldContain "#JAVA_OPTS=" content shouldContain ".werkator.yml" } + test("the htaccess proxies everything to the configured localhost port") { + val content = SystemdServiceFiles.htaccessContent(18088) + + content shouldContain "DirectoryIndex disabled" + content shouldContain "RewriteRule .* http://127.0.0.1:18088%{REQUEST_URI} [proxy]" + } } } diff --git a/src/test/kotlin/de/hoennig/werkator/config/ConfigLoaderTest.kt b/src/test/kotlin/de/hoennig/werkator/config/ConfigLoaderTest.kt index 018f689..da6bc43 100644 --- a/src/test/kotlin/de/hoennig/werkator/config/ConfigLoaderTest.kt +++ b/src/test/kotlin/de/hoennig/werkator/config/ConfigLoaderTest.kt @@ -110,6 +110,63 @@ class ConfigLoaderTest : FunSpec() { "./gradlew fromBranch" } + test("the applied instance fragment layers above the project config and below the machine config") { + val dir = Files.createTempDirectory("werkator-test") + dir.resolve(".werkator.yml").toFile().writeText("server:\n port: 1000\n publicBaseUrl: \"https://project/\"\n") + Files.createDirectories(dir.resolve(".git/werkator")) + dir.resolve(ConfigFiles.APPLIED).toFile().writeText("server:\n port: 2000\n bindAddress: 0.0.0.0\n") + dir + .resolve(".git/werkator/.werkator.yml") + .toFile() + .writeText("server:\n port: 3000\n") + + val server = loader.load(dir).server + + // machine wins over applied wins over project; untouched keys fall through + server.port shouldBe 3000 + server.bindAddress shouldBe "0.0.0.0" + server.publicBaseUrl shouldBe "https://project/" + } + + test("applyInstanceFragment installs a valid fragment verbatim, and re-applying replaces it") { + val dir = Files.createTempDirectory("werkator-test") + val fragment = dir.resolve("mih34.yml") + fragment.toFile().writeText("# instance mih34\nserver:\n port: 18088\n") + + val target = loader.applyInstanceFragment(dir, fragment) + + target shouldBe dir.resolve(ConfigFiles.APPLIED) + // verbatim copy: the comment survives + target.toFile().readText() shouldContain "# instance mih34" + loader.load(dir).server.port shouldBe 18088 + + fragment.toFile().writeText("server:\n port: 19099\n") + loader.applyInstanceFragment(dir, fragment) + loader.load(dir).server.port shouldBe 19099 + } + + test("applyInstanceFragment refuses an unknown key loudly instead of installing a silent no-op") { + val dir = Files.createTempDirectory("werkator-test") + val fragment = dir.resolve("typo.yml") + fragment.toFile().writeText("server:\n prot: 18088\n") + + val exception = + shouldThrow { + loader.applyInstanceFragment(dir, fragment) + } + + exception.message shouldContain "typo.yml" + Files.exists(dir.resolve(ConfigFiles.APPLIED)).shouldBeFalse() + } + + test("applyInstanceFragment refuses a missing or empty fragment") { + val dir = Files.createTempDirectory("werkator-test") + + shouldThrow { + loader.applyInstanceFragment(dir, dir.resolve("absent.yml")) + } + } + test("reads executor.maxConcurrent and defaults it to 1") { val dir = Files.createTempDirectory("werkator-test") loader.load(dir).executor.maxConcurrent shouldBe 1 diff --git a/tools/remote b/tools/remote index 7ce3faf..c208cb4 100755 --- a/tools/remote +++ b/tools/remote @@ -13,33 +13,38 @@ # repository are the running instance's job (or `bin/werkator build` on the # host — the werkator CLI, not this script). # -# Usage: -# tools/remote werkator check-prerequisites bwrap capability + disk/quota on the host -# tools/remote werkator instance-install first-time: upload + unpack bundle and werkdock -# tools/remote werkator instance-update redeploy bundle + werkdock, restart the service -# tools/remote werkator instance-start server config, Apache proxy, systemd unit -# tools/remote werkator repo-init clone the watched repo, init, rootfs, bwrap config -# tools/remote werkator control-token -# tools/remote port-forward start background tunnel to the Werkator UI -# tools/remote port-forward stop +# Usage (step 23: `--env-file` selects the target instance, default `.env`): +# tools/remote [--env-file FILE] werkator check-prerequisites werkdock doctor on the host +# tools/remote [--env-file FILE] werkator instance-install first-time: upload + unpack bundle and werkdock +# tools/remote [--env-file FILE] werkator instance-update redeploy bundle + werkdock, restart the service +# tools/remote [--env-file FILE] werkator instance-start apply fragment, Apache proxy, systemd unit +# tools/remote [--env-file FILE] werkator repo-init clone the watched repo, init --apply, rootfs +# tools/remote [--env-file FILE] werkator control-token +# tools/remote [--env-file FILE] port-forward start background tunnel to the Werkator UI +# tools/remote [--env-file FILE] port-forward stop # -# Required in .env: +# The env file carries TRANSPORT values only; everything that is Werkator +# configuration travels as a YAML fragment in the config schema, named by +# WERKATOR_INIT_CONFIG and installed remotely via `werkator init --apply` +# (docs/plan/23-init-owns-the-files.md). Pair the files per instance, e.g. +# `.env.mih34` + `.env.mih34.yml` (both gitignored). +# +# Required in the env file: # WERKATOR_REMOTE user@host to operate on, e.g. mih34-werkator@mih34.hostsharing.net # WERKATOR_PATH target directory on that host, e.g. /home/storage/mih34/users/werkator # # Required for `instance-start`: -# WERKATOR_PORT the localhost port assigned by Hostsharing (eigener Serverdienst) -# WERKATOR_DOMAIN the domain served by the managed Apache, e.g. ci.example.de +# WERKATOR_DOMAIN the domain served by the managed Apache (docroot location for the +# generated .htaccess); the port lives in the fragment (server.port) # # Required for `port-forward`: -# WERKATOR_LOCAL_PORT the local port the browser uses -# Optional in .env: -# WERKATOR_REPO_URL https clone URL of the watched repository -# (default: https://github.com/mhoennig/werkator.git) -# WERKATOR_MEMORY_MAX systemd MemoryMax for the unit, e.g. 1G (instance-start) -# WERKATOR_TASKS_MAX systemd TasksMax for the unit, e.g. 512 (instance-start) -# WERKATOR_ROOTFS rootfs archive path for repo-init -# (default: /build/werkator-buildenv-trixie-java-go-node.tar.zst) +# WERKATOR_LOCAL_PORT the local port the browser uses +# Optional in the env file: +# WERKATOR_INIT_CONFIG the init fragment to apply (repo-init, instance-start) +# WERKATOR_REPO_URL https clone URL of the watched repository +# (default: https://github.com/mhoennig/werkator.git) +# WERKATOR_ROOTFS rootfs archive path for repo-init +# (default: /build/werkator-buildenv-trixie-java-go-node.tar.zst) # # Install layout on the host: # $WERKATOR_PATH/werkator/ the watched repository (clone) @@ -52,11 +57,16 @@ set -euo pipefail die() { echo "ERROR: $*" >&2; exit 1; } +ENV_FILE="" +if [ "${1:-}" = "--env-file" ]; then + ENV_FILE="${2:?missing value for --env-file}" + shift 2 +fi + REPO="${1:-}" COMMAND="${2:-}" REPO_ROOT="$(cd "$(dirname "${BASH_SOURCE[0]}")/.." && pwd)" -PREREQ_SCRIPT="$REPO_ROOT/tools/werkator-build-prerequisites.sh" RUNTIME_BUNDLE="$REPO_ROOT/build/distributions/werkator-runtime-linux-x64.tar.gz" WERKDOCK_BINARY="$REPO_ROOT/werkdock/dist/werkdock" PID_FILE="/tmp/werkator-port-forward-$(id -u).pid" @@ -71,7 +81,7 @@ require_env() { local missing=0 for name in "$@"; do if [ -z "${!name:-}" ]; then - echo "ERROR: $name is not set — define it in $REPO_ROOT/.env" >&2 + echo "ERROR: $name is not set — define it in $ENV_FILE" >&2 missing=1 fi done @@ -80,10 +90,11 @@ require_env() { [ -n "$REPO" ] && [ -n "$COMMAND" ] || usage -# Load the connection and deployment values; explicit environment wins, the -# .env in the repository root fills the rest. +# Load the transport values; explicit environment wins, the selected env file +# (default: the .env in the repository root) fills the rest. +ENV_FILE="${ENV_FILE:-$REPO_ROOT/.env}" set -a -[ -f "$REPO_ROOT/.env" ] && source "$REPO_ROOT/.env" +[ -f "$ENV_FILE" ] && source "$ENV_FILE" set +a require_env WERKATOR_REMOTE WERKATOR_PATH @@ -109,16 +120,37 @@ ensure_ssh() { fi } -# Run the prerequisites script remotely by piping it over stdin; TARGET_DIR and -# ROOTFS_ARCHIVE are passed as arguments to `bash -s --`. +# Werkdock owns the host checks (`werkdock doctor` ports the old prerequisites +# script); the binary is uploaded first, so the check works pre-install. check_prerequisites() { - echo "==> Checking prerequisites on $HOST (target dir: $TARGET_DIR)" - local rootfs_remote="$TARGET_DIR/.werkator/$(basename "$ROOTFS")" - if ! ssh "$HOST" "WERKATOR_SSH_TARGET='$HOST' bash -s -- '$TARGET_DIR' '$rootfs_remote'" < "$PREREQ_SCRIPT"; then - die "prerequisites failed on $HOST — install aborted" + ensure_werkdock_binary + echo "==> Uploading werkdock and running its doctor on $HOST (target dir: $TARGET_DIR)" + ssh "$HOST" "mkdir -p '$TARGET_DIR/.werkator/bin'" + scp -q "$WERKDOCK_BINARY" "$HOST:$TARGET_DIR/.werkator/bin/werkdock.new" + ssh "$HOST" "mv '$TARGET_DIR/.werkator/bin/werkdock.new' '$TARGET_DIR/.werkator/bin/werkdock' && chmod 755 '$TARGET_DIR/.werkator/bin/werkdock'" + if ! ssh "$HOST" "'$TARGET_DIR/.werkator/bin/werkdock' doctor '$TARGET_DIR'"; then + die "werkdock doctor failed on $HOST — install aborted" fi } +ensure_werkdock_binary() { + if [ ! -f "$WERKDOCK_BINARY" ]; then + echo "==> werkdock binary not found; building it locally (go build)" + (cd "$REPO_ROOT/werkdock" && CGO_ENABLED=0 go build -o dist/werkdock .) + fi + [ -f "$WERKDOCK_BINARY" ] || die "werkdock binary missing: $WERKDOCK_BINARY" +} + +# Uploads the init fragment named by WERKATOR_INIT_CONFIG and echoes its remote +# path; empty when no fragment is configured. +upload_fragment() { + [ -n "${WERKATOR_INIT_CONFIG:-}" ] || { echo ""; return 0; } + [ -f "$WERKATOR_INIT_CONFIG" ] || die "init fragment missing: $WERKATOR_INIT_CONFIG" + local remote="$TARGET_DIR/.werkator/$(basename "$WERKATOR_INIT_CONFIG")" + scp -q "$WERKATOR_INIT_CONFIG" "$HOST:$remote" + echo "$remote" +} + # The instance artifacts are built locally (ADR 0006): the runtime bundle via # Gradle, the werkdock binary via the Go toolchain. Both are rebuilt when # missing, never on the target. @@ -128,11 +160,7 @@ ensure_instance_artifacts() { (cd "$REPO_ROOT" && ./gradlew runtimeBundle --console=plain -q) fi [ -f "$RUNTIME_BUNDLE" ] || die "runtime bundle missing: $RUNTIME_BUNDLE" - if [ ! -f "$WERKDOCK_BINARY" ]; then - echo "==> werkdock binary not found; building it locally (go build)" - (cd "$REPO_ROOT/werkdock" && CGO_ENABLED=0 go build -o dist/werkdock .) - fi - [ -f "$WERKDOCK_BINARY" ] || die "werkdock binary missing: $WERKDOCK_BINARY" + ensure_werkdock_binary } # Uploads and unpacks the instance artifacts. The previous runtime stays as @@ -168,7 +196,7 @@ instance_install() { # Refuse to swap the runtime under a running build; FORCE=1 overrides. require_idle() { local port - port="$(ssh "$HOST" "awk '/^server:/{f=1;next} f && /^ port:/{print \$2; exit}' '$MACHINE_CONFIG' 2>/dev/null" || true)" + port="$(ssh "$HOST" "cd '$TARGET_DIR/werkator' 2>/dev/null && '$WERKATOR_BIN' config:print 2>/dev/null" | awk '/^server:/{f=1;next} f && /^ port:/{print $2; exit}' | tr -d '"' || true)" [ -n "$port" ] || return 0 local current current="$(ssh "$HOST" "curl -s --max-time 5 http://127.0.0.1:$port/api/builds/current" || true)" @@ -202,8 +230,9 @@ instance_update() { # Sets up the WATCHED repository: an anonymous https clone (a private origin # gets its credentials via git.account/git.token in the machine config that -# `werkator init` creates), the werkator init, the rootfs archive for the -# sandbox builds, and the machine-local bwrap configuration. +# `werkator init` creates), the werkator init with the instance fragment +# applied, and the rootfs archive for the sandbox builds. All configuration +# writing is init's — this script transports and invokes (step 23). repo_init() { ensure_ssh [ -f "$ROOTFS" ] || die "rootfs archive missing: $ROOTFS — build it with tools/build-bwrap-rootfs.sh or set WERKATOR_ROOTFS" @@ -216,9 +245,6 @@ repo_init() { ssh "$HOST" "git clone '$REPO_URL' '$TARGET_DIR/werkator'" fi - echo "==> Running werkator init" - ssh "$HOST" "cd '$TARGET_DIR/werkator' && '$WERKATOR_BIN' init" - echo "==> Uploading the rootfs archive (skipped when unchanged)" local rootfs_remote="$TARGET_DIR/.werkator/$(basename "$ROOTFS")" local local_sha remote_sha @@ -232,25 +258,10 @@ repo_init() { [ "$local_sha" = "$remote_sha" ] || die "rootfs upload checksum mismatch" fi - echo "==> Writing the machine-local bwrap configuration" - # NOTE: the guard must match the block's real indentation — a mismatch here - # once appended the block on every run. - if ssh "$HOST" "grep -q '^ bwrap:' '$MACHINE_CONFIG' 2>/dev/null"; then - echo " (bwrap block present, skipping)" - else - ssh "$HOST" "cat >> '$MACHINE_CONFIG' <<'CFG' - -# Build in the bubblewrap sandbox instead of natively (ADR 0008), executed by -# the werkdock CLI (step 21 session C). All three keys are pinned: read from -# this machine config even if a branch sets its own values. -builds: - default: - bwrap: - enabled: true - rootfs: $rootfs_remote - werkdock: $TARGET_DIR/.werkator/bin/werkdock -CFG" - fi + echo "==> Running werkator init${WERKATOR_INIT_CONFIG:+ --apply $(basename "${WERKATOR_INIT_CONFIG}")}" + local fragment_remote + fragment_remote="$(upload_fragment)" + ssh "$HOST" "cd '$TARGET_DIR/werkator' && '$WERKATOR_BIN' init ${fragment_remote:+--apply '$fragment_remote'}" echo "==> Verifying the effective configuration" ssh "$HOST" "cd '$TARGET_DIR/werkator' && '$WERKATOR_BIN' config:print 2>/dev/null | grep -A4 'bwrap:' | head -5" @@ -262,47 +273,28 @@ CFG" echo " then tools/remote werkator instance-start" } -# Start the server as a systemd user unit behind the managed Apache. -# WERKATOR_MEMORY_MAX / WERKATOR_TASKS_MAX (optional) are written into the -# machine config so `init --systemd` bakes them into the unit. +# Start the server as a systemd user unit behind the managed Apache. All +# configuration comes from the instance fragment (server.port, publicBaseUrl, +# systemd limits); init generates the units AND the .htaccess — this script +# only places and activates them (step 23). instance_start() { ensure_ssh - require_env WERKATOR_PORT WERKATOR_DOMAIN + require_env WERKATOR_DOMAIN + + echo "==> Applying the instance fragment and generating the host integration (init --systemd)" + local fragment_remote + fragment_remote="$(upload_fragment)" + ssh "$HOST" "cd '$TARGET_DIR/werkator' && '$WERKATOR_BIN' init ${fragment_remote:+--apply '$fragment_remote'} --systemd" + + local htaccess_src="$TARGET_DIR/werkator/.git/werkator/werkator.htaccess" local htaccess="$TARGET_DIR/doms/$WERKATOR_DOMAIN/subs/www/.htaccess" - - echo "==> Writing server settings to the machine config" - if ssh "$HOST" "grep -q '^server:' '$MACHINE_CONFIG' 2>/dev/null"; then - # re-run: update port and publicBaseUrl in place (systemd limits stay as written) - ssh "$HOST" "sed -i 's/^ port: .*/ port: $WERKATOR_PORT/; s|^ publicBaseUrl: .*| publicBaseUrl: \"https://$WERKATOR_DOMAIN/\"|' '$MACHINE_CONFIG'" + if ssh "$HOST" "test -f '$htaccess_src'"; then + echo "==> Placing the generated Apache reverse proxy at $htaccess" + ssh "$HOST" "mkdir -p '$TARGET_DIR/doms/$WERKATOR_DOMAIN/subs/www' && cp '$htaccess_src' '$htaccess'" else - ssh "$HOST" "cat >> '$MACHINE_CONFIG' <<'CFG' - -# Web access: the managed Apache terminates TLS and proxies to the localhost -# port assigned by Hostsharing (eigener Serverdienst); TLS is the domain's -# Let's Encrypt certificate, so Werkator itself stays on 127.0.0.1. -server: - port: $WERKATOR_PORT - bindAddress: 127.0.0.1 - publicBaseUrl: \"https://$WERKATOR_DOMAIN/\" - nginx: - enabled: false - systemd: - memoryMax: \"${WERKATOR_MEMORY_MAX:-}\" - tasksMax: \"${WERKATOR_TASKS_MAX:-}\" -CFG" + echo "==> No werkator.htaccess generated (no server.publicBaseUrl configured) — skipping the Apache proxy" fi - echo "==> Writing the Apache reverse proxy to $htaccess" - ssh "$HOST" "mkdir -p '$TARGET_DIR/doms/$WERKATOR_DOMAIN/subs/www' && cat > '$htaccess' <<'HT' -DirectoryIndex disabled -RewriteEngine On -RewriteBase / -RewriteRule .* http://127.0.0.1:$WERKATOR_PORT%{REQUEST_URI} [proxy] -HT" - - echo "==> Generating the systemd user unit (init --systemd)" - ssh "$HOST" "cd '$TARGET_DIR/werkator' && '$WERKATOR_BIN' init --systemd" - echo "==> Linking the units into ~/.config/systemd/user and enabling the service" ssh "$HOST" "mkdir -p ~/.config/systemd/user && \ ln -sf '$TARGET_DIR/werkator/.git/werkator/$UNIT' ~/.config/systemd/user/ && \ @@ -322,9 +314,11 @@ HT" # `start` runs ssh -N -L detached with a pid file; `stop` kills it. port_forward() { require_env WERKATOR_LOCAL_PORT + # the effective port, wherever it is configured (machine config or applied + # fragment) — config:print is the single answer, not this script's parser local remote_port - remote_port="$(ssh "$HOST" "awk '/^server:/{f=1;next} f && /^ port:/{print \$2; exit}' '$MACHINE_CONFIG'")" - [ -n "$remote_port" ] || die "no server.port in the machine config — run 'tools/remote werkator instance-start' first" + remote_port="$(ssh "$HOST" "cd '$TARGET_DIR/werkator' && '$WERKATOR_BIN' config:print 2>/dev/null" | awk '/^server:/{f=1;next} f && /^ port:/{print $2; exit}' | tr -d '"')" + [ -n "$remote_port" ] || die "no server.port configured — run 'tools/remote werkator instance-start' first" case "$COMMAND" in start) @@ -361,14 +355,11 @@ port_forward() { esac } -# Print the control token guarding the mutating build endpoints. If the server -# has not created it yet (it does so on first use), generate one in place — the -# server reads the file lazily, so a pre-created token is equivalent. +# Print the control token guarding the mutating build endpoints — the werkator +# CLI owns creation and format (step 23), this script only invokes it. control_token() { ensure_ssh - local token_file="$TARGET_DIR/werkator/.git/werkator/control-token" - ssh "$HOST" "if [ -f '$token_file' ]; then cat '$token_file'; else \ - umask 077 && head -c 32 /dev/urandom | od -An -tx1 | tr -d ' \\n' > '$token_file' && cat '$token_file'; fi" + ssh "$HOST" "cd '$TARGET_DIR/werkator' && '$WERKATOR_BIN' control-token" } case "$REPO" in diff --git a/tools/werkator-build-prerequisites.sh b/tools/werkator-build-prerequisites.sh deleted file mode 100755 index 8534448..0000000 --- a/tools/werkator-build-prerequisites.sh +++ /dev/null @@ -1,164 +0,0 @@ -#!/usr/bin/env bash -# -# Verify the bwrap (bubblewrap) build precondition on a target host before -# running Werkator's bwrap build runtime there (step 17 / ADR 0007). -# -# The whole "build Werkator inside bubblewrap on a Managed Webspace" approach -# hinges on one hard precondition: unprivileged user namespaces with a uid-0 -# mapping and read-only root binds must work. This script runs the exact -# command line recorded in docs/plan/17-bwrap-build-runtime.md, checks the -# expected signals, and additionally verifies the disk/quota situation: -# a bwrap build unpacks the rootfs (a zstd archive expands to several GiB) -# plus a Gradle distribution and per-branch caches, so the host needs both -# raw free space and enough group-quota headroom. -# -# Run this ON the target host (the webspace), no root needed. -# -# Optional: the rootfs archive to size the disk/quota check against, e.g. -# werkator-build-prerequisites.sh /path/to/werkator-buildenv-trixie.tar.zst -# When omitted, the check runs against a conservative default footprint. -# -# Usage: werkator-build-prerequisites.sh [TARGET_DIR] [ROOTFS_ARCHIVE] -# -# TARGET_DIR is the directory the build workspace will live in (default: $HOME). -# The check verifies it sits on the home filesystem and has enough free space. -# ROOTFS_ARCHIVE, when given, is the rootfs archive that will be used there. -# -# Output is one PASS/FAIL line per check plus a final RESULT line, e.g.: -# PASS: bwrap version: bubblewrap 0.8.0 -# PASS: build runs as root inside the namespace (uid 0) -# PASS: uid_map maps root back to the unprivileged user (uid 120957) -# PASS: read-only root bind is enforced -# PASS: at least 5 GiB free space on the build working filesystem -# FAIL: group quota headroom below the 5 GiB build footprint ... -# RESULT: FAIL (4/5) — Werkator bubblewrap builds are not usable on this host. -# - -set -euo pipefail - -target_dir_arg="${1:-}" -rootfs_arg="${2:-}" - -die() { echo "ERROR: $*" >&2; exit 1; } - -# Disk footprint a bwrap build needs headroom for, in 1K blocks: unpacked -# rootfs (zstd expands roughly 3-4x), Gradle distribution + per-branch cache, -# build output and artifacts. ~5 GiB. -MIN_FREE_BLOCKS=$((5 * 1024 * 1024)) - -# Reference filesystem: the one the invoking user's home directory lives on. -# Builds (repo clone, buildenv, caches) must run there — other mounts, such as -# a slow mass-storage volume, are rejected. -HOME_FS="$(df -Pk "$HOME" 2>/dev/null | awk 'NR==2 {print $1}')" - -pass=0 -fail=0 -result() { # result PASS|FAIL "message" - echo "$1: $2" - if [ "$1" = "PASS" ]; then pass=$((pass+1)); else fail=$((fail+1)); fi -} - -command -v bwrap >/dev/null 2>&1 || die "bwrap is not installed on this host" - -output="$(bwrap --unshare-user --unshare-pid --die-with-parent --uid 0 --gid 0 \ - --ro-bind / / --dev /dev --proc /proc --tmpfs /tmp \ - sh -c 'id -u && cat /proc/self/uid_map && (touch /usr/ro-test 2>&1 || true)' 2>&1)" || - die "bwrap invocation failed (no user namespace support?): $output" - -# Signal 0: bwrap itself is usable (version as a visible marker). -result PASS "bwrap version: $(bwrap --version 2>&1)" - -# Signal 1: runs as root (uid 0) inside the namespace. -first="$(printf '%s\n' "$output" | sed -n '1p')" -if [ "$first" = "0" ]; then - result PASS "build runs as root inside the namespace (uid 0)" -else - result FAIL "expected uid 0 inside the namespace, got: $first" -fi - -# Signal 2: uid_map maps root to the invoking unprivileged user. -uid_line="$(printf '%s\n' "$output" | sed -n '2p')" -self_uid="$(id -u)" -if printf '%s\n' "$uid_line" | grep -E "^[[:space:]]*0[[:space:]]+${self_uid}[[:space:]]+1" >/dev/null; then - result PASS "uid_map maps root back to the unprivileged user (uid $self_uid)" -else - result FAIL "expected uid_map '0 $self_uid 1', got: $uid_line" -fi - -# Signal 3: the read-only root bind is enforced (a write to /usr fails). -if printf '%s\n' "$output" | grep -qi "read-only file system"; then - result PASS "read-only root bind is enforced" -else - result FAIL "the read-only root bind did not reject a write to /usr" -fi - -# --- Disk / quota checks ------------------------------------------------ - -target_dir="${target_dir_arg:-$HOME}" -target_dir="$(realpath -m "$target_dir")" -min_gib=$((MIN_FREE_BLOCKS / 1024 / 1024)) - -if [ -n "$rootfs_arg" ] && [ ! -f "$rootfs_arg" ]; then - echo "WARNING: rootfs archive not found: $rootfs_arg (continuing without it)" -fi - -target_fs="$(df -Pk "$target_dir" 2>/dev/null | awk 'NR==2 {print $1}')" -df_output="$(df -Pk "$target_dir" 2>/dev/null | awk 'NR==2 {print int($4) " " $6}')" -if [ -n "$df_output" ]; then - avail_k="${df_output%% *}" - mount="${df_output##* }" - if [ -n "$HOME_FS" ] && [ "$target_fs" != "$HOME_FS" ]; then - # An explicitly chosen foreign filesystem is allowed (e.g. for testing) - # but flagged: builds there will be slow. - echo "WARNING: target dir is on $target_fs (mounted at $mount), not the home filesystem ($HOME_FS) — builds will run on slower storage" - fi - if [ "${avail_k:-0}" -lt "$MIN_FREE_BLOCKS" ]; then - result FAIL "less than ${min_gib} GiB free space on the build working filesystem ($mount)" - else - result PASS "at least ${min_gib} GiB free space on the build working filesystem ($mount, device $target_fs)" - fi -else - echo "WARNING: could not measure free space on $target_dir — only the quota check below applies" -fi - -if quota_output="$(quota -g 2>/dev/null)" && [ -n "$quota_output" ]; then - quota_ok=1 - quota_seen=0 - detail="" - while read -r fs blocks quota_limit; do - quota_seen=1 - # Only the quota of the target filesystem counts — other volumes may - # legitimately be full or unquota'd without affecting the build. - if [ -n "$target_fs" ] && [ "$(basename "$fs")" != "$(basename "$target_fs")" ] && [ "$fs" != "$target_fs" ]; then - continue - fi - headroom=$((quota_limit - blocks)) - if [ "$headroom" -lt "$MIN_FREE_BLOCKS" ]; then - quota_ok=0 - detail+=" $(basename "$fs"): $(awk -v b="$headroom" 'BEGIN{printf "%.1f", b/1024/1024}') GiB free of quota;" - fi - done < <(printf '%s\n' "$quota_output" | awk ' - NF==1 && $1 ~ /^\// { pending_fs=$1; next } - $1 ~ /^\// && $2 ~ /^[0-9]+$/ { print $1, $2, $4; pending_fs=""; next } - $1 ~ /^[0-9]+[*]?/ && pending_fs != "" { gsub(/\*/, "", $1); print pending_fs, $1, $3; pending_fs="" }') - if [ "$quota_seen" -eq 0 ]; then - echo "WARNING: quota tooling present but no group quota lines could be parsed — only free space was checked" - elif [ "$quota_ok" -eq 1 ]; then - result PASS "group quota headroom covers the ${min_gib} GiB build footprint" - else - result FAIL "group quota headroom below the ${min_gib} GiB build footprint (rootfs + Gradle cache); raise the quota before building.$detail" - fi -else - echo "WARNING: no readable group quota tooling on this host — only free space was checked" -fi - -total=$((pass + fail)) -echo -if [ "$fail" -eq 0 ]; then - echo "RESULT: PASS ($pass/$total) — Werkator bubblewrap builds are usable on this host." - echo "Next: install the Werkator instance with: tools/remote werkator install ${WERKATOR_SSH_TARGET:-@} '$target_dir'" - exit 0 -else - echo "RESULT: FAIL ($pass/$total) — Werkator bubblewrap builds are not usable on this host." - exit 1 -fi