werkator/AGENTS.md

# Werkator — Agent Instructions

This file holds the shared, tool-agnostic instructions for all AI coding agents.
Claude Code imports it from `CLAUDE.md` via `@AGENTS.md`; Claude-Code-specific instructions belong in `CLAUDE.md`, everything else here.
Detailed guides live as Agent Skills under `.claude/skills/` ([SKILL.md format](https://agentskills.io)); they load on demand — see [Skills](#skills).

## Build and Test Commands

```bash
./gradlew build          # compile + ktlintCheck + test
./gradlew ktlintFormat   # auto-format before committing
./gradlew test           # run all tests (can be slow, prefer single test)
./gradlew unitTest       # only *UnitTest and *RestTest: no full Spring context, no real git or process
./gradlew integrationTest  # only *IntegrationTest
./gradlew pitest mutationCrapCheck mutationCrapCodeCharta  # mutation testing, hours on a first run; -PpitestThreads=4 for more minions
./gradlew test --tests "de.javagil.werkator.ApplicationContextIntegrationTest"  # example for running a single test class
```

Run the JAR directly:

```bash
java -jar build/libs/werkator.jar --help
java -jar build/libs/werkator.jar init
```

`ktlintFormat` must be run before `build` passes — the formatter is enforced as part of the `check` lifecycle.

## Architecture Overview

Werkator is a lightweight, declarative CI/CD build system: git-centric, one instance per repository, builds native or in Docker, statuses reported to Gitea.
It is a dual-mode application: **CLI** (interactive, status, config) and **Server** (HTTP, persistent web UI + JSON API).
IMPORTANT: Before designing or modifying code in any production package, load the [architecture skill](.claude/skills/architecture/SKILL.md) — it holds the subsystem details (CLI wiring, server mode, web UI, config system, git access, build execution, watcher, metrics).

### Package Structure

All production code lives under `de.javagil.werkator`, with sub-packages `commands` (picocli subcommands), `config` (YAML config loading and schema), `git` (git CLI access), `gitea` (Gitea commit-status API client), `build` (build execution, results, workspaces), `artifacts` (filesystem artifact store), `repo` (the `RepoContext` a repository is worked on through: checkout, results, artifact store, name), `watcher` (branch polling, auto-builds, startup recovery), `metrics` (system resource sampling and aggregation), `forgejo` (converting the committed build definitions into Forgejo Actions workflows, RFC 0006), and `server` (JSON API controllers, Thymeleaf UI, artifact serving, control token, watcher and metrics lifecycles). Tests mirror this structure under `src/test/kotlin`.

### Hard Invariants

- `exitProcess` is called only from `main()` — never inside `CliRunner.run()`; this keeps the Spring context alive during tests.
- Nothing is scheduled during CLI runs or tests: the watcher poll loop and metrics sampling start only via an explicit `start()` in the `server` profile.
- Builds run detached in worktrees under `.git/werkator/worktrees/<branchKey>`; the primary checkout is never used for builds; never assume a single running build.
- Everything repository-scoped (results, artifacts, worktrees, git and config access) goes through a `RepoContext`, never through an implicit current directory: the executor serializes per (context, branch) under one global `maxConcurrent` and the optional per-repository `maxConcurrentPerRepository` (a build takes its repository's slot before a global one); pull-request target builds take no slot of these but one of their own pool, `maxConcurrentPullRequestTargets` and `maxConcurrentPullRequestTargetsPerRepository` (`BuildPool`, PR#119), each pool with its own queue, and they are serialized per (context, `<head-branch>@<build>`) instead of with their head branch, since they run in a fresh workspace; the watcher polls every context in its own guard. `RepoRegistry` opens one context per entry of the instance configuration `~/.werkator.yml` (ADR 0009), or the current directory without one; the instance-level keys (`server`, `executor`, `watcher.pollInterval`) and the `defaults` block are folded into every repository's effective config by `ConfigLoader` itself, so no consumer reads the home file directly. Server routes carry the repository as `/repos/<name>/…` and `/api/repos/<name>/…`, with the unscoped form (`/latest`, `/branches`, `/history`, `/api/builds/…`) permanently meaning the served repository; `/` belongs to the instance since RFC 0003 and carries the home page; the pages stay per repository; what makes them one UI is the home page, the Repositories page (`/repos`, `GET /api/repos`, newest build first) and the repository line below the header naming the current and the conspicuous repositories (RFC 0001).
- When config keys change, three places must stay in sync: the `WerkatorConfig` data classes, the `InitCommand` templates, and `docs/configuration.md`.
- The build configuration stays expressible as a Forgejo Actions workflow (ADR 0010): a new key or concept in a build definition needs a workflow counterpart, named in its design; a feature request without one is reshaped until it has one, or stays out. Host-side keys (instance, server, executor, watcher, credentials, the pinned sandbox and trust policy) are exempt. The first step towards one workflow for both systems is RFC 0006.
- Every config file may declare `werkator.version.since`/`below` (the Werkator it is written for, never a format version — no API is involved). `since` is enforced in both directions, using `ConfigVersions.FORMAT_BROKE_IN` for "file predates a breaking change"; `below` only warns. A violation aborts the start for the machine and project config, but fails only that branch's builds for a branch config.
- A branch describes its own CI: its committed `.werkator.yml` is the branch layer (`ConfigLoader.loadWithBranchLayer`, used by the watcher per origin branch and by `loadForWorktree` at build time) and takes precedence over `.git`/project — including the whole `builds` section, so a new configuration can be tried out on a branch without affecting other branches. Only the pinned set is stripped from that layer: secrets (`git`), host/repository sections (`server`, `gitea`, `executor`, `watcher`), the docker (`docker.enabled`, `docker.network`, `docker.socket`) and werkdock (`werkdock.enabled`, `werkdock.rootfs`, `werkdock.binary`, `werkdock.gradleHome`) sandbox policies, the trust gate (`requirePullRequest`), the restart of the instance itself (`restartsInstance`), and explicitly host-authored triggers and privileged follow-up triggers (host execution, Docker socket access, or instance restart); ordinary sandboxed `trigger.afterSuccessOf` definitions remain branch-configurable. A branch must never reach credentials, disable its container or sandbox, change its network, hand itself the Docker socket, substitute a foreign rootfs, raise global concurrency, bypass its own pull-request gate, end the instance, or deploy itself; a branch's definitions apply to that branch alone. Stripping alone did not carry that: a branch layer writing a non-map where a block belongs (`docker: null`) had nothing to strip from while the merge replaced the host's block wholesale, so the shape is refused (`checkLayerShape`) and the pinned values are written back from the host's layers after merging (`enforcePinned`). `builds.<name>: null` stays allowed — a branch may say it runs no such build — and the host's `builds.default` is the fallback the pinned values are taken from when the branch empties it.
- A pull-request target build (`trigger.onPullRequestTarget`, Forgejo's `pull_request_target`) is pinned whole: it holds the host's secrets while it judges a change, so a branch can never change its command, environment or secrets, switch it off, add one, or fake its status. `ConfigLoader.stripPinned` drops every such definition from the branch layer (the host's and the branch's own) and `enforcePinnedPullRequestTargets` puts the host's back as the primary configuration resolves them, so not even a branch-rewritten `builds.default` reaches it; the executor resolves its settings from the primary configuration alone (`BuildSettingsSource`) and refuses the definition without its pull request. It runs at the base commit the forge reports, in a fresh workspace of its own (`BranchWorkspaces.prepareFresh`), never on the head's files, and publishes its status on the head commit; the watcher decides it per open, non-draft, non-fork pull request and head commit (`PullRequestTargets`), and re-runs keep the recorded `BuildResult.pullRequest`. Combined with another trigger key, `requirePullRequest`, `builds.default` or a follow-up it is refused at start.
  It shares nothing the repository's other builds write: the werkdock runner binds a fresh home beside its fresh workspace as `/root` (`GitWorktreeWorkspaces.pullRequestHomeOf`), never `.git/werkator/buildenv/home`, and no `werkdock.gradleHome`; the Docker runner mounts neither the Gradle volume nor the socket; it never runs natively (`BuildRunner.start(pullRequestTarget = true)`). A target without a sandbox or with `docker.socket` is refused at start.
- A build definition carries the complete description of its build, split in two: the `trigger` block (`onPush`, `atTimes`, `afterSuccessOf`, `onPullRequestTarget`, `branches`, `activeWithin`) says when and for which branches it runs, everything else what it does. `afterSuccessOf` makes a definition the follow-up of another one — a deployment is a build that follows a green build (PR#23): it runs at the predecessor's commit after every green run of it, whoever started that run, and the `FollowUpTrigger` that enqueues it is armed only by `Watcher.start()`. `builds.default` is the base every other definition inherits its settings — never its `trigger` — from. The split is structural so that a selector added to `TriggerConfig` later is non-inheritable by construction; writing a trigger key flat is refused, never ignored, because ignoring it leaves a build that silently stops running. A `!` prefix in `trigger.branches` excludes and always wins.
- The inheritance is applied after all layers are merged: that order is what makes a build a branch invents inherit the host's sandbox policy instead of the data-class default, so the pinning also holds for a build the host has never heard of. Pinned are `requirePullRequest`, `statusContext`, `restartsInstance`, `docker.enabled`, `docker.network`, `docker.socket`, `werkdock.enabled`, `werkdock.rootfs`, `werkdock.binary`, `werkdock.gradleHome`, and explicitly host-authored or privileged follow-up triggers.
`restartsInstance` says this build installs the version the instance is to run next (RFC 0005): on `SUCCESS` the executor drains — nothing new starts, everything executing is finished and recorded — and then closes the server context, so `Restart=always` brings the new version up.
It is refused at start on `builds.default` and on a definition without `trigger.branches`, because a restart must never come from whatever branch went green last, and — with a registry — in every repository whose entry in `~/.werkator.yml` lacks `selfDeploys: true` (PR#87), because a deployment block copied into a foreign repository must not restart the instance. Docker and werkdock are mutually exclusive per branch — enabling both is rejected at start. The section was called `bwrap` until v1.2.0 and is still read under that name, with a warning; the hard refusal waits for the release that sets `ConfigVersions.FORMAT_BROKE_IN`.
- `builds` or the legacy `branches`, never both: `branches` is read only while the merged config defines no build at all (`builds.maxConcurrent` is not one), and ignored with a warning as soon as one exists. The section is deprecated and goes away once the repositories have migrated; then `ConfigVersions.FORMAT_BROKE_IN` gets set and a leftover `branches:` key must be rejected by name — the version check alone cannot catch a file that declares no version.
- A resource is sized by whether anything *waited* for it, never by how full it is (RFC 0004): CPU from load above the threads, RAM from the share of quarter hours whose average `memoryPressure` reached `SystemHealth.MEMORY_OVERLOAD_PERCENT` — the two share the constant, so a machine the health verdict calls healthy cannot be red in the sizing. A fill level is named in the reason and may still prove slack for a downsize, but it never asks for a raise; swap occupancy asks for nothing at all. A disk is sized by the level it holds — the 95th percentile of the last week's quarter-hour maxima — and by the growth it shows on most days — the median of the daily deltas —, never by a single quarter hour or a step (PR#78); its window maximum is named and may veto a downsize, never ask for a raise. Every advised size is one that can be booked (`BookableSteps`, derived from the `HostingEnvironment` by the caller, no configuration key), and a downsize is advised only when the next bookable size down would still leave the target headroom.
- A build result outlives its artifacts and the branch it was built from (PR#59, PR#84): `retentionPerBranch` and `retentionMaxAge` decide over the artifacts only. A result beyond them is marked with `BuildResult.artifactsPrunedAt`, a result whose branch is gone from origin with `BuildResult.branchGoneAt`, and both stay in the history as the record of a build that ran; records go only beyond the newest `BuildResultRepository.RECORDS_PER_NAME` (100) of a name. The gone mark is cleared should the branch return; the pruned mark never is. What goes with the mark is what needs the artifacts: the stored artifacts and the log (the artifact store is pruned to `history().withArtifacts()`), the row's artifact link, and for a gone branch the exemptions that keep a dormant branch's last status and its latest green build, and the Gitea branch link. `latestFor`, `latestGreenFor` and `latestPerName` skip marked entries; `history()` is the one place that returns them, and the History answers with its newest 200 rows.
- A private repository gives out its metadata, never its content (RFC 0003, D8). Content is everything that names the work — the repository's name, its branch and build-definition names, commits, artifacts, logs and what stands in them — and it is redacted, or the route answers 404. Metadata says nothing about the content — that a build ran, when, how long, with what outcome, plus storage sizes and every other aggregate figure of the machine — and it is served to anyone. A route named after a repository (`/repos/<name>/…`, `/api/repos/<name>/…`) therefore answers a visitor without the control token exactly like an unserved name, since the name itself is content (`RepoRouting`); the unscoped form means the served repository and answers redacted.
- Web UI: server-rendered Thymeleaf plus one hand-written `static/werkator.js` — no SPA framework, no frontend build pipeline; every fetch has a timeout and an explicit error badge; `UiFormats` and `werkator.js` must produce identical display formats.
- Git and Docker access shells out to the CLIs (`GitCommandRunner`, `docker`) — no JGit, no Docker SDK.

## Testing

Tests use Kotest `FunSpec` with MockK; `SpringExtension` is registered globally in `io.kotest.provided.ProjectConfig` — do not add it per-spec.
Every spec name ends in what it needs: `*UnitTest` (no Spring, no real git or process), `*RestTest` (a `@WebMvcTest` slice) or `*IntegrationTest` (full context, real git, processes, WireMock, containers); `TestNamingConventionUnitTest` enforces it.
IMPORTANT: Before writing or changing tests, load the [writing-tests skill](.claude/skills/writing-tests/SKILL.md) — it holds the spec structure and the Spring-slice mocking patterns.

The nightly build `mutation` measures every method's Mutation-CRAP (werkcrappit): `C² × (1 − MS)³ + C`, C the complexity, MS the share of killed mutants.
The hard limit is 15 (a method may score 15, not more); the gate `maxMutationCrap` in `build.gradle.kts` is a ratchet, lowered with every refactoring until it reaches 15, and never raised.
Red in the code city from 12 is an urgent warning, not a limit: a method there is to be looked at, and a new or changed method should not land there without a reason.
Since the score never falls below C, a method above 15 needs a smaller method, not more tests: split it along its domain steps, never by technical cuts such as "first half, second half".

## File-Formatting

### Markdown

Write documentation in English in Markdown files.
In Markdown, use a single line per sentence.
Keep sentences short.
Column-align Markdown tables in the file itself (pad cells, align the separator row), so a table reads as a table in plain text and in diffs.

## Documentation

- `docs/Werkator-Konzept.md` — product concept and target architecture (in German): git-centric CI, builds in Docker, one instance per repository, status reported back to Gitea.
- `docs/configuration.md` — configuration reference; keep in sync with `WerkatorConfig` and the `init` templates.
- `docs/bootstrapping.md` — how `init` prepares a repository.
- `docs/deployment.md` — running Werkator as a systemd user service behind an existing reverse proxy (`init --systemd` generates the unit).
- `docs/werkator-migrationsplan.md` — renaming a running installation from GitTally to Werkator: what the name fallback covers and what has to be moved by hand.
- `docs/plan/` — the step-by-step rewrite plan; `docs/plan/README.md` explains how to execute a step, `docs/plan/00-legacy-analysis.md` summarizes the legacy bash script.
- `docs/rfcs/` — requests for comments: proposals that are larger than one PR and not yet a decision (an accepted RFC becomes an ADR or a plan step).
- `docs/prs/` — one document per pull request; every PR needs one, except a pure release PR (version bump plus release notes, no code). IMPORTANT: Before opening or finishing a pull request, load the [pr-doc skill](.claude/skills/pr-doc/SKILL.md) and write the PR-doc.

Every PR bumps `version` in `build.gradle.kts` and adds its own entry to the release notes (see [Workflow](#workflow)).
The two must agree — `ReleaseVersionConsistencyUnitTest` fails the build otherwise.
A version is what identifies a running instance, so no change reaches main without one of its own.

## Workflow

Follow [docs/agentic-workflow.md](docs/agentic-workflow.md), the workflow of all werk repositories:
roles, steps of a PR, review rounds, PR documents, versions and the writing style.
In addition, for werkator:

- The branch protection of `main` requires the statuses `werkator`, `ai-review/gate` and `ai-review/codex` and one approval,
  from the review agent `ai-reviewer` or from `mi`.
- Every PR adds its own `<h2>vX.Y.Z</h2>` entry at the top of `src/main/resources/templates/releases.html`;
  a change that users do not notice gets an entry that starts with "Internal:".

## Key Architectural Decisions

All major decisions are in `docs/adrs/`. Run `adr-status` (after `source .envrc`) for a one-line summary of each. Decisions in force:

- **Test framework**: Kotest + MockK + WireMock + Testcontainers (ADR 0001)
- **Gradle**: 8.14.5 (ADR 0002)
- **Spring Boot**: 4.0.6 (ADR 0003)
- **Rewrite architecture**: JSON-file persistence behind a repository interface, server-rendered UI with JSON polling, no managed nginx — systemd unit behind the host's reverse proxy (ADR 0004)
- **Managed nginx/TLS**: revises ADR 0004 — an opt-in nginx+certbot container for hosts without a reverse proxy (e.g. Hostsharing), planned as `docs/plan/13-nginx-tls.md` (ADR 0005)
- **Runtime bundle distribution**: `./gradlew runtimeBundle` builds a jlink-trimmed JRE + jar tarball for hosts without a Java runtime; GraalVM native image and a containerized runtime were rejected (ADR 0006)
- **Build definitions**: a top-level `builds` section of named builds with `trigger` blocks replaces the branch-owned `autoBuild` schedules (ADR 0007)
- **bwrap build runtime**: on hosts without root and without Docker (Hostsharing Managed Webspaces), builds run in a `bwrap` user-namespace sandbox over a prepared rootfs — filesystem isolation only, network and uid shared with the host; proot/fakechroot and unisolated native builds were rejected (ADR 0008)
- **Multi-repo instance**: one instance serves a registry of self-contained repositories (instance config in `~/.werkator.yml`, repo config in each repo); revises the one-instance-per-repository tenet, implementation planned as `docs/plan/22-multi-repo.md` (ADR 0009)
- **Forgejo Actions compatible configuration**: the build configuration moves towards the Forgejo Actions workflow format; no new build-definition key without a workflow counterpart; one workflow for both systems as the first step (RFC 0006) (ADR 0010)

## Skills

On-demand guides in the cross-tool [SKILL.md format](https://agentskills.io); agents with skill support load them automatically by description, all others should read the linked files when the topic comes up:

- [architecture](.claude/skills/architecture/SKILL.md) — subsystem details: CLI wiring, server mode, web UI, config, git access, build execution, watcher, metrics.
- [writing-tests](.claude/skills/writing-tests/SKILL.md) — Kotest/MockK conventions and Spring slice-test mocking patterns.
- [pr-doc](.claude/skills/pr-doc/SKILL.md) — how to write the mandatory per-PR documentation in `docs/prs/`.