# Werkator for LLM Agents This document describes Werkator's features, configuration, and interfaces for AI coding agents working with the project. ## System Overview Werkator is a lightweight, declarative CI/CD build system. It is single-purpose: monitor git repositories for new commits, trigger builds based on rules, execute them (native, Docker, or bubblewrap sandbox), archive artifacts, and report status back to Gitea. **Core capabilities:** - Git-centric design: one instance per repository (or one instance serving multiple via ADR 0009) - Dual-mode operation: CLI for interactive use, Server for persistent background operation - Three build runtimes: native execution, Docker containers, or bubblewrap sandbox (`werkdock`) - Branch-aware: each branch's committed config can override defaults for its own CI - Status reporting to Gitea commit status checks - Artifact archival and retention policies - Automated branch polling with configurable triggers (push-based, scheduled, branch age) ## Architecture ### Package Structure (all under `de.javagil.werkator`) | Package | Purpose | |---------|---------| | `commands` | picocli CLI subcommands: `init`, `build`, `retry`, `status`, `server`, `config:print`, `export:forgejo` | | `config` | YAML schema (`WerkatorConfig` data classes), version validation, layer merging | | `git` | Git CLI access (`GitCommandRunner`) for fetch, checkout, status, symbolic-ref | | `gitea` | HTTP client for Gitea commit-status API and branch listing | | `build` | Build execution: `BuildRunner`, `WorkspacePreparer`, result tracking, artifact collection | | `artifacts` | Filesystem artifact store with retention policy enforcement | | `repo` | `RepoContext` — the central repository interface (git access, artifacts, results, worktrees, config) | | `watcher` | Branch polling loop (`Watcher`), auto-build triggering, startup recovery, per-context concurrency guard | | `metrics` | System resource sampling (CPU, memory, disk) aggregated per build | | `forgejo` | Converts the committed build definitions into Forgejo Actions workflows (RFC 0006) | | `server` | Spring Boot app: JSON API controllers, Thymeleaf UI, artifact serving, lifecycle control, HTTP endpoints | ### Key Invariants - **No single primary checkout for builds**: each branch builds in its own worktree under `.git/werkator/worktrees/`, sharing only the `.git/` directory. The primary checkout is never used for builds. - **One build per branch at a time**: global concurrency is capped at `executor.maxConcurrent`, but within that, only one build per branch can run; others queue. - **No process exit from CLI runner**: `exitProcess()` is called only from `main()`, keeping the Spring context alive during tests. - **Repository scope through RepoContext**: all filesystem operations (builds, artifacts, worktrees, config, git access) go through a `RepoContext`; never assume an implicit current directory. - **Config merging with branch layer**: five layers merge in order: built-in defaults, instance config (home), project config (`.werkator.yml`), machine config (`.git/werkator/.werkator.yml`), branch layer (committed `.werkator.yml` on the branch). Branch layer takes precedence for everything **except** pinned keys (host secrets, sandbox policies, trust gate). A branch can never reach credentials, disable its sandbox, or bypass its own CI gate. ## Configuration Configuration is **purely YAML-based**, with no API config, no environment variables for CI settings, and no runtime UI for adding jobs. ### Config File Locations (layered) | Layer | Path | Committed | Purpose | |-------|------|-----------|---------| | Built-in | — | — | Werkator's own hardcoded defaults | | Instance | `~/.werkator.yml` | No | Registry of served repositories, instance-level settings (port, max concurrency, poll interval) | | Project | `.werkator.yml` | Yes | Team-shared CI settings (Gitea connection, builds, docker/werkdock policy, retention) | | Applied fragment | `.git/werkator/.werkator.applied.yml` | No | Deployment wrapper's parameters; validated strictly before installing | | Machine | `.git/werkator/.werkator.yml` | No | Host/user secrets (`git.token`, `git.account`) and overrides; wins on any key present | | Branch | `.werkator.yml` committed on branch | Yes | That branch's own build definitions and `buildCommand`/`docker.image`/artifact overrides | **Key property**: a branch cannot reach pinned keys (secrets, sandbox policy, trust gate, status context, pull-request gate). Trying to set them in the branch layer has no effect; they resolve from machine or project config. ### Configuration Reference (Top-level Keys) ```yaml werkator: version: since: "0.9.18" # enforced floor: Werkator refuses if it is older below: "2.0" # optional caution marker server: # instance-level HTTP server (not in branch config) publicBaseUrl: "https://ci.example.org/" port: 18080 bindAddress: "127.0.0.1" # default loopback for unauthenticated UI/API nginx: # opt-in managed nginx+certbot for HTTPS (ADR 0005) enabled: false serverName: "" httpPort: 8080 httpsPort: 8443 gitea: # repository side (project or machine config) baseUrl: "https://git.example.org" owner: "my-org" # user or org owning the repo repo: "my-repo" statusContext: "werkator" # label on commit status checks; pinned (cannot be overridden by branch) git: # credentials (machine config only; never committed) account: "ci-bot" token: "glpat-xxxxx" executor: # global build concurrency (instance-level) maxConcurrent: 1 # changes require restart maxConcurrentPerRepository: 1 # optional cap per repository below the global one builds: # named build definitions (jobs); not inherited per-branch default: # base every other definition inherits settings from trigger: # **only trigger** is NOT inherited onPush: true # build new commits branches: ["*"] # glob patterns; empty = all; "!" prefix excludes atTimes: ["01:00"] # daily UTC times HH:MM or ??:MM (hourly) activeWithin: "24h" # only branches with commits in the last N time cleanCommand: "rm -rf build" buildCommand: "./gradlew build" artifactDirs: - "build/reports" - "build/doc" stdoutLog: "build.stdout.log" stderrLog: "build.stderr.log" requirePullRequest: false # gate to refs/pull/*/head on origin (pinned) statusContext: "" # empty = gitea.statusContext (pinned) docker: # optional Docker build runtime enabled: false # (pinned) image: "openjdk:21-slim" dockerfile: "" # empty = pull as-is; set to rebuild when content changes context: "." network: "" # empty = Docker default (pinned) env: {} werkdock: # optional bubblewrap sandbox (for hosts without root or Docker) enabled: false # (pinned) rootfs: "/path/to/rootfs.tar" # or http(s) URL, downloaded once to .git/werkator/buildenv/ binary: "werkdock" # PATH-resolved CLI for the sandbox env: {} # environment inside sandbox (cleared; nothing leaks in) pitest: # every other definition inherits default's settings, brings its own trigger trigger: atTimes: ["01:00"] branches: ["main", "release/*"] buildCommand: "./gradlew piTestFull" statusContext: "werkator/pitest" artifacts: # build artifact storage and retention rootDir: "" # empty = $XDG_STATE_HOME/werkator/artifacts/ retentionPerBranch: 5 # keep the artifacts of the N newest builds per branch retentionMaxAge: "" # additionally drop the artifacts of builds older than this (e.g. "30d") keepLatestGreen: true # keep the latest successful build even beyond retention limits watcher: # branch polling (not instance-level in server config) pollInterval: "10s" newBranchMaxAge: "5d" # only auto-pull origin branches younger than this pullRequestGate: true # honor requirePullRequest gates (false for plain git without PR refs) fastForwardLocalRefs: true # at end of poll, ff local refs/heads/* to origin ``` ### Runtime Config Inspection ```bash java -jar build/libs/werkator.jar config:print # only explicitly set values java -jar build/libs/werkator.jar config:print --full # all values including defaults java -jar build/libs/werkator.jar config:print --show-secrets # include git.token in plaintext ``` ## Features ### Build Execution - **Commit-based**: builds are always keyed to a specific commit SHA, never a branch name - **Branch worktrees**: each branch reuses its own persistent worktree (`.git/werkator/worktrees/`) across builds, preserving incremental caches - **Three runtimes**: 1. **Native**: commands run directly on the host 2. **Docker** (`docker.enabled`): clean/build commands run inside a container with the build worktree mounted, Gradle caches in a named volume, `.git` mounted read-only (so builds can run read-only git commands but cannot reach `git.token`) 3. **Werkdock** (`werkdock.enabled`): bubblewrap sandbox (bwrap) for hosts without root or Docker (e.g., Hostsharing webspaces), filesystem-only isolation, network and UID shared with host - **Isolated `.git` access**: in all runtimes, `.git/werkator/` is masked so builds cannot read machine config or control tokens - **Concurrent builds**: global cap of `executor.maxConcurrent`, optional cap per repository `executor.maxConcurrentPerRepository`, one build per branch at a time (others queue) ### Build Definitions and Triggering - **Named builds** (`builds.`): each definition has a `trigger` block (when/where) and everything else (what to do) - **Trigger types**: `onPush: true` (every new commit), `atTimes: ["HH:MM", "??:MM"]` (scheduled daily or hourly) - **Branch selectors**: glob patterns (`*` matches across `/`); prefix with `!` to exclude (exclusions always win) - **Inheritance**: a definition inherits all settings from `builds.default` except its `trigger` block - **Status context**: commit statuses are reported under `gitea.statusContext` or a per-build override; two builds of the same commit with the same context overwrite each other - **Pull-request links**: with a Gitea token the pages show a branch's open pull request as `PR#`, linked to Gitea; read by the watcher once per poll cycle, no configuration key - **Pull-request gate** (`requirePullRequest`): limit builds to commits that are pull-request heads (checked via `git ls-remote` without API token); pinned at machine/project level, cannot be overridden by branch ### Watcher (Auto-Build) - Polls origin every `watcher.pollInterval` (default 10s) - Detects new commits on tracked branches and enqueues builds matching the `trigger` rules - Pulls new branches on origin younger than `watcher.newBranchMaxAge` (default 5d) - At end of each poll cycle, fast-forwards local `refs/heads/*` to origin (if `watcher.fastForwardLocalRefs: true`) - Per-context concurrency guard: one polling loop per `RepoContext`, serialized across the instance - Startup recovery: resumes unfinished builds and re-evaluates pending ones on restart ### Artifacts - Stored on the filesystem under `artifacts.rootDir` (default `$XDG_STATE_HOME/werkator/artifacts/`) - Per-branch retention: keep the artifacts of the `retentionPerBranch` newest builds; additionally drop the artifacts of builds older than `retentionMaxAge`; older builds and those of branches deleted from origin stay in the history as records without artifacts, up to 100 per build name - **Permanent latest-green link**: when `keepLatestGreen: true`, the latest successful build is kept and served at `/branches//…` even if newer builds fail - Artifact directories are archived at their committed path; `build/reports` is special (archived as `reports/` and browsed via the artifact page's report index) - Directory ownership is repaired after Docker builds to match the host user ### Metrics - System resource sampling: CPU (load), memory (resident, virtual), disk usage per repository - Sampled once per poll cycle and aggregated - Exposed via JSON API at `/api/repos//metrics` ### Web UI - Server-rendered Thymeleaf (no SPA framework) - Static assets + one hand-written `static/werkator.js` (no frontend build pipeline) - Per-repository view (drop-down in page title to switch between registered repositories) - Branches view: branch name, latest build status, latest timestamp; artifact links; rebuild button - Build detail: commit SHA, branch, trigger time, runtime, command output, artifact list - Artifact serving: direct downloads and in-browser viewing (logs, reports) - No authentication on read (intended; statuses and artifacts can be linked from anywhere) - Runs behind a reverse proxy on a loopback interface (or managed nginx container for opt-in HTTPS) ## CLI Commands ### `init` Prepare a git repository for Werkator. Runs once per repository from a checked-out working tree. ```bash java -jar werkator.jar init [--systemd] [--apply FILE]... ``` Actions: 1. Detect repository root (`git rev-parse --show-toplevel`) 2. Auto-detect Gitea connection from `origin` URL 3. Create `.git/werkator/.werkator.yml` (machine config with auto-detected stubs) 4. Create `.werkator.yml` (project config template with defaults and Gitea connection) 5. If `--apply FILE`: install a YAML fragment as the applied layer (see config reference); repeated, the fragments are merged in order 6. If `--systemd`: generate a systemd user unit for persistent server operation ### `build [--repo NAME] BRANCH` Trigger a manual build of a branch, regardless of `requirePullRequest` gate. ```bash java -jar werkator.jar build feature/foo java -jar werkator.jar build --repo my-repo feature/foo ``` ### `retry [--repo NAME] BRANCH [BUILD-NAME]` Re-run the latest build of a branch (or a named build, e.g., `feature/foo@pitest`). ```bash java -jar werkator.jar retry main java -jar werkator.jar retry main pitest ``` ### `status [--repo NAME] [BRANCH]` Print the status of builds. If no branch given, show all branches. ```bash java -jar werkator.jar status java -jar werkator.jar status feature/foo ``` ### `server` Start the HTTP server on the configured port. Serves registered repositories (or the current one without `~/.werkator.yml`). ```bash java -jar werkator.jar server ``` The server listens on `server.bindAddress:server.port` (default `127.0.0.1:18080`). With `server.nginx.enabled: true`, also starts a managed nginx container for HTTPS. The server runs until killed; no daemon mode. ### `config:print [--full] [--show-secrets]` Print the effective merged configuration. ```bash java -jar werkator.jar config:print java -jar werkator.jar config:print --full # include defaults java -jar werkator.jar config:print --show-secrets # unmask git.token ``` ### `export:forgejo [REPO] [--stdout] [--strict] [--output-dir DIR] [--runs-on LABEL]` Convert the committed `.werkator.yml` into Forgejo Actions workflows (RFC 0006, step 1); `tools/werkator-to-forgejo` wraps it. Reads the committed file only, never the `.git` layers or `~/.werkator.yml`. `REPO` is the target repository or a directory inside it; the `.werkator.yml` in its root is read, and the workflows are written into it. Writes `.forgejo/workflows/werkator-.yml` (or below `--output-dir`, relative to the root), one per build definition that triggers on its own; follow-ups become jobs with `needs`. What has no workflow counterpart is a comment in the workflow and a line on stderr; `--strict` writes nothing and exits 1 then. ## JSON API The server exposes a JSON API at `/api/repos//…`: | Endpoint | Method | Purpose | |----------|--------|---------| | `/api/repos//status` | GET | Build status of all branches | | `/api/repos//status/` | GET | Build status of a single branch | | `/api/repos//status//` | GET | Status of a named build (e.g., `feature/foo@pitest`) | | `/api/repos//build` | POST | Trigger a manual build (JSON body: `{"branch": "…", "buildName": "…"}`) | | `/api/repos//retry` | POST | Re-run the latest build (JSON body: `{"branch": "…", "buildName": "…"}`) | | `/api/repos//metrics` | GET | System metrics (CPU, memory, disk) | | `/api/repos//artifact//` | GET | Download an artifact | Response format (status endpoints): ```json { "repo": "my-repo", "branches": [ { "name": "main", "latestBuild": { "status": "SUCCESS", "startTime": "2024-01-01T12:00:00Z", "endTime": "2024-01-01T12:05:30Z", "duration": "PT5M30S", "commitSha": "abc123...", "artifacts": [ {"path": "build/reports/", "size": 12345}, {"path": "build/doc/", "size": 67890} ] } } ] } ``` ## Common Patterns for Agents ### How to Detect Build Failures - Query `/api/repos//status/` and check `latestBuild.status` - Values: `PENDING`, `RUNNING`, `SUCCESS`, `FAILURE` - A branch with no builds yet has `latestBuild: null` ### How to Retrieve Artifacts - Artifacts are stored and served at `/branches///` - For the default build: `/branches//build/reports/` (report index) - For a named build: `/branches/@/` - Permanent latest-green link: `/branches//latest/…` (always serves the latest successful build, if any) ### How to Check Build Logs - Logs are captured to `stdoutLog` and `stderrLog` in the build worktree - Served as artifacts at `/branches//build.stdout.log` and `.../build.stderr.log` - Tail the log via the file download or read it from the artifact store directly ### How to Understand Build Times - Each build records `startTime` and `endTime` (ISO 8601 UTC) - `duration` is the elapsed time (ISO 8601 Duration, e.g., `PT5M30S`) - Metrics endpoint exposes system load during builds ### How to Work with Multiple Builds per Branch - Define multiple named builds in `builds` with different `trigger` blocks and `statusContext` values - A branch can have one "default" build and several named builds (e.g., `feature/foo` and `feature/foo@pitest`) - Query `/api/repos//status//` for the named build's status - Artifact paths include the pool name: `/branches/@/…` ### How to Add CI to a Branch The branch's own `.werkator.yml` (committed on the branch) can: - Override `buildCommand`, `cleanCommand`, `artifactDirs`, `docker.image`, `werkdock.env` - Add new build definitions with their own triggers - **Cannot** change: secrets, sandbox policy (`docker.enabled`, `werkdock.enabled`), trust gate (`requirePullRequest`), or status context labels (those are pinned) Example: try a new Docker image on a feature branch: ```yaml # .werkator.yml on feature/test-docker docker: image: "openjdk:21-alpine" # override the default image for this branch buildCommand: "./gradlew build -x slowTest" # faster iteration ``` Push and the next poll will build with the new image; other branches are unaffected. ## Deployment Werkator is designed for deployment on Hostsharing managed containers, Managed Webspaces, and other hosts with limited privileges. It includes no reverse proxy of its own; instead, it runs behind the host's existing nginx, Apache, or HAProxy. See [docs/deployment.md](deployment.md) for: - Systemd user service setup - Integration with Hostsharing hsadmin - Nginx/Apache proxy configuration - Managed nginx container (opt-in HTTPS via ADR 0005) - Runtime bundle for hosts without Java (ADR 0006) ## Testing and Debugging ### Run Tests ```bash ./gradlew test ./gradlew test --tests "de.javagil.werkator.ApplicationContextIntegrationTest" # single test class ``` Tests use Kotest FunSpec + MockK; SpringExtension is globally registered. See [.claude/skills/writing-tests/SKILL.md](.claude/skills/writing-tests/SKILL.md) for patterns. ### Debug Configuration ```bash java -jar werkator.jar config:print --full --show-secrets ``` ### View Effective Build Settings ```bash java -jar werkator.jar config:print | grep -A 20 "builds:" ``` ### Monitor Build Execution - Web UI: http://127.0.0.1:18080/ (or configured port) - API: `curl http://127.0.0.1:18080/api/repos//status` - CLI: `java -jar werkator.jar status` ### Check Repository Registry (Multi-Repo Instance) ```bash cat ~/.werkator.yml | grep -A 10 "repositories:" ``` ## Architecture Decision Records Major decisions are documented in [docs/adrs/](docs/adrs/): - **ADR 0001**: Test framework (Kotest + MockK + WireMock + Testcontainers) - **ADR 0002**: Gradle 8.14.5 - **ADR 0003**: Spring Boot 4.0.6 - **ADR 0004**: Rewrite architecture (JSON-file persistence, server-rendered UI, systemd) - **ADR 0005**: Managed nginx+certbot for opt-in HTTPS - **ADR 0006**: Runtime bundle distribution (jlink-trimmed JRE + jar) - **ADR 0007**: Build definitions with trigger blocks (ADR 0007) - **ADR 0008**: Bubblewrap build runtime (bwrap) for unprivileged hosts (ADR 0008) - **ADR 0009**: Multi-repo instance (one service, multiple repositories) - **ADR 0010**: The build configuration stays expressible as a Forgejo Actions workflow See [docs/adrs/](docs/adrs/) for full texts. ## Related Documentation - [docs/configuration.md](configuration.md) — complete configuration reference - [docs/bootstrapping.md](bootstrapping.md) — repository initialization workflow - [docs/deployment.md](deployment.md) — production deployment and systemd setup - [AGENTS.md](../AGENTS.md) — shared agent instructions (package structure, hard invariants, testing patterns)