werkator/docs/llms.md

# Werkator for LLM Agents

This document describes Werkator's features, configuration, and interfaces for AI coding agents working with the project.

## System Overview

Werkator is a lightweight, declarative CI/CD build system.
It is single-purpose: monitor git repositories for new commits, trigger builds based on rules, execute them (native, Docker, or bubblewrap sandbox), archive artifacts, and report status back to Gitea.

**Core capabilities:**
- Git-centric design: one instance per repository (or one instance serving multiple via ADR 0009)
- Dual-mode operation: CLI for interactive use, Server for persistent background operation
- Three build runtimes: native execution, Docker containers, or bubblewrap sandbox (`werkdock`)
- Branch-aware: each branch's committed config can override defaults for its own CI
- Status reporting to Gitea commit status checks
- Artifact archival and retention policies
- Automated branch polling with configurable triggers (push-based, scheduled, branch age)

## Architecture

### Package Structure (all under `de.javagil.werkator`)

| Package | Purpose |
|---------|---------|
| `commands` | picocli CLI subcommands: `init`, `build`, `retry`, `status`, `server`, `config:print`, `export:forgejo` |
| `config` | YAML schema (`WerkatorConfig` data classes), version validation, layer merging |
| `git` | Git CLI access (`GitCommandRunner`) for fetch, checkout, status, symbolic-ref |
| `gitea` | HTTP client for Gitea commit-status API and branch listing |
| `build` | Build execution: `BuildRunner`, `WorkspacePreparer`, result tracking, artifact collection |
| `artifacts` | Filesystem artifact store with retention policy enforcement |
| `repo` | `RepoContext` — the central repository interface (git access, artifacts, results, worktrees, config) |
| `watcher` | Branch polling loop (`Watcher`), auto-build triggering, startup recovery, per-context concurrency guard |
| `metrics` | System resource sampling (CPU, memory, disk) aggregated per build |
| `forgejo` | Converts the committed build definitions into Forgejo Actions workflows (RFC 0006) |
| `server` | Spring Boot app: JSON API controllers, Thymeleaf UI, artifact serving, lifecycle control, HTTP endpoints |

### Key Invariants

- **No single primary checkout for builds**: each branch builds in its own worktree under `.git/werkator/worktrees/<branchKey>`, sharing only the `.git/` directory. The primary checkout is never used for builds.
- **One build per branch at a time**: global concurrency is capped at `executor.maxConcurrent`, but within that, only one build per branch can run; others queue.
- **No process exit from CLI runner**: `exitProcess()` is called only from `main()`, keeping the Spring context alive during tests.
- **Repository scope through RepoContext**: all filesystem operations (builds, artifacts, worktrees, config, git access) go through a `RepoContext`; never assume an implicit current directory.
- **Config merging with branch layer**: five layers merge in order: built-in defaults, instance config (home), project config (`.werkator.yml`), machine config (`.git/werkator/.werkator.yml`), branch layer (committed `.werkator.yml` on the branch). Branch layer takes precedence for everything **except** pinned keys (host secrets, sandbox policies, trust gate). A branch can never reach credentials, disable its sandbox, or bypass its own CI gate.

## Configuration

Configuration is **purely YAML-based**, with no API config, no environment variables for CI settings, and no runtime UI for adding jobs.

### Config File Locations (layered)

| Layer | Path | Committed | Purpose |
|-------|------|-----------|---------|
| Built-in | — | — | Werkator's own hardcoded defaults |
| Instance | `~/.werkator.yml` | No | Registry of served repositories, instance-level settings (port, max concurrency, poll interval) |
| Project | `.werkator.yml` | Yes | Team-shared CI settings (Gitea connection, builds, docker/werkdock policy, retention) |
| Applied fragment | `.git/werkator/.werkator.applied.yml` | No | Deployment wrapper's parameters; validated strictly before installing |
| Machine | `.git/werkator/.werkator.yml` | No | Host/user secrets (`git.token`, `git.account`) and overrides; wins on any key present |
| Branch | `.werkator.yml` committed on branch | Yes | That branch's own build definitions and `buildCommand`/`docker.image`/artifact overrides |

**Key property**: a branch cannot reach pinned keys (secrets, sandbox policy, trust gate, status context, pull-request gate). Trying to set them in the branch layer has no effect; they resolve from machine or project config.

### Configuration Reference (Top-level Keys)

```yaml
werkator:
  version:
    since: "0.9.18"      # enforced floor: Werkator refuses if it is older
    below: "2.0"         # optional caution marker

server:                   # instance-level HTTP server (not in branch config)
  publicBaseUrl: "https://ci.example.org/"
  port: 18080
  bindAddress: "127.0.0.1"  # default loopback for unauthenticated UI/API
  nginx:                 # opt-in managed nginx+certbot for HTTPS (ADR 0005)
    enabled: false
    serverName: ""
    httpPort: 8080
    httpsPort: 8443

gitea:                   # repository side (project or machine config)
  baseUrl: "https://git.example.org"
  owner: "my-org"        # user or org owning the repo
  repo: "my-repo"
  statusContext: "werkator"  # label on commit status checks; pinned (cannot be overridden by branch)

git:                     # credentials (machine config only; never committed)
  account: "ci-bot"
  token: "glpat-xxxxx"

executor:                # global build concurrency (instance-level)
  maxConcurrent: 1       # changes require restart
  maxConcurrentPerRepository: 1   # optional cap per repository below the global one

builds:                  # named build definitions (jobs); not inherited per-branch
  default:               # base every other definition inherits settings from
    trigger:             # **only trigger** is NOT inherited
      onPush: true       # build new commits
      branches: ["*"]    # glob patterns; empty = all; "!" prefix excludes
      atTimes: ["01:00"] # daily UTC times HH:MM or ??:MM (hourly)
      activeWithin: "24h" # only branches with commits in the last N time
    cleanCommand: "rm -rf build"
    buildCommand: "./gradlew build"
    artifactDirs:
      - "build/reports"
      - "build/doc"
    stdoutLog: "build.stdout.log"
    stderrLog: "build.stderr.log"
    requirePullRequest: false  # gate to refs/pull/*/head on origin (pinned)
    statusContext: ""     # empty = gitea.statusContext (pinned)
    docker:              # optional Docker build runtime
      enabled: false     # (pinned)
      image: "openjdk:21-slim"
      dockerfile: ""     # empty = pull as-is; set to rebuild when content changes
      context: "."
      network: ""        # empty = Docker default (pinned)
      env: {}
    werkdock:            # optional bubblewrap sandbox (for hosts without root or Docker)
      enabled: false     # (pinned)
      rootfs: "/path/to/rootfs.tar" # or http(s) URL, downloaded once to .git/werkator/buildenv/
      binary: "werkdock" # PATH-resolved CLI for the sandbox
      env: {}            # environment inside sandbox (cleared; nothing leaks in)

  pitest:                # every other definition inherits default's settings, brings its own trigger
    trigger:
      atTimes: ["01:00"]
      branches: ["main", "release/*"]
    buildCommand: "./gradlew piTestFull"
    statusContext: "werkator/pitest"

artifacts:               # build artifact storage and retention
  rootDir: ""            # empty = $XDG_STATE_HOME/werkator/artifacts/<repo-key>
  retentionPerBranch: 5  # keep the artifacts of the N newest builds per branch
  retentionMaxAge: ""    # additionally drop the artifacts of builds older than this (e.g. "30d")
  keepLatestGreen: true  # keep the latest successful build even beyond retention limits

watcher:                 # branch polling (not instance-level in server config)
  pollInterval: "10s"
  newBranchMaxAge: "5d"  # only auto-pull origin branches younger than this
  pullRequestGate: true  # honor requirePullRequest gates (false for plain git without PR refs)
  fastForwardLocalRefs: true  # at end of poll, ff local refs/heads/* to origin

```

### Runtime Config Inspection

```bash
java -jar build/libs/werkator.jar config:print         # only explicitly set values
java -jar build/libs/werkator.jar config:print --full  # all values including defaults
java -jar build/libs/werkator.jar config:print --show-secrets  # include git.token in plaintext
```

## Features

### Build Execution

- **Commit-based**: builds are always keyed to a specific commit SHA, never a branch name
- **Branch worktrees**: each branch reuses its own persistent worktree (`.git/werkator/worktrees/<branchKey>`) across builds, preserving incremental caches
- **Three runtimes**:
  1. **Native**: commands run directly on the host
  2. **Docker** (`docker.enabled`): clean/build commands run inside a container with the build worktree mounted, Gradle caches in a named volume, `.git` mounted read-only (so builds can run read-only git commands but cannot reach `git.token`)
  3. **Werkdock** (`werkdock.enabled`): bubblewrap sandbox (bwrap) for hosts without root or Docker (e.g., Hostsharing webspaces), filesystem-only isolation, network and UID shared with host
- **Isolated `.git` access**: in all runtimes, `.git/werkator/` is masked so builds cannot read machine config or control tokens
- **Concurrent builds**: global cap of `executor.maxConcurrent`, optional cap per repository `executor.maxConcurrentPerRepository`, one build per branch at a time (others queue)

### Build Definitions and Triggering

- **Named builds** (`builds.<name>`): each definition has a `trigger` block (when/where) and everything else (what to do)
- **Trigger types**: `onPush: true` (every new commit), `atTimes: ["HH:MM", "??:MM"]` (scheduled daily or hourly)
- **Branch selectors**: glob patterns (`*` matches across `/`); prefix with `!` to exclude (exclusions always win)
- **Inheritance**: a definition inherits all settings from `builds.default` except its `trigger` block
- **Status context**: commit statuses are reported under `gitea.statusContext` or a per-build override; two builds of the same commit with the same context overwrite each other
- **Pull-request links**: with a Gitea token the pages show a branch's open pull request as `PR#<number>`, linked to Gitea; read by the watcher once per poll cycle, no configuration key
- **Pull-request gate** (`requirePullRequest`): limit builds to commits that are pull-request heads (checked via `git ls-remote` without API token); pinned at machine/project level, cannot be overridden by branch

### Watcher (Auto-Build)

- Polls origin every `watcher.pollInterval` (default 10s)
- Detects new commits on tracked branches and enqueues builds matching the `trigger` rules
- Pulls new branches on origin younger than `watcher.newBranchMaxAge` (default 5d)
- At end of each poll cycle, fast-forwards local `refs/heads/*` to origin (if `watcher.fastForwardLocalRefs: true`)
- Per-context concurrency guard: one polling loop per `RepoContext`, serialized across the instance
- Startup recovery: resumes unfinished builds and re-evaluates pending ones on restart

### Artifacts

- Stored on the filesystem under `artifacts.rootDir` (default `$XDG_STATE_HOME/werkator/artifacts/<repo-key>`)
- Per-branch retention: keep the artifacts of the `retentionPerBranch` newest builds; additionally drop the artifacts of builds older than `retentionMaxAge`; older builds and those of branches deleted from origin stay in the history as records without artifacts, up to 100 per build name
- **Permanent latest-green link**: when `keepLatestGreen: true`, the latest successful build is kept and served at `/branches/<branch-key>/…` even if newer builds fail
- Artifact directories are archived at their committed path; `build/reports` is special (archived as `reports/` and browsed via the artifact page's report index)
- Directory ownership is repaired after Docker builds to match the host user

### Metrics

- System resource sampling: CPU (load), memory (resident, virtual), disk usage per repository
- Sampled once per poll cycle and aggregated
- Exposed via JSON API at `/api/repos/<name>/metrics`

### Web UI

- Server-rendered Thymeleaf (no SPA framework)
- Static assets + one hand-written `static/werkator.js` (no frontend build pipeline)
- Per-repository view (drop-down in page title to switch between registered repositories)
- Branches view: branch name, latest build status, latest timestamp; artifact links; rebuild button
- Build detail: commit SHA, branch, trigger time, runtime, command output, artifact list
- Artifact serving: direct downloads and in-browser viewing (logs, reports)
- No authentication on read (intended; statuses and artifacts can be linked from anywhere)
- Runs behind a reverse proxy on a loopback interface (or managed nginx container for opt-in HTTPS)

## CLI Commands

### `init`

Prepare a git repository for Werkator. Runs once per repository from a checked-out working tree.

```bash
java -jar werkator.jar init [--systemd] [--apply FILE]...
```

Actions:
1. Detect repository root (`git rev-parse --show-toplevel`)
2. Auto-detect Gitea connection from `origin` URL
3. Create `.git/werkator/.werkator.yml` (machine config with auto-detected stubs)
4. Create `.werkator.yml` (project config template with defaults and Gitea connection)
5. If `--apply FILE`: install a YAML fragment as the applied layer (see config reference); repeated, the fragments are merged in order
6. If `--systemd`: generate a systemd user unit for persistent server operation

### `build [--repo NAME] BRANCH`

Trigger a manual build of a branch, regardless of `requirePullRequest` gate.

```bash
java -jar werkator.jar build feature/foo
java -jar werkator.jar build --repo my-repo feature/foo
```

### `retry [--repo NAME] BRANCH [BUILD-NAME]`

Re-run the latest build of a branch (or a named build, e.g., `feature/foo@pitest`).

```bash
java -jar werkator.jar retry main
java -jar werkator.jar retry main pitest
```

### `status [--repo NAME] [BRANCH]`

Print the status of builds. If no branch given, show all branches.

```bash
java -jar werkator.jar status
java -jar werkator.jar status feature/foo
```

### `server`

Start the HTTP server on the configured port. Serves registered repositories (or the current one without `~/.werkator.yml`).

```bash
java -jar werkator.jar server
```

The server listens on `server.bindAddress:server.port` (default `127.0.0.1:18080`).
With `server.nginx.enabled: true`, also starts a managed nginx container for HTTPS.
The server runs until killed; no daemon mode.

### `config:print [--full] [--show-secrets]`

Print the effective merged configuration.

```bash
java -jar werkator.jar config:print
java -jar werkator.jar config:print --full  # include defaults
java -jar werkator.jar config:print --show-secrets  # unmask git.token
```

### `export:forgejo [REPO] [--stdout] [--strict] [--output-dir DIR] [--runs-on LABEL]`

Convert the committed `.werkator.yml` into Forgejo Actions workflows (RFC 0006, step 1); `tools/werkator-to-forgejo` wraps it.
Reads the committed file only, never the `.git` layers or `~/.werkator.yml`.
`REPO` is the target repository or a directory inside it; the `.werkator.yml` in its root is read, and the workflows are written into it.
Writes `.forgejo/workflows/werkator-<build>.yml` (or below `--output-dir`, relative to the root), one per build definition that triggers on its own; follow-ups become jobs with `needs`.
What has no workflow counterpart is a comment in the workflow and a line on stderr; `--strict` writes nothing and exits 1 then.

## JSON API

The server exposes a JSON API at `/api/repos/<name>/…`:

| Endpoint | Method | Purpose |
|----------|--------|---------|
| `/api/repos/<name>/status` | GET | Build status of all branches |
| `/api/repos/<name>/status/<branch>` | GET | Build status of a single branch |
| `/api/repos/<name>/status/<branch>/<build-name>` | GET | Status of a named build (e.g., `feature/foo@pitest`) |
| `/api/repos/<name>/build` | POST | Trigger a manual build (JSON body: `{"branch": "…", "buildName": "…"}`) |
| `/api/repos/<name>/retry` | POST | Re-run the latest build (JSON body: `{"branch": "…", "buildName": "…"}`) |
| `/api/repos/<name>/metrics` | GET | System metrics (CPU, memory, disk) |
| `/api/repos/<name>/artifact/<branch>/<file-path>` | GET | Download an artifact |

Response format (status endpoints):

```json
{
  "repo": "my-repo",
  "branches": [
    {
      "name": "main",
      "latestBuild": {
        "status": "SUCCESS",
        "startTime": "2024-01-01T12:00:00Z",
        "endTime": "2024-01-01T12:05:30Z",
        "duration": "PT5M30S",
        "commitSha": "abc123...",
        "artifacts": [
          {"path": "build/reports/", "size": 12345},
          {"path": "build/doc/", "size": 67890}
        ]
      }
    }
  ]
}
```

## Common Patterns for Agents

### How to Detect Build Failures

- Query `/api/repos/<name>/status/<branch>` and check `latestBuild.status`
- Values: `PENDING`, `RUNNING`, `SUCCESS`, `FAILURE`
- A branch with no builds yet has `latestBuild: null`

### How to Retrieve Artifacts

- Artifacts are stored and served at `/branches/<branch-key>/<pool-name>/<file-path>`
- For the default build: `/branches/<branch-key>/build/reports/` (report index)
- For a named build: `/branches/<branch-key>@<name>/<file-path>`
- Permanent latest-green link: `/branches/<branch-key>/latest/…` (always serves the latest successful build, if any)

### How to Check Build Logs

- Logs are captured to `stdoutLog` and `stderrLog` in the build worktree
- Served as artifacts at `/branches/<branch-key>/build.stdout.log` and `.../build.stderr.log`
- Tail the log via the file download or read it from the artifact store directly

### How to Understand Build Times

- Each build records `startTime` and `endTime` (ISO 8601 UTC)
- `duration` is the elapsed time (ISO 8601 Duration, e.g., `PT5M30S`)
- Metrics endpoint exposes system load during builds

### How to Work with Multiple Builds per Branch

- Define multiple named builds in `builds` with different `trigger` blocks and `statusContext` values
- A branch can have one "default" build and several named builds (e.g., `feature/foo` and `feature/foo@pitest`)
- Query `/api/repos/<name>/status/<branch>/<build-name>` for the named build's status
- Artifact paths include the pool name: `/branches/<branch-key>@<name>/…`

### How to Add CI to a Branch

The branch's own `.werkator.yml` (committed on the branch) can:
- Override `buildCommand`, `cleanCommand`, `artifactDirs`, `docker.image`, `werkdock.env`
- Add new build definitions with their own triggers
- **Cannot** change: secrets, sandbox policy (`docker.enabled`, `werkdock.enabled`), trust gate (`requirePullRequest`), or status context labels (those are pinned)

Example: try a new Docker image on a feature branch:

```yaml
# .werkator.yml on feature/test-docker
docker:
  image: "openjdk:21-alpine"  # override the default image for this branch
buildCommand: "./gradlew build -x slowTest"  # faster iteration
```

Push and the next poll will build with the new image; other branches are unaffected.

## Deployment

Werkator is designed for deployment on Hostsharing managed containers, Managed Webspaces, and other hosts with limited privileges.
It includes no reverse proxy of its own; instead, it runs behind the host's existing nginx, Apache, or HAProxy.

See [docs/deployment.md](deployment.md) for:
- Systemd user service setup
- Integration with Hostsharing hsadmin
- Nginx/Apache proxy configuration
- Managed nginx container (opt-in HTTPS via ADR 0005)
- Runtime bundle for hosts without Java (ADR 0006)

## Testing and Debugging

### Run Tests

```bash
./gradlew test
./gradlew test --tests "de.javagil.werkator.ApplicationContextIntegrationTest"  # single test class
```

Tests use Kotest FunSpec + MockK; SpringExtension is globally registered.
See [.claude/skills/writing-tests/SKILL.md](.claude/skills/writing-tests/SKILL.md) for patterns.

### Debug Configuration

```bash
java -jar werkator.jar config:print --full --show-secrets
```

### View Effective Build Settings

```bash
java -jar werkator.jar config:print | grep -A 20 "builds:"
```

### Monitor Build Execution

- Web UI: http://127.0.0.1:18080/ (or configured port)
- API: `curl http://127.0.0.1:18080/api/repos/<name>/status`
- CLI: `java -jar werkator.jar status`

### Check Repository Registry (Multi-Repo Instance)

```bash
cat ~/.werkator.yml | grep -A 10 "repositories:"
```

## Architecture Decision Records

Major decisions are documented in [docs/adrs/](docs/adrs/):

- **ADR 0001**: Test framework (Kotest + MockK + WireMock + Testcontainers)
- **ADR 0002**: Gradle 8.14.5
- **ADR 0003**: Spring Boot 4.0.6
- **ADR 0004**: Rewrite architecture (JSON-file persistence, server-rendered UI, systemd)
- **ADR 0005**: Managed nginx+certbot for opt-in HTTPS
- **ADR 0006**: Runtime bundle distribution (jlink-trimmed JRE + jar)
- **ADR 0007**: Build definitions with trigger blocks (ADR 0007)
- **ADR 0008**: Bubblewrap build runtime (bwrap) for unprivileged hosts (ADR 0008)
- **ADR 0009**: Multi-repo instance (one service, multiple repositories)
- **ADR 0010**: The build configuration stays expressible as a Forgejo Actions workflow

See [docs/adrs/](docs/adrs/) for full texts.

## Related Documentation

- [docs/configuration.md](configuration.md) — complete configuration reference
- [docs/bootstrapping.md](bootstrapping.md) — repository initialization workflow
- [docs/deployment.md](deployment.md) — production deployment and systemd setup
- [AGENTS.md](../AGENTS.md) — shared agent instructions (package structure, hard invariants, testing patterns)