werkator/docs/llms.md

Werkator for LLM Agents

This document describes Werkator's features, configuration, and interfaces for AI coding agents working with the project.

System Overview

Werkator is a lightweight, declarative CI/CD build system. It is single-purpose: monitor git repositories for new commits, trigger builds based on rules, execute them (native, Docker, or bubblewrap sandbox), archive artifacts, and report status back to Gitea.

Core capabilities:

• Git-centric design: one instance per repository (or one instance serving multiple via ADR 0009)
• Dual-mode operation: CLI for interactive use, Server for persistent background operation
• Three build runtimes: native execution, Docker containers, or bubblewrap sandbox (werkdock)
• Branch-aware: each branch's committed config can override defaults for its own CI
• Status reporting to Gitea commit status checks
• Artifact archival and retention policies
• Automated branch polling with configurable triggers (push-based, scheduled, branch age)

Architecture

Package Structure (all under de.javagil.werkator)

Package	Purpose
commands	picocli CLI subcommands: init, build, retry, status, server, config:print, export:forgejo
config	YAML schema (WerkatorConfig data classes), version validation, layer merging
git	Git CLI access (GitCommandRunner) for fetch, checkout, status, symbolic-ref
gitea	HTTP client for Gitea commit-status API and branch listing
build	Build execution: BuildRunner, WorkspacePreparer, result tracking, artifact collection
artifacts	Filesystem artifact store with retention policy enforcement
repo	RepoContext — the central repository interface (git access, artifacts, results, worktrees, config)
watcher	Branch polling loop (Watcher), auto-build triggering, startup recovery, per-context concurrency guard
metrics	System resource sampling (CPU, memory, disk) aggregated per build
forgejo	Converts the committed build definitions into Forgejo Actions workflows (RFC 0006)
server	Spring Boot app: JSON API controllers, Thymeleaf UI, artifact serving, lifecycle control, HTTP endpoints

Key Invariants

• No single primary checkout for builds: each branch builds in its own worktree under .git/werkator/worktrees/<branchKey>, sharing only the .git/ directory. The primary checkout is never used for builds.
• One build per branch at a time: global concurrency is capped at executor.maxConcurrent, but within that, only one build per branch can run; others queue.
• No process exit from CLI runner: exitProcess() is called only from main(), keeping the Spring context alive during tests.
• Repository scope through RepoContext: all filesystem operations (builds, artifacts, worktrees, config, git access) go through a RepoContext; never assume an implicit current directory.
• Config merging with branch layer: five layers merge in order: built-in defaults, instance config (home), project config (.werkator.yml), machine config (.git/werkator/.werkator.yml), branch layer (committed .werkator.yml on the branch). Branch layer takes precedence for everything except pinned keys (host secrets, sandbox policies, trust gate). A branch can never reach credentials, disable its sandbox, or bypass its own CI gate.

Configuration

Configuration is purely YAML-based, with no API config, no environment variables for CI settings, and no runtime UI for adding jobs.

Config File Locations (layered)

Layer	Path	Committed	Purpose
Built-in	—	—	Werkator's own hardcoded defaults
Instance	~/.werkator.yml	No	Registry of served repositories, instance-level settings (port, max concurrency, poll interval)
Project	.werkator.yml	Yes	Team-shared CI settings (Gitea connection, builds, docker/werkdock policy, retention)
Applied fragment	.git/werkator/.werkator.applied.yml	No	Deployment wrapper's parameters; validated strictly before installing
Machine	.git/werkator/.werkator.yml	No	Host/user secrets (git.token, git.account) and overrides; wins on any key present
Branch	.werkator.yml committed on branch	Yes	That branch's own build definitions and buildCommand/docker.image/artifact overrides

Key property: a branch cannot reach pinned keys (secrets, sandbox policy, trust gate, status context, pull-request gate). Trying to set them in the branch layer has no effect; they resolve from machine or project config.

Configuration Reference (Top-level Keys)

werkator:
  version:
    since: "0.9.18"      # enforced floor: Werkator refuses if it is older
    below: "2.0"         # optional caution marker

server:                   # instance-level HTTP server (not in branch config)
  publicBaseUrl: "https://ci.example.org/"
  port: 18080
  bindAddress: "127.0.0.1"  # default loopback for unauthenticated UI/API
  nginx:                 # opt-in managed nginx+certbot for HTTPS (ADR 0005)
    enabled: false
    serverName: ""
    httpPort: 8080
    httpsPort: 8443

gitea:                   # repository side (project or machine config)
  baseUrl: "https://git.example.org"
  owner: "my-org"        # user or org owning the repo
  repo: "my-repo"
  statusContext: "werkator"  # label on commit status checks; pinned (cannot be overridden by branch)

git:                     # credentials (machine config only; never committed)
  account: "ci-bot"
  token: "glpat-xxxxx"

executor:                # global build concurrency (instance-level)
  maxConcurrent: 1       # changes require restart
  maxConcurrentPerRepository: 1   # optional cap per repository below the global one

builds:                  # named build definitions (jobs); not inherited per-branch
  default:               # base every other definition inherits settings from
    trigger:             # **only trigger** is NOT inherited
      onPush: true       # build new commits
      branches: ["*"]    # glob patterns; empty = all; "!" prefix excludes
      atTimes: ["01:00"] # daily UTC times HH:MM or ??:MM (hourly)
      activeWithin: "24h" # only branches with commits in the last N time
    cleanCommand: "rm -rf build"
    buildCommand: "./gradlew build"
    artifactDirs:
      - "build/reports"
      - "build/doc"
    stdoutLog: "build.stdout.log"
    stderrLog: "build.stderr.log"
    requirePullRequest: false  # gate to refs/pull/*/head on origin (pinned)
    statusContext: ""     # empty = gitea.statusContext (pinned)
    docker:              # optional Docker build runtime
      enabled: false     # (pinned)
      image: "openjdk:21-slim"
      dockerfile: ""     # empty = pull as-is; set to rebuild when content changes
      context: "."
      network: ""        # empty = Docker default (pinned)
      env: {}
    werkdock:            # optional bubblewrap sandbox (for hosts without root or Docker)
      enabled: false     # (pinned)
      rootfs: "/path/to/rootfs.tar" # or http(s) URL, downloaded once to .git/werkator/buildenv/
      binary: "werkdock" # PATH-resolved CLI for the sandbox
      env: {}            # environment inside sandbox (cleared; nothing leaks in)

  pitest:                # every other definition inherits default's settings, brings its own trigger
    trigger:
      atTimes: ["01:00"]
      branches: ["main", "release/*"]
    buildCommand: "./gradlew piTestFull"
    statusContext: "werkator/pitest"

artifacts:               # build artifact storage and retention
  rootDir: ""            # empty = $XDG_STATE_HOME/werkator/artifacts/<repo-key>
  retentionPerBranch: 5  # keep the artifacts of the N newest builds per branch
  retentionMaxAge: ""    # additionally drop the artifacts of builds older than this (e.g. "30d")
  keepLatestGreen: true  # keep the latest successful build even beyond retention limits

watcher:                 # branch polling (not instance-level in server config)
  pollInterval: "10s"
  newBranchMaxAge: "5d"  # only auto-pull origin branches younger than this
  pullRequestGate: true  # honor requirePullRequest gates (false for plain git without PR refs)
  fastForwardLocalRefs: true  # at end of poll, ff local refs/heads/* to origin

Runtime Config Inspection

java -jar build/libs/werkator.jar config:print         # only explicitly set values
java -jar build/libs/werkator.jar config:print --full  # all values including defaults
java -jar build/libs/werkator.jar config:print --show-secrets  # include git.token in plaintext

Features

Build Execution

• Commit-based: builds are always keyed to a specific commit SHA, never a branch name
• Branch worktrees: each branch reuses its own persistent worktree (.git/werkator/worktrees/<branchKey>) across builds, preserving incremental caches
• Three runtimes:
  1. Native: commands run directly on the host
  2. Docker (docker.enabled): clean/build commands run inside a container with the build worktree mounted, Gradle caches in a named volume, .git mounted read-only (so builds can run read-only git commands but cannot reach git.token)
  3. Werkdock (werkdock.enabled): bubblewrap sandbox (bwrap) for hosts without root or Docker (e.g., Hostsharing webspaces), filesystem-only isolation, network and UID shared with host
• Isolated .git access: in all runtimes, .git/werkator/ is masked so builds cannot read machine config or control tokens
• Concurrent builds: global cap of executor.maxConcurrent, optional cap per repository executor.maxConcurrentPerRepository, one build per branch at a time (others queue)

Build Definitions and Triggering

• Named builds (builds.<name>): each definition has a trigger block (when/where) and everything else (what to do)
• Trigger types: onPush: true (every new commit), atTimes: ["HH:MM", "??:MM"] (scheduled daily or hourly)
• Branch selectors: glob patterns (* matches across /); prefix with ! to exclude (exclusions always win)
• Inheritance: a definition inherits all settings from builds.default except its trigger block
• Status context: commit statuses are reported under gitea.statusContext or a per-build override; two builds of the same commit with the same context overwrite each other
• Pull-request links: with a Gitea token the pages show a branch's open pull request as PR#<number>, linked to Gitea; read by the watcher once per poll cycle, no configuration key
• Pull-request gate (requirePullRequest): limit builds to commits that are pull-request heads (checked via git ls-remote without API token); pinned at machine/project level, cannot be overridden by branch

Watcher (Auto-Build)

• Polls origin every watcher.pollInterval (default 10s)
• Detects new commits on tracked branches and enqueues builds matching the trigger rules
• Pulls new branches on origin younger than watcher.newBranchMaxAge (default 5d)
• At end of each poll cycle, fast-forwards local refs/heads/* to origin (if watcher.fastForwardLocalRefs: true)
• Per-context concurrency guard: one polling loop per RepoContext, serialized across the instance
• Startup recovery: resumes unfinished builds and re-evaluates pending ones on restart

Artifacts

• Stored on the filesystem under artifacts.rootDir (default $XDG_STATE_HOME/werkator/artifacts/<repo-key>)
• Per-branch retention: keep the artifacts of the retentionPerBranch newest builds; additionally drop the artifacts of builds older than retentionMaxAge; older builds and those of branches deleted from origin stay in the history as records without artifacts, up to 100 per build name
• Permanent latest-green link: when keepLatestGreen: true, the latest successful build is kept and served at /branches/<branch-key>/… even if newer builds fail
• Artifact directories are archived at their committed path; build/reports is special (archived as reports/ and browsed via the artifact page's report index)
• Directory ownership is repaired after Docker builds to match the host user

Metrics

• System resource sampling: CPU (load), memory (resident, virtual), disk usage per repository
• Sampled once per poll cycle and aggregated
• Exposed via JSON API at /api/repos/<name>/metrics

Web UI

• Server-rendered Thymeleaf (no SPA framework)
• Static assets + one hand-written static/werkator.js (no frontend build pipeline)
• Per-repository view (drop-down in page title to switch between registered repositories)
• Branches view: branch name, latest build status, latest timestamp; artifact links; rebuild button
• Build detail: commit SHA, branch, trigger time, runtime, command output, artifact list
• Artifact serving: direct downloads and in-browser viewing (logs, reports)
• No authentication on read (intended; statuses and artifacts can be linked from anywhere)
• Runs behind a reverse proxy on a loopback interface (or managed nginx container for opt-in HTTPS)

CLI Commands

init

Prepare a git repository for Werkator. Runs once per repository from a checked-out working tree.

java -jar werkator.jar init [--systemd] [--apply FILE]...

Actions:

1. Detect repository root (git rev-parse --show-toplevel)
2. Auto-detect Gitea connection from origin URL
3. Create .git/werkator/.werkator.yml (machine config with auto-detected stubs)
4. Create .werkator.yml (project config template with defaults and Gitea connection)
5. If --apply FILE: install a YAML fragment as the applied layer (see config reference); repeated, the fragments are merged in order
6. If --systemd: generate a systemd user unit for persistent server operation

build [--repo NAME] BRANCH

Trigger a manual build of a branch, regardless of requirePullRequest gate.

java -jar werkator.jar build feature/foo
java -jar werkator.jar build --repo my-repo feature/foo

retry [--repo NAME] BRANCH [BUILD-NAME]

Re-run the latest build of a branch (or a named build, e.g., feature/foo@pitest).

java -jar werkator.jar retry main
java -jar werkator.jar retry main pitest

status [--repo NAME] [BRANCH]

Print the status of builds. If no branch given, show all branches.

java -jar werkator.jar status
java -jar werkator.jar status feature/foo

server

Start the HTTP server on the configured port. Serves registered repositories (or the current one without ~/.werkator.yml).

java -jar werkator.jar server

The server listens on server.bindAddress:server.port (default 127.0.0.1:18080). With server.nginx.enabled: true, also starts a managed nginx container for HTTPS. The server runs until killed; no daemon mode.

config:print [--full] [--show-secrets]

Print the effective merged configuration.

java -jar werkator.jar config:print
java -jar werkator.jar config:print --full  # include defaults
java -jar werkator.jar config:print --show-secrets  # unmask git.token

export:forgejo [REPO] [--stdout] [--strict] [--output-dir DIR] [--runs-on LABEL]

Convert the committed .werkator.yml into Forgejo Actions workflows (RFC 0006, step 1); tools/werkator-to-forgejo wraps it. Reads the committed file only, never the .git layers or ~/.werkator.yml. REPO is the target repository or a directory inside it; the .werkator.yml in its root is read, and the workflows are written into it. Writes .forgejo/workflows/werkator-<build>.yml (or below --output-dir, relative to the root), one per build definition that triggers on its own; follow-ups become jobs with needs. What has no workflow counterpart is a comment in the workflow and a line on stderr; --strict writes nothing and exits 1 then.

JSON API

The server exposes a JSON API at /api/repos/<name>/…:

Endpoint	Method	Purpose
/api/repos/<name>/status	GET	Build status of all branches
/api/repos/<name>/status/<branch>	GET	Build status of a single branch
/api/repos/<name>/status/<branch>/<build-name>	GET	Status of a named build (e.g., feature/foo@pitest)
/api/repos/<name>/build	POST	Trigger a manual build (JSON body: {"branch": "…", "buildName": "…"})
/api/repos/<name>/retry	POST	Re-run the latest build (JSON body: {"branch": "…", "buildName": "…"})
/api/repos/<name>/metrics	GET	System metrics (CPU, memory, disk)
/api/repos/<name>/artifact/<branch>/<file-path>	GET	Download an artifact

Response format (status endpoints):

{
  "repo": "my-repo",
  "branches": [
    {
      "name": "main",
      "latestBuild": {
        "status": "SUCCESS",
        "startTime": "2024-01-01T12:00:00Z",
        "endTime": "2024-01-01T12:05:30Z",
        "duration": "PT5M30S",
        "commitSha": "abc123...",
        "artifacts": [
          {"path": "build/reports/", "size": 12345},
          {"path": "build/doc/", "size": 67890}
        ]
      }
    }
  ]
}

Common Patterns for Agents

How to Detect Build Failures

• Query /api/repos/<name>/status/<branch> and check latestBuild.status
• Values: PENDING, RUNNING, SUCCESS, FAILURE
• A branch with no builds yet has latestBuild: null

How to Retrieve Artifacts

• Artifacts are stored and served at /branches/<branch-key>/<pool-name>/<file-path>
• For the default build: /branches/<branch-key>/build/reports/ (report index)
• For a named build: /branches/<branch-key>@<name>/<file-path>
• Permanent latest-green link: /branches/<branch-key>/latest/… (always serves the latest successful build, if any)

How to Check Build Logs

• Logs are captured to stdoutLog and stderrLog in the build worktree
• Served as artifacts at /branches/<branch-key>/build.stdout.log and .../build.stderr.log
• Tail the log via the file download or read it from the artifact store directly

How to Understand Build Times

• Each build records startTime and endTime (ISO 8601 UTC)
• duration is the elapsed time (ISO 8601 Duration, e.g., PT5M30S)
• Metrics endpoint exposes system load during builds

How to Work with Multiple Builds per Branch

• Define multiple named builds in builds with different trigger blocks and statusContext values
• A branch can have one "default" build and several named builds (e.g., feature/foo and feature/foo@pitest)
• Query /api/repos/<name>/status/<branch>/<build-name> for the named build's status
• Artifact paths include the pool name: /branches/<branch-key>@<name>/…

How to Add CI to a Branch

The branch's own .werkator.yml (committed on the branch) can:

• Override buildCommand, cleanCommand, artifactDirs, docker.image, werkdock.env
• Add new build definitions with their own triggers
• Cannot change: secrets, sandbox policy (docker.enabled, werkdock.enabled), trust gate (requirePullRequest), or status context labels (those are pinned)

Example: try a new Docker image on a feature branch:

# .werkator.yml on feature/test-docker
docker:
  image: "openjdk:21-alpine"  # override the default image for this branch
buildCommand: "./gradlew build -x slowTest"  # faster iteration

Push and the next poll will build with the new image; other branches are unaffected.

Deployment

Werkator is designed for deployment on Hostsharing managed containers, Managed Webspaces, and other hosts with limited privileges. It includes no reverse proxy of its own; instead, it runs behind the host's existing nginx, Apache, or HAProxy.

See docs/deployment.md for:

• Systemd user service setup
• Integration with Hostsharing hsadmin
• Nginx/Apache proxy configuration
• Managed nginx container (opt-in HTTPS via ADR 0005)
• Runtime bundle for hosts without Java (ADR 0006)

Testing and Debugging

Run Tests

./gradlew test
./gradlew test --tests "de.javagil.werkator.ApplicationContextIntegrationTest"  # single test class

Tests use Kotest FunSpec + MockK; SpringExtension is globally registered. See .claude/skills/writing-tests/SKILL.md for patterns.

Debug Configuration

java -jar werkator.jar config:print --full --show-secrets

View Effective Build Settings

java -jar werkator.jar config:print | grep -A 20 "builds:"

Monitor Build Execution

• Web UI: http://127.0.0.1:18080/ (or configured port)
• API: curl http://127.0.0.1:18080/api/repos/<name>/status
• CLI: java -jar werkator.jar status

Check Repository Registry (Multi-Repo Instance)

cat ~/.werkator.yml | grep -A 10 "repositories:"

Architecture Decision Records

Major decisions are documented in docs/adrs/:

• ADR 0001: Test framework (Kotest + MockK + WireMock + Testcontainers)
• ADR 0002: Gradle 8.14.5
• ADR 0003: Spring Boot 4.0.6
• ADR 0004: Rewrite architecture (JSON-file persistence, server-rendered UI, systemd)
• ADR 0005: Managed nginx+certbot for opt-in HTTPS
• ADR 0006: Runtime bundle distribution (jlink-trimmed JRE + jar)
• ADR 0007: Build definitions with trigger blocks (ADR 0007)
• ADR 0008: Bubblewrap build runtime (bwrap) for unprivileged hosts (ADR 0008)
• ADR 0009: Multi-repo instance (one service, multiple repositories)
• ADR 0010: The build configuration stays expressible as a Forgejo Actions workflow

See docs/adrs/ for full texts.

Related Documentation

• docs/configuration.md — complete configuration reference
• docs/bootstrapping.md — repository initialization workflow
• docs/deployment.md — production deployment and systemd setup
• AGENTS.md — shared agent instructions (package structure, hard invariants, testing patterns)