werkator/.claude/skills/architecture/SKILL.md

name, description

name

description

architecture

Detailed Werkator subsystem architecture — CLI wiring and exit codes, server mode, web UI, configuration system, git access, build execution (native, Docker, and the werkdock sandbox), watcher poll cycle, and system metrics. Use when designing or modifying code in the commands, config, git, gitea, build, artifacts, watcher, metrics, or server packages, or when a question goes beyond the overview in AGENTS.md.

Werkator Architecture

Werkator is a lightweight, declarative CI/CD build system. It is a dual-mode application: CLI (interactive, status, config) and Server (HTTP, persistent).

Entry Point and CLI Wiring

Spring Boot starts via WerkatorApplication. A separate CliRunner component (in the same file) implements both CommandLineRunner (runs picocli) and ExitCodeGenerator (returns the exit code). exitProcess is called only from main() via SpringApplication.exit() — never inside run(). This keeps the Spring context alive during tests.

Picocli commands are Spring @Component beans. The root command (WerkatorCommand) declares subcommands as class references in @Command(subcommands = [...]). Picocli resolves them from the Spring context via the auto-configured IFactory bean.

WerkatorApplication   ← @SpringBootApplication
CliRunner             ← CommandLineRunner + ExitCodeGenerator
WerkatorCommand       ← root @Command, delegates to subcommands
commands/
  InitCommand         ← "init [--systemd] [--apply FILE]..."
  ServerCommand       ← "server"
  StatusCommand       ← "status [--history]"
  BuildCommand        ← "build [<branch>]"
  RetryCommand        ← "retry"
  ConfigPrintCommand  ← "config:print [--full]"
  ControlTokenCommand ← "control-token"
  ExportForgejoCommand ← "export:forgejo [REPO] [--stdout] [--strict]" (RFC 0006 step 1, forgejo package)

status, build, and retry implement Callable<Int> for their exit codes (0 success, 1 build failure, 2 usage/config errors). build and retry run builds through the async BuildExecutor but block until completion via ConsoleBuildRunner, which streams the live log to stdout and waits for the artifact persist before the JVM exits. Branch arguments resolve legacy-style name fragments (BranchNameResolution); the CLI reuses UiFormats so console and web UI display the same formats.

The web application type is set to none in application.yml, so plain CLI runs never start a web server. The server subcommand launches a second SpringApplication with WebApplicationType.SERVLET and the server profile, then blocks until shutdown. application-server.yml switches the web type (spring.main.* properties beat programmatic builder settings), CliRunner is @Profile("!server") so the second context does not run picocli again, and the watcher poll loop starts only in the server profile (ServerWatcherLifecycle). The JSON API, artifact serving, and the web UI live in the server package; mutating endpoints are guarded by a generated control token under .git/werkator/control-token.

Web UI

The UI is server-rendered Thymeleaf (UiController, templates under src/main/resources/templates/) plus one hand-written JavaScript file (static/werkator.js) — no SPA framework, no frontend build pipeline. Pages render the full state server-side; the script then polls the JSON API and re-renders table bodies from data. Every fetch has a timeout and failures flip an explicit error badge — never re-fetch and diff whole HTML pages, and never leave a spinner without an error path (the legacy defect). Polling pauses while the tab is hidden. UiFormats/werkator.js must produce the same display formats (timestamps, durations). Two independent staleness signals, never merged: the live-indicator chip in the site header says whether this browser reaches the server, and the watcher-banner (fed from /api/watcher, in the shared viewHead fragment, which also fills the view-meta line next to the view's title) says whether the server reaches origin — a watcher that cannot fetch leaves the server perfectly reachable and every row stale.

Wherever a branch name is shown — the Latest, Branches, History and Current Builds views, the artifact page, the Repositories page and the home cards — its open pull request is shown as PR#<number>, linked to the pull request in Gitea: in a column of its own in the build tables (a line of its own on a phone card, none when empty, PR#82), apart from the branch on the Current Builds cards and the build page, beside the name on the Repositories page, and as plain text on a home card, which is a link already. The pages never ask Gitea: PullRequestIndex holds the open pull requests per repository by head branch, refreshed by the watcher once per poll cycle after a successful fetch (GiteaClient.readOpenPullRequests, token required, pull requests from forks left out), and a failed read keeps the last known list. The DTOs carry it as pullRequest (number, url); Redaction drops it with the branch, and a gone branch has none.

The History page carries the build-duration trend above its table (BuildDurationTrends, PR#49): one card per build definition — never per branch, since a branch leaves two or three measurements and disappears — over the last 30 successful builds, however old (PR#84; a time window left rarely built repositories without a trend and let a slow drift become the median), showing the series, the median with min and max, and the latest run's distance from that median, amber from a quarter above it. Derived on read from the recorded history, so nothing is stored; since records outlive their artifacts (up to 100 per build name), a single-branch build such as a nightly or release build has a series too; the median rather than the average is what makes pooling every branch of one definition affordable. Served as GET /api/builds/durations (repository-scoped like every route), which answers a private repository with an empty list rather than a redacted one — not because the timings are secret (they are metadata, and latest/branches/history hand them out) but because a card is grouped by build definition, and redacting that name to n/a leaves every card labelled alike and the list saying nothing. UiFormats.trendDelta and its werkator.js twin must word the distance identically.

A failed or timed-out run is outdated once its branch is gone from origin or a newer run of its name ended with a verdict (OutdatedRuns, PR#156). A verdict is success, failed or timed_out, never pending, running, cancelled or interrupted. It is derived on read over the whole history and carried as BuildResultDto.outdated and RecentBuildDto.outdated. The History shows such a run grey with an outdated tag, the home strip as a grey bar; UiFormats.outdatedTitle and its werkator.js twin word the reason.

The visual language is RFC 0001, option G ("Kin"): the author's blog as the family — white paper, IBM Plex, a blue-shifted petrol (--accent) used only as an accent, clay (--clay) for failures, a neutral dark mode, the browser's preference by default and overridable by the reader's own choice (RFC 0003) — which means the dark palette stands twice in werkator.css, guarded by prefers-color-scheme and by [data-theme="dark"], and DarkModeTokensUnitTest holds the two blocks together. Everything is custom properties at the top of werkator.css; the drafts the stylesheet was lifted from live under docs/rfcs/0001-web-ui-instrument-panel/drafts/. The page skeleton is three fragments in fragments.html: siteHeader (wordmark with the inline logo, leading to the home page /, the repository crumb — omitted on the instance pages home, system, release notes and repositories —, the instance navigation with Home, Repositories (several repositories only) and System, the live chip, the token, theme and reload buttons), viewHead (the repo-line on repository pages of a multi-repository instance, the repository views' tabs, then the view's <h1> and its one-line explanation keyed by view), and footer (the version as the link to the release notes, imprint).

What makes several repositories one UI (ADR 0009) is the Repositories page and the repository line (RFC 0001), not a switcher: RepoOverviews derives one RepoOverviewDto per served repository — the newest build per latestPerName, running and pending counts, the failed latest builds of the last seven days (RECENT_FAILURE_WINDOW), the watcher's per-repository problem in the banner's words — sorted newest build first, never-built last; GET /api/repos serves the list, /repos renders it (redirecting to the home page with one served repository), and UiController.baseModel puts the RepoLineView (the current repository plus the conspicuous ones, and the link to the rest) on every repository page. werkator.js re-renders both from the endpoint on the table interval; UiFormats.repoStatusLine/count and the script's repoStatusLine/countText, and RepoLineView.chipOf/renderRepoChip, must produce the same texts. Fragment names are deliberately not element names — Thymeleaf resolves fragments :: nav to the first <nav> element before the fragment of that name. Statuses are a dot and a Mono word (.status::before, outlined for pending, pulsing for running), so status status-<name> stays the only markup. Icons are stroke glyphs as CSS masks (.icon.icon-<name>, --icon data URIs appended at the end of werkator.css) colored by currentColor; werkator.js builds them with icon(name)/iconLink(...), the templates write the <span class="icon icon-…"> by hand — no emoji anywhere. IBM Plex Sans and Mono are bundled as WOFF2 under static/fonts/ (latin subset, six faces) and declared by @font-face in the stylesheet: nothing is loaded from a third party.

The home page (/, RFC 0003) is the instance's entrance, served with one repository as well as with twelve: a 288 px system rail left — the verdict, one row per gauge of the system page and four smaller tiles, each a link into /system — and the repository mosaic right, one card per repository in the Repositories page's own order (newest activity first, so attention is a tint and never a new position) with the bar strip of its last twelve builds, height for duration and colour for outcome. HomeViews.kt derives all of it from GET /api/repos (the strip is RepoOverviewDto.recentBuilds) and GET /api/system; only the sparklines are drawn client-side, by the system page's own drawSpark from GET /api/system/history. The page does not use viewHead — no tabs, no h1, no repository line — but it carries the watcher banner, and the footer sits outside its two-column grid so it spans both. HomeBarView.strip/barStrip, HomeCardView.toneOf/cardTone and HomeTexts/homeMeta are twins like every other pair.

The Builds panel stands above the mosaic (PR#160). Across all repositories it shows the running builds, longest running first, then the queued ones, longest waiting first, then the failures, newest first, at most BuildActivity.LIMIT (5). A failure there is failed or timed out, not outdated (PR#156), started within RECENT_FAILURE_WINDOW, and of a name that does not run or wait again. BuildActivity derives the list on read from each repository's history and redacts a private repository's build like RepoOverviews; GET /api/activity serves it. A tile leads to the Current Builds view while its build runs or waits, and to the build page once it failed. HomeBuildView.toneOf/buildTone and the tile markup of home.html/renderHomeBuild are twins. A tile carries the ticker's data attributes, so its time counts on between polls. Five tiles fit one row on a desktop screen: every page is up to 1440 px wide, and a tile at least 180 px.

A private repository gives out its metadata, never its content (RFC 0003, D8), and its name is content: RepoRouting is the single place a route's repository segment is resolved, and it answers a visitor without the control token for a private repository with the same UnknownRepositoryException an unserved name gets — same status, same body, and a cached privacy answer so not the same request takes longer either. Redaction keeps its old job of filling the list, the line and the mosaic without naming anything.

Configuration System

Werkator is configured by three YAML files, deep-merged by ConfigLoader (later wins):

1. .werkator.yml at the repo root — committed, shared team settings.
2. .git/werkator/.werkator.applied.yml — not committed; the instance fragment installed verbatim by init --apply (step 23), validated strictly (unknown keys refused) and replaced wholesale on re-apply. --apply repeats (PR#87): the fragments are merged in order and the merge is written as YAML, so the self-deployment can travel apart from the sandbox policy.
3. .git/werkator/.werkator.yml — not committed; machine-specific overrides and secrets (git.account, git.token); hand-edited, always wins.

A key present in a layer wins even when blank — an empty gitea.baseUrl says "report nowhere" — with one exception: a blank git.account/git.token is dropped from a repository layer (withoutBlankCredentials, PR#51) instead of shadowing the instance defaults.git below it, because a credential left empty is an unfilled placeholder and never a choice; the templates init wrote until then put token: "" into every machine config and silently disabled the inherited credentials, which is why init now writes that block commented out and loadFile treats a comment-only file as a layer that sets nothing.

Every lookup falls back to the pre-rename name (ConfigFiles): .gittally.yml, and .git/gittally/.gittally.yml for the machine layer. Current name first, and where both exist the old one is ignored rather than merged — a missing config is not an error, so an un-renamed installation would otherwise start on defaults without a single failure.

On top of those comes the branch layer: the .werkator.yml committed on a branch, applied by loadWithBranchLayer (the watcher passes the content read via git show, loadForWorktree the file in the build worktree). A branch describes its own CI and wins over both layers — the whole builds section — so a configuration can be tried out on a branch without touching other branches' builds. stripPinned removes what is not a description of this branch's build: git, server, gitea, executor, watcher, and — inside every builds definition as well as every legacy branches entry — requirePullRequest, statusContext, docker.enabled/docker.network/docker.socket, and werkdock.enabled/werkdock.rootfs/werkdock.binary/werkdock.gradleHome (the section was called bwrap until v1.2.0; renameLegacySandbox maps the old name onto the new one on every raw layer, before merging). Subtraction is not the whole pinning: checkLayerShape refuses a branch layer that writes a non-map where a block belongs — docker: null left nothing to strip while deepMerge still replaced the host's block — and enforcePinned writes the pinned values back from the host's layers after the merge, falling back to the host's builds.default for a definition the host does not know or the branch emptied. builds.<name>: null remains a branch's way of saying it runs no such build.

Sandboxed follow-ups may be defined or changed in a branch's trigger.afterSuccessOf. ConfigLoader records which trigger blocks came from instance defaults, the applied fragment or the repository's machine config; these stay pinned even in a sandbox. Follow-ups with native host execution, Docker socket access or restartsInstance also stay pinned to the primary configuration. With a registry, restartsInstance is refused in every repository whose entry lacks selfDeploys: true (checkSelfDeploys in loadPrimary, PR#87): what a repository's files say cannot decide whether the instance runs from it, so the instance file does. The permission to add a follow-up is derived from the primary sandbox policy after default inheritance, never from branch-supplied flags.

A pull-request target build (trigger.onPullRequestTarget) is pinned whole, because it holds the host's secrets while it judges a change. stripPinned drops every such definition from the branch layer — the host's ones, which the branch must not change in any key or set to null, and the ones the branch writes itself — and enforcePinnedPullRequestTargets replaces them in the merged result with the primary configuration's bound definitions, so a builds.default the branch rewrote cannot reach them through inheritance. checkPullRequestTargets refuses one combined with another trigger key, with requirePullRequest, on builds.default, or as a predecessor of a follow-up.

Each file is version-checked before merging (werkator.version.since/below, ConfigVersions.verdict), so the message can name the file to fix: since is hard in both directions — too old a Werkator, or a file written before ConfigVersions.FORMAT_BROKE_IN and read after it — while below only warns. There is no format version (apiVersion) on purpose: only one configuration generation is supported, and the declared version exists to make the incompatibility nameable.

After merging, resolveBuildSections decides which section describes the builds: builds or the legacy branches, never both. With no real build definition (builds.maxConcurrent is not one, dropNonDefinitionBuilds already drops it) the legacy path runs and branches.default is merged into every other branch entry; otherwise branches is dropped with a warning and mergeBuildDefaults applies builds.default as the base of every other definition — its settings only, never its trigger block (TRIGGER_KEYS, a single key so that a selector added to TriggerConfig later is non-inheritable by construction). checkTriggerBlocks refuses a definition still writing onPush/atTimes/branches/activeWithin flat, per file and scoped like the version check. Deciding this on the merged map is deliberate: a build defined on a branch and unknown to the host still inherits the host's builds.default, sandbox policy included, which is what keeps the pinned keys effective for it. The result is bound to the WerkatorConfig data classes (config/WerkatorConfig.kt), which define the schema and all defaults; WerkatorConfig.buildSettings(branch, build) is the single answer to "what does this build run".

Three places must stay in sync when config keys change: the WerkatorConfig data classes, the commented templates generated by InitCommand, and the reference in docs/configuration.md.

Git Access

GitService shells out to the git CLI via GitCommandRunner (a thin ProcessBuilder wrapper; no JGit). Commands that need repo information take it as a constructor dependency so tests can mock it. HTTPS fetches authenticate via a temporary, secret-free GIT_ASKPASS script (GitAskPass) with credentials from config passed through environment variables.

Repository Context

Everything repository-scoped goes through a RepoContext (repo package, ADR 0009): the primary checkout (workingDir), the repository's BuildResultRepository (.git/werkator/build-results.json), its ArtifactStore (keyed by the repository path), and a short name defaulting to the directory basename — the future route segment. RepoContexts.open(dir) builds one (running the pre-rename state-dir migration for that repository on the way); RepoRegistry opens one per entry of the instance configuration's repositories — or the current directory without a registry — lazily on first use and loudly: a non-repository entry or a duplicate name aborts the start naming the home file, a repository whose config must not be read (ConfigException) is skipped with an error. RepoConfiguration provides registry.current() (the cwd when served, else the first entry) as the RepoContext bean; the controllers no longer take it — they resolve per request from the {repo} path segment, and registry.current() is what the unscoped routes mean. The --repo mixin (RepoOption) selects by name in build, retry, and status. Git access and config loading stay path-based services taking repo.workingDir; the instance configuration (~/.werkator.yml, ConfigLoader.homeDir/WERKATOR_HOME, bound as InstanceConfig) is folded in by ConfigLoader.loadRaw itself — its defaults below every repository layer, its server/executor/watcher.pollInterval overlaid on top and stripped from the repository files with one warning — so every consumer of load(dir) sees the instance values without knowing the file. The context object is the identity (executor pools, watcher memory are keyed by it), so exactly one is opened per repository — RunningBuild carries it too, so currentBuilds() says which repository a running build belongs to: the current-builds view and API serve only the served repository's builds, and the watcher's worktree pruning is protected by its own repository's builds alone. Routes, pages, and artifact files are repository-scoped (session D): every mapping exists twice, /api/repos/<name>/… and /repos/<name>/… beside the unscoped form, an unknown name is a 404, and the link prefix follows the number of served repositories — one repository keeps its existing URLs, several make every link name its repository and replace the repository name in the page title with a drop-down that switches repositories, keeping the current view.

Build Execution

BuildExecutor runs builds asynchronously: startBuild(repo, branch, commit, build) takes the RepoContext first; up to executor.maxConcurrent builds run concurrently across all repositories (default 1, sized once from the first build's config — an instance-level setting), but never more than one build per (repository, branch) at a time, and with executor.maxConcurrentPerRepository never more than that many per repository (PR#103) — a build takes its repository's slot before a global one, so it never sits on a global slot while it waits. Pull-request target builds (trigger.onPullRequestTarget) take their slots from a second pool (BuildPool, PR#119), sized by executor.maxConcurrentPullRequestTargets (default 2) and maxConcurrentPullRequestTargetsPerRepository (default 1): BuildSlots holds one set of fair semaphores per pool, so neither pool's waiting builds queue behind the other's, and a review never waits for the CI it reviews. Such a build also runs in its own serial lane, keyed by its pool name <head-branch>@<build> rather than by its head branch, because it runs in a fresh workspace (BranchWorkspaces.prepareFresh) named after that pool; two heads of one pull request still run one after the other. The current-builds view tags it PR target, and the System page counts it in its own PR targets tile (RunningBuildDto.pullRequestTarget, SystemDto.maxConcurrentPullRequestTargets). Each branch builds in its own reusable git worktree at .git/werkator/worktrees/<branchKey> (BranchWorkspaces), checked out detached at the requested commit — the primary checkout is never used for builds. Status transitions are persisted in the build's own RepoContext.results (JSON file under that repository's .git/werkator/), published to Gitea non-fatally — with the build's artifact page as target_url when server.publicBaseUrl is set, repository-scoped through RepoLinks — and emitted as BuildStatusChangedEvents. Every run belongs to a named build definition (job, ADR 0007): the YAML builds section defines triggers (onPush, atTimes), branch selectors (branches globs, activeWithin), and build-setting overrides applied last over the merged branch config; the implicit default build (onPush, all branches) preserves the job-less behavior. Definitions are part of the branch layer — a branch may add and override its own, and they apply to that branch alone (its selectors are evaluated for it only) — while executor.maxConcurrent stays pinned. BuildResult.build records the job; restart, retry, and startup recovery re-run by that name, resolving settings from the current config. BuildResult.name — the pool, <branch>@<build> for non-default builds — keys everything display- and retention-side (repository grouping via latestPerName, retention pools, branches-view rows, permanent latest-green links), while BuildResult.branch keys everything git-side: origin lookups, the gone-from-origin mark of retention, worktrees (every build but a pull-request target runs in its branch's worktree, serialized per branch), and Gitea links/statuses. branches.*.autoBuild survives as a deprecated alias for a scheduled default-pool rebuild. Retention (artifacts.retentionPerBranch, retentionMaxAge, keepLatestGreen) is applied by the watcher's prune per name and decides over the artifacts only (PR#84): a result beyond it is stamped with BuildResult.artifactsPrunedAt and stays as a record, up to BuildResultRepository.RECORDS_PER_NAME (100) per name, skipped by every "latest" lookup and shown without artifacts (artifactsPruned in the DTO and the row view); a result whose branch is gone from origin is not dropped for that but stamped with BuildResult.branchGoneAt (PR#59) — it stays in the history as the record of the build, while latestFor/latestGreenFor/latestPerName skip it, the artifact store is pruned to the results withArtifacts(), and the row offers neither artifacts nor a Gitea branch link (branchGone in the DTO and the row view). The mark is cleared should the branch come back. Cancellation addresses a build by artifact key and terminates the whole process tree; so does the clock: BuildTimeouts.effective combines the host's executor.buildTimeout (instance-level, and executor is stripped from a branch layer) with the definition's own builds.<name>.timeout by taking the smaller, so a build may give itself less time and never more. One deadline covers the whole build, not each command, and a build that reaches it is recorded as TIMED_OUT — its own status, terminal and not restartable on its own, published to Gitea as failure. What a build spent its time on is recorded beside the duration (plan step 14, PR#70): BuildResult.phases holds one BuildPhase per stretch — queue, workspace, prepare (everything a BuildRunner does before the command's process exists, so the figure means the same for all three runtimes), clean, build, archive — a phase entered twice adding up its time; the complete list, archive included, is written by one silent update after the artifacts, since the terminal status has to be stored first. executor.overheadBudget (default 30s, instance-level like the rest of executor) warns once per build when workspace + prepare + archive exceed it and names the slowest of them; the queue wait is measured but never counted as overhead. Future code (watcher, server, UI) must not assume a single running build.

Beside the wall-clock duration, every run records the processor time its process tree consumed (BuildResult.cpuTime, PR#67): a sampler follows the tree through /proc/<pid>/task/<tid>/children once a second and keeps the largest sum of utime+stime+cutime+cstime over the live processes, summed over the build's commands. ProcessHandle.descendants() is deliberately not used — it walks the whole of /proc, which on a shared host is thousands of processes for a tree of a few dozen. The number says whether a long build worked or waited, which a duration alone cannot; it is null where nothing was measured, never zero.

On context close (e.g. systemd SIGTERM), a ContextClosedEvent listener in BuildExecutor terminates the process trees of all executing builds and waits (bounded) until their results are persisted as INTERRUPTED — a shutdown is never recorded as FAILED. Builds still queued stay PENDING and start no process. Both are re-enqueued by the watcher's startup recovery; INTERRUPTED therefore publishes as Gitea state pending, not failure (GiteaStateMapping).

The runtime is selected per build behind the BuildRunner interface: DispatchingBuildRunner (@Primary) routes to native ProcessBuildRunner (the default), to DockerBuildRunner when docker.enabled, or to WerkdockBuildRunner when werkdock.enabled — docker and werkdock are mutually exclusive per build and rejected in buildSettings, never picked silently. The Docker runner shells out to the docker CLI (no SDK): it (re)builds the configured image when the Dockerfile inputs changed (tracked via the org.werkator.build-inputs-sha256 image label), maintains a per-repo Gradle cache volume, mounts the worktree and the Docker socket into a labelled (org.hoennig.werkator) --rm --init container (with TESTCONTAINERS_HOST_OVERRIDE from docker.env if it names one, else the child address a rootless daemon's rootlesskit reports for the slirp4netns port driver — RootlessKitPorts, asked over dockerd-rootless/api.sock on every build, PR#86 — else localhost under network: host and host.docker.internal otherwise), and repairs workspace ownership in-container after each command (under a rootless daemon the container runs as root, which is the host user, and the repair degenerates to 0:0). Git works inside the container: the primary .git is mounted read-only with .git/werkator/ masked by an empty tmpfs (credential isolation) and the worktree's admin dir mounted read-write (gitMetadataMounts). The returned Process is the attached docker run client, so log streaming and termination work exactly like native builds. The image is built in the build's workspace from the real paths of the Dockerfile and the context, which must stay inside it, under Werkator's own tag werkator-build/<scope>:current per repository, branch and image; a pull-request target build builds none (PR#168). Since PR#168 it runs docker build before every command instead of on changed inputs; Docker's layer cache keeps that short, and the label only records the inputs.

No sandbox runner puts the value of a build's variable on its CLI's argv (PR#166). The werkdock runner passes a bare -e KEY with the value in werkdock's environment (RuntimeEnvironment); that environment holds only werkdock's own variables from Werkator's, and a build variable named like one of them keeps KEY=VALUE on the argv. The Docker runner writes the values to a 0600 file for --env-file (DockerEnvFile), since the docker client starts host programs with its environment; a value with a line break stays --env KEY=VALUE.

WerkdockBuildRunner (ADR 0008) is the third runtime, for hosts without root and without Docker — Hostsharing Managed Webspaces. It shells out to the bwrap CLI (no library): a prepared rootfs archive (werkdock.rootfs, built by tools/build-bwrap-rootfs.sh) is unpacked on demand into .git/werkator/buildenv/<envKey>/rootfs and bound read-only at /, with uid 0 inside mapped to the calling user; isolation is filesystem-only — network, uid, /proc, /dev are the host's by contract. It reuses the Docker runner's gitMetadataMounts; mount order matters (repo dir read-write before the metadata mounts and the workspace), and bind mountpoints missing from the rootfs are pre-created there, since the rootfs is a plain host directory while bwrap cannot mkdir against the read-only sandbox root. werkdock.enabled/werkdock.rootfs are pinned like the docker sandbox policy. The persistent toolchain home under .git/werkator/buildenv/home is bound as /root; with werkdock.gradleHome (PR#76, pinned, opt-in) a host directory is mounted after it as /gradle-user-home with GRADLE_USER_HOME set, so every sandbox naming the same path shares one dependency cache and one provisioned JDK — the same pair the Docker runner uses with its per-repository volume. The returned Process is the attached bwrap process, so streaming and cancellation are unchanged. The generic sandbox machinery is the standalone tool Werkdock (plan step 21: grown in werkdock/, consumed via the CLI since session C, its own repository since session E); the runner delegates to the werkdock CLI and this repository no longer carries its source.

Pull-Request Target Builds

The watcher reads the open pull requests once per cycle (PullRequestIndex.refresh, which returns this cycle's answer) and PullRequestTargets.decide names the due builds: one per primary pull-request target definition and open, non-draft, non-fork pull request whose head commit has no recorded result of that definition, and only while the forge's base commit equals the fetched origin head of the base branch. Gitea's draft flag is read, or the WIP:/[WIP] title prefix where the field is absent. BuildExecutor.startPullRequestTargetBuild records the build under the head branch (<head>@<build>, serialized with the branch's other builds) with BuildResult.pullRequest, checks out the base commit in a fresh worktree under .git/werkator/pull-request-worktrees/ (prepareFresh; the watcher removes those of finished builds), resolves the settings from the primary configuration alone (BuildSettingsSource, which also refuses the definition without a pull request), sets WERKATOR_PR_* and WERKATOR_GITEA_*, and publishes every status on the head commit (RunningBuild.statusCommit); the duplicate check goes by the head as well. Recovery, UI restart and retry re-run the recorded pull request; FollowUpTrigger ignores such results. The runners get pullRequestTarget = true for such a build and give it nothing other builds write: WerkdockBuildRunner binds <workspace>.home (GitWorktreeWorkspaces.pullRequestHomeOf, removed with the workspace by prepareFresh and the watcher) as /root instead of the shared toolchain home and drops werkdock.gradleHome; DockerBuildRunner mounts neither the per-repository Gradle volume nor the socket; ProcessBuildRunner refuses it. checkPullRequestTargetSandboxes refuses, in the primary configuration with builds.default applied, a target without docker or werkdock and one with docker.socket.

Watcher

Watcher replaces the legacy blocking main loop with a non-blocking fixed-delay poll cycle over the registry's contexts (start(repos), pollAll(repos), poll(repo) for one; recoverOnStartup(repo) per repository, each in its own guard). Every repository is polled in its own guard — a crash or an unreachable origin is that repository's report in WatcherState.repositories, and the next one is polled regardless; the top-level WatcherState fields aggregate the reports, reading exactly as before with one repository and prefixed by the repository name with several. What the watcher remembers per repository — the logged fetch error, the deprecation warning, the cached branch definitions — lives in a RepoWatch keyed by context. Per repository the cycle is: fetch origin, enqueue due branches (changed local, recent new origin, due auto-build slots) via BuildExecutor, then prune results, artifacts, and stale worktrees. After all repositories are polled, pruneBuildEnvironmentsSafely runs once per cycle for the whole instance (PR#77) — at most once an hour and never while a build executes: BuildEnvironmentPruner removes the werkator-buildenv-* images from werkdock's store whose source no repository's host configuration names (werkdock rmi; the store is shared per OS user, so this cannot be per repository) and the .git/werkator/buildenv/<hash>/ download caches of unnamed sources. werkdock.rootfs is pinned, so the host layers are the whole set of references. Branches whose build has requirePullRequest are enqueued only while their head commit matches a pull-request head, detected without an API token by listing refs/pull/*/head via git ls-remote (lazily, at most once per poll cycle); manual build commands bypass this gate, and watcher.pullRequestGate: false disables it globally for plain-git origins without pull-request refs. Which builds are due is decided per branch from that branch's own definitions (definitionsFor): the primary config with the branch's committed .werkator.yml merged on top, cached per branch by its head commit and the primary config it was merged with, so the git show runs only when the branch moved while an edited machine or project config still takes effect on the next poll, and falling back to the primary definitions when that config is unreadable. Nothing is scheduled until Watcher.start() is called explicitly (server/watch mode) — CLI commands and tests never start the loop. "Already built" is tracked via the result repository, not by moving local branch refs. After the enqueue decision — and only after it, because a local ref lagging behind origin is the change signal — the cycle fast-forwards the primary checkout's local branch refs to their origin counterparts (watcher.fastForwardLocalRefs, GitService.fastForwardLocalBranches), so build tools reading the shared .git from a worktree see the refs they expect; diverged or ahead branches are never touched. Auto-build slot state lives in .git/werkator/auto-builds.json; watcher health is exposed via Watcher.state().

System Metrics

SystemMetricsCollector samples CPU (/proc/stat deltas), RAM and swap (/proc/meminfo), the swap-out (pswpout of /proc/vmstat over five minutes, PR#146), the load average (/proc/loadavg), the pressure-stall information (/proc/pressure/cpu, /proc/pressure/memory and /proc/pressure/io, PSI some avg60), disk, and repository size every 15s, but only after ServerMetricsLifecycle (server profile) calls start() — like the watcher, nothing is scheduled in CLI runs or tests. Min/max/avg aggregation state persists as JSON in the instance's state home (${XDG_STATE_HOME:-~/.local/state}/werkator/system-metrics-state.json, moved there from the artifact root once, PR#33), so restarts continue the series. Unavailable sources (e.g. no /proc outside Linux) yield null metrics served as HTTP 200 by GET /api/system — the /system page shows n/a, never an error.

The same state file carries the sample history (MetricsHistory, PR#27), three tiers: every 15 s sample for four hours, one point per minute for a day and one per quarter hour for a month, the aggregated tiers with average and maximum, column-wise (one timestamp list, one value list per metric key) so the file stays a few hundred kilobytes. A minute is aggregated from the samples when the next one opens, a quarter from the minutes, so no accumulator is persisted, and the file is written once a minute (PERSIST_EVERY_SAMPLES) and on shutdown. GET /api/system/history?range=1h|4h|24h|7d|30d serves a cut of one tier (samples up to four hours, minutes up to a day, quarters beyond) with its peaks (HistoryPeaks: the highest value, when, and the UtilizationLevel it reached against the current total) — the evidence for a capacity decision. The verdict is computed on the server (SystemHealth.assess, healthy/pressure/overloaded with the rules that fired as reasons) and only rendered by the page, so a later executor change can act on the very same rules; CPU utilization alone never counts, waiting does (load above the core count, pressure, swap). CapacitySizing.of (PR#30, corrected in PR#42 and PR#58) sizes the machine from the quarter-hour tier. RAM is judged by waiting, never by a fill level (RFC 0004): a quarter hour counts as stalled when its average memoryPressure reached SystemHealth.MEMORY_OVERLOAD_PERCENT — shared, so a machine the health verdict calls healthy cannot be red in the sizing — and RAM is tight from 1 % and upgrade from 5 % of stalled quarter hours, the share CPU already uses; swap occupancy plays no part. The peak is named in the reason and never judged, with the single exception that it may still prove slack for a downsize — without the 40 % gate CPU and disk keep (PR#79), since the smaller bookable GiB that holds peak, headroom and reserve is the whole proof; it sizes a raise when one is due, rounded up and at least one step above what is booked, because a stall proves the peak was the ceiling. Disk is judged by the level it holds — the 95th percentile of the quarter-hour maxima of the last seven days (DISK_LEVEL_DAYS), the percentile the CPU watch band already uses, so a single quarter hour is a moment and not a level (PR#78; upgrade from 90 %, tight from 80 % where there is something to raise; the recommendation leaves a fifth free, a third for disks). Disk is also judged by its growth, the median of the daily deltas of the last week, so a step counts on one day and a trend on most, extrapolated thirty days ahead from the level, only from seven days of history on. The window maximum is named in the reason and vetoes a downsize (under 40 %), never asks for a raise. Every size is rounded to what can be booked (BookableSteps, a parameter of of(...) that the controllers derive from HostingEnvironments.bookableSteps — for Hostsharing 10 GiB per quota, 250 GB per volume, 1 GiB RAM; elsewhere the whole unit), and a downsize is advised only when the next bookable size down would still leave the target headroom, otherwise fits with a reason saying why. The steps, unlike the hosting itself, reach every viewer: they name nothing, and one machine must not advise two different things depending on who asks. ioPressure (PR#69) is sampled and shown like the other two but read by neither the health verdict nor the sizing, in RFC 0004's order — the series has to exist before anyone can say what a number means, and on a shared hive the disks are the part shared hardest, so it will say more about the neighbours than about this instance until somebody has watched it. oomKills is sampled into the history from /proc/vmstat as the count per bucket but deliberately not read by the sizing — the counter is machine-wide, so on a shared hive it counts every webspace's kills. It is a counted series (MetricsHistory.COUNTED_KEYS, PR#147): its buckets hold the sum, not the average, it has no averages entry, and GET /api/system/history serves its range total as totals for the OOM kills tile. CPU is sized from the quarter-hour averages, never the maxima — waiting (load above the thread count in more than 5 % of the quarters, the 95th percentile of those loads being the recommendation) and, below that, the 95th percentile of sustained utilization for the watch band. A CPU is counted in hardware threads (CapacitySizing.CPU_UNIT), which is what availableProcessors reports and what a webspace is billed by. RAM is advised in bookable whole GiB: the visible total rounded up is the booked size, and the difference the reserve the host keeps. Nothing is advised from less than a day. Served as capacity on GET /api/system and shown as colour-coded chips in the verdict panel; where the instance can tell it runs on a Hostsharing Managed Webspace (HostingEnvironments.detect, served as hosting) the chips carry an order button that composes a German mail to the service desk from the changes the user ticks — the page sends nothing itself. The /system page is a verdict line with the sizing chips beneath it, one gauge card per resource (percentage, bar, level pill, inline-SVG sparkline drawn by werkator.js, peak line of the selected range), and the tiles — on a desktop screen (from 1100 px) the cards in one row (PR#146), the eight tiles always in full rows of four, two on a phone (PR#147). The swap-out card is the one card without a total: its value is MiB/s, its bar and chart are scaled against metrics.swapOutCriticalMibPerSecond, its dashed lines and peak level follow the two swap-out thresholds instead of 80/90 %, and its detail line is the swap in use; UiFormats.percent/levelLabel/healthLabel and their werkator.js twins must stay identical.

The disk metric is the budget this installation manages (DiskQuota, PR#16; RFC 0002): readDisk() shells out quota -u -g --no-wrap --raw-grace next to the Files.getFileStore read, parses it into candidate budgets, and takes the tightest quota where one applies — a Hostsharing Managed Webspace — else the volume itself (werkdock doctor already reads the same number in Go, as a one-off pre-build check). diskTotalGib/diskUsedGib/diskFreeGib always come from that one source; SystemMetrics.diskSource names it (UiFormats.diskTotal/werkator.js's formatDiskTotal render the same info-line suffix), and a source change resets the disk min/max/avg (PersistedMetricsState.diskSourceKey) so a stale volume-sized ceiling never survives a newly detected quota. The volume is a constraint on the shown budget, never a competitor for the same number: when it will run out first, physicalFreeGib/physicalSource carry its headroom, the gauge marks it on the bar and names it in the note, and the level is judged against that tighter ceiling (DiskMetrics.levelCeilingGib) while the scale stays put. What fills a disk is measured in the same throttled walk that produces repoSizeGib, tallied per subtree into workspaces, build environments, artifacts and the repositories themselves and keyed by file store (DiskUsage); whatever the budget holds beyond them is one honest other slice, and the page renders it as a donut in a dialog per disk card.

Since PR#33 the page measures a list of disks: metrics.disks of the instance file (an instance-level section like server, unreachable for a branch) names one filesystem per entry with a label and a key; without it the served checkout's filesystem is the one disk (disk/Disk). SystemMetrics.disks carries one DiskMetrics per entry, the series are disk.<key>.usedGib/freeGib in the aggregates, the history tiers (whose columns are the union of what was sampled, not a fixed list), and the history API; SystemHealth, HistoryPeaks, CapacitySizing (capacity.disks[]), the gauges, chips, and footer totals iterate the list. A pre-PR#33 state file is migrated on load (PersistedMetricsState.migratedTo): the single disk's series continue as the first configured disk's.