implemented 04-build-executor.md incl. concurrency amendment: async builds in per-branch worktrees, builds.maxConcurrent, cancellation, live logs; fix .gitignore build/ rule that silently excluded the de.hoennig.gittally.build package (also recovers the step-01 domain files)

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
This commit is contained in:
Michael Hoennig
2026-07-07 08:43:28 +02:00
co-authored by Claude Fable 5
parent ae3ae7aa04
commit b379bc0a6b
31 changed files with 1790 additions and 12 deletions
+20 -4
View File
@@ -88,11 +88,11 @@ flowchart LR
### BuildWorker
- Pending-Build übernehmen
- Worktree anlegen
- Branch-Worktree anlegen oder wiederverwenden
- Commit auschecken
- Build starten
- Artefakte sammeln
- Worktree entfernen
- Worktrees entfallener Branches entfernen
### GitStatusPublisher
@@ -118,10 +118,26 @@ flowchart LR
### Commit-basierte Builds
GitTally baut immer einen konkreten Commit und niemals nur einen Branchnamen.
Buildergebnisse werden intern trotzdem pro Branch geführt: derselbe Commit auf zwei Branches ergibt zwei getrennte Builds mit eigenem Status, eigenen Artefakten und eigenem Worktree.
Das ist gewollt, weil Buildläufe den Branchnamen einbeziehen können (Umgebungsvariable `branch`).
In Gitea hängt der Commit-Status dagegen am Commit-SHA: zeigen zwei Branches auf denselben Commit, überschreiben sich ihre Statusmeldungen gegenseitig (der zuletzt gemeldete gewinnt).
Falls das je stört, kann der Branchname später in den Status-Context aufgenommen werden (z. B. `GitTally/main`), sodass ein Commit mehrere unabhängige Statuszeilen bekommt.
### Worktree pro Build
### Worktree pro Branch
Jeder Build erhält einen eigenen temporären Worktree.
Jeder Branch erhält einen eigenen, wiederverwendeten Worktree unter `.git/gittally/worktrees/<branchKey>`.
Der Branch-Key ist der dateisystem-sicher bereinigte Branchname plus 12 Zeichen SHA-256 des Originalnamens (z. B. `main-0d6e4079e367`).
Der Hash schützt nur vor Kollisionen durch die Bereinigung (`feature/x` vs. `feature_x`); der Commit ist bewusst nicht Teil des Keys, damit das Verzeichnis über alle Builds des Branches stabil bleibt.
Der zu bauende Commit wird darin detached ausgecheckt.
Der primäre Checkout wird niemals für Builds verwendet.
Die Wiederverwendung erhält inkrementelle Build-Caches; das `cleanCommand` des Branches bestimmt, wie viel davon überlebt.
### Parallele Builds
Mehrere Branches können gleichzeitig bauen (`builds.maxConcurrent`, Default 1).
Pro Branch läuft höchstens ein Build gleichzeitig.
Ein neuer Build desselben Branches wartet, bis der laufende fertig ist.
Ob ein neuer Commit den laufenden Build seines Branches stattdessen abbrechen soll, wird später entschieden und ist möglicherweise konfigurierbar.
## Konfigurationsmodell
+8
View File
@@ -34,6 +34,14 @@ gitea:
repo: my-repo # repository name
statusContext: GitTally # label shown on Gitea commit status checks (default: GitTally)
# Build execution.
builds:
# How many branches may build at the same time.
# At most one build per branch runs regardless; each branch builds in its own
# git worktree under .git/gittally/worktrees/, never in the primary checkout.
# Changing this value requires a restart.
maxConcurrent: 1
# Build artifact retention.
artifacts:
# number of builds to keep per branch
+34
View File
@@ -47,3 +47,37 @@ Consider `builds.timeout` only if trivial; otherwise defer.
- `./gradlew ktlintFormat` then `./gradlew build` is green.
- While a test build sleeps, the repository reports `RUNNING` — proven by a test.
## Execution Notes (done 2026-07-07)
Implemented as designed in `de.hoennig.gittally.build`; build green, 18 new tests
(`BuildExecutorTest`, `ProcessBuildRunnerTest`, `ArtifactKeysTest`, plus two new `FileBuildResultRepositoryTest` cases).
Deviations and decisions:
- One-build-at-a-time uses a single-thread worker executor; a second `startBuild` queues and stays `PENDING` until the first finishes.
`PENDING` is persisted synchronously in `startBuild`, so a queued build is immediately visible.
- `BuildResultRepository` gained `updateByArtifactKey(...)` (extends step 01) so transitions always hit the exact entry, even when a newer `PENDING` entry of the same branch was queued meanwhile.
- `cancel()` sets the flag and destroys the process tree directly (TERM, 2s wait, KILL via `ProcessHandle.descendants()`); no separate monitor thread.
The flag is checked before each command and after `waitFor`, so a cancel between clean and build commands still records `CANCELLED`.
- Artifact key naming (`ArtifactKeys`) was needed here because `BuildResult` requires a key; it follows the legacy scheme (sanitized name + 12-char SHA-256 prefix + sanitized ISO timestamp + hash) using the UTC `Instant`, not local time.
Step 05 should reuse it rather than re-implement.
- `ArtifactStore` is an interface in the `build` package with a logging `NoOpArtifactStore` placeholder; step 05 replaces the placeholder and implements the real store in `de.hoennig.gittally.artifacts`.
- The combined live log is `build.log` inside the per-build staging directory (a temp dir exposed via `RunningBuild.stagingDir`/`liveLogFile`); output is flushed per read chunk so the log grows while the build runs.
- Commands run via `bash -c` with the branch name in the environment as `branch`, like legacy `run_build_command`; a failing `cleanCommand` fails the build without running `buildCommand`.
- `BuildResultRepository` is wired as a Spring bean (`BuildConfiguration`) at `.git/gittally/build-results.json` relative to the working directory, matching how `ConfigLoader` resolves the override file; `git rev-parse --git-path` style worktree resolution can come later if needed.
- Gitea `target_url` is not published yet; the artifact page URL scheme only exists from step 07 on.
- `builds.timeout` was deferred (not trivial alongside cancellation semantics); no config keys were added or changed.
## Amendment: Concurrent Builds and Per-Branch Worktrees (2026-07-07)
Refactored on request, superseding parts of the notes above:
- Builds now run concurrently up to the new config key `builds.maxConcurrent` (default 1), enforced by a global semaphore sized on first use (changing it requires a restart).
- At most one build per branch at a time, enforced by one serial worker per branch; a second build of the same branch queues as `PENDING` and runs afterwards ("finish, then next").
Whether a new commit should instead cancel the branch's running build is a later, possibly configurable decision (see step 06).
- Each branch builds in its own reusable git worktree at `.git/gittally/worktrees/<branchKey>` (`BranchWorkspaces`/`GitWorktreeWorkspaces`), checked out detached at the requested commit — the primary checkout is never touched.
Reuse keeps incremental build caches; `cleanCommand` decides how much of them survives.
`GitService` gained `worktreeAdd`, `worktreePrune`, and `checkoutDetached` for this.
- API change: `currentBuild()` became `currentBuilds(): List<RunningBuild>`, and `cancel()` became `cancel(artifactKey)`; a queued build can be cancelled too and is recorded `CANCELLED` when its worker picks it up.
- The Gitea `PENDING` status is now published synchronously in `startBuild`, so queued builds are visible in Gitea while they wait for a slot.
- Branch config is still loaded from the primary repository directory, not from the branch's checked-out `.gittally.yml`; honoring the branch's own committed config would be a separate decision.
+5 -2
View File
@@ -16,8 +16,11 @@ Create package `de.hoennig.gittally.watcher`:
1. `fetchOrigin()` (errors: log, publish watcher health state, retry next cycle — no internal retry-sleep loops like legacy `retry_origin_change_check`).
2. Determine candidate branches: changed local branches, plus new origin branches within `watcher.newBranchMaxAge` (step 02 operations).
3. Check auto-build time slots (below).
4. If the executor is idle, dequeue the next branch: checkout, reset to origin, `startBuild` (async).
5. Run repository retention pruning and artifact pruning.
4. Start builds for due branches via `startBuild(branch, commit)` (async).
The executor prepares a per-branch worktree itself (step 04 amendment) — the watcher must never check out or reset the primary worktree.
Multiple branches may build concurrently (`builds.maxConcurrent`); the executor already serializes builds of the same branch, so the watcher only has to avoid enqueueing a branch that is already pending or running.
5. Run repository retention pruning and artifact pruning; also remove worktrees under `.git/gittally/worktrees/` of branches no longer on origin (`git worktree remove` or delete + `worktreePrune`).
- Decide here (or defer with a note): when a new commit arrives for a branch whose build is still running, keep the current queue-behind behavior or cancel the running build and start fresh — this may become a per-branch config option.
- Startup sequence (port of legacy recovery): mark stale running builds interrupted, then enqueue restartable branches.
- Auto-builds: per-branch `autoBuild.enabled` + `times` (UTC HH:MM) from the merged `branches` config.
Persist "already triggered for slot/day" state via a small JSON file next to the build results (replaces `auto-builds.tsv`).
+2 -2
View File
@@ -22,9 +22,9 @@ JSON API (package `de.hoennig.gittally.server`), replacing the legacy `/control/
- `GET /api/builds/latest` — latest build per branch.
- `GET /api/builds/history` — all builds, newest first.
- `GET /api/builds/current` — running build, its live status, and log tail (`?offset=` for incremental log fetch).
- `GET /api/builds/current` the list of running builds (there can be several, one per branch, up to `builds.maxConcurrent`), each with live status and log tail (`?offset=` for incremental log fetch, addressed by artifact key).
- `GET /api/status/{commit}` — effective status including Gitea lookup (replaces `/control/status`); must return an explicit error state on Gitea failure, never hang.
- `POST /api/builds/{branch}/restart`, `POST /api/builds/current/cancel`, `DELETE /api/builds/{artifactKey}` — guarded by a simple token like the legacy cancel token; wire into executor/watcher/repository.
- `POST /api/builds/{branch}/restart`, `POST /api/builds/{artifactKey}/cancel` (cancel takes the artifact key because multiple builds can run concurrently), `DELETE /api/builds/{artifactKey}` — guarded by a simple token like the legacy cancel token; wire into executor/watcher/repository.
- `GET /api/watcher` — watcher health (last poll, last error).
- Artifact serving: `GET /artifacts/{artifactKey}/**` streaming from the artifact store, with no-cache headers for html/json/log.
+2 -1
View File
@@ -17,7 +17,8 @@ Views (ported from legacy, see analysis for columns and behavior):
- `/` (Latest): latest build per branch — status badge, branch and commit with Gitea links and copy buttons, times, duration, artifact link, restart/delete actions.
- `/history`: all builds.
- `/current`: running build with live log view (incremental fetch via `/api/builds/current?offset=`), cancel button, and a clear "no build running" state.
- `/current`: running builds with live log view (incremental fetch via the step 07 current-builds API), a cancel button per build, and a clear "no build running" state.
Remember there can be several running builds (one per branch, up to `builds.maxConcurrent`); the view must list all of them, not assume a single one.
- `/builds/{artifactKey}`: artifact index — build command, logs, links into archived report directories (rendered from the artifact store, not pre-generated HTML).
- Shared layout: view toggle nav, footer with version and optional impressum link, `prefers-color-scheme` support (port the legacy CSS look loosely, keep it simple).
+4 -1
View File
@@ -30,6 +30,9 @@ These are proposals baked into the steps.
Revisit them in an ADR if a step uncovers problems.
- Build results are persisted as a JSON file under `.git/gittally/`, behind a `BuildResultRepository` interface (no database, but replaceable).
- Builds run concurrently up to `builds.maxConcurrent` (default 1), but never more than one build per branch at a time.
Each branch builds in its own reusable git worktree under `.git/gittally/worktrees/`, checked out detached at the requested commit — never in the primary checkout.
A later step must decide (possibly per config) whether a new commit on a branch cancels that branch's running build or waits for it; for now new builds queue behind the running one.
- Artifacts stay on the filesystem, served by the Spring server.
- The web UI is server-rendered HTML plus small JavaScript polling JSON endpoints (no SPA framework).
- The watcher is a Spring-managed scheduled component, decoupled from the build executor via the result repository and events.
@@ -45,7 +48,7 @@ Foundation:
Core engine:
- [ ] `04-build-executor.md` — async build execution with logs, cancellation, status transitions
- [x] `04-build-executor.md` — async build execution with logs, cancellation, status transitions
- [ ] `05-artifact-store.md` — artifact persistence, naming, retention
- [ ] `06-watcher.md` — branch watching, scheduling, auto-builds