werkator/docs/plan/15-runtime-bundle-distribution.md

# Step 15: Self-Contained Runtime Bundle Distribution

Prerequisites: steps 12, 13.
Read `README.md` first.
This step revises the "Future: Docker-based Deployment" section of `docs/bootstrapping.md` — see the ADR requirement below.

## Goal

Deploy GitTally on hosts that provide Docker and git but no Java runtime (Hostsharing container servers, e.g. `tallyman@vm4006`).
GitTally is distributed as a self-contained runtime bundle: a jlink-trimmed JRE plus `gittally.jar` plus a launcher script, packed as one tarball.
The JAR stays the primary artifact for development and for hosts that already have a JRE.

## Distribution Format Decision (ADR 0006)

Three formats were considered; write ADR 0006 recording the decision and this rationale:

- **jlink runtime bundle (chosen)** — no production-code changes, plain JVM semantics, one tarball to `scp`.
  git and docker CLIs are used from the host, worktree paths stay host paths, and the `init --systemd` unit works unchanged because `java.home` and the running-jar path resolve into the bundle.
- **GraalVM native image (rejected)** — Spring AOT evaluates bean conditions at build time.
  GitTally's dual-context design (CLI context without web, second `SpringApplication` with the `server` profile and `WebApplicationType.SERVLET`, `@Profile("!server")` `CliRunner`, `@Profile("server")` lifecycles) cannot be represented in a single AOT arrangement.
  Supporting it would require replacing the profile wiring with runtime guards and collapsing the two context shapes — an invasive rewrite with regression risk for the JVM path.
- **Containerized GitTally runtime (rejected, was the `docs/bootstrapping.md` sketch)** — needs git and docker CLIs inside the image, a same-path `$HOME` mount plus docker-socket mount and uid/gid mapping so that `DockerBuildRunner`'s `--volume $workspace:$workspace` sibling mounts keep working, and a hand-edited systemd unit.
  Kept as the documented fallback if the bundle approach ever becomes unworkable.

## Target Host Facts (verified 2026-08-10)

- `vm4006.hostsharing.net`: Debian 13, x86_64, glibc 2.41, git 2.47.3, docker 26.1.5, GNU make, systemd user session running with `Linger=yes`, no Java runtime.
- The jlink image contains natively linked JVM libs, so it must be built on glibc ≤ 2.41 for the same architecture; the Ubuntu 24.04 dev machine (glibc 2.39, x86_64) qualifies.
  For reproducible builds elsewhere, the bundle can be built inside an `eclipse-temurin:21-jdk` container (glibc 2.39 base).

## Design

Gradle:

- Add a `runtimeBundle` task (depends on `bootJar`); the normal `./gradlew build` stays unchanged.
- The task runs `jlink` from the configured Java toolchain (every JDK 21 ships jlink; no new toolchain requirement).
- The JDK module list is pinned in the build script, computed once via `jdeps` on the exploded boot jar and its `BOOT-INF/lib`; document the jdeps command next to the list and re-check it when dependencies change.
- Bundle layout: `gittally/jre/` (jlink image), `gittally/lib/gittally.jar`, `gittally/bin/gittally` (sh launcher: `exec "$DIR/../jre/bin/java" $JAVA_OPTS -jar "$DIR/../lib/gittally.jar" "$@"`).
- Output: `build/distributions/gittally-runtime-linux-x64.tar.gz` with preserved execute permissions.

Deployment (no code changes expected):

- Unpack to `~/opt/gittally/` on the target host; run everything via `~/opt/gittally/bin/gittally`.
- `init --systemd` already generates `ExecStart=<java> $JAVA_OPTS -jar <jar> server` from `java.home` and the running jar path — from the bundle both resolve into `~/opt/gittally/`, so the unit points at the bundle without changes.
  Verify this instead of adapting code; adapt only if the resolution fails.
- Updating GitTally = unpack a new bundle over `~/opt/gittally/` (or switch a symlink) and restart the service.

Documentation:

- `docs/deployment.md`: prerequisites become "JRE 21 **or** the runtime bundle"; add a section "Hosts Without a Java Runtime (Runtime Bundle)" with build, `scp`, unpack, and systemd setup.
- `docs/bootstrapping.md`: replace the "Future: Docker-based Deployment" section with the runtime bundle and a pointer to ADR 0006.
- `docs/adrs/0006-…`: the distribution-format decision (see above).
- `docs/migration-from-legacy.md`: add a note that migrating to a different host allows parallel operation, with a distinct `gitea.statusContext` per instance so the two CIs do not overwrite each other's commit statuses.

## Tests

- No production code changes are expected, so no new unit tests; existing tests must stay green.
- Smoke-verify the bundle manually: `bin/gittally --help`, `init` in a scratch repo, `config:print --full`, a short `server` run, and `init --systemd` unit content pointing into the bundle; document the results in this file.
- Verify on vm4006 (which has no Java): copy the bundle, run `bin/gittally --help` and `config:print`; document the results in this file.

## Acceptance Criteria

- `./gradlew ktlintFormat` then `./gradlew build` is green, with unchanged toolchain requirements.
- `./gradlew runtimeBundle` produces a tarball whose `bin/gittally` runs `--help`, `init`, and `server` on a machine without any Java runtime.
- A fresh deployment to vm4006 following `docs/deployment.md` and `docs/migration-from-legacy.md` reaches a running service: web UI reachable, a Docker build succeeds, commit status arrives in Gitea, managed nginx/TLS works (`server.nginx.enabled: true`, DNS for `serverName` pointing at vm4006).
- vm2176 (legacy) keeps running in parallel during the migration; the legacy service is only retired after vm4006 is verified.
- Docs and ADR 0006 written as described; document deviations in this file.

## Result (2026-08-10)

Implemented as designed; no production-code change was needed.
The step was originally drafted for a GraalVM native image; it was re-planned to the jlink bundle after the Spring-AOT build-time condition evaluation turned out to be incompatible with the dual-context CLI/server wiring (see ADR 0006).

- `runtimeBundle` task in `build.gradle.kts` with the pinned module list (jdeps output plus java.logging, jdk.crypto.ec, jdk.management, jdk.zipfs); launcher script in `packaging/gittally`; tarball ~66 MB.
- Smoke tests on the dev machine (with `JAVA_HOME` unset and a stripped `PATH`): `--help`, `init --systemd` in a scratch repo, `config:print --full`, and a `server` run all passed; `/` served HTTP 200 and `/api/branches` returned JSON.
- The `init --systemd` unit generated from the bundle points at `<bundle>/jre/bin/java` and `<bundle>/lib/gittally.jar` as predicted — no detection code needed.
- Verified on vm4006 (no Java installed): bundle unpacked to `~/opt/gittally`, `--version`, `--help`, and `status` in a scratch repo (host git via `GitCommandRunner`) all worked.

Production deployment to vm4006 (2026-08-10, same session):

- `hs.hsadmin.ng` cloned to `~/hs.hsadmin.ng` on vm4006; legacy configuration from vm2176 (repo `.gitTally` + `gitTally.env`) migrated to `.gittally.yml` per `docs/migration-from-legacy.md`; Gitea token moved (the token in vm2176's `gitTally.env` file was stale — the valid one came from the running legacy process environment).
- `statusContext: GitTally@vm4006` for the parallel phase; rename to `GitTally` after vm2176 is retired.
- systemd user service installed via `init --systemd` from the bundle and running; watcher fetches origin branches with the migrated credentials.
- Managed nginx/TLS live: Let's Encrypt certificate for `vm4006.hostsharing.net` obtained, `https://vm4006.hostsharing.net/` serves the UI with a valid chain, HTTP 301s to HTTPS (Hostsharing routes public 80/443 to `httpPort`/`httpsPort`, same as on vm2176).
- Fix discovered during rollout: certbot removed `ssl-dhparams.pem` from its repository, so the first nginx start failed with HTTP 404.
  The DH parameters (RFC 7919 ffdhe2048) are now bundled as the classpath resource `nginx/ssl-dhparams.pem` instead of downloaded; the download seam and `NginxConfigFiles.DH_PARAMS_URL` were removed (revises the step-13 note about the download).
  Legacy on vm2176 only kept working because its state dir cached the file.

Second fix discovered by the first real build: under a **rootless** Docker daemon, `DockerBuildRunner` ran the build container as `--user <host-uid>` (ported from legacy).
With rootless identity mapping the host user is container root, and the host uid inside the container falls into the subuid range — the container could not create `.gradle` in the freshly created worktree ("Failed to create parent directory").
Legacy on vm2176 only worked because its ownership-repair chown had (unintentionally) moved `build/` and `.gradle/` into subuid ownership on the host (verified: owned by uid 166536 there) — a stable but wrong equilibrium tied to its reused primary checkout.
The rewrite now always runs the build container as `--user 0`: under rootless that IS the unprivileged host user (files stay host-owned, the repair chown degenerates to `0:0`); under rootful daemons the behavior is unchanged (root + chown to the host ids).

Third finding (config, not code): the hsadmin-ng Liquibase migration tests (`LiquibaseCompatibilityIntegrationTest`, `ImportHostingAssets.liquibaseMigrationForBookingAndHosting`) failed on vm4006 with "environment variable HSADMINNG_POSTGRES_ADMIN_USERNAME not set".
These tests run Liquibase programmatically without Spring's `spring.liquibase.parameters`, so the changelog parameters resolve only via Liquibase's env-var substitution; the legacy script had a host-env passthrough for exactly these variables.
The values `admin`/`restricted` are hsadmin-ng's committed defaults, but only for the other execution paths: `.tc-environment` for the documented dev workflow (`. .tc-environment; ./gradlew …`) and the `${…:admin}` fallbacks in `src/test/resources/application.yml` for the Spring-managed Liquibase path.
The programmatic path deliberately has no fallback — `009-check-environment.sql` exists to verify the environment is configured — so a CI environment must export the variables itself.
Fix: `HSADMINNG_POSTGRES_ADMIN_USERNAME=admin` and `HSADMINNG_POSTGRES_RESTRICTED_USERNAME=restricted` in `branches.default.docker.env` on vm4006 — the mechanism `docs/migration-from-legacy.md` prescribes for the legacy passthrough list; alternatively the build command could source `.tc-environment` like the dev workflow.
Verified by running both test classes in the build container with the variables set: green.
Open oddity: the same commit passed on vm2176 although neither its daemon environment, build image, Gradle volume, nor any build-script mechanism supplies these variables there (an unused git-ignored `.environment` file exists in its primary checkout, but nothing in the build reads it); the loading path on vm2176 remains unidentified.

Fourth finding (GitTally limitation, worked around in config): with all tests green, the build then failed in hsadmin-ng's `:prQuickCheck` — "fatal: not a git repository".
GitTally builds in a git worktree whose `.git` is a pointer file into the primary repository's `.git/worktrees/…`, and the Docker build container (deliberately, credentials live under `.git/gittally/`) only mounts the worktree — so build steps that call git fail; the legacy script avoided this by building in the primary checkout.
Workaround: `prQuickCheck` removed from the vm4006 build command — it is a PR quality gate against a base branch and has no meaning in a post-merge master build (on vm2176 it only passed as an accidental no-op).
The underlying question (safe git availability inside Docker build containers without exposing `.git/gittally/` secrets) is left as a follow-up design task.

Cutover completed (2026-08-10, same day): after three green master builds and verified Gitea statuses from vm4006, the legacy service on vm2176 was disabled and removed from systemd.
vm4006's `statusContext` was switched to the canonical `GitTally` (effective without a restart — the Gitea client loads the config per call), and the branches still carrying red statuses from the buggy first hours were re-queued.
vm2176 now runs only a redirect nginx container (`gittally-redirect`, ports 8080/8443 like before): HTTP and HTTPS answer 301 to `https://vm4006.hostsharing.net$request_uri`, the ACME webroot keeps serving so the `nginx-letsencrypt-renew.timer` continues to renew the old host's certificate (the renew unit gained an `ExecStartPost` nginx reload).
GitTally answers the legacy static page names (`/index.html`, `/branches.html`, `/history.html`, `/system.html`, `/about.html`, `/license.html`) with permanent redirects to the new routes, so pre-rewrite links survive the host redirect.

Update to v0.9.8 (2026-08-10): the running build was awaited first (a restart would have killed it), then service stopped, `~/opt/gittally` backed up to `~/opt/gittally.v0.9.7.bak` and the new bundle unpacked over it, service started.
Verified live: `/` reports v0.9.8, the nav has no `Current` entry, the permanent `🔗` link appears only on branches whose latest build is their latest green one, and the newly linked reports answer 200 — including the stable `/branches/<branch>/reports/profile/`.
Master has no profile report yet because its `.gittally.yml` still carries the pre-PR#282 build command; it appears once that PR merges.

Update to v0.9.9 (2026-08-11): no build was running, service stopped, `~/opt/gittally` backed up to `~/opt/gittally.v0.9.8.bak` and replaced by the new bundle, service started.
The changed `server.bindAddress` default was harmless here because vm4006 sets `0.0.0.0` explicitly in `.git/gittally/.gittally.yml` — which the managed nginx container needs.
Found and fixed on the host: `.git/gittally/.gittally.yml` (the Gitea token) was still `0644` and its directory `0755` from the pre-0.9.9 `init`; both were tightened to `0600`/`0700` manually, as the new code only sets the mode for files it creates.
Verified live: `/` reports v0.9.9, HTTP 301s to HTTPS with a valid certificate, `/api/builds/latest` answers, a control token in the query string is rejected with 403, `config:print` masks `git.token`, and the mobile header stacks title over repository name.

Update to v0.9.10 (2026-08-11): same procedure, `~/opt/gittally.v0.9.9.bak` as the rollback copy.
Verified live: `/` reports v0.9.10, the pages no longer contain `gittally-control-token`, `/api/builds/latest` and `/branches` still answer 200 without any credential, and a mutation without the token is rejected with 403.
The operator has to paste the token from `~/hs.hsadmin.ng/.git/gittally/control-token` once per browser now.

Update to v0.9.11 (2026-08-14): same procedure, no build was running, `~/opt/gittally.0.9.10.bak` as the rollback copy.
Verified live: `bin/gittally --version` reports v0.9.11 before the start, the service is `active`, `/` answers 200 with v0.9.11 in the footer, `/releases` shows the v0.9.11 entry, and `/api/builds/current` is empty.
The watcher's new local-ref fast-forward logged nothing, because `~/hs.hsadmin.ng` had already been reset to `origin/master` by hand — it only acts on a branch that actually lags behind.

Update to v0.9.12 (2026-08-26): same procedure, `~/opt/gittally.0.9.11.bak` as the rollback copy; a build was running — interrupted and re-enqueued by the startup recovery as designed.
Verified live: `bin/gittally --version` reports v0.9.12 before the start, the service is `active`, `/` shows v0.9.12, and the recovery re-enqueued exactly one build per affected branch (the pre-fix duplicate queue entries were collapsed by `markStaleRunningAsInterrupted` + `latestPerBranch`).
Shipped fixes: prune never removes queued/running results (a branch deleted mid-build stays visible), and manual triggers dedup against an already active build of the same branch and commit.

Update to v0.9.13 (2026-08-28): same procedure, no build was running, `~/opt/gittally.0.9.12.bak` as the rollback copy.
Verified live: `bin/gittally --version` reports v0.9.13 before the start, the service is `active`, `/` answers 200 with v0.9.13 in the footer, `/api/builds/latest` carries the new `name` field, and the watcher polls without errors.
Shipped feature: an `autoBuild.times` entry can carry its own `buildCommand` and `name` — a named nightly slot (e.g. `master@nightly`) gets its own branches-view row, retention pool, and permanent latest-green link; restart/retry/recovery repeat a build with its original command and name.

Update to v0.9.14 (2026-08-28): same procedure, `~/opt/gittally.0.9.13.bak` as the rollback copy; a build was running — interrupted and re-enqueued by the startup recovery as designed (now under its recorded build definition `default`).
Verified live: `bin/gittally --version` reports v0.9.14 before the start, the service is `active`, `/` answers 200 with v0.9.14 in the footer, the watcher polls without errors, and the deprecation warning for `branches.master.autoBuild` appears once in the log.
Shipped feature: named build definitions (ADR 0007) — the `builds` section defines jobs with `onPush`/`atTimes` triggers, branch selectors (globs, `activeWithin`), and setting overrides; named builds record under `<branch>@<build>` pools; the v0.9.13 per-slot syntax was removed again.

Update to v0.9.15 (2026-08-29): same procedure, `~/opt/gittally.0.9.14.bak` as the rollback copy, no build was running.
Verified live: `bin/gittally --version` reports v0.9.15 before the start, the service is `active`, the API answers, and the watcher polls without errors.
The leftover `builds.maxConcurrent` in the repository's committed `master` config is ignored with exactly one warning per context instead of failing the configuration — the reason it is tolerated rather than rejected: that config needs a colleague's approval to change, and the installation must not be stuck on a key it is meant to forget.
Shipped feature: the `.gittally.yml` committed on a branch takes precedence for its `builds` section as well, and the watcher reads it per origin branch (`git show`, cached by head commit) to decide which of that branch's builds are due.
Verified live within seconds of the start: the build definition `reactivate-pi-test-with-full-pitest`, which exists only on the branch `mihoe/reactivate-pi-test` and not in the `master` config, fired its due 05:00 UTC slot, recorded under the pool `mihoe/reactivate-pi-test@reactivate-pi-test-with-full-pitest`, and ran the branch's `pitestFull` command in the build container inherited from the `master` config.

Update to v0.9.16 (2026-08-29): same procedure, `~/opt/gittally.0.9.15.bak` as the rollback copy.
A build was running at the first attempt, so the deployment aborted itself before stopping anything; the service was then stopped in the first idle window (waiting rather than interrupting was the operator's call).
Verified live: `bin/gittally --version` reports v0.9.16 before the start, the service is `active`, `/releases` lists v0.9.16, and the watcher polls without errors.
Shipped features: hourly scheduled builds (`atTimes: ["??:05"]`) and the artifact page showing the command a build actually runs.
Verified live: the branch's `atTimes: ["??:00"]` — rejected by v0.9.15 with a warning on every poll cycle — is accepted, and its 06:00 UTC slot fired right after the restart.

Update to v0.9.17 (2026-08-29): same procedure, `~/opt/gittally.0.9.16.bak` as the rollback copy.
Deployed under the operator's condition "only if no build is running": the deploy script re-checks `/api/builds/current` immediately before the stop and exits without touching anything when a build is executing.
Verified live: `bin/gittally --version` reports v0.9.17 before the start, the service is `active`, `/releases` lists v0.9.17, the served `gittally.js` carries the resume listeners, and no errors in the log.
Shipped fix: a page returning from the background fetches the current state immediately instead of showing (and ticking) the state it was left in.

Update to v0.9.18 (2026-08-29): same procedure, `~/opt/gittally.0.9.17.bak` as the rollback copy, no build was running.
Verified live: `bin/gittally --version` reports v0.9.18 before the start, the service is `active`, `/releases` lists v0.9.18, the watcher polls without fetch or poll errors, and the only warnings are the two known `builds.maxConcurrent` lines from the repository's committed config.
Shipped feature: a configuration file can declare the GitTally it is written for (`gitTally.version.since`/`below`), so an incompatibility is named instead of silently ignored.
The configs of the watched repository declare nothing yet and are unaffected — a missing declaration is never an error.