Merge remote-tracking branch 'origin/main' into 20-build-duration-tracking

# Conflicts:
#	docs/plan/README.md

docs/prs/2026-09-03-PR#16-quota-aware-disk-metrics.md

@@ -1,237 +0,0 @@
-> **WARNING:** This document describes only the change applied in this PR.
-> It may already be outdated once the next PR is merged.
-> Historic PR-documentation is not maintained along with new PRs — treat it as a snapshot, not as current documentation.
-
-## Related Links
-
-- The live system page: <https://werkator.javagil.de/system> (instance `mih09-werkator` on `mih09.hostsharing.net`, a Hostsharing Managed Webspace).
-- [Step 09: System Metrics](../plan/09-system-metrics.md) — where the disk metric comes from.
-- [ADR 0008](../adrs/0008-2026-09-01.bwrap-build-runtime.md) and [`werkdock doctor`](../../werkdock/internal/doctor/doctor.go) — the one place that already reads the group quota, as a one-off check before the first build.
-- Hostsharing-internal: the package's quota is set in HSAdmin; the values below were read on the host with `quota(1)`.
-
-## The Problem
-
-The system page shows the disk of the *host*, not the disk the instance can use.
-On `mih09` it reads `Disk total: 70.99 GiB`, `Disk used 37.13 GiB`, `Disk free 33.86 GiB` — the numbers of the whole `/` volume, shared by every package on that machine.
-What actually limits Werkator there is a **group quota** of the package `mih09`: 8 GiB soft limit, 12 GiB hard limit, 1.04 GiB used, measured on 2026-09-03 (see [Attachments](#measured-on-mih09-2026-09-03)).
-So the page promises 33 GiB of headroom where 7 GiB exist, and its warn/critical highlighting (80 %/90 % of the total) can never fire before a build fails with "Disk quota exceeded".
-
-The disk metric is read via `java.nio.file.FileStore` with `df` semantics (step 09).
-That is right on a host Werkator owns (vm4006, a Docker host) and wrong on every host where a user or group quota is the binding limit — the Hostsharing Managed Webspace being the deployment variant we now run in production.
-`werkdock doctor` already knows this: it checks the group-quota headroom before the first build, because the free space of the volume said nothing (a 1 GiB quota blocked a build mid-flight on `h68`, step 17).
-The system page should show the same truth, continuously.
-
-## Non-Goals
-
-- Implementing the change — this PR is the plan only; the implementation follows in the next PR.
-- Inode (file-count) quotas: `quota(1)` reports them, but the Gradle caches on `mih09` use 14 226 of 16.7 M files; a follow-up if it ever matters.
-- Alerting or refusing to start a build on a full quota — the page only shows; `werkdock doctor` keeps the one-off pre-build check.
-- A configuration switch: the quota is detected, never declared (see Open Questions).
-- Changing the JSON field names of `GET /api/system` or the metric rows of the page: `diskTotalGib`, `diskUsedGib`, `diskFreeGib` keep their names and their meaning "the budget the instance can fill".
-
-## The Scenarios
-
-### Feature: the disk metric is the tightest budget — user quota, group quota, or the volume
-
-#### Background
-
-- `quota(1)` prints one block per subject (`-u` the user, `-g` its groups), each with one line per filesystem: `blocks` (1 KiB units, currently used), `quota` (soft limit), `limit` (hard limit), grace, then the same four for files.
-  A subject without quota prints `… : none`.
-  A `*` after `blocks` marks "over the soft limit".
-- Three **candidates** can limit what a directory may still take: the user quota, the group quota (each the lines of the directory's filesystem) and the volume itself (the file store, `df` semantics).
-  Each has a headroom: `soft − blocks` for a quota, the usable space for the volume.
-- The **binding candidate** is the one with the smallest headroom; its numbers are the disk metric — total, used and free come from one source, never mixed.
-  On `mih09` today that is the group quota (6.96 GiB left against 33.86 GiB on the volume); a user quota, once the webspace introduces one, joins the comparison without any change.
-- For a quota the **total** is its **soft** limit, the hard limit is shown alongside (see Open Questions for the choice).
-- Where no quota line matches the directory's filesystem, the volume is the only candidate — Docker hosts and developer machines render exactly as today.
-- The directory is the first served repository's, as for the file-store metric today.
-
-#### Scenario#16.01: A group quota replaces the volume numbers
-
-So that the operator of a Managed Webspace sees the budget the package can fill, not the size of the host's disk.
-
-- **Given** `quota -u -g` reports no user quota and a group quota on the repository's filesystem with 8 GiB soft limit, 12 GiB hard limit and 1.04 GiB used
-  - **and** the file store reports 71 GiB total and 34 GiB free
-- **When** a sample is taken
-- **Then** `diskTotalGib` is 8.00, `diskUsedGib` 1.04 and `diskFreeGib` 6.96
-  - **and** the snapshot names the quota: group `mih09`, its filesystem, and the hard limit 12.00 GiB.
-
-##### Verified by
-
-- [SystemMetricsCollectorTest — "a group quota on the repository's filesystem replaces the file-store disk numbers"](../../src/test/kotlin/de/hoennig/werkator/metrics/SystemMetricsCollectorTest.kt) (planned)
-- [DiskQuotaTest — "the mih09 output parses into one group line per filesystem and no user line"](../../src/test/kotlin/de/hoennig/werkator/metrics/DiskQuotaTest.kt) (planned, fixture: the attachment below)
-
-#### Scenario#16.02: The tightest of user quota, group quota and volume binds
-
-So that neither a user quota below the group's, nor a nearly full host volume below both, is hidden by the wider budgets.
-
-- **Given** a user quota with 2 GiB headroom and a group quota with 7 GiB headroom on the same filesystem
-  - **and** the volume has 34 GiB usable
-- **When** a sample is taken
-- **Then** the user quota binds: its soft limit is the total, its blocks the used value
-  - **and** with the user quota reporting `none`, the group quota binds
-  - **and** with the volume down to 1 GiB usable, the volume binds and the metric shows the file-store numbers, quotas or not.
-
-##### Verified by
-
-- [DiskQuotaTest — "among user quota, group quota and volume the smallest headroom binds"](../../src/test/kotlin/de/hoennig/werkator/metrics/DiskQuotaTest.kt) (planned)
-
-#### Scenario#16.03: Only the quota of the repository's filesystem counts
-
-So that a full quota on another volume (on `mih09`: `/dev/sdb1`) does not shrink the metric of the volume Werkator writes to.
-
-- **Given** quota lines for two filesystems
-  - **and** the repository directory's file store is named like the first one
-- **When** the binding quota is chosen
-- **Then** the second filesystem's lines are ignored
-  - **and** the name is matched exactly, or by its last path segment when the file store reports a resolved device path (`/dev/sdb1` vs `/dev/disk/by-id/…`).
-
-##### Verified by
-
-- [DiskQuotaTest — "only the lines of the directory's file store are considered, matched exactly or by device name"](../../src/test/kotlin/de/hoennig/werkator/metrics/DiskQuotaTest.kt) (planned)
-
-#### Scenario#16.04: Without a quota the volume stays the source
-
-So that hosts without quota tooling, without a quota, or with an unreadable `quota` output render exactly as before.
-
-- **Given** `quota` is absent, fails, prints `none` for user and group, or prints only other filesystems
-- **When** a sample is taken
-- **Then** `diskTotalGib`, `diskUsedGib` and `diskFreeGib` are the file-store values
-  - **and** the snapshot names the volume as the source and no quota
-  - **and** an absent or failing `quota` is logged once, not every 60 s, like every other source.
-
-##### Verified by
-
-- [SystemMetricsCollectorTest — "without a quota the volume stays the disk source"](../../src/test/kotlin/de/hoennig/werkator/metrics/SystemMetricsCollectorTest.kt) (planned)
-- [SystemMetricsCollectorTest — "unreadable sources degrade to null metrics, never fail the sample"](../../src/test/kotlin/de/hoennig/werkator/metrics/SystemMetricsCollectorTest.kt) (existing, extended by the quota source)
-
-#### Scenario#16.05: A changed disk source restarts the disk series
-
-So that the min/max/avg of a 1 GiB quota metric are not poisoned by the 37 GiB volume history of the previous binary — the max would otherwise read 37.13 GiB forever.
-
-- **Given** a persisted aggregation state whose disk series were recorded from the file store
-- **When** the first sample after the update finds a binding quota
-- **Then** the `diskUsedGib` and `diskFreeGib` series start afresh
-  - **and** every other series continues
-  - **and** the state records the disk source, so the next restart continues the quota series.
-
-##### Verified by
-
-- [SystemMetricsCollectorTest — "a changed disk source restarts the disk series and keeps the others"](../../src/test/kotlin/de/hoennig/werkator/metrics/SystemMetricsCollectorTest.kt) (planned)
-
-#### Scenario#16.06: The page says which budget it shows
-
-So that `Disk total: 8.00 GiB` on a 71 GiB host is not mistaken for a broken metric.
-
-- **Given** a snapshot with a binding group quota
-- **When** the system page renders or polls
-- **Then** the info line reads `Disk total: 8.00 GiB (group quota mih09, hard limit 12.00 GiB)`
-  - **and** with a binding user quota `Disk total: 4.00 GiB (user quota mih09-werkator, hard limit 6.00 GiB)`
-  - **and** with the volume binding although quotas exist `Disk total: 70.99 GiB (volume, tighter than the quotas)`
-  - **and** without any quota it reads `Disk total: 70.99 GiB` as today
-  - **and** the server-rendered line and the polled line are identical.
-
-##### Verified by
-
-- [UiViewsTest — "the disk total names the binding source: user quota, group quota, or the volume"](../../src/test/kotlin/de/hoennig/werkator/server/UiViewsTest.kt) (planned)
-- `werkator.js` mirrors `UiFormats.diskTotal` (manual: the polled line must equal the rendered one after the first refresh)
-
-#### Scenario#16.07: The highlighting follows the quota
-
-So that the warn/critical colours fire before a build hits "Disk quota exceeded".
-
-- **Given** a binding quota with 8 GiB soft limit and 6.6 GiB used
-- **When** the system page renders
-- **Then** the `Disk used` cell is highlighted `metric-warn` (82 %), and `metric-crit` from 7.2 GiB.
-
-##### Verified by
-
-- [UiViewsTest — "utilization highlights warn from 80% and crit from 90% of the total"](../../src/test/kotlin/de/hoennig/werkator/server/UiViewsTest.kt) (existing — the total is now the quota, nothing else changes)
-
-## The Solution
-
-This PR records the plan; the code lands in the next PR.
-
-**Read the quota through the CLI, like git and Docker.**
-Linux exposes quotas only through the `quotactl` syscall, which Java cannot reach without JNI/JNA — a new runtime dependency and a native layer for one number.
-`quota(1)` is installed wherever quotas are set (Debian's `quota` package on the Managed Webspaces, version 4.06 on `mih09`), so the collector shells out: `quota -u -g --no-wrap --raw-grace` (`-ugwp`).
-`--no-wrap` keeps long device names such as `/dev/disk/by-id/wwn-…-part2` on one line, `--raw-grace` prints the grace columns as numbers instead of leaving them empty, so every filesystem line has the same nine fields and the parser needs no column heuristics.
-The exit status is not read as a failure signal — `quota` also uses it to say "over quota" — only the parsed output counts; an absent binary or an empty output means "no quota", never a failed sample.
-This is the same decision `werkdock doctor` took in Go; the two parsers stay separate because the tools are different binaries in different languages, but the fixture is the same real output.
-
-**Choose the binding candidate in a pure function.**
-`DiskQuota` (new, package `de.hoennig.werkator.metrics`) parses the output into lines `{kind user|group, subject, filesystem, blocksKib, softKib, hardKib}` and keeps the lines of the directory's file store (`Files.getFileStore(dir).name()` is the mount's device string, the same string `quota` prints; a resolved device path is matched by its last segment as `werkdock` does).
-Each remaining line becomes a candidate `DiskSpace` with `total = soft`, `used = blocks`, `free = max(0, total − blocks)`; a line whose soft limit is 0 (unset) uses the hard limit as total, a line with both 0 is no candidate.
-The volume's `fileStoreDiskSpace(dir)` is the last candidate, and `bindingDiskSpace(candidates)` returns the one with the smallest `free` — the tightest budget wins, and total, used and free always come from that one source.
-All of it is pure over strings and numbers, so the whole matrix — user only, group only, both, none, volume tighter than the quotas, two filesystems, `*` marker, `none` line — is a Kotest table.
-
-**The collector gets one more injectable source.**
-`SystemMetricsCollector` gains `quotaOutput: () -> String?` next to `diskSpace` (the process call with a 5 s timeout in production, a string in tests); `readDisk()` collects the quota candidates plus the file store and takes the binding one — a failing `quota` simply leaves the volume as the only candidate.
-`DiskSpace` gains a `source: DiskSource` (`kind` `volume|user|group`, `subject`, `filesystem`, and for a quota `softLimitGib`/`hardLimitGib`), carried into `SystemMetrics.diskSource` — an additive JSON field, the three existing disk fields keep their names; `quotasPresent: Boolean` says whether a quota lost against the volume, for the info line.
-The persisted state gains `diskSource` (`"volume"` or `"quota:<kind>:<subject>:<filesystem>"`); a mismatch drops the two disk series before the sample is recorded (Scenario#16.05).
-That reset also fires when the binding candidate switches at runtime, e.g. from the group quota to a newly introduced user quota — the series then describe one budget at a time.
-The quota is read every sample: it is one syscall behind a small process, cheaper than the repo-size walk, and a raised quota should show within a minute.
-
-**The page names the budget.**
-`UiFormats.diskTotal(metrics)` formats `8.00 GiB (group quota mih09, hard limit 12.00 GiB)`, `… (user quota …)`, `70.99 GiB (volume, tighter than the quotas)` or the plain total when no quota exists; `werkator.js` gets the identical function for the poll — the UI invariant that server-rendered and polled output match.
-Rows, labels and the highlighting stay as they are: `utilizationClass(used, total)` simply receives the quota as the total.
-
-**Where it is verified live.**
-After the deployment on `mih09` the page must read `Disk total: 8.00 GiB (group quota mih09, hard limit 12.00 GiB)`, `Disk used` about 1.04 GiB and `Disk free` about 6.96 GiB, with the `Repo size` row unchanged at about 0.73 GiB — the used value is the whole package's usage (every user of group `mih09`), which is what counts against the budget, while the repo size stays Werkator's own share.
-On `vm4006` (Docker host, no quota) the page must render exactly as before.
-The first sample after the update restarts the disk min/max/avg, visible as `Max` dropping from 37.13 GiB to the current value.
-
-**Order of work for the implementing PR:**
-
-1. `DiskQuota` parser and selection with the table test and the `mih09` fixture.
-2. `SystemMetricsCollector`: the quota source, the fallback, `diskSource` in the state, the series reset.
-3. `SystemMetrics.diskSource`, `UiFormats.diskTotal`, `SystemMetricsView`, `werkator.js`, `UiViewsTest`.
-4. Docs: the metrics paragraph of the architecture skill, one sentence in `docs/deployment.md` (Hostsharing section) and in `docs/plan/09-system-metrics.md` (implementation note), and this PR-doc's "Verified by" links turned from planned into real.
-5. Deploy to `mih09` via `tools/remote --env-file .env.mih09 werkator instance-update`, check the page and the journal for the one-time source log line.
-
-## Open Questions
-
-- **Soft or hard limit as the total?** Planned: the soft limit, with the hard limit in the info line.
-  Beyond the soft limit the grace period starts and writes fail once it expires, so for a service that runs for weeks the soft limit is the effective one; and a page that turns critical *before* the hard stop is the point of the highlighting.
-  The reviewer may prefer the hard limit as the total and the soft limit as the warn threshold instead — that would need a third highlighting rule, hence not planned.
-- **The volume as a candidate — decided.** The owner asked for the smallest of user quota, group quota and free disk space, so the volume competes on equal terms instead of being only the fallback; that is what Scenario#16.02 and `bindingDiskSpace` describe.
-- **A configuration switch?** Planned: none.
-  The quota is detected and the volume is always a candidate; a key `metrics.disk: quota|filestore` would be a fourth place to keep in sync (`WerkatorConfig`, `init` templates, `docs/configuration.md`) for a choice nobody is expected to make.
-- **Which directory's filesystem?** Planned: the first served repository's, as today.
-  The artifact root and the worktrees live under it by default; an artifact root on another volume would need its own line — noted as a follow-up.
-
-## Prerequisite PRs
-
-- None; the change is confined to the metrics package and the system page.
-  It applies equally on top of PR #12 (the registry), which only changed the directory the disk metric is read for from `workingDir` to `repoDirs().first()`.
-
-## Follow-up PRs
-
-- The implementation of this plan.
-- Inode quotas, if a host ever runs into the file limit before the block limit.
-- A per-volume disk metric when the artifact root is configured onto another file store than the repository.
-
-## Attachments
-
-### Measured on mih09 (2026-09-03)
-
-Read as `mih09-werkator` (uid 120974, gid 102180 `mih09`); this output is the parser fixture.
-
-```
-$ quota -u -g -w -p
-Disk quotas for user mih09-werkator (uid 120974): none
-Disk quotas for group mih09 (gid 102180):
-     Filesystem  blocks   quota   limit   grace   files   quota   limit   grace
-/dev/disk/by-id/wwn-0x0000000000000001-part2 1088116  8388608 12582912       0   14226  16777216 25165824       0
-      /dev/sdb1 1456376  10485760 15728640       0   52983  20971520 31457280       0
-
-$ df -Pk "$HOME"
-Filesystem                                   1024-blocks     Used Available Capacity Mounted on
-/dev/disk/by-id/wwn-0x0000000000000001-part2    74436100 38933504  35502596      53% /
-
-$ du -sk "$HOME"
-954544  /home/pacs/mih09/users/werkator
-```
-
-In GiB: the group's soft limit is 8.00, the hard limit 12.00, the usage 1.04 (of which Werkator's home is 0.91); the volume is 70.99 with 33.86 free — the numbers the page shows today.
-Without `--raw-grace` the empty grace columns vanish and the lines have seven fields instead of nine; without `--no-wrap` the long device name is printed on a line of its own.