From 22a1b28824783abf12967306f400119d1b81394f Mon Sep 17 00:00:00 2001 From: mhoennig Date: Thu, 3 Sep 2026 12:35:29 +0200 Subject: [PATCH] =?UTF-8?q?docs(prs):=20Plan=20f=C3=BCr=20die=20quota-bewu?= =?UTF-8?q?sste=20Plattenplatz-Anzeige=20der=20System=C3=BCbersicht=20?= =?UTF-8?q?=E2=80=94=20PR#13?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Auf mih09 zeigt /system die 71-GiB-Platte des Hosts, gemessen bindet aber die Gruppen-Quota des Pakets (8 GiB soft, 12 GiB hard, 1,04 GiB belegt). Das PR-Doc plant: quota(1) per CLI lesen, die bindende Zeile (User/Gruppe, Dateisystem des Repos) als Disk-Budget nehmen, ohne Quota unverändert der File Store, Serien-Reset bei Quellenwechsel, Budget in der Info-Zeile benannt. Nur Plan, kein Code. Co-Authored-By: Claude Fable 5.1 --- ...26-09-03-PR#13-quota-aware-disk-metrics.md | 228 ++++++++++++++++++ 1 file changed, 228 insertions(+) create mode 100644 docs/prs/2026-09-03-PR#13-quota-aware-disk-metrics.md diff --git a/docs/prs/2026-09-03-PR#13-quota-aware-disk-metrics.md b/docs/prs/2026-09-03-PR#13-quota-aware-disk-metrics.md new file mode 100644 index 0000000..02fda2f --- /dev/null +++ b/docs/prs/2026-09-03-PR#13-quota-aware-disk-metrics.md @@ -0,0 +1,228 @@ +> **WARNING:** This document describes only the change applied in this PR. +> It may already be outdated once the next PR is merged. +> Historic PR-documentation is not maintained along with new PRs — treat it as a snapshot, not as current documentation. + +## Related Links + +- The live system page: (instance `mih09-werkator` on `mih09.hostsharing.net`, a Hostsharing Managed Webspace). +- [Step 09: System Metrics](../plan/09-system-metrics.md) — where the disk metric comes from. +- [ADR 0008](../adrs/0008-2026-09-01.bwrap-build-runtime.md) and [`werkdock doctor`](../../werkdock/internal/doctor/doctor.go) — the one place that already reads the group quota, as a one-off check before the first build. +- Hostsharing-internal: the package's quota is set in HSAdmin; the values below were read on the host with `quota(1)`. + +## The Problem + +The system page shows the disk of the *host*, not the disk the instance can use. +On `mih09` it reads `Disk total: 70.99 GiB`, `Disk used 37.13 GiB`, `Disk free 33.86 GiB` — the numbers of the whole `/` volume, shared by every package on that machine. +What actually limits Werkator there is a **group quota** of the package `mih09`: 8 GiB soft limit, 12 GiB hard limit, 1.04 GiB used, measured on 2026-09-03 (see [Attachments](#measured-on-mih09-2026-09-03)). +So the page promises 33 GiB of headroom where 7 GiB exist, and its warn/critical highlighting (80 %/90 % of the total) can never fire before a build fails with "Disk quota exceeded". + +The disk metric is read via `java.nio.file.FileStore` with `df` semantics (step 09). +That is right on a host Werkator owns (vm4006, a Docker host) and wrong on every host where a user or group quota is the binding limit — the Hostsharing Managed Webspace being the deployment variant we now run in production. +`werkdock doctor` already knows this: it checks the group-quota headroom before the first build, because the free space of the volume said nothing (a 1 GiB quota blocked a build mid-flight on `h68`, step 17). +The system page should show the same truth, continuously. + +## Non-Goals + +- Implementing the change — this PR is the plan only; the implementation follows in the next PR. +- Inode (file-count) quotas: `quota(1)` reports them, but the Gradle caches on `mih09` use 14 226 of 16.7 M files; a follow-up if it ever matters. +- Alerting or refusing to start a build on a full quota — the page only shows; `werkdock doctor` keeps the one-off pre-build check. +- A configuration switch: the quota is detected, never declared (see Open Questions). +- Changing the JSON field names of `GET /api/system` or the metric rows of the page: `diskTotalGib`, `diskUsedGib`, `diskFreeGib` keep their names and their meaning "the budget the instance can fill". + +## The Scenarios + +### Feature: the disk metric is the quota when a quota binds + +#### Background + +- `quota(1)` prints one block per subject (`-u` the user, `-g` its groups), each with one line per filesystem: `blocks` (1 KiB units, currently used), `quota` (soft limit), `limit` (hard limit), grace, then the same four for files. + A subject without quota prints `… : none`. + A `*` after `blocks` marks "over the soft limit". +- The **binding quota** of a directory is, among the user and group lines of the directory's filesystem, the one with the smallest headroom (`soft − blocks`, its total minus its usage). +- The **total** the page reports is the binding quota's **soft** limit, the hard limit is shown alongside (see Open Questions for the choice). +- Where no quota line matches the directory's filesystem, the file-store numbers apply unchanged — Docker hosts and developer machines render exactly as today. +- The directory is the first served repository's, as for the file-store metric today. + +#### Scenario#13.01: A group quota replaces the volume numbers + +So that the operator of a Managed Webspace sees the budget the package can fill, not the size of the host's disk. + +- **Given** `quota -u -g` reports no user quota and a group quota on the repository's filesystem with 8 GiB soft limit, 12 GiB hard limit and 1.04 GiB used + - **and** the file store reports 71 GiB total and 34 GiB free +- **When** a sample is taken +- **Then** `diskTotalGib` is 8.00, `diskUsedGib` 1.04 and `diskFreeGib` 6.96 + - **and** the snapshot names the quota: group `mih09`, its filesystem, and the hard limit 12.00 GiB. + +##### Verified by + +- [SystemMetricsCollectorTest — "a group quota on the repository's filesystem replaces the file-store disk numbers"](../../src/test/kotlin/de/hoennig/werkator/metrics/SystemMetricsCollectorTest.kt) (planned) +- [DiskQuotaTest — "the mih09 output parses into one group line per filesystem and no user line"](../../src/test/kotlin/de/hoennig/werkator/metrics/DiskQuotaTest.kt) (planned, fixture: the attachment below) + +#### Scenario#13.02: The binding quota is the tighter one of user and group + +So that a user quota below the group's (or the other way round) never hides the limit that fails first. + +- **Given** a user quota with 2 GiB headroom and a group quota with 7 GiB headroom on the same filesystem +- **When** a sample is taken +- **Then** the user quota binds: its soft limit is the total, its blocks the used value + - **and** with the user quota reporting `none`, the group quota binds. + +##### Verified by + +- [DiskQuotaTest — "among user and group quotas of one filesystem the smallest headroom binds"](../../src/test/kotlin/de/hoennig/werkator/metrics/DiskQuotaTest.kt) (planned) + +#### Scenario#13.03: Only the quota of the repository's filesystem counts + +So that a full quota on another volume (on `mih09`: `/dev/sdb1`) does not shrink the metric of the volume Werkator writes to. + +- **Given** quota lines for two filesystems + - **and** the repository directory's file store is named like the first one +- **When** the binding quota is chosen +- **Then** the second filesystem's lines are ignored + - **and** the name is matched exactly, or by its last path segment when the file store reports a resolved device path (`/dev/sdb1` vs `/dev/disk/by-id/…`). + +##### Verified by + +- [DiskQuotaTest — "only the lines of the directory's file store are considered, matched exactly or by device name"](../../src/test/kotlin/de/hoennig/werkator/metrics/DiskQuotaTest.kt) (planned) + +#### Scenario#13.04: Without a binding quota the file store stays the source + +So that hosts without quota tooling, without a quota, or with an unreadable `quota` output render exactly as before. + +- **Given** `quota` is absent, fails, prints `none` for user and group, or prints only other filesystems +- **When** a sample is taken +- **Then** `diskTotalGib`, `diskUsedGib` and `diskFreeGib` are the file-store values + - **and** the snapshot carries no quota name + - **and** an absent or failing `quota` is logged once, not every 60 s, like every other source. + +##### Verified by + +- [SystemMetricsCollectorTest — "without a binding quota the file store stays the disk source"](../../src/test/kotlin/de/hoennig/werkator/metrics/SystemMetricsCollectorTest.kt) (planned) +- [SystemMetricsCollectorTest — "unreadable sources degrade to null metrics, never fail the sample"](../../src/test/kotlin/de/hoennig/werkator/metrics/SystemMetricsCollectorTest.kt) (existing, extended by the quota source) + +#### Scenario#13.05: A changed disk source restarts the disk series + +So that the min/max/avg of a 1 GiB quota metric are not poisoned by the 37 GiB volume history of the previous binary — the max would otherwise read 37.13 GiB forever. + +- **Given** a persisted aggregation state whose disk series were recorded from the file store +- **When** the first sample after the update finds a binding quota +- **Then** the `diskUsedGib` and `diskFreeGib` series start afresh + - **and** every other series continues + - **and** the state records the disk source, so the next restart continues the quota series. + +##### Verified by + +- [SystemMetricsCollectorTest — "a changed disk source restarts the disk series and keeps the others"](../../src/test/kotlin/de/hoennig/werkator/metrics/SystemMetricsCollectorTest.kt) (planned) + +#### Scenario#13.06: The page says which budget it shows + +So that `Disk total: 8.00 GiB` on a 71 GiB host is not mistaken for a broken metric. + +- **Given** a snapshot with a binding group quota +- **When** the system page renders or polls +- **Then** the info line reads `Disk total: 8.00 GiB (group quota mih09, hard limit 12.00 GiB)` + - **and** without a quota it reads `Disk total: 70.99 GiB` as today + - **and** the server-rendered line and the polled line are identical. + +##### Verified by + +- [UiViewsTest — "the disk total names the binding quota and its hard limit"](../../src/test/kotlin/de/hoennig/werkator/server/UiViewsTest.kt) (planned) +- `werkator.js` mirrors `UiFormats.diskTotal` (manual: the polled line must equal the rendered one after the first refresh) + +#### Scenario#13.07: The highlighting follows the quota + +So that the warn/critical colours fire before a build hits "Disk quota exceeded". + +- **Given** a binding quota with 8 GiB soft limit and 6.6 GiB used +- **When** the system page renders +- **Then** the `Disk used` cell is highlighted `metric-warn` (82 %), and `metric-crit` from 7.2 GiB. + +##### Verified by + +- [UiViewsTest — "utilization highlights warn from 80% and crit from 90% of the total"](../../src/test/kotlin/de/hoennig/werkator/server/UiViewsTest.kt) (existing — the total is now the quota, nothing else changes) + +## The Solution + +This PR records the plan; the code lands in the next PR. + +**Read the quota through the CLI, like git and Docker.** +Linux exposes quotas only through the `quotactl` syscall, which Java cannot reach without JNI/JNA — a new runtime dependency and a native layer for one number. +`quota(1)` is installed wherever quotas are set (Debian's `quota` package on the Managed Webspaces, version 4.06 on `mih09`), so the collector shells out: `quota -u -g --no-wrap --raw-grace` (`-ugwp`). +`--no-wrap` keeps long device names such as `/dev/disk/by-id/wwn-…-part2` on one line, `--raw-grace` prints the grace columns as numbers instead of leaving them empty, so every filesystem line has the same nine fields and the parser needs no column heuristics. +The exit status is not read as a failure signal — `quota` also uses it to say "over quota" — only the parsed output counts; an absent binary or an empty output means "no quota", never a failed sample. +This is the same decision `werkdock doctor` took in Go; the two parsers stay separate because the tools are different binaries in different languages, but the fixture is the same real output. + +**Choose the binding line in a pure function.** +`DiskQuota` (new, package `de.hoennig.werkator.metrics`) parses the output into lines `{kind user|group, subject, filesystem, blocksKib, softKib, hardKib}` and picks the binding one for a directory: lines of the directory's file store (`Files.getFileStore(dir).name()` is the mount's device string, the same string `quota` prints; a resolved device path is matched by its last segment as `werkdock` does), then the smallest `soft − blocks` among them. +A line whose soft limit is 0 (unset) uses the hard limit as total; a line with both 0 is no quota. +`total = soft`, `used = blocks`, `free = max(0, total − blocks)`, all in KiB and converted to GiB like the file-store numbers. +The function is pure over strings, so the whole matrix — user only, group only, both, none, two filesystems, `*` marker, `none` line — is a Kotest table. + +**The collector gets one more injectable source.** +`SystemMetricsCollector` gains `quotaOutput: () -> String?` next to `diskSpace` (the process call with a 5 s timeout in production, a string in tests); `readDisk()` asks the quota first and falls back to `fileStoreDiskSpace` when nothing binds. +`DiskSpace` gains a nullable `quota: DiskQuotaInfo` (`kind`, `subject`, `filesystem`, `softLimitGib`, `hardLimitGib`), carried into `SystemMetrics.diskQuota` — an additive JSON field, the three existing disk fields keep their names. +The persisted state gains `diskSource` (`"filestore"` or `"quota:::"`); a mismatch drops the two disk series before the sample is recorded (Scenario#13.05). +The quota is read every sample: it is one syscall behind a small process, cheaper than the repo-size walk, and a raised quota should show within a minute. + +**The page names the budget.** +`UiFormats.diskTotal(metrics)` formats `8.00 GiB (group quota mih09, hard limit 12.00 GiB)` or the plain total; `werkator.js` gets the identical function for the poll — the UI invariant that server-rendered and polled output match. +Rows, labels and the highlighting stay as they are: `utilizationClass(used, total)` simply receives the quota as the total. + +**Where it is verified live.** +After the deployment on `mih09` the page must read `Disk total: 8.00 GiB (group quota mih09, hard limit 12.00 GiB)`, `Disk used` about 1.04 GiB and `Disk free` about 6.96 GiB, with the `Repo size` row unchanged at about 0.73 GiB — the used value is the whole package's usage (every user of group `mih09`), which is what counts against the budget, while the repo size stays Werkator's own share. +On `vm4006` (Docker host, no quota) the page must render exactly as before. +The first sample after the update restarts the disk min/max/avg, visible as `Max` dropping from 37.13 GiB to the current value. + +**Order of work for the implementing PR:** + +1. `DiskQuota` parser and selection with the table test and the `mih09` fixture. +2. `SystemMetricsCollector`: the quota source, the fallback, `diskSource` in the state, the series reset. +3. `SystemMetrics.diskQuota`, `UiFormats.diskTotal`, `SystemMetricsView`, `werkator.js`, `UiViewsTest`. +4. Docs: the metrics paragraph of the architecture skill, one sentence in `docs/deployment.md` (Hostsharing section) and in `docs/plan/09-system-metrics.md` (implementation note), and this PR-doc's "Verified by" links turned from planned into real. +5. Deploy to `mih09` via `tools/remote --env-file .env.mih09 werkator instance-update`, check the page and the journal for the one-time source log line. + +## Open Questions + +- **Soft or hard limit as the total?** Planned: the soft limit, with the hard limit in the info line. + Beyond the soft limit the grace period starts and writes fail once it expires, so for a service that runs for weeks the soft limit is the effective one; and a page that turns critical *before* the hard stop is the point of the highlighting. + The reviewer may prefer the hard limit as the total and the soft limit as the warn threshold instead — that would need a third highlighting rule, hence not planned. +- **A configuration switch?** Planned: none. + The quota is detected and falls back to the file store; a key `metrics.disk: quota|filestore` would be a fourth place to keep in sync (`WerkatorConfig`, `init` templates, `docs/configuration.md`) for a choice nobody is expected to make. +- **Which directory's filesystem?** Planned: the first served repository's, as today. + The artifact root and the worktrees live under it by default; an artifact root on another volume would need its own line — noted as a follow-up. + +## Prerequisite PRs + +- None; the change is confined to the metrics package and the system page. + It applies equally on top of PR #12 (the registry), which only changed the directory the disk metric is read for from `workingDir` to `repoDirs().first()`. + +## Follow-up PRs + +- The implementation of this plan. +- Inode quotas, if a host ever runs into the file limit before the block limit. +- A per-volume disk metric when the artifact root is configured onto another file store than the repository. + +## Attachments + +### Measured on mih09 (2026-09-03) + +Read as `mih09-werkator` (uid 120974, gid 102180 `mih09`); this output is the parser fixture. + +``` +$ quota -u -g -w -p +Disk quotas for user mih09-werkator (uid 120974): none +Disk quotas for group mih09 (gid 102180): + Filesystem blocks quota limit grace files quota limit grace +/dev/disk/by-id/wwn-0x0000000000000001-part2 1088116 8388608 12582912 0 14226 16777216 25165824 0 + /dev/sdb1 1456376 10485760 15728640 0 52983 20971520 31457280 0 + +$ df -Pk "$HOME" +Filesystem 1024-blocks Used Available Capacity Mounted on +/dev/disk/by-id/wwn-0x0000000000000001-part2 74436100 38933504 35502596 53% / + +$ du -sk "$HOME" +954544 /home/pacs/mih09/users/werkator +``` + +In GiB: the group's soft limit is 8.00, the hard limit 12.00, the usage 1.04 (of which Werkator's home is 0.91); the volume is 70.99 with 33.86 free — the numbers the page shows today. +Without `--raw-grace` the empty grace columns vanish and the lines have seven fields instead of nine; without `--no-wrap` the long device name is printed on a line of its own.