werkator/docs/prs/2026-09-03-PR#16-quota-aware-disk-metrics.md

> **WARNING:** This document describes only the change applied in this PR.
> It may already be outdated once the next PR is merged.
> Historic PR-documentation is not maintained along with new PRs — treat it as a snapshot, not as current documentation.

## Related Links

- The live system page: <https://werkator.javagil.de/system> (instance `mih09-werkator` on `mih09.hostsharing.net`, a Hostsharing Managed Webspace).
- [Step 09: System Metrics](../plan/09-system-metrics.md) — where the disk metric comes from.
- [ADR 0008](../adrs/0008-2026-09-01.bwrap-build-runtime.md) and [`werkdock doctor`](../../werkdock/internal/doctor/doctor.go) — the one place that already reads the group quota, as a one-off check before the first build.
- Hostsharing-internal: the package's quota is set in HSAdmin; the values below were read on the host with `quota(1)`.

## The Problem

The system page shows the disk of the *host*, not the disk the instance can use.
On `mih09` it reads `Disk total: 70.99 GiB`, `Disk used 37.13 GiB`, `Disk free 33.86 GiB` — the numbers of the whole `/` volume, shared by every package on that machine.
What actually limits Werkator there is a **group quota** of the package `mih09`: 8 GiB soft limit, 12 GiB hard limit, 1.04 GiB used, measured on 2026-09-03 (see [Attachments](#measured-on-mih09-2026-09-03)).
So the page promises 33 GiB of headroom where 7 GiB exist, and its warn/critical highlighting (80 %/90 % of the total) can never fire before a build fails with "Disk quota exceeded".

The disk metric is read via `java.nio.file.FileStore` with `df` semantics (step 09).
That is right on a host Werkator owns (vm4006, a Docker host) and wrong on every host where a user or group quota is the binding limit — the Hostsharing Managed Webspace being the deployment variant we now run in production.
`werkdock doctor` already knows this: it checks the group-quota headroom before the first build, because the free space of the volume said nothing (a 1 GiB quota blocked a build mid-flight on `h68`, step 17).
The system page should show the same truth, continuously.

## Non-Goals

- Implementing the change — this PR is the plan only; the implementation follows in the next PR.
- Inode (file-count) quotas: `quota(1)` reports them, but the Gradle caches on `mih09` use 14 226 of 16.7 M files; a follow-up if it ever matters.
- Alerting or refusing to start a build on a full quota — the page only shows; `werkdock doctor` keeps the one-off pre-build check.
- A configuration switch: the quota is detected, never declared (see Open Questions).
- Changing the JSON field names of `GET /api/system` or the metric rows of the page: `diskTotalGib`, `diskUsedGib`, `diskFreeGib` keep their names and their meaning "the budget the instance can fill".

## The Scenarios

### Feature: the disk metric is the tightest budget — user quota, group quota, or the volume

#### Background

- `quota(1)` prints one block per subject (`-u` the user, `-g` its groups), each with one line per filesystem: `blocks` (1 KiB units, currently used), `quota` (soft limit), `limit` (hard limit), grace, then the same four for files.
  A subject without quota prints `… : none`.
  A `*` after `blocks` marks "over the soft limit".
- Three **candidates** can limit what a directory may still take: the user quota, the group quota (each the lines of the directory's filesystem) and the volume itself (the file store, `df` semantics).
  Each has a headroom: `soft − blocks` for a quota, the usable space for the volume.
- The **binding candidate** is the one with the smallest headroom; its numbers are the disk metric — total, used and free come from one source, never mixed.
  On `mih09` today that is the group quota (6.96 GiB left against 33.86 GiB on the volume); a user quota, once the webspace introduces one, joins the comparison without any change.
- For a quota the **total** is its **soft** limit, the hard limit is shown alongside (see Open Questions for the choice).
- Where no quota line matches the directory's filesystem, the volume is the only candidate — Docker hosts and developer machines render exactly as today.
- The directory is the first served repository's, as for the file-store metric today.

#### Scenario#16.01: A group quota replaces the volume numbers

So that the operator of a Managed Webspace sees the budget the package can fill, not the size of the host's disk.

- **Given** `quota -u -g` reports no user quota and a group quota on the repository's filesystem with 8 GiB soft limit, 12 GiB hard limit and 1.04 GiB used
  - **and** the file store reports 71 GiB total and 34 GiB free
- **When** a sample is taken
- **Then** `diskTotalGib` is 8.00, `diskUsedGib` 1.04 and `diskFreeGib` 6.96
  - **and** the snapshot names the quota: group `mih09`, its filesystem, and the hard limit 12.00 GiB.

##### Verified by

- [SystemMetricsCollectorTest — "a group quota on the repository's filesystem replaces the file-store disk numbers"](../../src/test/kotlin/de/hoennig/werkator/metrics/SystemMetricsCollectorTest.kt) (planned)
- [DiskQuotaTest — "the mih09 output parses into one group line per filesystem and no user line"](../../src/test/kotlin/de/hoennig/werkator/metrics/DiskQuotaTest.kt) (planned, fixture: the attachment below)

#### Scenario#16.02: The tightest of user quota, group quota and volume binds

So that neither a user quota below the group's, nor a nearly full host volume below both, is hidden by the wider budgets.

- **Given** a user quota with 2 GiB headroom and a group quota with 7 GiB headroom on the same filesystem
  - **and** the volume has 34 GiB usable
- **When** a sample is taken
- **Then** the user quota binds: its soft limit is the total, its blocks the used value
  - **and** with the user quota reporting `none`, the group quota binds
  - **and** with the volume down to 1 GiB usable, the volume binds and the metric shows the file-store numbers, quotas or not.

##### Verified by

- [DiskQuotaTest — "among user quota, group quota and volume the smallest headroom binds"](../../src/test/kotlin/de/hoennig/werkator/metrics/DiskQuotaTest.kt) (planned)

#### Scenario#16.03: Only the quota of the repository's filesystem counts

So that a full quota on another volume (on `mih09`: `/dev/sdb1`) does not shrink the metric of the volume Werkator writes to.

- **Given** quota lines for two filesystems
  - **and** the repository directory's file store is named like the first one
- **When** the binding quota is chosen
- **Then** the second filesystem's lines are ignored
  - **and** the name is matched exactly, or by its last path segment when the file store reports a resolved device path (`/dev/sdb1` vs `/dev/disk/by-id/…`).

##### Verified by

- [DiskQuotaTest — "only the lines of the directory's file store are considered, matched exactly or by device name"](../../src/test/kotlin/de/hoennig/werkator/metrics/DiskQuotaTest.kt) (planned)

#### Scenario#16.04: Without a quota the volume stays the source

So that hosts without quota tooling, without a quota, or with an unreadable `quota` output render exactly as before.

- **Given** `quota` is absent, fails, prints `none` for user and group, or prints only other filesystems
- **When** a sample is taken
- **Then** `diskTotalGib`, `diskUsedGib` and `diskFreeGib` are the file-store values
  - **and** the snapshot names the volume as the source and no quota
  - **and** an absent or failing `quota` is logged once, not every 60 s, like every other source.

##### Verified by

- [SystemMetricsCollectorTest — "without a quota the volume stays the disk source"](../../src/test/kotlin/de/hoennig/werkator/metrics/SystemMetricsCollectorTest.kt) (planned)
- [SystemMetricsCollectorTest — "unreadable sources degrade to null metrics, never fail the sample"](../../src/test/kotlin/de/hoennig/werkator/metrics/SystemMetricsCollectorTest.kt) (existing, extended by the quota source)

#### Scenario#16.05: A changed disk source restarts the disk series

So that the min/max/avg of a 1 GiB quota metric are not poisoned by the 37 GiB volume history of the previous binary — the max would otherwise read 37.13 GiB forever.

- **Given** a persisted aggregation state whose disk series were recorded from the file store
- **When** the first sample after the update finds a binding quota
- **Then** the `diskUsedGib` and `diskFreeGib` series start afresh
  - **and** every other series continues
  - **and** the state records the disk source, so the next restart continues the quota series.

##### Verified by

- [SystemMetricsCollectorTest — "a changed disk source restarts the disk series and keeps the others"](../../src/test/kotlin/de/hoennig/werkator/metrics/SystemMetricsCollectorTest.kt) (planned)

#### Scenario#16.06: The page says which budget it shows

So that `Disk total: 8.00 GiB` on a 71 GiB host is not mistaken for a broken metric.

- **Given** a snapshot with a binding group quota
- **When** the system page renders or polls
- **Then** the info line reads `Disk total: 8.00 GiB (group quota mih09, hard limit 12.00 GiB)`
  - **and** with a binding user quota `Disk total: 4.00 GiB (user quota mih09-werkator, hard limit 6.00 GiB)`
  - **and** with the volume binding although quotas exist `Disk total: 70.99 GiB (volume, tighter than the quotas)`
  - **and** without any quota it reads `Disk total: 70.99 GiB` as today
  - **and** the server-rendered line and the polled line are identical.

##### Verified by

- [UiViewsTest — "the disk total names the binding source: user quota, group quota, or the volume"](../../src/test/kotlin/de/hoennig/werkator/server/UiViewsTest.kt) (planned)
- `werkator.js` mirrors `UiFormats.diskTotal` (manual: the polled line must equal the rendered one after the first refresh)

#### Scenario#16.07: The highlighting follows the quota

So that the warn/critical colours fire before a build hits "Disk quota exceeded".

- **Given** a binding quota with 8 GiB soft limit and 6.6 GiB used
- **When** the system page renders
- **Then** the `Disk used` cell is highlighted `metric-warn` (82 %), and `metric-crit` from 7.2 GiB.

##### Verified by

- [UiViewsTest — "utilization highlights warn from 80% and crit from 90% of the total"](../../src/test/kotlin/de/hoennig/werkator/server/UiViewsTest.kt) (existing — the total is now the quota, nothing else changes)

## The Solution

This PR records the plan; the code lands in the next PR.

**Read the quota through the CLI, like git and Docker.**
Linux exposes quotas only through the `quotactl` syscall, which Java cannot reach without JNI/JNA — a new runtime dependency and a native layer for one number.
`quota(1)` is installed wherever quotas are set (Debian's `quota` package on the Managed Webspaces, version 4.06 on `mih09`), so the collector shells out: `quota -u -g --no-wrap --raw-grace` (`-ugwp`).
`--no-wrap` keeps long device names such as `/dev/disk/by-id/wwn-…-part2` on one line, `--raw-grace` prints the grace columns as numbers instead of leaving them empty, so every filesystem line has the same nine fields and the parser needs no column heuristics.
The exit status is not read as a failure signal — `quota` also uses it to say "over quota" — only the parsed output counts; an absent binary or an empty output means "no quota", never a failed sample.
This is the same decision `werkdock doctor` took in Go; the two parsers stay separate because the tools are different binaries in different languages, but the fixture is the same real output.

**Choose the binding candidate in a pure function.**
`DiskQuota` (new, package `de.hoennig.werkator.metrics`) parses the output into lines `{kind user|group, subject, filesystem, blocksKib, softKib, hardKib}` and keeps the lines of the directory's file store (`Files.getFileStore(dir).name()` is the mount's device string, the same string `quota` prints; a resolved device path is matched by its last segment as `werkdock` does).
Each remaining line becomes a candidate `DiskSpace` with `total = soft`, `used = blocks`, `free = max(0, total − blocks)`; a line whose soft limit is 0 (unset) uses the hard limit as total, a line with both 0 is no candidate.
The volume's `fileStoreDiskSpace(dir)` is the last candidate, and `bindingDiskSpace(candidates)` returns the one with the smallest `free` — the tightest budget wins, and total, used and free always come from that one source.
All of it is pure over strings and numbers, so the whole matrix — user only, group only, both, none, volume tighter than the quotas, two filesystems, `*` marker, `none` line — is a Kotest table.

**The collector gets one more injectable source.**
`SystemMetricsCollector` gains `quotaOutput: () -> String?` next to `diskSpace` (the process call with a 5 s timeout in production, a string in tests); `readDisk()` collects the quota candidates plus the file store and takes the binding one — a failing `quota` simply leaves the volume as the only candidate.
`DiskSpace` gains a `source: DiskSource` (`kind` `volume|user|group`, `subject`, `filesystem`, and for a quota `softLimitGib`/`hardLimitGib`), carried into `SystemMetrics.diskSource` — an additive JSON field, the three existing disk fields keep their names; `quotasPresent: Boolean` says whether a quota lost against the volume, for the info line.
The persisted state gains `diskSource` (`"volume"` or `"quota:<kind>:<subject>:<filesystem>"`); a mismatch drops the two disk series before the sample is recorded (Scenario#16.05).
That reset also fires when the binding candidate switches at runtime, e.g. from the group quota to a newly introduced user quota — the series then describe one budget at a time.
The quota is read every sample: it is one syscall behind a small process, cheaper than the repo-size walk, and a raised quota should show within a minute.

**The page names the budget.**
`UiFormats.diskTotal(metrics)` formats `8.00 GiB (group quota mih09, hard limit 12.00 GiB)`, `… (user quota …)`, `70.99 GiB (volume, tighter than the quotas)` or the plain total when no quota exists; `werkator.js` gets the identical function for the poll — the UI invariant that server-rendered and polled output match.
Rows, labels and the highlighting stay as they are: `utilizationClass(used, total)` simply receives the quota as the total.

**Where it is verified live.**
After the deployment on `mih09` the page must read `Disk total: 8.00 GiB (group quota mih09, hard limit 12.00 GiB)`, `Disk used` about 1.04 GiB and `Disk free` about 6.96 GiB, with the `Repo size` row unchanged at about 0.73 GiB — the used value is the whole package's usage (every user of group `mih09`), which is what counts against the budget, while the repo size stays Werkator's own share.
On `vm4006` (Docker host, no quota) the page must render exactly as before.
The first sample after the update restarts the disk min/max/avg, visible as `Max` dropping from 37.13 GiB to the current value.

**Order of work for the implementing PR:**

1. `DiskQuota` parser and selection with the table test and the `mih09` fixture.
2. `SystemMetricsCollector`: the quota source, the fallback, `diskSource` in the state, the series reset.
3. `SystemMetrics.diskSource`, `UiFormats.diskTotal`, `SystemMetricsView`, `werkator.js`, `UiViewsTest`.
4. Docs: the metrics paragraph of the architecture skill, one sentence in `docs/deployment.md` (Hostsharing section) and in `docs/plan/09-system-metrics.md` (implementation note), and this PR-doc's "Verified by" links turned from planned into real.
5. Deploy to `mih09` via `tools/remote --env-file .env.mih09 werkator instance-update`, check the page and the journal for the one-time source log line.

## Open Questions

- **Soft or hard limit as the total?** Planned: the soft limit, with the hard limit in the info line.
  Beyond the soft limit the grace period starts and writes fail once it expires, so for a service that runs for weeks the soft limit is the effective one; and a page that turns critical *before* the hard stop is the point of the highlighting.
  The reviewer may prefer the hard limit as the total and the soft limit as the warn threshold instead — that would need a third highlighting rule, hence not planned.
- **The volume as a candidate — decided.** The owner asked for the smallest of user quota, group quota and free disk space, so the volume competes on equal terms instead of being only the fallback; that is what Scenario#16.02 and `bindingDiskSpace` describe.
- **A configuration switch?** Planned: none.
  The quota is detected and the volume is always a candidate; a key `metrics.disk: quota|filestore` would be a fourth place to keep in sync (`WerkatorConfig`, `init` templates, `docs/configuration.md`) for a choice nobody is expected to make.
- **Which directory's filesystem?** Planned: the first served repository's, as today.
  The artifact root and the worktrees live under it by default; an artifact root on another volume would need its own line — noted as a follow-up.

## Prerequisite PRs

- None; the change is confined to the metrics package and the system page.
  It applies equally on top of PR #12 (the registry), which only changed the directory the disk metric is read for from `workingDir` to `repoDirs().first()`.

## Follow-up PRs

- The implementation of this plan.
- Inode quotas, if a host ever runs into the file limit before the block limit.
- A per-volume disk metric when the artifact root is configured onto another file store than the repository.

## Attachments

### Measured on mih09 (2026-09-03)

Read as `mih09-werkator` (uid 120974, gid 102180 `mih09`); this output is the parser fixture.

```
$ quota -u -g -w -p
Disk quotas for user mih09-werkator (uid 120974): none
Disk quotas for group mih09 (gid 102180):
     Filesystem  blocks   quota   limit   grace   files   quota   limit   grace
/dev/disk/by-id/wwn-0x0000000000000001-part2 1088116  8388608 12582912       0   14226  16777216 25165824       0
      /dev/sdb1 1456376  10485760 15728640       0   52983  20971520 31457280       0

$ df -Pk "$HOME"
Filesystem                                   1024-blocks     Used Available Capacity Mounted on
/dev/disk/by-id/wwn-0x0000000000000001-part2    74436100 38933504  35502596      53% /

$ du -sk "$HOME"
954544  /home/pacs/mih09/users/werkator
```

In GiB: the group's soft limit is 8.00, the hard limit 12.00, the usage 1.04 (of which Werkator's home is 0.91); the volume is 70.99 with 33.86 free — the numbers the page shows today.
Without `--raw-grace` the empty grace columns vanish and the lines have seven fields instead of nine; without `--no-wrap` the long device name is printed on a line of its own.