werkator/docs/prs/2026-09-03-PR#13-quota-aware-disk-metrics.md

WARNING: This document describes only the change applied in this PR. It may already be outdated once the next PR is merged. Historic PR-documentation is not maintained along with new PRs — treat it as a snapshot, not as current documentation.

Related Links

• The live system page: https://werkator.javagil.de/system (instance mih09-werkator on mih09.hostsharing.net, a Hostsharing Managed Webspace).
• Step 09: System Metrics — where the disk metric comes from.
• ADR 0008 and werkdock doctor — the one place that already reads the group quota, as a one-off check before the first build.
• Hostsharing-internal: the package's quota is set in HSAdmin; the values below were read on the host with quota(1).

The Problem

The system page shows the disk of the host, not the disk the instance can use. On mih09 it reads Disk total: 70.99 GiB, Disk used 37.13 GiB, Disk free 33.86 GiB — the numbers of the whole / volume, shared by every package on that machine. What actually limits Werkator there is a group quota of the package mih09: 8 GiB soft limit, 12 GiB hard limit, 1.04 GiB used, measured on 2026-09-03 (see Attachments). So the page promises 33 GiB of headroom where 7 GiB exist, and its warn/critical highlighting (80 %/90 % of the total) can never fire before a build fails with "Disk quota exceeded".

The disk metric is read via java.nio.file.FileStore with df semantics (step 09). That is right on a host Werkator owns (vm4006, a Docker host) and wrong on every host where a user or group quota is the binding limit — the Hostsharing Managed Webspace being the deployment variant we now run in production. werkdock doctor already knows this: it checks the group-quota headroom before the first build, because the free space of the volume said nothing (a 1 GiB quota blocked a build mid-flight on h68, step 17). The system page should show the same truth, continuously.

Non-Goals

• Implementing the change — this PR is the plan only; the implementation follows in the next PR.
• Inode (file-count) quotas: quota(1) reports them, but the Gradle caches on mih09 use 14 226 of 16.7 M files; a follow-up if it ever matters.
• Alerting or refusing to start a build on a full quota — the page only shows; werkdock doctor keeps the one-off pre-build check.
• A configuration switch: the quota is detected, never declared (see Open Questions).
• Changing the JSON field names of GET /api/system or the metric rows of the page: diskTotalGib, diskUsedGib, diskFreeGib keep their names and their meaning "the budget the instance can fill".

The Scenarios

Feature: the disk metric is the quota when a quota binds

Background

• quota(1) prints one block per subject (-u the user, -g its groups), each with one line per filesystem: blocks (1 KiB units, currently used), quota (soft limit), limit (hard limit), grace, then the same four for files. A subject without quota prints … : none. A * after blocks marks "over the soft limit".
• The binding quota of a directory is, among the user and group lines of the directory's filesystem, the one with the smallest headroom (soft − blocks, its total minus its usage).
• The total the page reports is the binding quota's soft limit, the hard limit is shown alongside (see Open Questions for the choice).
• Where no quota line matches the directory's filesystem, the file-store numbers apply unchanged — Docker hosts and developer machines render exactly as today.
• The directory is the first served repository's, as for the file-store metric today.

Scenario#13.01: A group quota replaces the volume numbers

So that the operator of a Managed Webspace sees the budget the package can fill, not the size of the host's disk.

• Given quota -u -g reports no user quota and a group quota on the repository's filesystem with 8 GiB soft limit, 12 GiB hard limit and 1.04 GiB used
  • and the file store reports 71 GiB total and 34 GiB free
• When a sample is taken
• Then diskTotalGib is 8.00, diskUsedGib 1.04 and diskFreeGib 6.96
  • and the snapshot names the quota: group mih09, its filesystem, and the hard limit 12.00 GiB.

Verified by

• SystemMetricsCollectorTest — "a group quota on the repository's filesystem replaces the file-store disk numbers" (planned)
• DiskQuotaTest — "the mih09 output parses into one group line per filesystem and no user line" (planned, fixture: the attachment below)

Scenario#13.02: The binding quota is the tighter one of user and group

So that a user quota below the group's (or the other way round) never hides the limit that fails first.

• Given a user quota with 2 GiB headroom and a group quota with 7 GiB headroom on the same filesystem
• When a sample is taken
• Then the user quota binds: its soft limit is the total, its blocks the used value
  • and with the user quota reporting none, the group quota binds.

Verified by

• DiskQuotaTest — "among user and group quotas of one filesystem the smallest headroom binds" (planned)

Scenario#13.03: Only the quota of the repository's filesystem counts

So that a full quota on another volume (on mih09: /dev/sdb1) does not shrink the metric of the volume Werkator writes to.

• Given quota lines for two filesystems
  • and the repository directory's file store is named like the first one
• When the binding quota is chosen
• Then the second filesystem's lines are ignored
  • and the name is matched exactly, or by its last path segment when the file store reports a resolved device path (/dev/sdb1 vs /dev/disk/by-id/…).

Verified by

• DiskQuotaTest — "only the lines of the directory's file store are considered, matched exactly or by device name" (planned)

Scenario#13.04: Without a binding quota the file store stays the source

So that hosts without quota tooling, without a quota, or with an unreadable quota output render exactly as before.

• Given quota is absent, fails, prints none for user and group, or prints only other filesystems
• When a sample is taken
• Then diskTotalGib, diskUsedGib and diskFreeGib are the file-store values
  • and the snapshot carries no quota name
  • and an absent or failing quota is logged once, not every 60 s, like every other source.

Verified by

• SystemMetricsCollectorTest — "without a binding quota the file store stays the disk source" (planned)
• SystemMetricsCollectorTest — "unreadable sources degrade to null metrics, never fail the sample" (existing, extended by the quota source)

Scenario#13.05: A changed disk source restarts the disk series

So that the min/max/avg of a 1 GiB quota metric are not poisoned by the 37 GiB volume history of the previous binary — the max would otherwise read 37.13 GiB forever.

• Given a persisted aggregation state whose disk series were recorded from the file store
• When the first sample after the update finds a binding quota
• Then the diskUsedGib and diskFreeGib series start afresh
  • and every other series continues
  • and the state records the disk source, so the next restart continues the quota series.

Verified by

• SystemMetricsCollectorTest — "a changed disk source restarts the disk series and keeps the others" (planned)

Scenario#13.06: The page says which budget it shows

So that Disk total: 8.00 GiB on a 71 GiB host is not mistaken for a broken metric.

• Given a snapshot with a binding group quota
• When the system page renders or polls
• Then the info line reads Disk total: 8.00 GiB (group quota mih09, hard limit 12.00 GiB)
  • and without a quota it reads Disk total: 70.99 GiB as today
  • and the server-rendered line and the polled line are identical.

Verified by

• UiViewsTest — "the disk total names the binding quota and its hard limit" (planned)
• werkator.js mirrors UiFormats.diskTotal (manual: the polled line must equal the rendered one after the first refresh)

Scenario#13.07: The highlighting follows the quota

So that the warn/critical colours fire before a build hits "Disk quota exceeded".

• Given a binding quota with 8 GiB soft limit and 6.6 GiB used
• When the system page renders
• Then the Disk used cell is highlighted metric-warn (82 %), and metric-crit from 7.2 GiB.

Verified by

• UiViewsTest — "utilization highlights warn from 80% and crit from 90% of the total" (existing — the total is now the quota, nothing else changes)

The Solution

This PR records the plan; the code lands in the next PR.

Read the quota through the CLI, like git and Docker. Linux exposes quotas only through the quotactl syscall, which Java cannot reach without JNI/JNA — a new runtime dependency and a native layer for one number. quota(1) is installed wherever quotas are set (Debian's quota package on the Managed Webspaces, version 4.06 on mih09), so the collector shells out: quota -u -g --no-wrap --raw-grace (-ugwp). --no-wrap keeps long device names such as /dev/disk/by-id/wwn-…-part2 on one line, --raw-grace prints the grace columns as numbers instead of leaving them empty, so every filesystem line has the same nine fields and the parser needs no column heuristics. The exit status is not read as a failure signal — quota also uses it to say "over quota" — only the parsed output counts; an absent binary or an empty output means "no quota", never a failed sample. This is the same decision werkdock doctor took in Go; the two parsers stay separate because the tools are different binaries in different languages, but the fixture is the same real output.

Choose the binding line in a pure function. DiskQuota (new, package de.hoennig.werkator.metrics) parses the output into lines {kind user|group, subject, filesystem, blocksKib, softKib, hardKib} and picks the binding one for a directory: lines of the directory's file store (Files.getFileStore(dir).name() is the mount's device string, the same string quota prints; a resolved device path is matched by its last segment as werkdock does), then the smallest soft − blocks among them. A line whose soft limit is 0 (unset) uses the hard limit as total; a line with both 0 is no quota. total = soft, used = blocks, free = max(0, total − blocks), all in KiB and converted to GiB like the file-store numbers. The function is pure over strings, so the whole matrix — user only, group only, both, none, two filesystems, * marker, none line — is a Kotest table.

The collector gets one more injectable source. SystemMetricsCollector gains quotaOutput: () -> String? next to diskSpace (the process call with a 5 s timeout in production, a string in tests); readDisk() asks the quota first and falls back to fileStoreDiskSpace when nothing binds. DiskSpace gains a nullable quota: DiskQuotaInfo (kind, subject, filesystem, softLimitGib, hardLimitGib), carried into SystemMetrics.diskQuota — an additive JSON field, the three existing disk fields keep their names. The persisted state gains diskSource ("filestore" or "quota:<kind>:<subject>:<filesystem>"); a mismatch drops the two disk series before the sample is recorded (Scenario#13.05). The quota is read every sample: it is one syscall behind a small process, cheaper than the repo-size walk, and a raised quota should show within a minute.

The page names the budget. UiFormats.diskTotal(metrics) formats 8.00 GiB (group quota mih09, hard limit 12.00 GiB) or the plain total; werkator.js gets the identical function for the poll — the UI invariant that server-rendered and polled output match. Rows, labels and the highlighting stay as they are: utilizationClass(used, total) simply receives the quota as the total.

Where it is verified live. After the deployment on mih09 the page must read Disk total: 8.00 GiB (group quota mih09, hard limit 12.00 GiB), Disk used about 1.04 GiB and Disk free about 6.96 GiB, with the Repo size row unchanged at about 0.73 GiB — the used value is the whole package's usage (every user of group mih09), which is what counts against the budget, while the repo size stays Werkator's own share. On vm4006 (Docker host, no quota) the page must render exactly as before. The first sample after the update restarts the disk min/max/avg, visible as Max dropping from 37.13 GiB to the current value.

Order of work for the implementing PR:

1. DiskQuota parser and selection with the table test and the mih09 fixture.
2. SystemMetricsCollector: the quota source, the fallback, diskSource in the state, the series reset.
3. SystemMetrics.diskQuota, UiFormats.diskTotal, SystemMetricsView, werkator.js, UiViewsTest.
4. Docs: the metrics paragraph of the architecture skill, one sentence in docs/deployment.md (Hostsharing section) and in docs/plan/09-system-metrics.md (implementation note), and this PR-doc's "Verified by" links turned from planned into real.
5. Deploy to mih09 via tools/remote --env-file .env.mih09 werkator instance-update, check the page and the journal for the one-time source log line.

Open Questions

• Soft or hard limit as the total? Planned: the soft limit, with the hard limit in the info line. Beyond the soft limit the grace period starts and writes fail once it expires, so for a service that runs for weeks the soft limit is the effective one; and a page that turns critical before the hard stop is the point of the highlighting. The reviewer may prefer the hard limit as the total and the soft limit as the warn threshold instead — that would need a third highlighting rule, hence not planned.
• A configuration switch? Planned: none. The quota is detected and falls back to the file store; a key metrics.disk: quota|filestore would be a fourth place to keep in sync (WerkatorConfig, init templates, docs/configuration.md) for a choice nobody is expected to make.
• Which directory's filesystem? Planned: the first served repository's, as today. The artifact root and the worktrees live under it by default; an artifact root on another volume would need its own line — noted as a follow-up.

Prerequisite PRs

• None; the change is confined to the metrics package and the system page. It applies equally on top of PR #12 (the registry), which only changed the directory the disk metric is read for from workingDir to repoDirs().first().

Follow-up PRs

• The implementation of this plan.
• Inode quotas, if a host ever runs into the file limit before the block limit.
• A per-volume disk metric when the artifact root is configured onto another file store than the repository.

Attachments

Measured on mih09 (2026-09-03)

Read as mih09-werkator (uid 120974, gid 102180 mih09); this output is the parser fixture.

$ quota -u -g -w -p
Disk quotas for user mih09-werkator (uid 120974): none
Disk quotas for group mih09 (gid 102180):
     Filesystem  blocks   quota   limit   grace   files   quota   limit   grace
/dev/disk/by-id/wwn-0x0000000000000001-part2 1088116  8388608 12582912       0   14226  16777216 25165824       0
      /dev/sdb1 1456376  10485760 15728640       0   52983  20971520 31457280       0

$ df -Pk "$HOME"
Filesystem                                   1024-blocks     Used Available Capacity Mounted on
/dev/disk/by-id/wwn-0x0000000000000001-part2    74436100 38933504  35502596      53% /

$ du -sk "$HOME"
954544  /home/pacs/mih09/users/werkator

In GiB: the group's soft limit is 8.00, the hard limit 12.00, the usage 1.04 (of which Werkator's home is 0.91); the volume is 70.99 with 33.86 free — the numbers the page shows today. Without --raw-grace the empty grace columns vanish and the lines have seven fields instead of nine; without --no-wrap the long device name is printed on a line of its own.