werkator/docs/prs/2026-09-09-PR#40-system-page-charts-and-disk-usage.md

> **WARNING:** This document describes only the change applied in this PR.
> It may already be outdated once the next PR is merged.
> Historic PR-documentation is not maintained along with new PRs — treat it as a snapshot, not as current documentation.

## Related Links

- [RFC 0002](../rfcs/0002-disk-usage-and-physical-headroom.md) — the evaluation and the drafts behind the last two changes; its decisions A1 and B1 are what this PR implements.

## The Problem

Five things the System page could not say, asked for in one breath and answered in one PR.
The first three are small; the last two needed an evaluation first, which is RFC 0002 and part of this branch.

The gauge charts only ever showed one colour for the value line.
What looked like a red line in the 24 h, 7 d and 30 d ranges was the *maxima* line, permanently drawn in the critical colour whatever it showed.
So the very ranges a capacity question is asked in suggested a level that was not being stated, and the 1 h and 4 h ranges — the ones looked at while something is going on — said nothing about level at all.

A gauge card answered "what is the machine doing now" and, one line further down, "what was the single worst moment".
The number in between, what the machine usually does over the shown range, was only readable by eyeballing the chart.

The sizing advice waited for seven days of history before saying anything.
On a freshly deployed instance that is a week of a grey "collecting" chip, although a single day already covers a full daily build cycle and a rough answer beats none.

The disk gauge said how full a disk is and nothing else.
When it turned yellow there was no lever: build workspaces, unpacked build environments, artifacts and the checkouts all live under directories Werkator created, and none of them was broken out.
The one existing number, `repoSizeGib`, is the sum of the primary checkouts, so it neither adds up to the gauge nor separates what can be pruned from what cannot.

The last gap was the budget itself.
Since PR#16 the gauge showed the *tightest* of the user quota, the group quota and the volume.
That is right for the number and wrong for the explanation: on a webspace the quota is normally binding, so the gauge shows the quota — and if the volume then fills up through other users of it, the volume becomes the tighter candidate and the gauge switches.
Three things happened at once, none of them announced: the total jumped from a few hundred GiB to a couple of TiB, the percentage *dropped* although the situation had got worse, and the disk's min/max/avg were reset on purpose so the old ceiling would not survive.

## Non-Goals

- The sizing formulas themselves are unchanged; only when they are applied and how the basis is labelled.
- No average for the disk gauges: a disk's mean over an hour is its current fill, and would only add a number that never differs from the one above it.
- No new sampling for the breakdown. Nothing is measured every 15 seconds that was not measured before.
- No per-repository split of the breakdown; the dialog answers "which category", not "which repository". RFC 0002 keeps that as a later addition inside the same dialog.
- Docker images and volumes and caches inside containers stay unattributed: they belong to the daemon and may not even be on the measured filesystem. They are part of "other", never guessed at.
- No new rule in the machine verdict (RFC 0002 option B3 was offered and rejected): the statement stays where the numbers are.

## The Scenarios

### Feature: reading a gauge chart

#### Background

- A gauge chart draws one series over the selected range; the y scale is the resource's total, so 80 % and 90 % of the height are where the gauge's own colours change.
- The aggregated ranges (24 h, 7 d, 30 d) draw a second, thinner line: the maximum within each bucket.

#### Scenario#40.01: A line above a threshold is drawn in that threshold's colour

So that the range in which a resource was tight can be seen without reading the peak line.

- **Given** a series that rises from 20 % of the total to 95 % and falls back
- **When** the chart is drawn, in any of the five ranges
- **Then** the line is cut where it crosses 80 % and where it crosses 90 %
  - **and** the piece below 80 % is drawn in the accent colour, the piece between the thresholds in the watch colour, the piece above 90 % in the critical colour
  - **and** the cut sits exactly on the threshold, interpolated between the two samples, so the pieces join without a gap
  - **and** the maxima line of an aggregated range is cut and coloured by the same rule

##### Verified by

- the pure part of `drawSpark` in [werkator.js](../../src/main/resources/static/werkator.js) — no JS test harness exists in this repository; verified in the browser against a seeded history, light and dark

### Feature: what a resource usually needs

#### Scenario#40.02: The mean of the shown range stands next to the current value

So that a spike can be told apart from a level.

- **Given** a history range with values recorded for CPU and RAM
- **When** `GET /api/system/history` is asked for that range
- **Then** the response carries an `averages` entry per series, the mean of the [values](../../src/main/kotlin/de/hoennig/werkator/metrics/MetricsHistory.kt) column
  - **and** the mean ignores gaps, and a series with no value at all has none
  - **and** on an aggregated range the bucket averages are averaged, never the maxima, because those answer a different question
- **And when** the system page renders a CPU or RAM card
- **Then** the mean appears right-aligned in the row of the current value, marked with the average sign and the range in its tooltip
  - **and** a disk card shows none

##### Verified by

- [HistoryAveragesTest](../../src/test/kotlin/de/hoennig/werkator/metrics/HistoryAveragesTest.kt)
- [SystemApiControllerTest](../../src/test/kotlin/de/hoennig/werkator/server/SystemApiControllerTest.kt)

### Feature: sizing advice earlier

#### Scenario#40.03: The advice starts after one day and says how much it is worth

So that a new instance is not silent for a week.

- **Given** less than a day of quarter-hour history
- **Then** every resource is `unknown` and the reason reads `collecting history: 0.5 of 1 day`
- **Given** at least one day but less than seven
- **Then** the advice is computed as usual
  - **and** `capacity.preliminary` is true
  - **and** the basis line reads `sizing from 3 d of history — preliminary, reliable from 7 d`
- **Given** seven days or more
- **Then** `capacity.preliminary` is false and the basis line reads as before

##### Verified by

- [CapacitySizingTest](../../src/test/kotlin/de/hoennig/werkator/metrics/CapacitySizingTest.kt)
- [UiViewsTest](../../src/test/kotlin/de/hoennig/werkator/server/UiViewsTest.kt)
- [UiControllerTest](../../src/test/kotlin/de/hoennig/werkator/server/UiControllerTest.kt)

### Feature: what fills a disk

#### Background

- Werkator owns four kinds of directory: the build worktrees and the unpacked build environments under a repository's `.git/werkator/`, the artifact store under the state home, and the checkouts themselves.
- The usage walk is throttled to every 40th sample, as the repository-size walk always was.

#### Scenario#40.04: One walk, tallied by subtree

So that a breakdown costs no more than the size number it replaces.

- **Given** a repository with files in its worktrees, in its build environments, and elsewhere
- **When** the usage walk runs
- **Then** each regular file is counted into the first subtree it lies under, and everything else under none
  - **and** a directory that does not exist tallies nothing rather than failing the sample

##### Verified by

- [DiskUsageTest](../../src/test/kotlin/de/hoennig/werkator/metrics/DiskUsageTest.kt)

#### Scenario#40.05: Every disk gets what lies on it, and one honest rest

So that the dialog never claims an authority it does not have.

- **Given** repositories on one file store and the artifact store on another
- **When** the breakdown of a disk is built
- **Then** it holds only the categories measured on that disk's own file store
  - **and** `other` is the used space minus what was explained, never negative
  - **and** a disk with nothing of Werkator's on it is all `other`
- **And when** the disk card's button is used
- **Then** a dialog shows the slices as a donut and as a legend with sizes and shares, naming when the walk ran

##### Verified by

- [DiskUsageTest](../../src/test/kotlin/de/hoennig/werkator/metrics/DiskUsageTest.kt)
- [SystemMetricsCollectorTest](../../src/test/kotlin/de/hoennig/werkator/metrics/SystemMetricsCollectorTest.kt)

### Feature: quota free, volume full

#### Scenario#40.06: The quota keeps the scale, the volume marks its ceiling

So that a filling volume is stated instead of silently moving every number.

- **Given** a group quota that still allows 342 GiB
  - **and** a volume behind it with 1 GiB left
- **When** a sample is taken
- **Then** the shown total, used and free stay the quota's
  - **and** `physicalFreeGib`/`physicalSource` carry the volume's headroom
  - **and** the level is judged against used plus that headroom, so the card turns critical
  - **and** the bar marks where the headroom ends and hatches the rest
  - **and** the note reads "the volume nearly full: 1.00 GiB left, the group quota mih09 still allows 342.00 GiB"
- **Given** a volume with more room than the quota allows
- **Then** none of this appears and the card is exactly as before

##### Verified by

- [SystemMetricsCollectorTest](../../src/test/kotlin/de/hoennig/werkator/metrics/SystemMetricsCollectorTest.kt)
- [UiViewsTest](../../src/test/kotlin/de/hoennig/werkator/server/UiViewsTest.kt)

## The Solution

`drawSpark` no longer maps a segment straight to screen coordinates.
It keeps the points in value space, cuts every gap-free segment at the two thresholds with an interpolated crossing point, and draws each piece as its own polyline carrying its level in the class name.
The maxima line goes through the same function, which is why it stops being red by decree.
The level thresholds stay the single ones `utilizationLevel` already owned; the CSS only adds the two colour overrides per line kind.

`HistoryAverages` is the counterpart of `HistoryPeaks` — the same shape, the same place in `GET /api/system/history`, computed on the server so it can be tested in Kotlin rather than only in a browser.
The card renders it in JavaScript alone, like the peak line: the server has no range to average over.

`CapacitySizing.MIN_DAYS` drops from seven to one and `RELIABLE_DAYS` takes over the seven, as a label rather than a gate.
`CapacityAdvice.preliminary` carries the distinction into the API, and `UiFormats.adviceBasis` and its JavaScript twin render the same three cases.

`DiskUsage` is the new pure part: `tallyBytes` walks a directory once and sums each file into the first of the given subtrees it lies under, and `breakdownOf` turns those sums plus the disk's used space into the slices, `other` included.
The collector's throttled probe now returns a `UsageMeasurement` — the repository size *and* the categories per file store — instead of one number, so the breakdown is a by-product of the walk that already happened; the artifact store is the one subtree that is new, and it is shallow.

`readDisk` stops treating the volume as one candidate among equals.
Where a quota applies it is the budget, and the tightest quota wins among quotas; the volume only *is* the budget where no quota applies, as before.
When the volume will run out before the shown budget does, it becomes the physical headroom instead: `DiskMetrics.levelCeilingGib` is what the level is judged against, while the total, the bar and every text stay on the budget being managed.

On the page the disk card gains three things: a marker and a hatched tail on the bar, a note in the critical colour, and a button opening one shared `<dialog>` that renders the donut and the legend from the polled snapshot.
The dialog never asks the server for anything; it shows what the last walk found and says when that was.
`UiFormats.physicalHeadroomNote`/`headroomMarkPercent` and their JavaScript twins render the same texts and positions, as the invariant requires.

## Open Questions

- The average is a plain mean over the range, not weighted by how long each bucket actually covered; with a gap in the history the two differ slightly. Weighting would be more correct and less obvious to read.
- The maxima line is now grey while a range stays below 80 %, where it used to be red. It is more honest and a little less conspicuous.
- The breakdown is attributed by file store, which is one level coarser than the mount point. Two monitored directories on the same file store would each be shown the same categories; there is no such configuration today.
- `bindingDiskSpace` is still the tightest-wins helper and is still used to pick among quotas; its name now describes less than it did.

## Additional Changes

- The `field(root, name)` helper moved from inside `initSystemPage` to module scope in `werkator.js`, so the new render functions can use it.
- The gauge-bar fill got a class of its own: the level colours had been applied by `.gauge-bar div`, which would have painted the new marker and tail as well.