werkator/docs/prs/2026-09-15-PR#63-an-unreadable-quota-is-not-the-volume.md

> **WARNING:** This document describes only the change applied in this PR.
> It may already be outdated once the next PR is merged.
> Historic PR-documentation is not maintained along with new PRs — treat it as a snapshot, not as current documentation.

## The Problem

On 2026-09-15 the production instance recorded a disk peak of 447.36 GiB against a quota of 20.00 GiB, and the System page began advising an upgrade with the reason "peak 447.36 of 20.00 GiB (2237 %) in 7 d".
The SSD showed the same shape, 44.46 GiB against 16.00.
Neither figure is possible: a quota cannot be exceeded by a factor of twenty, and the disks were in fact around a third full.

Both spikes are a single sample, at 07:52:12 UTC.

What that sample hit was a window in which the Java virtual machine could not start any child process at all.
A deployment had just replaced the installed runtime directory underneath the running instance, so the old process's `java.home` pointed at a newer runtime's `jspawnhelper`, and the handshake between the two failed.
Every repository's `git fetch` failed at the same moment with `Failed to exec spawn helper`, which is how the window was noticed at all.

The disk measurement, however, did not fail.
It quietly measured something else:

```kotlin
val quotaLines = readSource("quota") { quotaOutput()?.let(DiskQuota::parse) ?: emptyList() } ?: emptyList()
…
val shown = quotaCandidates.takeIf { it.isNotEmpty() }?.let(DiskQuota::bindingDiskSpace) ?: volume
```

Both an absent quota and an unaskable one arrive as an empty list, and an empty list means "the volume is the budget".
That rule is right for a machine without quotas, and it is what makes such a machine measurable at all.
It is wrong the moment the reading merely failed, because the code then cannot tell "there is no quota" from "I could not ask".
The volume figure reaching the series is the shared Hostsharing disk this installation has a slice of — hence 447 GiB where 7 GiB belonged.

The damage outlives the outage.
`CapacitySizing` judges a disk by the peak of the retention window, so one unreadable sample produces a false upgrade recommendation for thirty days.
The measurement that failed lasted a few seconds.

## Non-Goals

- **No change to the deployment.** The window that triggered this is a defect of its own and is fixed separately; this one happens whenever the quota command fails for any reason, and it would outlive that fix.
- **No change for machines without quotas.** They keep being measured against their volume, unchanged and untouched by this PR.
- **No plausibility check in the sizing.** Refusing a peak that exceeds its own total would have hidden this symptom without fixing the cause, and it would quietly discard real readings on a disk whose quota was lowered.
- **No repair of the recorded history.** The spike on the production instance stays until it ages out of the window, unless it is removed by hand; that is an operational decision, not a code change.

## The Scenarios

### Feature: a disk is measured against its own budget or not at all

#### Background

- A **quota** is this installation's slice of a disk; the **volume** is the whole disk, shared with everyone else on the machine. On the production host they differ by more than a factor of twenty.
- The **disk source key** persisted per disk records which of the two a disk was last measured against, and a change of it already restarts that disk's series (Scenario#16.05).
- `quotaOutput()` returns the quota command's output, `null` or an exception when there is none to be had.

#### Scenario#63.01: A quota that cannot be asked for yields no measurement

So that an outage of a few seconds does not recommend an upgrade for a month.

- **Given** a disk that has been measured against a group quota
- **When** the quota command cannot be run, fails, or times out
- **Then** the sample carries no size, no source and no series value for that disk
  - **and** the disk's series keeps the values it had, with no peak from the volume
  - **and** the rest of the sample — memory, CPU, the other disks — is recorded as usual

##### Verified by

- [SystemMetricsCollectorTest: "a quota that cannot be asked for is missing, never the whole volume (Scenario#63.01)"](../../src/test/kotlin/de/hoennig/werkator/metrics/SystemMetricsCollectorTest.kt)

#### Scenario#63.02: A machine without quotas is measured against its volume

So that the fix does not blind the installations the volume rule exists for.

- **Given** a machine that has no quota at all, and a disk never bound to one
- **When** the quota command returns nothing, or does not exist
- **Then** the disk is measured against its volume, exactly as before
  - **and** its source is reported as the volume

##### Verified by

- [SystemMetricsCollectorTest: "without a quota the volume stays the disk source"](../../src/test/kotlin/de/hoennig/werkator/metrics/SystemMetricsCollectorTest.kt)
- [SystemMetricsCollectorTest: "a failing quota command degrades to the volume, logged once, the sample still succeeds"](../../src/test/kotlin/de/hoennig/werkator/metrics/SystemMetricsCollectorTest.kt)

## The Solution

The quota reading keeps its third state instead of collapsing it.

`readSource("quota") { … }` no longer ends in `?: emptyList()`: `null` now means the quota could not be asked for, and an empty list means it was asked and had nothing to say.
`readDiskAgainst` then decides per disk, using the source key that disk was last bound to — state the collector already persists for the series reset:

| quota reading | disk was last bound to | measured against |
| --- | --- | --- |
| lines present | anything | the binding quota, as before |
| asked, no lines | anything | the volume, as before |
| could not ask | a quota | nothing; the sample carries no value |
| could not ask | the volume, or nothing yet | the volume, as before |

The last row is what keeps a machine without quotas measurable: there the command fails on every sample, permanently, and falling back is the only sensible reading.
The third row is the fix.

The persisted key's spelling is unchanged, which matters more than it looks: it is compared against stored state, and a new spelling would restart every installation's disk series once.

## Open Questions

- **The recorded spike on the production instance stays** until it leaves the thirty-day window, and the System page will keep advising an upgrade for both disks until then. Removing it means editing the persisted metrics state on the host by hand.
- **Whether the sizing should also refuse an impossible peak** — one above its own total — is left open deliberately. It would have masked this bug, and a disk whose quota was genuinely lowered produces the same shape honestly.

## Additional Changes

- The release notes for v1.14.2 explain the confusion in the terms a reader of the release page needs, without the code.

## Prerequisite PRs

- None. The change stands on its own and is independent of the deployment that exposed it.

## Follow-up PRs

- The deployment swaps the installed runtime while the old process is still using it, which is what broke process spawning for thirty seconds and triggered this. The swap belongs in the transient restart unit — stop, swap, start — so the running process keeps its own runtime until the service is down. Decided on 2026-09-15, not yet written.
- [Step 2 of RFC 0005](../rfcs/0005-werkator-deploys-itself.md#step-2--restartsinstance-as-a-build-setting-option-b), and the two measurements decided on 2026-09-14: sampling `/proc/pressure/io`, and recording the CPU time a build consumed beside its wall-clock duration.