werkator/docs/prs/2026-09-12-PR#58-sizing-by-waiting-in-bookable-steps.md

> **WARNING:** This document describes only the change applied in this PR.
> It may already be outdated once the next PR is merged.
> Historic PR-documentation is not maintained along with new PRs — treat it as a snapshot, not as current documentation.

## Related Links

- [RFC 0004: Sizing by Waiting, in Bookable Steps](../rfcs/0004-sizing-by-waiting-in-bookable-steps.md) — the evidence and the four decisions this PR implements (A1, B1, C3, D1).

## The Problem

On 2026-09-12 the System page of the production host gave two recommendations that do not hold.

**RAM: "upgrade to 8 GiB", red.**
The reason read "peak 5.51 of 5.79 GiB usable (95 %) in 4 d, 6 GiB booked".
The measured value is `MemTotal − MemAvailable`, so page cache is not even in it; what filled the memory was a Gradle build, and it was allowed to.
A fill level is not a shortage: JVMs and the kernel take what is there, and nothing is wrong as long as nobody waits.

The history already holds the number that says whether anybody waited — PSI `some avg60` of `/proc/pressure/memory`, sampled as `memoryPressure` since PR#27 — and it said no.
Over 30 days of quarter hours, 1 % averaged a stall of at least 5 % and none reached 10 %.
On the same machine 24.7 % of the quarter hours averaged a CPU stall of at least 5 % and 16.2 % at least 10 % — and the CPU was judged `fits`, because it is sized from waiting and the load never exceeded the threads.
The resource that was waited for eight times less was the one shown in red.
That is not a wrong threshold; it is two different yardsticks for two resources.

**HDD: "downsize to 11 GiB", amber.**
The reason read "peak 7.57 of 20.00 GiB (38 %) in 4 d" on a group quota.
There is no 11 GiB at Hostsharing: a quota is booked in steps of 10 GB, the physical storage of a Managed Server in steps of 250 GB.
The next bookable size below, 10 GiB, would put the same peak at 76 % — above `DISK_TARGET_RATIO` (70 %) and inside the watch band the advice itself would then flag.
A downsize that immediately produces a `tight` is not advice.

## Non-Goals

- No change to the CPU sizing of PR#42: it was already sized from waiting, and it is the yardstick RAM is being brought onto.
- No change to the hosting detection or to the order mail: what the page composes and whom it names stays as PR#42 left it.
- No configuration key for the bookable steps: the caller already knows where the instance runs, and a step maintained by hand per instance would produce quietly wrong advice for values the detection knows.
- `oom_kill` is sampled but not judged: the counter is machine-wide, so on a shared hive it counts every webspace's kills, and reading it into a verdict waits for experience with it.
- No change to `SystemHealth`: its live thresholds stay, and the sizing shares one of them rather than inventing a second scale.

## The Scenarios

### Feature: a machine is sized by what it waited for, in sizes that can be booked

#### Background

- A *quarter hour* is one bucket of the 30-day tier of the metrics history; the sizing reads its averages.
- A quarter hour is *stalled* when its average PSI memory stall reached `MEMORY_STALL_PERCENT` (10 %, shared with `SystemHealth.MEMORY_OVERLOAD_PERCENT`).
- A *bookable step* is the size a resource is actually ordered in: at Hostsharing 10 GiB per quota, 250 GB per volume, 1 GiB of RAM, one hardware thread.
- The *fill peak* is the highest `ramUsedGib` of the window — `MemTotal − MemAvailable`, page cache excluded.

#### Scenario#58.01: memory is judged by waiting, not by fill level

So that a build using the memory it was given is not read as a machine running out of it.

- **Given** a 30-day window whose fill peak reached 95 % of the usable RAM
  - **and** not one quarter hour was stalled
- **When** the sizing runs
- **Then** the verdict is `fits`
  - **and** the reason names the peak and says that no quarter hour stalled
- **And when** at least 1 % of the quarter hours were stalled
- **Then** the verdict is `tight`
- **And when** at least 5 % of them were stalled
- **Then** the verdict is `upgrade`
  - **and** the recommended size is the peak plus its headroom plus the host's reserve, rounded to the bookable step and at least one step above what is booked

##### Verified by

- [CapacitySizingTest: a RAM peak at 90 % of the total that nothing waited for is not a shortage](../../src/test/kotlin/de/hoennig/werkator/metrics/CapacitySizingTest.kt)
- [CapacitySizingTest: memory stalls in a twentieth of the quarter hours are an upgrade to the peak with headroom](../../src/test/kotlin/de/hoennig/werkator/metrics/CapacitySizingTest.kt)
- [CapacitySizingTest: a hundredth of the quarter hours stalled is tight, not an alarm](../../src/test/kotlin/de/hoennig/werkator/metrics/CapacitySizingTest.kt)
- [CapacitySizingTest: a quarter hour below the overload level is a build evicting page cache, not a shortage](../../src/test/kotlin/de/hoennig/werkator/metrics/CapacitySizingTest.kt)
- [CapacitySizingTest: a history without the pressure column says so instead of judging the fill level](../../src/test/kotlin/de/hoennig/werkator/metrics/CapacitySizingTest.kt)
- [CapacitySizingTest: the production host of 2026-09-12, both verdicts it got wrong](../../src/test/kotlin/de/hoennig/werkator/metrics/CapacitySizingTest.kt)

#### Scenario#58.02: a recommendation names a size that can be booked

So that the order dialog offers a line the service desk can act on.

- **Given** the instance runs on a recognised Hostsharing webspace
- **When** a raise is due for a disk whose budget is a quota
- **Then** the recommended size is rounded up to the next multiple of 10 GiB
- **And when** the disk's budget is the volume itself
- **Then** it is rounded up to the next multiple of 250 GB, named in the GiB the rest of the page is in
- **And when** no hosting is recognised
- **Then** every resource keeps the whole unit it was advised in before

##### Verified by

- [CapacitySizingTest: a raise is rounded to the size the hosting books in](../../src/test/kotlin/de/hoennig/werkator/metrics/CapacitySizingTest.kt)
- [CapacitySizingTest: a volume is advised in the steps its physical storage is booked in](../../src/test/kotlin/de/hoennig/werkator/metrics/CapacitySizingTest.kt)
- [CapacitySizingTest: RAM is booked and raised in whole GiB](../../src/test/kotlin/de/hoennig/werkator/metrics/CapacitySizingTest.kt)
- [UiViewsTest: advice chips read like werkator.js adviceLabel](../../src/test/kotlin/de/hoennig/werkator/server/UiViewsTest.kt)

#### Scenario#58.03: a downsize is only advice when the smaller size still fits

So that no recommendation produces the very warning it was meant to avoid.

- **Given** a 20 GiB group quota whose peak was 7.57 GiB
- **When** the sizing runs with a bookable step of 10 GiB
- **Then** the verdict is `fits`
  - **and** the reason says that the 10 GiB below would be tight
- **And when** the same peak sits on a 40 GiB quota, where 20 GiB would still keep the target headroom
- **Then** the verdict is `downsize` to 20 GiB
- **And when** the budget is a volume booked in 250 GB pieces and holds 2.2 GiB
- **Then** the verdict is `fits` with a reason saying there is no smaller size to book

##### Verified by

- [CapacitySizingTest: a downsize the next smaller size would flag is no downsize](../../src/test/kotlin/de/hoennig/werkator/metrics/CapacitySizingTest.kt)
- [CapacitySizingTest: a downsize that still leaves the headroom stays advice](../../src/test/kotlin/de/hoennig/werkator/metrics/CapacitySizingTest.kt)
- [CapacitySizingTest: a volume whose next step is 250 GB away is simply fits](../../src/test/kotlin/de/hoennig/werkator/metrics/CapacitySizingTest.kt)
- [CapacitySizingTest: the production host of 2026-09-12, both verdicts it got wrong](../../src/test/kotlin/de/hoennig/werkator/metrics/CapacitySizingTest.kt)

#### Scenario#58.04: OOM kills are recorded so that a later verdict can read them

So that the one memory shortage nobody waits for leaves a trace.

- **Given** the kernel counts `oom_kill` in `/proc/vmstat`
- **When** a sample is taken
- **Then** the history records the kills since the previous sample, not the counter since boot
  - **and** the first sample of a process records no point at all
  - **and** a counter that went backwards — a reboot — counts as no kill
- **And when** the kernel has no such counter, or the persisted history predates the column
- **Then** the column is simply absent or starts where the sampling did, and no sample fails

##### Verified by

- [SystemMetricsCollectorTest: OOM kills are recorded as the count per sample, not as the counter since boot](../../src/test/kotlin/de/hoennig/werkator/metrics/SystemMetricsCollectorTest.kt)
- [SystemMetricsCollectorTest: a rebooted counter counts as no kill, and a kernel without the counter as no column](../../src/test/kotlin/de/hoennig/werkator/metrics/SystemMetricsCollectorTest.kt)
- [SystemMetricsCollectorTest: a persisted history from before the OOM column simply starts it](../../src/test/kotlin/de/hoennig/werkator/metrics/SystemMetricsCollectorTest.kt)

## The Solution

**RAM is judged by waiting (RFC 0004 A1, C3).**
`CapacitySizing.ram` reads the `memoryPressure` column of the quarter-hour tier — its averages, never its maxima, exactly as the CPU is sized — and counts the quarter hours whose average reached `MEMORY_STALL_PERCENT`.
That constant *is* `SystemHealth.MEMORY_OVERLOAD_PERCENT`: sharing it is what keeps a machine the health verdict calls healthy from being red in the sizing.
Health's lower *pressure* level (2 %) is deliberately not the mark — eighteen seconds of waiting per quarter hour during a build is a build evicting page cache, not a machine short of memory.
`tight` follows from 1 % of stalled quarter hours in the window, `upgrade` from 5 %, which is the share `CPU_WAITING_SHARE` already uses.

The fill peak keeps two jobs and loses one.
It is named in the reason, it sizes a raise when one is due — the PR#42 formula, rounded to the bookable step and at least one step above what is booked, because a stall proves the peak was the ceiling — but it no longer produces `tight` or `upgrade`.
The one asymmetry is deliberate: a fill level cannot prove a shortage, but it can prove slack, so a *downsize* stays peak-based and additionally requires that not one quarter hour stalled.
Swap occupancy plays no part at all; two gigabytes of cold pages held for days are not a shortage, which is why `SWAP_PRESSURE_RATIO` stays a live signal of `SystemHealth` and nothing more.
A history without the column says so in its reason instead of quietly falling back to judging the fill level.

**Sizes are the ones that can be booked (RFC 0004 B1).**
`CapacitySizing.of` takes a `BookableSteps` parameter and `ram`, `byPeak` and `disk` round to it.
The caller derives it — `HostingEnvironments.bookableSteps` — so the sizing object stays pure and free of a dependency on the `server` package, no configuration key joins the three places that must stay in sync, and the steps and the order dialog come from one source that cannot drift apart.
Which of the two disk steps applies follows from what a disk's budget is, a quota or the volume.
Everything is carried in GiB, the unit the metrics and the gauges are in, so the 250 GB step is converted once on the way in and renders through the existing `UiFormats`/`werkator.js` twins unchanged.
Unlike the hosting itself, the steps reach every viewer: they name neither webspace nor machine, and one instance advising two different sizes depending on who asks would be worse than either.

**A downsize must survive its own advice (RFC 0004 D1).**
Beyond `DOWNSIZE_RATIO`, a downsize now requires that the recommendation fits into the next bookable size below the current one — the largest step multiple strictly below it, so a 12 GiB quota steps down to 10 rather than to 2.
Otherwise the verdict is `fits`, and the reason says which of the two reasons applied: the size below would be tight, or there is no smaller size to book at all.
The second is the volume case the RFC names, where the next step is 250 GB away from an 8 GiB peak.

**`oom_kill` is sampled, not judged (RFC 0004 A1).**
PSI measures waiting, and an OOM kill is the one memory shortage nobody waits for: the process does not stall, it dies.
`SystemMetricsCollector` records the difference to the previous sample — the count per bucket rather than a number that only grows — under the `oomKills` history column.
The sizing does not read it: the counter is machine-wide, so on a shared hive it counts every webspace's kills, and a verdict on it waits until there is experience with the numbers.

## Additional Changes

- `MetricsHistory.BASE_KEYS` gains the new column, so a history written before it simply starts it — the tiers carry the union of what was sampled and need no migration.
- The advice-chip twin test pins how a size booked in GB renders: Kotlin's `%.2f` and `werkator.js`'s `toFixed(2)` both produce `232.83 GiB`, the first recommendation that is not a whole number.

## Follow-up PRs

- Reading `oom_kill` into the verdict, once there is experience with the counter on a shared hive.
- Whether an amber order line should be preselected in the order dialog.