# RFC 0004: Sizing by Waiting, in Bookable Steps **Status:** - proposed: 2026-09-12 - accepted: 2026-09-12 (A1, B1, C3, D1; see Decisions) - rejected: - - implemented: 2026-09-12 (PR#58) **Proposal:** The sizing advice of the System page judges RAM the way it already judges CPU — by whether anything *waited* for it — and names only sizes that can actually be booked. A downsize is offered only when the next smaller bookable size would still leave the headroom the advice itself demands. The decisions were taken in conversation; this RFC records the evidence and the reasoning so the implementation can be handed over as a whole. ## Context and Problem Statement `CapacitySizing` ([source](../../src/main/kotlin/de/javagil/werkator/metrics/CapacitySizing.kt)) sizes the machine from the quarter-hour tier of the metrics history: CPU from sustained load above the thread count (PR#42), RAM and disk from the 30-day peak against the total (PR#30). On the production host it gave two recommendations on 2026-09-12 that do not hold. **RAM: "upgrade to 8 GiB", red.** The reason reads "peak 5.51 of 5.79 GiB usable (95 %) in 4 d, 6 GiB booked". The measured value is `MemTotal − MemAvailable`, so page cache is not in it; what filled the memory was a Gradle build, and it was allowed to. A fill level is not a shortage: JVMs and the kernel take what is there, and nothing is wrong as long as nobody waits. The history already holds the number that says whether anybody waited — PSI `some avg60` of `/proc/pressure/memory`, sampled as `memoryPressure` — and it says no. Over 30 days of quarter hours (421 buckets), the share whose *average* memory stall was at least the given level: | average stall ≥ | 0.5 % | 1 % | 2 % | 5 % | 10 % | | --- | --- | --- | --- | --- | --- | | share of quarter hours | 22.8 % | 14.5 % | 6.7 % | 1.0 % | 0.0 % | Two percent of a quarter hour is 18 seconds of some task waiting, spread over a running build. For comparison, `cpuPressure` on the same machine: 24.7 % of the quarter hours averaged at least 5 %, 16.2 % at least 10 % — and the CPU is judged `fits`, because it is sized from waiting and the load never exceeded the threads. The resource that was waited for eight times less is the one shown in red. That is not a wrong threshold; it is two different yardsticks for two resources. Swap adds a false witness: the host has held around 2 GiB of its 4 GiB swap for days at a stall level of well under one percent. Those are cold pages, not a shortage. `SystemHealth` reports them as pressure (`SWAP_PRESSURE_RATIO`), which PR#42 left open for the sizing. **HDD: "downsize to 11 GiB", amber.** The reason reads "peak 7.57 of 20.00 GiB (38 %) in 4 d" on a group quota. There is no 11 GiB at Hostsharing: a quota is booked in steps of 10 GB, the physical storage of a Managed Server in steps of 250 GB. The next bookable size below, 10 GiB, would put the same peak at 76 % — above `DISK_TARGET_RATIO` (70 %) and inside the watch band the advice itself would then flag. A downsize that immediately produces a `tight` is not advice. ## A — What says RAM is short ### The candidates The kernel measures exactly the question being asked. PSI `memory` counts the time some task spent waiting for memory: direct reclaim, swap-in, refault of evicted page cache. Swap *traffic* is therefore already inside PSI, as the waiting it causes. What PSI does not see is an OOM kill: the process does not wait, it dies. - **A1 — PSI alone.** The share of quarter hours whose average memory stall exceeds a level, exactly as `CPU_WAITING_SHARE` counts the quarter hours the load exceeded the threads. Swap occupancy is ignored. No new data; applies to the existing history immediately; one yardstick for CPU and RAM. Blind to OOM kills only. - **A2 — PSI plus swap occupancy.** As A1, but swap held above `SWAP_PRESSURE_RATIO` lifts `fits` to `tight`. Consistent with `SystemHealth` today — and wrong for the same reason the peak is wrong: occupancy says nothing about waiting. The production host would be permanently `tight` while nobody waits. - **A3 — PSI plus new sampling.** Additionally sample `pswpin`/`pswpout` and `oom_kill` from `/proc/vmstat`; an OOM kill inside the window is an `upgrade`. The hardest evidence there is, at the price of new history columns, a persistence migration, and advice that only becomes available once enough new history exists. Swap traffic is redundant with PSI; only `oom_kill` adds something. ### Recommendation A1, and start sampling `oom_kill` now so that the data exists when a follow-up reads it. The counter is machine-wide — on a shared hive it counts every webspace's kills, not Werkator's — which is one more reason to keep it out of the verdict until there is experience with it. ## B — Where the bookable steps come from Four numbers are needed: the quota step (10 GiB), the physical storage step (250 GB), the RAM step (1 GB), the CPU step (1 thread). Today "1 GiB" is a constant in the code. - **B1 — from `HostingEnvironment`.** When the instance recognises Hostsharing, its steps apply; otherwise today's (1 GiB, 1 thread). `CapacitySizing.of(...)` takes the steps as a parameter — the object stays pure and free of a dependency on `server` — and the controller derives them from the hosting. No configuration key, none of the three places that must stay in sync (`WerkatorConfig`, `InitCommand`, `docs/configuration.md`) is touched, and the order dialog and the steps come from one source and cannot drift apart. - **B2 — from configuration.** `metrics.disks[].step`, `metrics.ramStep`, `metrics.cpuStep`, default 1. Generic, but four keys in three places, maintained per instance by hand — for values the hosting detection already knows — and a wrong step on a volume produces quietly wrong advice. - **B3 — both, configuration overriding hosting.** Covers everything, costs the maintenance of B2 plus a precedence rule, and pays off only once a second provider with other steps exists. ### Recommendation B1. A direct interface to Hostsharing is planned; it will replace the constants behind the same parameter, so nothing is gained by making them configurable now. ## C — From when on "something waited" Two numbers: from which average stall a quarter hour counts as *stalled*, and which share of stalled quarter hours in the 30-day window makes `tight` and `upgrade`. `SystemHealth` already has two levels for the live value: 2 % is pressure, 10 % is overload (`MEMORY_PRESSURE_PERCENT`, `MEMORY_OVERLOAD_PERCENT`). The table above gives what each candidate says about the production host today. - **C1 — one level, Health's pressure.** Stalled = average ≥ 2 %; `upgrade` from a 5 % share (as `CPU_WAITING_SHARE`), `tight` from 1 %. Production host: **upgrade**. One level puts "noticeable" and "in trouble" into the same pot. - **C2 — two levels, both Health's.** `upgrade` when ≥ 5 % of the quarter hours averaged ≥ 10 % (overload was routine); `tight` when ≥ 5 % averaged ≥ 2 % (pressure was routine). Production host: **tight**, and an order line for 8 GB, preselected, for a machine that copes. - **C3 — only overload counts.** Stalled = average ≥ 10 %; `tight` from a 1 % share, `upgrade` from 5 %. Production host: **fits**. `tight` from about 7 overloaded quarter hours in 30 days, `upgrade` from about 36. Under C3 the fill-level peak cannot remain a second signal, or the production host is `tight` from its 95 % peak after all. The peak is therefore only *named* — "peak 5.51 of 5.79 GiB, no stalls" — and never judged. It keeps one job: when PSI does demand a raise, the size is the peak-based formula of PR#42 (peak ÷ `TARGET_RATIO` + reserve), rounded up to the bookable step, and at least one step above the current size, because a stall proves that the peak was the ceiling. ### Recommendation C3. Eighteen seconds of waiting per quarter hour during a build is a build evicting page cache, not a machine short of memory. ## D — When a downsize is advice - **D1 — only when the next smaller bookable size still fits.** `DOWNSIZE` requires, beyond today's `DOWNSIZE_RATIO` (peak below 40 %), that the peak stays below the target ratio — `DISK_TARGET_RATIO` for a disk, `TARGET_RATIO` for RAM — of the next bookable size down. Otherwise `fits`. Production host: HDD `fits`, since 7.57 GiB is 76 % of 10 GiB. - **D2 — as today.** Recommend the computed size, rounded to the step. Rounding 11 down to 10 recommends a size the advice itself would flag next; rounding up to 20 recommends the size already booked. A volume (physical storage, RFC 0002 B1) is advised in its 250 GB steps. Where that step is absurd against the peak — 8 GiB used, the next step 250 GB away — the verdict is `fits` with a reason saying so, not an order nobody will place. ### Recommendation D1. ## Decisions Taken on 2026-09-12: 1. **A1** — RAM is judged from `memoryPressure` alone; swap occupancy plays no part. `oom_kill` from `/proc/vmstat` is sampled from now on, as the count per bucket, and not yet read by the sizing. 2. **B1** — the bookable steps are a parameter of `CapacitySizing.of(...)`, derived from `HostingEnvironment` by the caller: for Hostsharing 10 GiB per quota, 250 GB per volume, 1 GB RAM, 1 thread; without a recognised hosting today's 1 GiB and 1 thread. No configuration key. 3. **C3** — a quarter hour is stalled when its average memory stall is at least 10 %; `tight` from 1 % stalled quarter hours in the window, `upgrade` from 5 %. The fill-level peak is named in the reason and never judged; it sizes the raise when one is due, rounded to the step and at least one step up. 4. **D1** — a downsize is advised only when the next smaller bookable size keeps the peak below the target ratio; otherwise `fits`. ## Consequences - `CapacitySizing.ram` changes its yardstick; `CapacitySizingTest` gets the production host's series as a regression case (peak 5.51 of 5.79 GiB, no quarter hour at 10 %: `fits`) beside the synthetic ones. - `CapacitySizing.of(...)` gains a steps parameter; `byPeak` and `disk` round to it and apply D1; the HDD case (peak 7.57 of 20 GiB group quota, step 10) becomes the second regression case: `fits`. - `SystemMetricsCollector` reads one more counter from `/proc/vmstat` and the history gets one more base column; the persisted state must accept a history without it. - `SystemHealth` keeps its live thresholds; the two share `MEMORY_OVERLOAD_PERCENT`, so a machine that is `healthy` cannot be red in the sizing. - `UiFormats.sizingChip` and the `sizingChip`/`orderItems`/`orderSize` twins in `werkator.js` keep producing the same texts; the order dialog thereby lists only bookable sizes. - Not touched: the CPU sizing of PR#42, the hosting detection, the mail, and any hosting API. - Follow-ups, not part of the implementation: reading `oom_kill` into the verdict once there is experience with it on a shared hive; whether an amber order line should be preselected in the dialog.