werkator/docs/rfcs/0004-sizing-by-waiting-in-bookable-steps.md

RFC 0004: Sizing by Waiting, in Bookable Steps

Status:

• proposed: 2026-09-12
• accepted: 2026-09-12 (A1, B1, C3, D1; see Decisions)
• rejected: -
• implemented: 2026-09-12 (PR#58)

Proposal: The sizing advice of the System page judges RAM the way it already judges CPU — by whether anything waited for it — and names only sizes that can actually be booked. A downsize is offered only when the next smaller bookable size would still leave the headroom the advice itself demands. The decisions were taken in conversation; this RFC records the evidence and the reasoning so the implementation can be handed over as a whole.

Context and Problem Statement

CapacitySizing (source) sizes the machine from the quarter-hour tier of the metrics history: CPU from sustained load above the thread count (PR#42), RAM and disk from the 30-day peak against the total (PR#30). On the production host it gave two recommendations on 2026-09-12 that do not hold.

RAM: "upgrade to 8 GiB", red. The reason reads "peak 5.51 of 5.79 GiB usable (95 %) in 4 d, 6 GiB booked". The measured value is MemTotal − MemAvailable, so page cache is not in it; what filled the memory was a Gradle build, and it was allowed to. A fill level is not a shortage: JVMs and the kernel take what is there, and nothing is wrong as long as nobody waits. The history already holds the number that says whether anybody waited — PSI some avg60 of /proc/pressure/memory, sampled as memoryPressure — and it says no. Over 30 days of quarter hours (421 buckets), the share whose average memory stall was at least the given level:

average stall ≥	0.5 %	1 %	2 %	5 %	10 %
share of quarter hours	22.8 %	14.5 %	6.7 %	1.0 %	0.0 %

Two percent of a quarter hour is 18 seconds of some task waiting, spread over a running build. For comparison, cpuPressure on the same machine: 24.7 % of the quarter hours averaged at least 5 %, 16.2 % at least 10 % — and the CPU is judged fits, because it is sized from waiting and the load never exceeded the threads. The resource that was waited for eight times less is the one shown in red. That is not a wrong threshold; it is two different yardsticks for two resources.

Swap adds a false witness: the host has held around 2 GiB of its 4 GiB swap for days at a stall level of well under one percent. Those are cold pages, not a shortage. SystemHealth reports them as pressure (SWAP_PRESSURE_RATIO), which PR#42 left open for the sizing.

HDD: "downsize to 11 GiB", amber. The reason reads "peak 7.57 of 20.00 GiB (38 %) in 4 d" on a group quota. There is no 11 GiB at Hostsharing: a quota is booked in steps of 10 GB, the physical storage of a Managed Server in steps of 250 GB. The next bookable size below, 10 GiB, would put the same peak at 76 % — above DISK_TARGET_RATIO (70 %) and inside the watch band the advice itself would then flag. A downsize that immediately produces a tight is not advice.

A — What says RAM is short

The candidates

The kernel measures exactly the question being asked. PSI memory counts the time some task spent waiting for memory: direct reclaim, swap-in, refault of evicted page cache. Swap traffic is therefore already inside PSI, as the waiting it causes. What PSI does not see is an OOM kill: the process does not wait, it dies.

• A1 — PSI alone. The share of quarter hours whose average memory stall exceeds a level, exactly as CPU_WAITING_SHARE counts the quarter hours the load exceeded the threads. Swap occupancy is ignored. No new data; applies to the existing history immediately; one yardstick for CPU and RAM. Blind to OOM kills only.
• A2 — PSI plus swap occupancy. As A1, but swap held above SWAP_PRESSURE_RATIO lifts fits to tight. Consistent with SystemHealth today — and wrong for the same reason the peak is wrong: occupancy says nothing about waiting. The production host would be permanently tight while nobody waits.
• A3 — PSI plus new sampling. Additionally sample pswpin/pswpout and oom_kill from /proc/vmstat; an OOM kill inside the window is an upgrade. The hardest evidence there is, at the price of new history columns, a persistence migration, and advice that only becomes available once enough new history exists. Swap traffic is redundant with PSI; only oom_kill adds something.

Recommendation

A1, and start sampling oom_kill now so that the data exists when a follow-up reads it. The counter is machine-wide — on a shared hive it counts every webspace's kills, not Werkator's — which is one more reason to keep it out of the verdict until there is experience with it.

B — Where the bookable steps come from

Four numbers are needed: the quota step (10 GiB), the physical storage step (250 GB), the RAM step (1 GB), the CPU step (1 thread). Today "1 GiB" is a constant in the code.

• B1 — from HostingEnvironment. When the instance recognises Hostsharing, its steps apply; otherwise today's (1 GiB, 1 thread). CapacitySizing.of(...) takes the steps as a parameter — the object stays pure and free of a dependency on server — and the controller derives them from the hosting. No configuration key, none of the three places that must stay in sync (WerkatorConfig, InitCommand, docs/configuration.md) is touched, and the order dialog and the steps come from one source and cannot drift apart.
• B2 — from configuration. metrics.disks[].step, metrics.ramStep, metrics.cpuStep, default 1. Generic, but four keys in three places, maintained per instance by hand — for values the hosting detection already knows — and a wrong step on a volume produces quietly wrong advice.
• B3 — both, configuration overriding hosting. Covers everything, costs the maintenance of B2 plus a precedence rule, and pays off only once a second provider with other steps exists.

Recommendation

B1. A direct interface to Hostsharing is planned; it will replace the constants behind the same parameter, so nothing is gained by making them configurable now.

C — From when on "something waited"

Two numbers: from which average stall a quarter hour counts as stalled, and which share of stalled quarter hours in the 30-day window makes tight and upgrade. SystemHealth already has two levels for the live value: 2 % is pressure, 10 % is overload (MEMORY_PRESSURE_PERCENT, MEMORY_OVERLOAD_PERCENT). The table above gives what each candidate says about the production host today.

• C1 — one level, Health's pressure. Stalled = average ≥ 2 %; upgrade from a 5 % share (as CPU_WAITING_SHARE), tight from 1 %. Production host: upgrade. One level puts "noticeable" and "in trouble" into the same pot.
• C2 — two levels, both Health's. upgrade when ≥ 5 % of the quarter hours averaged ≥ 10 % (overload was routine); tight when ≥ 5 % averaged ≥ 2 % (pressure was routine). Production host: tight, and an order line for 8 GB, preselected, for a machine that copes.
• C3 — only overload counts. Stalled = average ≥ 10 %; tight from a 1 % share, upgrade from 5 %. Production host: fits. tight from about 7 overloaded quarter hours in 30 days, upgrade from about 36.

Under C3 the fill-level peak cannot remain a second signal, or the production host is tight from its 95 % peak after all. The peak is therefore only named — "peak 5.51 of 5.79 GiB, no stalls" — and never judged. It keeps one job: when PSI does demand a raise, the size is the peak-based formula of PR#42 (peak ÷ TARGET_RATIO + reserve), rounded up to the bookable step, and at least one step above the current size, because a stall proves that the peak was the ceiling.

Recommendation

C3. Eighteen seconds of waiting per quarter hour during a build is a build evicting page cache, not a machine short of memory.

D — When a downsize is advice

• D1 — only when the next smaller bookable size still fits. DOWNSIZE requires, beyond today's DOWNSIZE_RATIO (peak below 40 %), that the peak stays below the target ratio — DISK_TARGET_RATIO for a disk, TARGET_RATIO for RAM — of the next bookable size down. Otherwise fits. Production host: HDD fits, since 7.57 GiB is 76 % of 10 GiB.
• D2 — as today. Recommend the computed size, rounded to the step. Rounding 11 down to 10 recommends a size the advice itself would flag next; rounding up to 20 recommends the size already booked.

A volume (physical storage, RFC 0002 B1) is advised in its 250 GB steps. Where that step is absurd against the peak — 8 GiB used, the next step 250 GB away — the verdict is fits with a reason saying so, not an order nobody will place.

Recommendation

D1.

Decisions

Taken on 2026-09-12:

1. A1 — RAM is judged from memoryPressure alone; swap occupancy plays no part. oom_kill from /proc/vmstat is sampled from now on, as the count per bucket, and not yet read by the sizing.
2. B1 — the bookable steps are a parameter of CapacitySizing.of(...), derived from HostingEnvironment by the caller: for Hostsharing 10 GiB per quota, 250 GB per volume, 1 GB RAM, 1 thread; without a recognised hosting today's 1 GiB and 1 thread. No configuration key.
3. C3 — a quarter hour is stalled when its average memory stall is at least 10 %; tight from 1 % stalled quarter hours in the window, upgrade from 5 %. The fill-level peak is named in the reason and never judged; it sizes the raise when one is due, rounded to the step and at least one step up.
4. D1 — a downsize is advised only when the next smaller bookable size keeps the peak below the target ratio; otherwise fits.

Consequences

• CapacitySizing.ram changes its yardstick; CapacitySizingTest gets the production host's series as a regression case (peak 5.51 of 5.79 GiB, no quarter hour at 10 %: fits) beside the synthetic ones.
• CapacitySizing.of(...) gains a steps parameter; byPeak and disk round to it and apply D1; the HDD case (peak 7.57 of 20 GiB group quota, step 10) becomes the second regression case: fits.
• SystemMetricsCollector reads one more counter from /proc/vmstat and the history gets one more base column; the persisted state must accept a history without it.
• SystemHealth keeps its live thresholds; the two share MEMORY_OVERLOAD_PERCENT, so a machine that is healthy cannot be red in the sizing.
• UiFormats.sizingChip and the sizingChip/orderItems/orderSize twins in werkator.js keep producing the same texts; the order dialog thereby lists only bookable sizes.
• Not touched: the CPU sizing of PR#42, the hosting detection, the mail, and any hosting API.
• Follow-ups, not part of the implementation: reading oom_kill into the verdict once there is experience with it on a shared hive; whether an amber order line should be preselected in the dialog.