claude-code
06e29e871d
See docs/prs/2026-10-15-PR#163-package-de-javagil.md 🤖 Generated with [Claude Code](https://claude.com/claude-code) Reviewed-on: #163 Reviewed-by: mi <1+mi@noreply.git.javagil.de> Co-authored-by: claude-code <claude-code@javagil.de>
136 lines
11 KiB
Markdown
136 lines
11 KiB
Markdown
# RFC 0004: Sizing by Waiting, in Bookable Steps
|
||
|
||
**Status:**
|
||
- proposed: 2026-09-12
|
||
- accepted: 2026-09-12 (A1, B1, C3, D1; see Decisions)
|
||
- rejected: -
|
||
- implemented: 2026-09-12 (PR#58)
|
||
|
||
**Proposal:** The sizing advice of the System page judges RAM the way it already judges CPU — by whether anything *waited* for it — and names only sizes that can actually be booked.
|
||
A downsize is offered only when the next smaller bookable size would still leave the headroom the advice itself demands.
|
||
The decisions were taken in conversation; this RFC records the evidence and the reasoning so the implementation can be handed over as a whole.
|
||
|
||
## Context and Problem Statement
|
||
|
||
`CapacitySizing` ([source](../../src/main/kotlin/de/javagil/werkator/metrics/CapacitySizing.kt)) sizes the machine from the quarter-hour tier of the metrics history: CPU from sustained load above the thread count (PR#42), RAM and disk from the 30-day peak against the total (PR#30).
|
||
On the production host it gave two recommendations on 2026-09-12 that do not hold.
|
||
|
||
**RAM: "upgrade to 8 GiB", red.**
|
||
The reason reads "peak 5.51 of 5.79 GiB usable (95 %) in 4 d, 6 GiB booked".
|
||
The measured value is `MemTotal − MemAvailable`, so page cache is not in it; what filled the memory was a Gradle build, and it was allowed to.
|
||
A fill level is not a shortage: JVMs and the kernel take what is there, and nothing is wrong as long as nobody waits.
|
||
The history already holds the number that says whether anybody waited — PSI `some avg60` of `/proc/pressure/memory`, sampled as `memoryPressure` — and it says no.
|
||
Over 30 days of quarter hours (421 buckets), the share whose *average* memory stall was at least the given level:
|
||
|
||
| average stall ≥ | 0.5 % | 1 % | 2 % | 5 % | 10 % |
|
||
| --- | --- | --- | --- | --- | --- |
|
||
| share of quarter hours | 22.8 % | 14.5 % | 6.7 % | 1.0 % | 0.0 % |
|
||
|
||
Two percent of a quarter hour is 18 seconds of some task waiting, spread over a running build.
|
||
For comparison, `cpuPressure` on the same machine: 24.7 % of the quarter hours averaged at least 5 %, 16.2 % at least 10 % — and the CPU is judged `fits`, because it is sized from waiting and the load never exceeded the threads.
|
||
The resource that was waited for eight times less is the one shown in red.
|
||
That is not a wrong threshold; it is two different yardsticks for two resources.
|
||
|
||
Swap adds a false witness: the host has held around 2 GiB of its 4 GiB swap for days at a stall level of well under one percent.
|
||
Those are cold pages, not a shortage.
|
||
`SystemHealth` reports them as pressure (`SWAP_PRESSURE_RATIO`), which PR#42 left open for the sizing.
|
||
|
||
**HDD: "downsize to 11 GiB", amber.**
|
||
The reason reads "peak 7.57 of 20.00 GiB (38 %) in 4 d" on a group quota.
|
||
There is no 11 GiB at Hostsharing: a quota is booked in steps of 10 GB, the physical storage of a Managed Server in steps of 250 GB.
|
||
The next bookable size below, 10 GiB, would put the same peak at 76 % — above `DISK_TARGET_RATIO` (70 %) and inside the watch band the advice itself would then flag.
|
||
A downsize that immediately produces a `tight` is not advice.
|
||
|
||
## A — What says RAM is short
|
||
|
||
### The candidates
|
||
|
||
The kernel measures exactly the question being asked.
|
||
PSI `memory` counts the time some task spent waiting for memory: direct reclaim, swap-in, refault of evicted page cache.
|
||
Swap *traffic* is therefore already inside PSI, as the waiting it causes.
|
||
What PSI does not see is an OOM kill: the process does not wait, it dies.
|
||
|
||
- **A1 — PSI alone.** The share of quarter hours whose average memory stall exceeds a level, exactly as `CPU_WAITING_SHARE` counts the quarter hours the load exceeded the threads. Swap occupancy is ignored.
|
||
No new data; applies to the existing history immediately; one yardstick for CPU and RAM. Blind to OOM kills only.
|
||
- **A2 — PSI plus swap occupancy.** As A1, but swap held above `SWAP_PRESSURE_RATIO` lifts `fits` to `tight`.
|
||
Consistent with `SystemHealth` today — and wrong for the same reason the peak is wrong: occupancy says nothing about waiting. The production host would be permanently `tight` while nobody waits.
|
||
- **A3 — PSI plus new sampling.** Additionally sample `pswpin`/`pswpout` and `oom_kill` from `/proc/vmstat`; an OOM kill inside the window is an `upgrade`.
|
||
The hardest evidence there is, at the price of new history columns, a persistence migration, and advice that only becomes available once enough new history exists. Swap traffic is redundant with PSI; only `oom_kill` adds something.
|
||
|
||
### Recommendation
|
||
|
||
A1, and start sampling `oom_kill` now so that the data exists when a follow-up reads it.
|
||
The counter is machine-wide — on a shared hive it counts every webspace's kills, not Werkator's — which is one more reason to keep it out of the verdict until there is experience with it.
|
||
|
||
## B — Where the bookable steps come from
|
||
|
||
Four numbers are needed: the quota step (10 GiB), the physical storage step (250 GB), the RAM step (1 GB), the CPU step (1 thread).
|
||
Today "1 GiB" is a constant in the code.
|
||
|
||
- **B1 — from `HostingEnvironment`.** When the instance recognises Hostsharing, its steps apply; otherwise today's (1 GiB, 1 thread). `CapacitySizing.of(...)` takes the steps as a parameter — the object stays pure and free of a dependency on `server` — and the controller derives them from the hosting.
|
||
No configuration key, none of the three places that must stay in sync (`WerkatorConfig`, `InitCommand`, `docs/configuration.md`) is touched, and the order dialog and the steps come from one source and cannot drift apart.
|
||
- **B2 — from configuration.** `metrics.disks[].step`, `metrics.ramStep`, `metrics.cpuStep`, default 1.
|
||
Generic, but four keys in three places, maintained per instance by hand — for values the hosting detection already knows — and a wrong step on a volume produces quietly wrong advice.
|
||
- **B3 — both, configuration overriding hosting.**
|
||
Covers everything, costs the maintenance of B2 plus a precedence rule, and pays off only once a second provider with other steps exists.
|
||
|
||
### Recommendation
|
||
|
||
B1.
|
||
A direct interface to Hostsharing is planned; it will replace the constants behind the same parameter, so nothing is gained by making them configurable now.
|
||
|
||
## C — From when on "something waited"
|
||
|
||
Two numbers: from which average stall a quarter hour counts as *stalled*, and which share of stalled quarter hours in the 30-day window makes `tight` and `upgrade`.
|
||
`SystemHealth` already has two levels for the live value: 2 % is pressure, 10 % is overload (`MEMORY_PRESSURE_PERCENT`, `MEMORY_OVERLOAD_PERCENT`).
|
||
The table above gives what each candidate says about the production host today.
|
||
|
||
- **C1 — one level, Health's pressure.** Stalled = average ≥ 2 %; `upgrade` from a 5 % share (as `CPU_WAITING_SHARE`), `tight` from 1 %.
|
||
Production host: **upgrade**. One level puts "noticeable" and "in trouble" into the same pot.
|
||
- **C2 — two levels, both Health's.** `upgrade` when ≥ 5 % of the quarter hours averaged ≥ 10 % (overload was routine); `tight` when ≥ 5 % averaged ≥ 2 % (pressure was routine).
|
||
Production host: **tight**, and an order line for 8 GB, preselected, for a machine that copes.
|
||
- **C3 — only overload counts.** Stalled = average ≥ 10 %; `tight` from a 1 % share, `upgrade` from 5 %.
|
||
Production host: **fits**. `tight` from about 7 overloaded quarter hours in 30 days, `upgrade` from about 36.
|
||
|
||
Under C3 the fill-level peak cannot remain a second signal, or the production host is `tight` from its 95 % peak after all.
|
||
The peak is therefore only *named* — "peak 5.51 of 5.79 GiB, no stalls" — and never judged.
|
||
It keeps one job: when PSI does demand a raise, the size is the peak-based formula of PR#42 (peak ÷ `TARGET_RATIO` + reserve), rounded up to the bookable step, and at least one step above the current size, because a stall proves that the peak was the ceiling.
|
||
|
||
### Recommendation
|
||
|
||
C3.
|
||
Eighteen seconds of waiting per quarter hour during a build is a build evicting page cache, not a machine short of memory.
|
||
|
||
## D — When a downsize is advice
|
||
|
||
- **D1 — only when the next smaller bookable size still fits.** `DOWNSIZE` requires, beyond today's `DOWNSIZE_RATIO` (peak below 40 %), that the peak stays below the target ratio — `DISK_TARGET_RATIO` for a disk, `TARGET_RATIO` for RAM — of the next bookable size down. Otherwise `fits`.
|
||
Production host: HDD `fits`, since 7.57 GiB is 76 % of 10 GiB.
|
||
- **D2 — as today.** Recommend the computed size, rounded to the step.
|
||
Rounding 11 down to 10 recommends a size the advice itself would flag next; rounding up to 20 recommends the size already booked.
|
||
|
||
A volume (physical storage, RFC 0002 B1) is advised in its 250 GB steps.
|
||
Where that step is absurd against the peak — 8 GiB used, the next step 250 GB away — the verdict is `fits` with a reason saying so, not an order nobody will place.
|
||
|
||
### Recommendation
|
||
|
||
D1.
|
||
|
||
## Decisions
|
||
|
||
Taken on 2026-09-12:
|
||
|
||
1. **A1** — RAM is judged from `memoryPressure` alone; swap occupancy plays no part. `oom_kill` from `/proc/vmstat` is sampled from now on, as the count per bucket, and not yet read by the sizing.
|
||
2. **B1** — the bookable steps are a parameter of `CapacitySizing.of(...)`, derived from `HostingEnvironment` by the caller: for Hostsharing 10 GiB per quota, 250 GB per volume, 1 GB RAM, 1 thread; without a recognised hosting today's 1 GiB and 1 thread. No configuration key.
|
||
3. **C3** — a quarter hour is stalled when its average memory stall is at least 10 %; `tight` from 1 % stalled quarter hours in the window, `upgrade` from 5 %. The fill-level peak is named in the reason and never judged; it sizes the raise when one is due, rounded to the step and at least one step up.
|
||
4. **D1** — a downsize is advised only when the next smaller bookable size keeps the peak below the target ratio; otherwise `fits`.
|
||
|
||
## Consequences
|
||
|
||
- `CapacitySizing.ram` changes its yardstick; `CapacitySizingTest` gets the production host's series as a regression case (peak 5.51 of 5.79 GiB, no quarter hour at 10 %: `fits`) beside the synthetic ones.
|
||
- `CapacitySizing.of(...)` gains a steps parameter; `byPeak` and `disk` round to it and apply D1; the HDD case (peak 7.57 of 20 GiB group quota, step 10) becomes the second regression case: `fits`.
|
||
- `SystemMetricsCollector` reads one more counter from `/proc/vmstat` and the history gets one more base column; the persisted state must accept a history without it.
|
||
- `SystemHealth` keeps its live thresholds; the two share `MEMORY_OVERLOAD_PERCENT`, so a machine that is `healthy` cannot be red in the sizing.
|
||
- `UiFormats.sizingChip` and the `sizingChip`/`orderItems`/`orderSize` twins in `werkator.js` keep producing the same texts; the order dialog thereby lists only bookable sizes.
|
||
- Not touched: the CPU sizing of PR#42, the hosting detection, the mail, and any hosting API.
|
||
- Follow-ups, not part of the implementation: reading `oom_kill` into the verdict once there is experience with it on a shared hive; whether an amber order line should be preselected in the dialog.
|