werkator/docs/prs/2026-09-08-PR#27-system-health-page.md

WARNING: This document describes only the change applied in this PR. It may already be outdated once the next PR is merged. Historic PR-documentation is not maintained along with new PRs — treat it as a snapshot, not as current documentation.

The Problem

The System page is a seven-row table of absolute values: CPU used and idle in cores, RAM used and free in GiB, disk used and free in GiB, repository size, each with min, max and average since the first server start. Whether the machine is fine takes reading the footer to find the totals and dividing in one's head. Half of the rows repeat the other half (idle is cores minus used, free is total minus used). The min/max/avg columns are all-time extremes, so after a few days they freeze and no longer say whether the current value is unusual. There is no history, no trend, and no signal that answers the question the page exists for: is this machine overloaded right now? Load average, swap and the kernel's pressure-stall counters, which answer exactly that, are not read at all.

Non-Goals

• The Instrument Panel redesign of the whole UI (RFC 0001) — this PR stays inside the current palette and layout vocabulary.
• Per-build resource accounting (what one build cost in CPU or RAM).
• Throttling or deferring builds on an overloaded machine — the verdict introduced here is a display; using it in the executor is a follow-up.
• A charting library or any frontend build pipeline — charts are inline SVG drawn by werkator.js.

The Scenarios

Feature: The System page answers "is this machine fine?" at a glance

Background

• Werkator sampled the machine every 60 seconds (SystemMetricsCollector) and serves the latest sample as GET /api/system; this PR makes it every 15 seconds, because many builds last less than a minute and would fall between two samples.
• Utilization thresholds are the existing ones: warn from 80 % of a total, critical from 90 % (UiFormats.utilizationClass, werkator.js).
• A level is ok, warn or crit; the machine verdict is healthy, pressure or overloaded.
• On Linux the kernel offers three overload signals the page did not read: the load average (/proc/loadavg), swap usage (/proc/meminfo), and the pressure-stall information (/proc/pressure/cpu, /proc/pressure/memory, PSI, kernel 4.20+).
• On a shared VM (a Hostsharing Managed Webspace) these numbers describe the whole machine, not only Werkator's builds — which is exactly the question.

Scenario#27.01: Load average, swap and pressure are sampled

So that the page can tell an overloaded machine from a busy one.

• Given a Linux host
• When a sample is taken
• Then the snapshot carries the 1, 5 and 15 minute load averages
  • and the swap total and the swap in use (with min/max/avg like the other metrics)
  • and the CPU and memory pressure (PSI some avg60, in percent)
  • and each of these is null where its source is unavailable, logged once, never failing the sample.

Verified by

• SystemMetricsCollectorTest

Scenario#27.02: The server pronounces a machine verdict with reasons

So that the page states overloaded instead of leaving the reader to combine seven numbers, and so that a later PR can let the executor act on the same verdict.

• Given a sample
• When SystemHealth.assess is applied
• Then the verdict is overloaded when RAM or disk are critical, the 1 minute load is at least twice the core count, or the memory pressure is at least 10 %
  • and pressure when RAM or disk are in the warn band, the 1 minute load exceeds the core count, the CPU pressure is at least 20 %, the memory pressure at least 2 %, or more than half of the swap is in use
  • and healthy otherwise
  • and every rule that fired is named in reasons, so the page can say why
  • and CPU utilization alone never makes the machine overloaded — a build is expected to use every core; waiting (load above cores, pressure) is what counts.

Verified by

• SystemHealthTest

Scenario#27.03: A history of samples survives restarts

So that the page can show the last day and the last month, even right after a deploy.

• Given the collector has taken samples
• When the state is persisted
• Then the last 4 hours are kept as they were sampled, every 15 seconds (960 points)
  • and the last 24 hours at one point per minute (1440 points), each carrying the average and the maximum of its samples
  • and the last 30 days at one point per quarter hour (2880 points), average and maximum of its minutes
  • and the state file is written once a minute and on shutdown, not on every sample
  • and the history is stored column-wise (one timestamp list, one value list per metric) in the existing state file
  • and a restart continues both tiers; a state file from before this PR simply starts an empty history.

Verified by

• MetricsHistoryTest
• SystemMetricsCollectorTest

Scenario#27.04: The history is served per range with its peaks

So that the charts and the peak line come from one request.

• Given a persisted history
• When GET /api/system/history?range=<r> is called with 1h or 4h (the samples), 24h (the minutes), 7d or 30d (the quarter hours)
• Then the response carries the timestamps, one value series per metric (and a maximum series for the quarter-hour ranges)
  • and for CPU, RAM, disk and swap the peak of the range: its value, when it happened, and the level it reached against the current total
  • and an unknown range is a 400, an empty history a 200 with empty series.

Verified by

• SystemApiControllerTest

Scenario#27.05: The page shows the verdict, gauges, charts and peaks

So that the current load is unmistakable and a capacity decision (more RAM, CPU or disk) has its evidence on the page.

• Given the System page is opened
• Then the machine verdict stands at the top with its reasons and the load average
  • and CPU, RAM and disk are one card each: percentage, used of total, a bar, a level pill, and a chart of the selected range
  • and every card names the peak of the selected range with the level it reached, so a resource that was ever critical in the last month is visible even while it is idle now
  • and the range of the charts and peaks is selectable: 1 h (the default, so the current state is what one sees first), 4 h, 24 h, 7 d, 30 d
  • and a strip below shows load 1/5/15, CPU and memory pressure, swap, repository size, and the builds running now
  • and the page keeps polling every 15 seconds with the existing timeout and error badge (the aggregated ranges are refetched once a minute); the charts are inline SVG drawn by werkator.js
  • and UiFormats and werkator.js render percentages, levels and the verdict identically.

Verified by

• UiControllerTest
• UiViewsTest

The Solution

Backend first, one commit per step, then the page:

1. SystemMetricsCollector reads three more kernel files — /proc/loadavg, the swap lines of /proc/meminfo, and /proc/pressure/{cpu,memory} — each through the existing readSource guard, so a host without them (no PSI before kernel 4.20, no swap) shows n/a.
2. SystemHealth.assess turns a snapshot into the verdict with reasons; SystemMetrics.health exposes it, so GET /api/system carries it without a second endpoint. The rules deliberately ignore CPU utilization: a build is supposed to use every core, and the page would otherwise shout overloaded during every build.
3. MetricsHistory is the ring buffer, persisted inside the existing PersistedMetricsState (a state file from before simply starts empty): three tiers, 15 s samples for four hours, minutes for a day, quarter hours for a month, the aggregated ones with average and maximum. Column-wise storage keeps all of it at a few hundred kilobytes; the state file lost its pretty-printing for the same reason, and it is written once a minute instead of on every sample. A minute is aggregated from the samples when the next one opens, a quarter from the minutes, so nothing but the tiers themselves is persisted. The sampling interval went from 60 to 15 seconds on the way: many builds last less than a minute and would otherwise fall between two samples.
4. GET /api/system/history?range=1h|4h|24h|7d|30d serves a cut of one tier (minutes up to a day, quarters beyond) plus its peaks (HistoryPeaks), and GET /api/system gained runningBuilds.
5. The page: a verdict line, three gauge cards (percentage, used of total, bar, level pill, chart with the 80/90 % lines, peak line of the selected range), a tile strip (load 1/5/15, CPU and memory pressure, swap, repository size, running builds), and the range toggle. The charts are inline SVG built by werkator.js from the history endpoint — no library, per the UI invariant; the quarter-hour ranges draw the maxima as a second line, because averages hide exactly the peaks a capacity decision is about. Each chart spans its whole range ending now (a fresh instance shows its first hour at the right edge, the axis below says so) and scales its height to the resource total, so the 80/90 % lines sit where the gauge changes colour. The min/max/avg-since-first-start columns are gone from the page (the API still serves them): the peaks of a range answer the same question with a date.

Open Questions

• The peak percentage of a range is computed against the current total; a quota raised last week makes older peaks look smaller than they were. Acceptable for a capacity decision, noted here.
• The verdict thresholds (SystemHealth constants) are a first guess from mih09 and vm4006; they are not configurable yet.

Additional Changes

• docs/deployment.md and the architecture skill describe the verdict and the history.
• Static resources (werkator.css, werkator.js) are served with Cache-Control: no-cache, must-revalidate: the first rollout of this PR to mih09 rendered the new page with the browser's cached old stylesheet and script, because without the header a browser keeps them for hours after a deploy.