implemented 09-system-metrics.md: added system metrics: introduced system monitoring with a metrics collector, REST API endpoint, Thymeleaf UI rendering, and lifecycle management

This commit is contained in:
Michael Hoennig
2026-07-07 12:55:11 +02:00
parent 67c9f0ade9
commit a6a2c9de0b
21 changed files with 1069 additions and 6 deletions
+5 -1
View File
@@ -70,9 +70,13 @@ Three places must stay in sync when config keys change: the `GitTallyConfig` dat
`Watcher` replaces the legacy blocking main loop with a non-blocking fixed-delay poll cycle: fetch origin, enqueue due branches (changed local, recent new origin, due auto-build slots) via `BuildExecutor`, then prune results, artifacts, and stale worktrees. Nothing is scheduled until `Watcher.start()` is called explicitly (server/watch mode) — CLI commands and tests never start the loop. "Already built" is tracked via the result repository, not by moving local branch refs. Auto-build slot state lives in `.git/gittally/auto-builds.json`; watcher health is exposed via `Watcher.state()`.
### System Metrics
`SystemMetricsCollector` samples CPU (`/proc/stat` deltas), RAM (`/proc/meminfo`), disk, and repository size every 60s, but only after `ServerMetricsLifecycle` (server profile) calls `start()` — like the watcher, nothing is scheduled in CLI runs or tests. Min/max/avg aggregation state persists as JSON in the artifact root (`ArtifactStore.rootDir()`), so restarts continue the series. Unavailable sources (e.g. no `/proc` outside Linux) yield null metrics served as HTTP 200 by `GET /api/system` — the `/system` page shows `n/a`, never an error.
### Package Structure
All production code lives under `de.hoennig.gittally`, with sub-packages `commands` (picocli subcommands), `config` (YAML config loading and schema), `git` (git CLI access), `gitea` (Gitea commit-status API client), `build` (build execution, results, workspaces), `artifacts` (filesystem artifact store), `watcher` (branch polling, auto-builds, startup recovery), and `server` (JSON API controllers, Thymeleaf UI, artifact serving, control token, watcher lifecycle). Tests mirror this structure under `src/test/kotlin`.
All production code lives under `de.hoennig.gittally`, with sub-packages `commands` (picocli subcommands), `config` (YAML config loading and schema), `git` (git CLI access), `gitea` (Gitea commit-status API client), `build` (build execution, results, workspaces), `artifacts` (filesystem artifact store), `watcher` (branch polling, auto-builds, startup recovery), `metrics` (system resource sampling and aggregation), and `server` (JSON API controllers, Thymeleaf UI, artifact serving, control token, watcher and metrics lifecycles). Tests mirror this structure under `src/test/kotlin`.
## Testing Conventions
+27
View File
@@ -33,3 +33,30 @@ Create package `de.hoennig.gittally.metrics`:
- `./gradlew ktlintFormat` then `./gradlew build` is green.
- `/system` renders live values on Linux (manual smoke test; document in this file).
## Implementation Notes (2026-07-07)
Implemented as designed: `SystemMetricsCollector` in `de.hoennig.gittally.metrics` samples every 60s once `ServerMetricsLifecycle` (server profile only) calls `start()`, following the watcher's start/stop pattern.
CPU comes from `/proc/stat` deltas, RAM from `/proc/meminfo`, disk from `java.nio.file.FileStore` (`df` semantics: used = total unallocated, free = usable), and the repository size from a file walk.
`GET /api/system` returns the snapshot plus aggregates, and `/system` renders the legacy system page in the step 08 layout, polling every 60s with the same timeout/error-badge rules.
Since the metric rows are fixed, `gittally.js` only updates the cell texts in place — nothing is rebuilt.
Deviations and decisions:
- The JSON uses camelCase fields with nested `{current, min, max, avg}` aggregates instead of the flat snake_case legacy `system.json`; the value set matches legacy.
- The aggregation state persists as `system-metrics-state.json` in the artifact root and restarts continue the series, as this step requires.
Legacy actually deleted `system_state.dat` on every start, so the footnote now reads "since the first server start" instead of "since script start".
`ArtifactStore` gained `rootDir()` so the state can live next to the stored builds.
- CPU load needs a counter delta, so the first sample after process start reports no CPU metric yet (`n/a`); legacy aggregated a meaningless near-zero first delta instead.
- The repository size is re-probed only every 10th sample (10 minutes) and reused in between — the throttle this step requires; legacy ran `du -sk` every cycle.
The file walk sums file sizes, not disk blocks like `du`, which is close enough for a trend metric.
- An unavailable source (no `/proc` outside Linux, unreadable file store) yields explicit `null` metrics over HTTP 200 and `n/a` cells; the failure is logged once, not every 60s.
- No new config keys: the 60s interval is fixed like legacy, so `GitTallyConfig`, the `init` templates, and `docs/configuration.md` are unchanged.
- The legacy `generation` field was not ported; it only guarded the legacy JS against monitor restarts.
- The CPU count comes from `Runtime.availableProcessors()` instead of `nproc`.
Manual smoke test (2026-07-07): scratch repository with a bare origin, server on port 18986, observed through a real browser tab (via a TCP proxy, so the tab outlived backend restarts).
The first sample rendered RAM/disk/repo values immediately with CPU `n/a` and the totals line (`8 cores`, RAM/disk GiB, updated time).
After the next 60s poll the open tab updated in place without reload: the updated time ticked, CPU used appeared (1.58 cores, idle 6.42 = 8 total), and min/max diverged.
Killing the server flipped the indicator to the red `error` badge and dimmed the table — zero spinners; after a restart the tab returned to `live` and the series continued from the persisted state (`sampleCount` 5, min/max from before the restart preserved).
The 375px viewport stacked the rows as labeled cards, and SIGINT shut the server down cleanly (exit 130, no exceptions).
+1 -1
View File
@@ -56,7 +56,7 @@ Server and UI:
- [x] `07-server-mode.md``server` subcommand, REST/JSON endpoints, artifact serving
- [x] `08-web-ui.md` — HTML views with robust live updates
- [ ] `09-system-metrics.md` — system resource monitoring page
- [x] `09-system-metrics.md` — system resource monitoring page
Completion:
@@ -102,7 +102,7 @@ class FileArtifactStore(
private fun branchesDir(): Path = rootDir().resolve("branches")
private fun rootDir(): Path {
override fun rootDir(): Path {
val configured =
configLoader
.load(workingDir)
@@ -28,4 +28,11 @@ interface ArtifactStore {
/** The stored artifact directory for [artifactKey], or null if none exists. */
fun artifactDir(artifactKey: String): Path?
/**
* The artifact root directory (which need not exist yet). Besides the stored
* builds it hosts sibling state like the system-metrics aggregation — legacy
* kept its `system_state.dat` in the artifact root, too.
*/
fun rootDir(): Path
}
@@ -0,0 +1,25 @@
package de.hoennig.gittally.metrics
import de.hoennig.gittally.build.ArtifactStore
import org.springframework.context.annotation.Bean
import org.springframework.context.annotation.Configuration
import java.time.Clock
@Configuration
class MetricsConfiguration {
/**
* The aggregation state lives in the artifact root (like the legacy
* `system_state.dat`), resolved lazily because the root comes from the config.
* Nothing is sampled or written until [SystemMetricsCollector.start], which only
* `ServerMetricsLifecycle` calls — the bean is inert in CLI runs and tests.
*/
@Bean
fun systemMetricsCollector(
artifactStore: ArtifactStore,
clock: Clock,
): SystemMetricsCollector =
SystemMetricsCollector(
stateFile = { artifactStore.rootDir().resolve(SystemMetricsCollector.STATE_FILE_NAME) },
clock = clock,
)
}
@@ -0,0 +1,54 @@
package de.hoennig.gittally.metrics
import java.time.Instant
/** Current value of one metric plus its min/max/avg over all recorded samples. */
data class MetricAggregate(
val current: Double,
val min: Double,
val max: Double,
val avg: Double,
)
/**
* One system snapshot with aggregates, the payload of `GET /api/system`
* (the legacy `system.json`, with camelCase names like the rest of the API).
* A metric is null when its source is unavailable — e.g. no `/proc` outside
* Linux — and the UI shows `n/a`; the endpoint itself works everywhere.
*/
data class SystemMetrics(
val timestamp: Instant?,
val sampleCount: Long,
val cpuCount: Int,
val ramTotalGib: Double?,
val diskTotalGib: Double?,
val cpuUsed: MetricAggregate?,
val cpuIdle: MetricAggregate?,
val ramUsedGib: MetricAggregate?,
val ramFreeGib: MetricAggregate?,
val diskUsedGib: MetricAggregate?,
val diskFreeGib: MetricAggregate?,
val repoSizeGib: MetricAggregate?,
)
/** Running aggregation of one metric; persisted so restarts continue the series. */
data class MetricSeries(
val min: Double,
val max: Double,
val sum: Double,
val count: Long,
) {
operator fun plus(value: Double) =
MetricSeries(
min = minOf(min, value),
max = maxOf(max, value),
sum = sum + value,
count = count + 1,
)
fun aggregate(current: Double) = MetricAggregate(current = current, min = min, max = max, avg = sum / count)
companion object {
fun of(value: Double) = MetricSeries(min = value, max = value, sum = value, count = 1)
}
}
@@ -0,0 +1,352 @@
package de.hoennig.gittally.metrics
import com.fasterxml.jackson.databind.DeserializationFeature
import com.fasterxml.jackson.databind.ObjectMapper
import com.fasterxml.jackson.databind.SerializationFeature
import com.fasterxml.jackson.module.kotlin.readValue
import com.fasterxml.jackson.module.kotlin.registerKotlinModule
import org.slf4j.LoggerFactory
import java.io.IOException
import java.nio.file.FileVisitResult
import java.nio.file.Files
import java.nio.file.Path
import java.nio.file.Paths
import java.nio.file.SimpleFileVisitor
import java.nio.file.StandardCopyOption
import java.nio.file.attribute.BasicFileAttributes
import java.time.Clock
import java.util.concurrent.Executors
import java.util.concurrent.ScheduledExecutorService
import java.util.concurrent.TimeUnit
/** The aggregation state persisted in the artifact root (the legacy `system_state.dat`, but as JSON). */
data class PersistedMetricsState(
val sampleCount: Long = 0,
val series: Map<String, MetricSeries> = emptyMap(),
)
/**
* Samples CPU (from `/proc/stat` deltas), RAM (from `/proc/meminfo`), disk, and
* repository size every 60 seconds and keeps running min/max/avg per metric.
* The aggregation state is persisted, so restarts continue the series.
* Every source degrades gracefully: an unreadable source makes its metric null
* (shown as `n/a`), never fails a sample. Like the [de.hoennig.gittally.watcher.Watcher],
* nothing is scheduled until [start] is called (server mode only).
*/
class SystemMetricsCollector(
private val stateFile: () -> Path,
private val workingDir: Path = Paths.get("."),
private val clock: Clock = Clock.systemUTC(),
private val procStat: Path = Paths.get("/proc/stat"),
private val procMeminfo: Path = Paths.get("/proc/meminfo"),
private val cpuCount: Int = Runtime.getRuntime().availableProcessors(),
private val diskSpace: (Path) -> DiskSpace = { dir -> fileStoreDiskSpace(dir) },
private val repoSizeBytes: (Path) -> Long = { dir -> directorySizeBytes(dir) },
) {
/** Disk usage in bytes; used/free follow `df` semantics (free is what a user can still allocate). */
data class DiskSpace(
val totalBytes: Long,
val usedBytes: Long,
val freeBytes: Long,
)
private val log = LoggerFactory.getLogger(SystemMetricsCollector::class.java)
private val json =
ObjectMapper()
.registerKotlinModule()
.configure(DeserializationFeature.FAIL_ON_UNKNOWN_PROPERTIES, false)
.configure(SerializationFeature.INDENT_OUTPUT, true)
private var scheduler: ScheduledExecutorService? = null
private var state: PersistedMetricsState? = null
private var previousCpu: CpuCounters? = null
private var lastRepoSizeGib: Double? = null
private var samplesSinceRepoSizeProbe = 0
private val warnedSources = mutableSetOf<String>()
@Volatile
private var snapshot: SystemMetrics? = null
/** The latest sample, or an empty snapshot (all metrics `null`) before the first one. */
fun snapshot(): SystemMetrics = snapshot ?: emptySnapshot()
/** Schedules the fixed-delay sampling loop; the first sample runs immediately. */
@Synchronized
fun start() {
check(scheduler == null) { "metrics collector is already running" }
scheduler =
Executors
.newSingleThreadScheduledExecutor { runnable ->
Thread(runnable, "gittally-metrics").apply { isDaemon = true }
}.also {
it.scheduleWithFixedDelay(::sampleSafely, 0, SAMPLE_INTERVAL_SECONDS, TimeUnit.SECONDS)
}
}
@Synchronized
fun stop() {
scheduler?.shutdownNow()
scheduler = null
}
/**
* Takes one sample, updates the aggregates, and persists the aggregation state.
* CPU needs a counter delta, so the very first sample after process start reports
* no CPU metric yet (legacy aggregated a meaningless near-zero first delta instead).
*/
@Synchronized
fun sample() {
val previousState = state ?: loadState()
val series = previousState.series.toMutableMap()
val sampleCount = previousState.sampleCount + 1
fun record(
key: String,
value: Double?,
): MetricAggregate? {
if (value == null) {
return null
}
val updated = series[key]?.plus(value) ?: MetricSeries.of(value)
series[key] = updated
return updated.aggregate(value)
}
val cpu = readCpuLoad()
val ram = readRam()
val disk = readDisk()
val repoSizeGib = readRepoSizeThrottled()
snapshot =
SystemMetrics(
timestamp = clock.instant(),
sampleCount = sampleCount,
cpuCount = cpuCount,
ramTotalGib = ram?.totalGib,
diskTotalGib = disk?.totalBytes?.let { it / BYTES_PER_GIB },
cpuUsed = record("cpuUsed", cpu?.usedCores),
cpuIdle = record("cpuIdle", cpu?.idleCores),
ramUsedGib = record("ramUsedGib", ram?.usedGib),
ramFreeGib = record("ramFreeGib", ram?.freeGib),
diskUsedGib = record("diskUsedGib", disk?.usedBytes?.let { it / BYTES_PER_GIB }),
diskFreeGib = record("diskFreeGib", disk?.freeBytes?.let { it / BYTES_PER_GIB }),
repoSizeGib = record("repoSizeGib", repoSizeGib),
)
state = PersistedMetricsState(sampleCount = sampleCount, series = series)
saveState(state!!)
}
private fun sampleSafely() {
try {
sample()
} catch (e: Exception) {
log.error("metrics sample failed", e)
}
}
private fun emptySnapshot() =
SystemMetrics(
timestamp = null,
sampleCount = 0,
cpuCount = cpuCount,
ramTotalGib = null,
diskTotalGib = null,
cpuUsed = null,
cpuIdle = null,
ramUsedGib = null,
ramFreeGib = null,
diskUsedGib = null,
diskFreeGib = null,
repoSizeGib = null,
)
// ---- sources, each null when unavailable --------------------------------
private data class CpuCounters(
val total: Long,
val idle: Long,
)
private data class CpuLoad(
val usedCores: Double,
val idleCores: Double,
)
/** Like the legacy monitor: used cores = `cpuCount * (totalDiff - idleDiff) / totalDiff`. */
private fun readCpuLoad(): CpuLoad? {
val counters = readCpuCounters() ?: return null
val previous = previousCpu
previousCpu = counters
if (previous == null) {
return null
}
val totalDiff = counters.total - previous.total
val idleDiff = counters.idle - previous.idle
val usedCores =
if (totalDiff > 0) {
cpuCount * (totalDiff - idleDiff).toDouble() / totalDiff
} else {
0.0
}
return CpuLoad(usedCores = usedCores, idleCores = cpuCount - usedCores)
}
/** The aggregate `cpu` line of `/proc/stat`: total is the sum of all fields, idle is the 4th. */
private fun readCpuCounters(): CpuCounters? =
readSource("cpu") {
val fields =
Files
.readAllLines(procStat)
.first { it.startsWith("cpu ") }
.split(Regex("\\s+"))
.drop(1)
.map { it.toLong() }
CpuCounters(total = fields.sum(), idle = fields[3])
}
private data class RamUsage(
val totalGib: Double,
val usedGib: Double,
val freeGib: Double,
)
/** `MemTotal` and `MemAvailable` (KiB) from `/proc/meminfo`; used = total - available. */
private fun readRam(): RamUsage? =
readSource("ram") {
val lines = Files.readAllLines(procMeminfo)
val totalKib = memValueKib(lines, "MemTotal")
val availableKib = memValueKib(lines, "MemAvailable")
RamUsage(
totalGib = totalKib / KIB_PER_GIB,
usedGib = (totalKib - availableKib) / KIB_PER_GIB,
freeGib = availableKib / KIB_PER_GIB,
)
}
private fun memValueKib(
lines: List<String>,
key: String,
): Long =
lines
.first { it.startsWith("$key:") }
.removePrefix("$key:")
.trim()
.removeSuffix(" kB")
.toLong()
private fun readDisk(): DiskSpace? = readSource("disk") { diskSpace(workingDir) }
/**
* The repository size is expensive to determine (a full file walk), so unlike
* legacy — which ran `du -sk` every cycle — it is re-probed only every
* [REPO_SIZE_PROBE_EVERY_SAMPLES] samples and reused in between.
*/
private fun readRepoSizeThrottled(): Double? {
if (lastRepoSizeGib != null && samplesSinceRepoSizeProbe < REPO_SIZE_PROBE_EVERY_SAMPLES) {
samplesSinceRepoSizeProbe++
return lastRepoSizeGib
}
lastRepoSizeGib = readSource("repo size") { repoSizeBytes(workingDir) / BYTES_PER_GIB }
samplesSinceRepoSizeProbe = 1
return lastRepoSizeGib
}
/** Wraps one source read; a failure yields null and is logged once, not every 60s. */
private fun <T> readSource(
source: String,
read: () -> T,
): T? =
try {
read().also { warnedSources.remove(source) }
} catch (e: Exception) {
if (warnedSources.add(source)) {
log.warn("cannot read {} metrics; showing n/a: {}", source, e.toString())
}
null
}
// ---- aggregation state persistence ---------------------------------------
private fun loadState(): PersistedMetricsState {
val file = stateFile()
if (!Files.exists(file)) {
return PersistedMetricsState()
}
return try {
json.readValue<PersistedMetricsState>(file.toFile())
} catch (e: Exception) {
log.warn("ignoring unreadable metrics state file {}; starting a fresh series: {}", file, e.message)
PersistedMetricsState()
}
}
private fun saveState(state: PersistedMetricsState) {
try {
val file = stateFile()
Files.createDirectories(file.parent)
val tempFile = Files.createTempFile(file.parent, file.fileName.toString(), ".tmp")
try {
json.writeValue(tempFile.toFile(), state)
Files.move(tempFile, file, StandardCopyOption.ATOMIC_MOVE, StandardCopyOption.REPLACE_EXISTING)
} finally {
Files.deleteIfExists(tempFile)
}
} catch (e: Exception) {
log.warn("cannot persist metrics state; aggregates restart on the next server start: {}", e.toString())
}
}
companion object {
const val SAMPLE_INTERVAL_SECONDS = 60L
/** With 60s samples, the repository size is re-measured every 10 minutes. */
const val REPO_SIZE_PROBE_EVERY_SAMPLES = 10
/** State file name in the artifact root, next to the `branches/` directory. */
const val STATE_FILE_NAME = "system-metrics-state.json"
private const val KIB_PER_GIB = 1_048_576.0
private const val BYTES_PER_GIB = 1_073_741_824.0
/** `df` semantics via [java.nio.file.FileStore]: free is the space a user can still allocate. */
fun fileStoreDiskSpace(dir: Path): DiskSpace {
val store = Files.getFileStore(dir)
return DiskSpace(
totalBytes = store.totalSpace,
usedBytes = store.totalSpace - store.unallocatedSpace,
freeBytes = store.usableSpace,
)
}
/** File-walk replacement for the legacy `du -sk`; unreadable subtrees are skipped, links are not followed. */
fun directorySizeBytes(dir: Path): Long {
var size = 0L
Files.walkFileTree(
dir,
object : SimpleFileVisitor<Path>() {
override fun visitFile(
file: Path,
attrs: BasicFileAttributes,
): FileVisitResult {
if (attrs.isRegularFile) {
size += attrs.size()
}
return FileVisitResult.CONTINUE
}
override fun visitFileFailed(
file: Path,
exc: IOException,
): FileVisitResult = FileVisitResult.CONTINUE
},
)
return size
}
}
}
@@ -0,0 +1,28 @@
package de.hoennig.gittally.server
import de.hoennig.gittally.metrics.SystemMetricsCollector
import jakarta.annotation.PreDestroy
import org.springframework.boot.context.event.ApplicationReadyEvent
import org.springframework.context.annotation.Profile
import org.springframework.context.event.EventListener
import org.springframework.stereotype.Component
/**
* Starts the system-metrics sampling loop once the server context is ready and
* stops it on shutdown. Only in the `server` profile, like [ServerWatcherLifecycle].
*/
@Component
@Profile("server")
class ServerMetricsLifecycle(
private val collector: SystemMetricsCollector,
) {
@EventListener(ApplicationReadyEvent::class)
fun onApplicationReady() {
collector.start()
}
@PreDestroy
fun onShutdown() {
collector.stop()
}
}
@@ -0,0 +1,18 @@
package de.hoennig.gittally.server
import de.hoennig.gittally.metrics.SystemMetrics
import de.hoennig.gittally.metrics.SystemMetricsCollector
import org.springframework.web.bind.annotation.GetMapping
import org.springframework.web.bind.annotation.RestController
@RestController
class SystemApiController(
private val collector: SystemMetricsCollector,
) {
/**
* The current system snapshot plus min/max/avg aggregates (the legacy
* `system.json`). Unavailable metrics are null, never an error.
*/
@GetMapping("/api/system")
fun system(): SystemMetrics = collector.snapshot()
}
@@ -5,6 +5,7 @@ import de.hoennig.gittally.build.BuildExecutor
import de.hoennig.gittally.build.BuildResultRepository
import de.hoennig.gittally.build.BuildStatus
import de.hoennig.gittally.config.ConfigLoader
import de.hoennig.gittally.metrics.SystemMetricsCollector
import org.springframework.beans.factory.ObjectProvider
import org.springframework.boot.info.BuildProperties
import org.springframework.http.HttpStatus
@@ -32,6 +33,7 @@ class UiController(
private val artifactStore: ArtifactStore,
private val controlTokens: ControlTokenService,
private val configLoader: ConfigLoader,
private val metricsCollector: SystemMetricsCollector,
private val buildProperties: ObjectProvider<BuildProperties>,
) {
var workingDir: Path = Paths.get(".")
@@ -80,6 +82,13 @@ class UiController(
return "current"
}
@GetMapping("/system")
fun system(model: Model): String {
baseModel(model, view = "system", pageTitle = "System Metrics")
model.addAttribute("metrics", SystemMetricsView.from(metricsCollector.snapshot()))
return "system"
}
/** Artifact index rendered from the artifact store — legacy pre-generated this page as static HTML. */
@GetMapping("/builds/{artifactKey}")
fun artifactIndex(
@@ -2,12 +2,15 @@ package de.hoennig.gittally.server
import de.hoennig.gittally.build.BuildResult
import de.hoennig.gittally.config.GiteaConfig
import de.hoennig.gittally.metrics.MetricAggregate
import de.hoennig.gittally.metrics.SystemMetrics
import java.net.URLEncoder
import java.nio.charset.StandardCharsets
import java.time.Duration
import java.time.Instant
import java.time.ZoneId
import java.time.format.DateTimeFormatter
import java.util.Locale
/** Links into the Gitea web UI, like legacy `gitea_branch_web_url`; null when Gitea is not configured. */
class GiteaWebLinks(
@@ -33,6 +36,8 @@ class GiteaWebLinks(
object UiFormats {
private val timestampFormat = DateTimeFormatter.ofPattern("yyyy-MM-dd HH:mm").withZone(ZoneId.systemDefault())
private val timeOfDayFormat = DateTimeFormatter.ofPattern("HH:mm:ss").withZone(ZoneId.systemDefault())
fun timestamp(instant: Instant): String = timestampFormat.format(instant)
/** `m:ss`, or `h:mm:ss` from one hour — like the legacy `MM:SS` duration column. */
@@ -50,6 +55,16 @@ object UiFormats {
"%d:%02d".format(minutes, rest)
}
}
/** Two-decimal metric value with a dot like JS `toFixed(2)`; `n/a` when the source is unavailable. */
fun metric(value: Double?): String =
if (value == null || !value.isFinite()) {
"n/a"
} else {
String.format(Locale.ROOT, "%.2f", value)
}
fun timeOfDay(instant: Instant): String = timeOfDayFormat.format(instant)
}
/** One row of the latest/history build tables. */
@@ -96,3 +111,60 @@ data class CurrentBuildView(
val branchUrl: String?,
val commitUrl: String?,
)
/**
* One row of the system-metrics table. The [key] matches the JSON field of
* `GET /api/system`, so `gittally.js` can update the cells in place.
*/
data class MetricRowView(
val key: String,
val label: String,
val current: String,
val min: String,
val max: String,
val avg: String,
) {
companion object {
fun from(
key: String,
label: String,
aggregate: MetricAggregate?,
) = MetricRowView(
key = key,
label = label,
current = UiFormats.metric(aggregate?.current),
min = UiFormats.metric(aggregate?.min),
max = UiFormats.metric(aggregate?.max),
avg = UiFormats.metric(aggregate?.avg),
)
}
}
/** The system view: the metric rows plus the totals/updated info line, like the legacy system page. */
data class SystemMetricsView(
val rows: List<MetricRowView>,
val cpuCount: String,
val ramTotal: String,
val diskTotal: String,
val updated: String,
) {
companion object {
fun from(metrics: SystemMetrics) =
SystemMetricsView(
rows =
listOf(
MetricRowView.from("cpuUsed", "CPU used (cores)", metrics.cpuUsed),
MetricRowView.from("cpuIdle", "CPU idle (cores)", metrics.cpuIdle),
MetricRowView.from("ramUsedGib", "RAM used (GiB)", metrics.ramUsedGib),
MetricRowView.from("ramFreeGib", "RAM free (GiB)", metrics.ramFreeGib),
MetricRowView.from("diskUsedGib", "Disk used (GiB)", metrics.diskUsedGib),
MetricRowView.from("diskFreeGib", "Disk free (GiB)", metrics.diskFreeGib),
MetricRowView.from("repoSizeGib", "Repo size (GiB)", metrics.repoSizeGib),
),
cpuCount = "${metrics.cpuCount} cores",
ramTotal = metrics.ramTotalGib?.let { "${UiFormats.metric(it)} GiB" } ?: "n/a",
diskTotal = metrics.diskTotalGib?.let { "${UiFormats.metric(it)} GiB" } ?: "n/a",
updated = metrics.timestamp?.let { UiFormats.timeOfDay(it) } ?: "n/a",
)
}
}
+8
View File
@@ -86,6 +86,12 @@ tbody.is-stale { opacity: 0.55; }
.duration-cell { white-space: nowrap; }
.empty { padding: 28px 14px; color: var(--muted); text-align: center; }
/* system metrics */
#system-table { min-width: 640px; }
.num { text-align: right; font-variant-numeric: tabular-nums; font-family: ui-monospace, SFMono-Regular, Menlo, Consolas, monospace; font-size: 13px; }
th.num { font-family: inherit; font-size: 12px; }
.meta { margin: 14px 0 0; color: var(--muted); font-size: 13px; display: flex; justify-content: space-between; align-items: baseline; flex-wrap: wrap; gap: 6px; }
/* status badges */
.status { display: inline-flex; align-items: center; min-width: 72px; justify-content: center; padding: 3px 10px; border-radius: 999px; font-size: 12px; font-weight: 700; text-transform: uppercase; }
.status-success { background: var(--success-bg); color: var(--success-text); }
@@ -144,4 +150,6 @@ tbody.is-stale { opacity: 0.55; }
td + td { border-top: 1px solid color-mix(in srgb, var(--border) 50%, transparent); }
td[data-label]::before { content: attr(data-label); width: 90px; flex-shrink: 0; font-size: 11px; font-weight: 700; text-transform: uppercase; color: var(--muted); }
.actions-cell { justify-content: center; background: color-mix(in srgb, var(--border) 18%, transparent); width: auto; }
#system-table { min-width: 0; }
.num { text-align: left; }
}
+49
View File
@@ -34,6 +34,20 @@ function abbrevCommit(commit) {
return (commit || "").slice(0, 12);
}
/** Two-decimal metric value like `UiFormats.metric`; "n/a" when the source is unavailable. */
function formatMetric(value) {
return typeof value === "number" && Number.isFinite(value) ? value.toFixed(2) : "n/a";
}
function formatTimeOfDay(iso) {
const date = new Date(iso);
if (!iso || Number.isNaN(date.getTime())) {
return "n/a";
}
const two = (n) => String(n).padStart(2, "0");
return `${two(date.getHours())}:${two(date.getMinutes())}:${two(date.getSeconds())}`;
}
const KNOWN_STATUSES = new Set(
["success", "failed", "running", "pending", "interrupted", "cancelled", "unknown", "error", "finished"],
);
@@ -47,6 +61,7 @@ function statusCssClass(status) {
const FETCH_TIMEOUT_MS = 8000;
const TABLE_POLL_MS = 10000;
const CURRENT_POLL_MS = 3000;
const SYSTEM_POLL_MS = 60000;
function metaContent(name) {
const element = document.querySelector(`meta[name="${name}"]`);
@@ -376,6 +391,39 @@ function initCurrentBuilds() {
startPolling(refresh, CURRENT_POLL_MS);
}
// ---- system metrics ----------------------------------------------------------
/** The metric rows are fixed, so only the cell texts are updated — never rebuilt. */
function initSystemTable() {
const table = document.getElementById("system-table");
if (!table) {
return;
}
function setText(id, text) {
const element = document.getElementById(id);
if (element) {
element.textContent = text;
}
}
async function refresh() {
const metrics = await fetchJson(table.dataset.api);
table.querySelectorAll("tbody tr[data-metric]").forEach((row) => {
const aggregate = metrics[row.dataset.metric];
row.querySelectorAll("[data-field]").forEach((cell) => {
cell.textContent = formatMetric(aggregate ? aggregate[cell.dataset.field] : null);
});
});
setText("info-cpu-count", metrics.cpuCount != null ? metrics.cpuCount + " cores" : "n/a");
setText("info-ram-total", metrics.ramTotalGib != null ? formatMetric(metrics.ramTotalGib) + " GiB" : "n/a");
setText("info-disk-total", metrics.diskTotalGib != null ? formatMetric(metrics.diskTotalGib) + " GiB" : "n/a");
setText("info-updated", formatTimeOfDay(metrics.timestamp));
}
startPolling(refresh, SYSTEM_POLL_MS);
}
// ---- running-duration ticking ------------------------------------------------
function tickRunningDurations() {
@@ -438,5 +486,6 @@ document.addEventListener("click", async (event) => {
initBuildsTable();
initCurrentBuilds();
initSystemTable();
setInterval(tickRunningDurations, 1000);
tickRunningDurations();
@@ -25,6 +25,8 @@
<a th:unless="${view == 'history'}" href="/history">History</a>
<span th:if="${view == 'current'}">Current</span>
<a th:unless="${view == 'current'}" href="/current">Current</a>
<span th:if="${view == 'system'}">System</span>
<a th:unless="${view == 'system'}" href="/system">System</a>
</nav>
<span class="view-row-actions">
<span id="live-indicator" class="status status-unknown" title="live-update state">static</span>
+44
View File
@@ -0,0 +1,44 @@
<!doctype html>
<html xmlns:th="http://www.thymeleaf.org" lang="en">
<head th:replace="~{fragments :: head(${pageTitle})}"></head>
<body>
<main>
<h1 th:replace="~{fragments :: header(${pageTitle})}"></h1>
<div th:replace="~{fragments :: nav(${view})}"></div>
<div class="table-wrap">
<table id="system-table" data-api="/api/system">
<thead>
<tr>
<th>Metric</th>
<th class="num">Current</th>
<th class="num">Min</th>
<th class="num">Max</th>
<th class="num">Avg</th>
</tr>
</thead>
<tbody id="system-rows">
<tr th:each="row : ${metrics.rows}" th:attr="data-metric=${row.key}">
<td data-label="Metric" th:text="${row.label}">CPU used (cores)</td>
<td class="num" data-label="Current" data-field="current" th:text="${row.current}">0.42</td>
<td class="num" data-label="Min" data-field="min" th:text="${row.min}">0.10</td>
<td class="num" data-label="Max" data-field="max" th:text="${row.max}">3.20</td>
<td class="num" data-label="Avg" data-field="avg" th:text="${row.avg}">0.80</td>
</tr>
</tbody>
</table>
</div>
<p class="meta">
<span>
CPU total: <strong id="info-cpu-count" th:text="${metrics.cpuCount}">8 cores</strong>
&nbsp;·&nbsp; RAM total: <strong id="info-ram-total" th:text="${metrics.ramTotal}">31.29 GiB</strong>
&nbsp;·&nbsp; Disk total: <strong id="info-disk-total" th:text="${metrics.diskTotal}">465.12 GiB</strong>
&nbsp;·&nbsp; Updated: <span id="info-updated" th:text="${metrics.updated}">12:00:00</span>
(updated every 60s)
</span>
<span>(*: min/max/avg since the first server start)</span>
</p>
</main>
<footer th:replace="~{fragments :: footer}"></footer>
<script src="/gittally.js"></script>
</body>
</html>
@@ -1,6 +1,7 @@
package de.hoennig.gittally
import com.ninjasquad.springmockk.MockkBean
import de.hoennig.gittally.metrics.SystemMetricsCollector
import de.hoennig.gittally.watcher.Watcher
import io.kotest.core.spec.style.FunSpec
import io.kotest.matchers.booleans.shouldBeTrue
@@ -12,8 +13,9 @@ import org.springframework.test.context.ActiveProfiles
import org.springframework.web.client.RestClient
/**
* Proves the `server` profile boots a real web server and starts the watcher.
* The watcher is mocked so the test never fetches origin or enqueues builds.
* Proves the `server` profile boots a real web server and starts the watcher and
* the metrics collector. Both are mocked so the test never fetches origin,
* enqueues builds, or walks the repository for its size.
*/
@SpringBootTest(webEnvironment = SpringBootTest.WebEnvironment.RANDOM_PORT)
@ActiveProfiles("server")
@@ -21,6 +23,9 @@ class ServerModeApplicationTest : FunSpec() {
@MockkBean(relaxUnitFun = true)
lateinit var watcher: Watcher
@MockkBean(relaxUnitFun = true)
lateinit var metricsCollector: SystemMetricsCollector
@LocalServerPort
var port: Int = 0
@@ -40,6 +45,7 @@ class ServerModeApplicationTest : FunSpec() {
.shouldBeTrue()
verify { watcher.start(any()) }
verify { metricsCollector.start() }
}
}
}
@@ -0,0 +1,217 @@
package de.hoennig.gittally.metrics
import io.kotest.core.spec.style.FunSpec
import io.kotest.matchers.nulls.shouldNotBeNull
import io.kotest.matchers.shouldBe
import java.nio.file.Files
import java.nio.file.Path
import java.time.Clock
import java.time.Instant
import java.time.ZoneOffset
class SystemMetricsCollectorTest : FunSpec() {
private lateinit var tempDir: Path
private val now = Instant.parse("2026-07-07T10:00:00Z")
private fun collector(
cpuCount: Int = 4,
diskSpace: (Path) -> SystemMetricsCollector.DiskSpace = { GIB_100_DISK },
repoSizeBytes: (Path) -> Long = { HALF_GIB_BYTES },
) = SystemMetricsCollector(
stateFile = { tempDir.resolve("system-metrics-state.json") },
workingDir = tempDir,
clock = Clock.fixed(now, ZoneOffset.UTC),
procStat = tempDir.resolve("stat"),
procMeminfo = tempDir.resolve("meminfo"),
cpuCount = cpuCount,
diskSpace = diskSpace,
repoSizeBytes = repoSizeBytes,
)
private fun writeStat(
total: Long,
idle: Long,
) {
// "cpu" totals across: user nice system idle iowait irq softirq steal guest guest_nice
val user = total - idle
Files.writeString(tempDir.resolve("stat"), "cpu $user 0 0 $idle 0 0 0 0 0 0\ncpu0 0 0 0 0 0 0 0 0 0 0\n")
}
private fun writeMeminfo(
totalKib: Long,
availableKib: Long,
) {
Files.writeString(
tempDir.resolve("meminfo"),
"MemTotal: $totalKib kB\nMemFree: 1000000 kB\nMemAvailable: $availableKib kB\n",
)
}
init {
beforeEach {
tempDir = Files.createTempDirectory("gittally-metrics-test")
}
afterEach {
tempDir.toFile().deleteRecursively()
}
test("CPU load is computed from /proc/stat deltas, so the first sample has no CPU metric yet") {
writeStat(total = 1000, idle = 700)
writeMeminfo(totalKib = 33_554_432, availableKib = 25_165_824)
val collector = collector()
collector.sample()
collector.snapshot().cpuUsed shouldBe null
writeStat(total = 1100, idle = 775)
collector.sample()
val snapshot = collector.snapshot()
// 4 cores * (100 total - 75 idle) / 100 total = 1 core used
snapshot.cpuUsed.shouldNotBeNull().current shouldBe 1.0
snapshot.cpuIdle.shouldNotBeNull().current shouldBe 3.0
}
test("RAM comes from MemTotal and MemAvailable in GiB") {
writeStat(total = 1000, idle = 700)
writeMeminfo(totalKib = 33_554_432, availableKib = 25_165_824)
val collector = collector()
collector.sample()
val snapshot = collector.snapshot()
snapshot.ramTotalGib shouldBe 32.0
snapshot.ramUsedGib.shouldNotBeNull().current shouldBe 8.0
snapshot.ramFreeGib.shouldNotBeNull().current shouldBe 24.0
}
test("disk and repository size are reported in GiB") {
writeStat(total = 1000, idle = 700)
writeMeminfo(totalKib = 33_554_432, availableKib = 25_165_824)
val collector = collector()
collector.sample()
val snapshot = collector.snapshot()
snapshot.diskTotalGib shouldBe 100.0
snapshot.diskUsedGib.shouldNotBeNull().current shouldBe 40.0
snapshot.diskFreeGib.shouldNotBeNull().current shouldBe 55.0
snapshot.repoSizeGib.shouldNotBeNull().current shouldBe 0.5
}
test("min, max, and avg aggregate over all samples") {
writeStat(total = 1000, idle = 700)
writeMeminfo(totalKib = 33_554_432, availableKib = 25_165_824)
val usedGibValues = mutableListOf(40L, 20L, 60L)
val collector =
collector(diskSpace = {
SystemMetricsCollector.DiskSpace(
totalBytes = 100L * GIB,
usedBytes = usedGibValues.removeFirst() * GIB,
freeBytes = 30L * GIB,
)
})
repeat(3) { collector.sample() }
val diskUsed = collector.snapshot().diskUsedGib.shouldNotBeNull()
diskUsed.current shouldBe 60.0
diskUsed.min shouldBe 20.0
diskUsed.max shouldBe 60.0
diskUsed.avg shouldBe 40.0
}
test("a restart loads the persisted aggregation state and continues the series") {
writeStat(total = 1000, idle = 700)
writeMeminfo(totalKib = 33_554_432, availableKib = 25_165_824)
collector(repoSizeBytes = { 4L * GIB }).sample()
val restarted = collector(repoSizeBytes = { 2L * GIB })
restarted.sample()
val snapshot = restarted.snapshot()
snapshot.sampleCount shouldBe 2
val repoSize = snapshot.repoSizeGib.shouldNotBeNull()
repoSize.current shouldBe 2.0
repoSize.min shouldBe 2.0
repoSize.max shouldBe 4.0
repoSize.avg shouldBe 3.0
}
test("a corrupt state file starts a fresh series instead of failing") {
writeStat(total = 1000, idle = 700)
writeMeminfo(totalKib = 33_554_432, availableKib = 25_165_824)
Files.writeString(tempDir.resolve("system-metrics-state.json"), "not json {")
val collector = collector()
collector.sample()
collector.snapshot().sampleCount shouldBe 1
}
test("unreadable sources degrade to null metrics, never fail the sample") {
// no stat/meminfo files written, disk and repo-size probes fail
val collector =
collector(
diskSpace = { error("no file store") },
repoSizeBytes = { error("walk failed") },
)
collector.sample()
val snapshot = collector.snapshot()
snapshot.timestamp shouldBe now
snapshot.sampleCount shouldBe 1
snapshot.cpuCount shouldBe 4
snapshot.cpuUsed shouldBe null
snapshot.ramTotalGib shouldBe null
snapshot.ramUsedGib shouldBe null
snapshot.diskTotalGib shouldBe null
snapshot.diskUsedGib shouldBe null
snapshot.repoSizeGib shouldBe null
}
test("the repository size probe is throttled to every 10th sample") {
writeStat(total = 1000, idle = 700)
writeMeminfo(totalKib = 33_554_432, availableKib = 25_165_824)
var probes = 0
val collector =
collector(repoSizeBytes = {
probes++
HALF_GIB_BYTES
})
repeat(11) { collector.sample() }
probes shouldBe 2
collector
.snapshot()
.repoSizeGib
.shouldNotBeNull()
.current shouldBe 0.5
}
test("the snapshot before the first sample is empty but well-formed") {
val snapshot = collector().snapshot()
snapshot.timestamp shouldBe null
snapshot.sampleCount shouldBe 0
snapshot.cpuCount shouldBe 4
snapshot.cpuUsed shouldBe null
}
}
companion object {
private const val GIB = 1_073_741_824L
private const val HALF_GIB_BYTES = GIB / 2
private val GIB_100_DISK =
SystemMetricsCollector.DiskSpace(
totalBytes = 100L * GIB,
usedBytes = 40L * GIB,
freeBytes = 55L * GIB,
)
}
}
@@ -0,0 +1,81 @@
package de.hoennig.gittally.server
import com.ninjasquad.springmockk.MockkBean
import de.hoennig.gittally.metrics.MetricAggregate
import de.hoennig.gittally.metrics.SystemMetrics
import de.hoennig.gittally.metrics.SystemMetricsCollector
import io.kotest.core.spec.style.FunSpec
import io.mockk.clearMocks
import io.mockk.every
import org.hamcrest.Matchers.nullValue
import org.springframework.beans.factory.annotation.Autowired
import org.springframework.boot.webmvc.test.autoconfigure.WebMvcTest
import org.springframework.test.web.servlet.MockMvc
import org.springframework.test.web.servlet.request.MockMvcRequestBuilders.get
import org.springframework.test.web.servlet.result.MockMvcResultMatchers.jsonPath
import org.springframework.test.web.servlet.result.MockMvcResultMatchers.status
import java.time.Instant
@WebMvcTest(SystemApiController::class, properties = ["spring.main.web-application-type=servlet"])
class SystemApiControllerTest : FunSpec() {
@Autowired
lateinit var mockMvc: MockMvc
@MockkBean
lateinit var collector: SystemMetricsCollector
private val emptySnapshot =
SystemMetrics(
timestamp = null,
sampleCount = 0,
cpuCount = 8,
ramTotalGib = null,
diskTotalGib = null,
cpuUsed = null,
cpuIdle = null,
ramUsedGib = null,
ramFreeGib = null,
diskUsedGib = null,
diskFreeGib = null,
repoSizeGib = null,
)
init {
beforeEach { clearMocks(collector) }
test("the system endpoint answers snapshot and aggregates") {
every { collector.snapshot() } returns
emptySnapshot.copy(
timestamp = Instant.parse("2026-07-07T10:00:00Z"),
sampleCount = 5,
ramTotalGib = 32.0,
cpuUsed = MetricAggregate(current = 1.0, min = 0.5, max = 2.0, avg = 1.25),
)
mockMvc
.perform(get("/api/system"))
.andExpect(status().isOk)
.andExpect(jsonPath("$.timestamp").value("2026-07-07T10:00:00Z"))
.andExpect(jsonPath("$.sampleCount").value(5))
.andExpect(jsonPath("$.cpuCount").value(8))
.andExpect(jsonPath("$.ramTotalGib").value(32.0))
.andExpect(jsonPath("$.cpuUsed.current").value(1.0))
.andExpect(jsonPath("$.cpuUsed.min").value(0.5))
.andExpect(jsonPath("$.cpuUsed.max").value(2.0))
.andExpect(jsonPath("$.cpuUsed.avg").value(1.25))
}
test("unavailable metrics are explicit nulls, and the endpoint still answers 200") {
every { collector.snapshot() } returns emptySnapshot
mockMvc
.perform(get("/api/system"))
.andExpect(status().isOk)
.andExpect(jsonPath("$.sampleCount").value(0))
.andExpect(jsonPath("$.timestamp").value(nullValue()))
.andExpect(jsonPath("$.cpuUsed").value(nullValue()))
.andExpect(jsonPath("$.ramUsedGib").value(nullValue()))
.andExpect(jsonPath("$.repoSizeGib").value(nullValue()))
}
}
}
@@ -11,6 +11,9 @@ import de.hoennig.gittally.config.ConfigLoader
import de.hoennig.gittally.config.GitTallyConfig
import de.hoennig.gittally.config.GiteaConfig
import de.hoennig.gittally.config.ServerConfig
import de.hoennig.gittally.metrics.MetricAggregate
import de.hoennig.gittally.metrics.SystemMetrics
import de.hoennig.gittally.metrics.SystemMetricsCollector
import io.kotest.core.spec.style.FunSpec
import io.mockk.clearMocks
import io.mockk.every
@@ -49,8 +52,27 @@ class UiControllerTest : FunSpec() {
@MockkBean
lateinit var configLoader: ConfigLoader
@MockkBean
lateinit var metricsCollector: SystemMetricsCollector
private val startedAt = Instant.parse("2026-07-07T10:00:00Z")
private val emptySystemMetrics =
SystemMetrics(
timestamp = null,
sampleCount = 0,
cpuCount = 8,
ramTotalGib = null,
diskTotalGib = null,
cpuUsed = null,
cpuIdle = null,
ramUsedGib = null,
ramFreeGib = null,
diskUsedGib = null,
diskFreeGib = null,
repoSizeGib = null,
)
private val successResult =
BuildResult(
branch = "main",
@@ -63,7 +85,7 @@ class UiControllerTest : FunSpec() {
init {
beforeEach {
clearMocks(repository, buildExecutor, artifactStore, controlTokens, configLoader)
clearMocks(repository, buildExecutor, artifactStore, controlTokens, configLoader, metricsCollector)
every { configLoader.load(any()) } returns
GitTallyConfig(
server = ServerConfig(impressumUrl = "https://example.org/imprint"),
@@ -191,6 +213,36 @@ class UiControllerTest : FunSpec() {
.andExpect(status().isNotFound)
}
test("system view renders metric rows, totals, and the polling hook") {
every { metricsCollector.snapshot() } returns
emptySystemMetrics.copy(
timestamp = Instant.parse("2026-07-07T10:00:00Z"),
sampleCount = 5,
ramTotalGib = 32.0,
cpuUsed = MetricAggregate(current = 1.0, min = 0.5, max = 2.0, avg = 1.25),
)
mockMvc
.perform(get("/system"))
.andExpect(status().isOk)
.andExpect(content().string(containsString("""data-api="/api/system"""")))
.andExpect(content().string(containsString("""data-metric="cpuUsed"""")))
.andExpect(content().string(containsString("CPU used (cores)")))
.andExpect(content().string(containsString("1.25")))
.andExpect(content().string(containsString("8 cores")))
.andExpect(content().string(containsString("32.00 GiB")))
}
test("system view renders n/a for unavailable metrics") {
every { metricsCollector.snapshot() } returns emptySystemMetrics
mockMvc
.perform(get("/system"))
.andExpect(status().isOk)
.andExpect(content().string(containsString("Repo size (GiB)")))
.andExpect(content().string(containsString("n/a")))
}
test("branch names with HTML metacharacters render escaped") {
val nasty = "feat/<script>alert('x')</script>"
every { repository.latestPerBranch() } returns listOf(successResult.copy(branch = nasty))
@@ -14,6 +14,14 @@ class UiViewsTest : FunSpec() {
UiFormats.duration(Duration.ofSeconds(3600 + 62)) shouldBe "1:01:02"
}
test("metric values format with two decimals and a dot, n/a when unavailable") {
UiFormats.metric(null) shouldBe "n/a"
UiFormats.metric(Double.NaN) shouldBe "n/a"
UiFormats.metric(0.0) shouldBe "0.00"
UiFormats.metric(1.234) shouldBe "1.23"
UiFormats.metric(31.288) shouldBe "31.29"
}
test("Gitea web links escape branch segments but keep slashes") {
val links =
GiteaWebLinks(GiteaConfig(baseUrl = "https://git.example.org/", owner = "acme", repo = "widget"))