3ca2e66d95
A nightly build an hour after `docker system prune -af` first rebuilt its image (233 s on vm4006) and ran into the build timeout. The cleanup now keeps images labelled org.werkator.build-inputs-sha256 on six nights and takes them in the night to Sunday, so the base image is at most a week old. A replaced build image loses its tag but keeps the label, so `docker image prune -f` follows. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Co-authored-by: mhoennig <michael@hoennig.de> Reviewed-on: #83
3.2 KiB
WARNING: This document describes only the change applied in this PR. It may already be outdated once the next PR is merged. Historic PR-documentation is not maintained along with new PRs — treat it as a snapshot, not as current documentation.
The Problem
On vm4006 the nightly master@release build of hs.hsadmin.ng timed out on 2026-09-29 at 30 minutes.
The cleanup timer runs docker system prune -af at 02:00 host time, which is 00:00 UTC.
It removes hsadmin-ng-build-env, because at that time no container uses the image.
The nightly build at 01:00 UTC then first rebuilt the image, in 233 seconds.
With pitestFull, the build itself takes about 28.5 minutes; together they did not fit.
Non-Goals
- No change to the build timeout; vm4006 raised its
executor.buildTimeoutseparately. - No keeping of other images, e.g. the ones Testcontainers pulls; they carry no label Werkator knows.
- No
--pullwhen Werkator builds an image.
The Scenarios
Feature: the nightly cleanup keeps Werkator's build images
Scenario#83.01: Build images survive six nights, the night to Sunday takes them
So that a nightly build does not spend its time on a rebuild, and the build environment is still at most a week old.
- Given a build image Werkator built, labelled
org.werkator.build-inputs-sha256, and no container using it - When the cleanup runs on Monday to Saturday, then the image stays
- When the cleanup runs on Sunday, then it is removed like any other unused image
- And unused images without the label are removed every night, as before
Verified by
- SystemdServiceFilesTest — the command runs with a faked
dateanddockerfor every weekday - By hand: the generated
ExecStartline in a real systemd user unit ran the filtered prune on a Tuesday, so systemd's$$and%%escapes arrive as intended.
Scenario#83.02: A replaced build image does not pile up
So that a changed Dockerfile does not leave a 4 GB image behind every time.
- Given a build image that was rebuilt, so its old image lost its tag but kept the label
- When the cleanup runs, then the untagged image is removed
Verified by
- SystemdServiceFilesTest —
docker image prune -ffollows the main step - By hand with Docker 27.3.1:
docker image prune -af --filter label!=…keeps a labelled untagged image, anddocker image prune -fremoves it.
The Solution
The prune service runs one sh -c line: on Sunday (date +%u is 7) the plain docker system prune -af, on other days the same with --filter label!=org.werkator.build-inputs-sha256.
The label is the one DockerBuildRunner already sets to detect stale images, so no new label is needed.
A second step, docker image prune -f, removes dangling images, which the label filter would keep.
The anonymous-volume step (PR#80) stays unchanged after these two.
Once a week is a trade-off: docker build runs without --pull, so a kept image keeps its base image and packages until it is removed.
Additional Changes
docs/deployment.mddescribes the new behaviour.- Version 1.23.2 with its release note.