werkator/docs/prs/2026-10-03-PR#93-nightly-mutation-testing.md

PR#93: Nightly Mutation Testing With Mutation-CRAP

This document describes the change of this pull request at the time it was made. Later pull requests may change the behaviour without updating it.

The Problem

Werkator's tests say which lines run, not whether a wrong line would be noticed. Mutation testing answers that: PIT changes the code in small ways and runs the tests against every change, the mutant. werkcrappit turns PIT's results into a Mutation-CRAP value per method, C² · (1 − MS)³ + C, so that complex and weakly tested methods stand out.

A run takes hours, so it cannot be part of the build on every push.

Non-Goals

• A gate: no maxMutationCrap yet; the first reports decide where a limit belongs.
• Unit tests for BuildExecutor: a follow-up PR (see below).
• The Testcontainers smoke test under PIT: it needs a container engine per minion and kills nothing in Werkator's code.

The Scenarios

Scenario#93.01: a nightly build measures main

• Given main has new commits in the last 24 hours
• When it is 02:00 UTC
• Then the build mutation runs PIT with every spec but the Testcontainers smoke test
• And keeps PIT's report, mutation-crap.json/.md and the CodeCharta code city as artifacts
• And reports its status as werkator/mutation

Verified by

• ConfigLoaderUnitTest, the trigger of mutation
• the local full run of 2026-10-03, see the Attachments

Scenario#93.02: only the mutation build gets the host's long limit

• Given the host allows hours in executor.buildTimeout, for the mutation build
• Then mutation gets that ceiling
• And every other build stays at 10 minutes, so a hanging normal build is still stopped soon

Verified by

• ConfigLoaderUnitTest, test Werkator's own mutation build takes the host's ceiling…

The Solution

• build.gradle.kts: the Gradle plugins info.solidsoft.pitest 1.19.0 and de.javagil.werkcrappit 0.15.0, PIT 1.30.0, Kotest's PIT extension. werkcrappit 0.15.0 writes the progress at most once a minute, so the build log shows how far the run is.
  • targetTests is every spec but TestcontainersSmokeIntegrationTest. A first probe with only *UnitTest and *RestTest left BuildExecutor, GitService and GiteaClient uncovered: only their integration specs reach them, and they topped the list with a mutation score of 0.
  • timeoutConstInMillis = 20000: at PIT's default of 4 s, 323 of 787 mutants of the package metrics timed out, because each Kotest run in a fresh minion starts slowly.
  • addJUnitPlatformLauncher = false: otherwise the plugin pins junit-platform-launcher to 1.13.4, against JUnit 6 of Spring Boot, and test discovery fails in every Test task.
  • Incremental analysis with PIT's history in .pitest/, outside build/: the clean command keeps it, and the build worktree is per branch. After a full run, a night with few changes re-runs only the changed classes.
• settings.gradle.kts: the Maven registry of git.javagil.de for the werkcrappit Gradle plugin.
• .werkator.yml: the definition mutation, nightly at 02:00 on main, only after new commits; builds.default sets timeout: 10m, about 3 times the longest normal build of the last 30 days (220 s), mutation sets it back to empty, the host's ceiling.

Open Questions

• The host's executor.buildTimeout on mih09 must allow a full run, at least 3h; without it, the first run is stopped and no history is written.
• Threads: the definition runs PIT with 2 minions, as measured; more may shorten the first run if mih09's load allows it (-PpitestThreads=4).

Additional Changes

• AGENTS.md names the mutation command; the writing-tests skill says why decision logic belongs where a *UnitTest reaches it.
• .gitignore: /.pitest/.

Follow-up PRs

• BuildExecutor: move its pure decisions (status description, timeout and status logic) into small classes with unit tests; its integration spec takes 29 s, paid per mutant.
• Tests for the tags of HomeCardView.from and for HomeTexts.meta, the highest real gaps of the first report.
• A maxMutationCrap gate, once the reports show a sensible limit.

Attachments

The local full runs of 2026-10-03

Both locally, 2 threads, werkcrappit with the policy timeouts-killed-v1 (a timeout counts as killed).

	Probe: *UnitTest and *RestTest	This PR: every spec but Testcontainers
Duration	1 h 40 min	2 h 24 min
Mutants	4368	4368
KILLED / SURVIVED / TIMED_OUT	2372 / 570 / 636	3046 / 928 / 79
NO_COVERAGE	790	315
Line coverage of mutated classes	82 %	94 %
Scored methods above 10 / 50	110 / 12	89 / 5

The probe's 636 timeouts counted as kills and made it look better than it was. The highest values of this PR's run, which are real test gaps:

• HomeCardView.from 224 (complexity 21, 17 of 22 mutants survive: no test checks its tags)
• BuildExecutor.execute 74 (complexity 29)
• HomeTexts.meta 66
• DockerSocketLocator.locate, InitCommand.runningJarPath 56 (no test reaches them)