werkator/docs/prs/2026-10-03-PR#93-nightly-mutation-testing.md

# PR#93: Nightly Mutation Testing With Mutation-CRAP

> This document describes the change of this pull request at the time it was made.
> Later pull requests may change the behaviour without updating it.

## The Problem

Werkator's tests say which lines run, not whether a wrong line would be noticed.
Mutation testing answers that: PIT changes the code in small ways and runs the tests against every change, the *mutant*.
werkcrappit turns PIT's results into a Mutation-CRAP value per method, `C² · (1 − MS)³ + C`,
so that complex and weakly tested methods stand out.

A run takes hours, so it cannot be part of the build on every push.

## Non-Goals

- A gate: no `maxMutationCrap` yet; the first reports decide where a limit belongs.
- Unit tests for `BuildExecutor`: a follow-up PR (see below).
- The Testcontainers smoke test under PIT: it needs a container engine per minion and kills nothing in Werkator's code.

## The Scenarios

#### Scenario#93.01: a nightly build measures `main`

- **Given** `main` has new commits in the last 24 hours
- **When** it is 02:00 UTC
- **Then** the build `mutation` runs PIT with every spec but the Testcontainers smoke test
- **And** keeps PIT's report, `mutation-crap.json`/`.md` and the CodeCharta code city as artifacts
- **And** reports its status as `werkator/mutation`

##### Verified by

- [ConfigLoaderUnitTest](../../src/test/kotlin/de/hoennig/werkator/config/ConfigLoaderUnitTest.kt), the trigger of `mutation`
- the local full run of 2026-10-03, see the Attachments

#### Scenario#93.02: only the mutation build gets the host's long limit

- **Given** the host allows hours in `executor.buildTimeout`, for the mutation build
- **Then** `mutation` gets that ceiling
- **And** every other build stays at 10 minutes, so a hanging normal build is still stopped soon

##### Verified by

- [ConfigLoaderUnitTest](../../src/test/kotlin/de/hoennig/werkator/config/ConfigLoaderUnitTest.kt), test `Werkator's own mutation build takes the host's ceiling…`

## The Solution

- `build.gradle.kts`: the Gradle plugins `info.solidsoft.pitest` 1.19.0 and `de.javagil.werkcrappit` 0.15.0, PIT 1.30.0, Kotest's PIT extension.
  werkcrappit 0.15.0 writes the progress at most once a minute, so the build log shows how far the run is.
  - `targetTests` is every spec but `TestcontainersSmokeIntegrationTest`.
    A first probe with only `*UnitTest` and `*RestTest` left `BuildExecutor`, `GitService` and `GiteaClient` uncovered:
    only their integration specs reach them, and they topped the list with a mutation score of 0.
  - `timeoutConstInMillis = 20000`: at PIT's default of 4 s, 323 of 787 mutants of the package `metrics` timed out, because each Kotest run in a fresh minion starts slowly.
  - `addJUnitPlatformLauncher = false`: otherwise the plugin pins `junit-platform-launcher` to 1.13.4, against JUnit 6 of Spring Boot, and test discovery fails in every `Test` task.
  - Incremental analysis with PIT's history in `.pitest/`, outside `build/`: the clean command keeps it, and the build worktree is per branch.
    After a full run, a night with few changes re-runs only the changed classes.
- `settings.gradle.kts`: the Maven registry of git.javagil.de for the werkcrappit Gradle plugin.
- `.werkator.yml`: the definition `mutation`, nightly at 02:00 on `main`, only after new commits; `builds.default` sets `timeout: 10m`, about 3 times the longest normal build of the last 30 days (220 s), `mutation` sets it back to empty, the host's ceiling.

## Open Questions

- The host's `executor.buildTimeout` on mih09 must allow a full run, at least 3h; without it, the first run is stopped and no history is written.
- Threads: the definition runs PIT with 2 minions, as measured; more may shorten the first run if mih09's load allows it (`-PpitestThreads=4`).

## Additional Changes

- AGENTS.md names the mutation command; the writing-tests skill says why decision logic belongs where a `*UnitTest` reaches it.
- `.gitignore`: `/.pitest/`.

## Follow-up PRs

- `BuildExecutor`: move its pure decisions (status description, timeout and status logic) into small classes with unit tests; its integration spec takes 29 s, paid per mutant.
- Tests for the tags of `HomeCardView.from` and for `HomeTexts.meta`, the highest real gaps of the first report.
- A `maxMutationCrap` gate, once the reports show a sensible limit.

## Attachments

### The local full runs of 2026-10-03

Both locally, 2 threads, werkcrappit with the policy `timeouts-killed-v1` (a timeout counts as killed).

| | Probe: `*UnitTest` and `*RestTest` | This PR: every spec but Testcontainers |
|---|---|---|
| Duration | 1 h 40 min | 2 h 24 min |
| Mutants | 4368 | 4368 |
| KILLED / SURVIVED / TIMED_OUT | 2372 / 570 / 636 | 3046 / 928 / 79 |
| NO_COVERAGE | 790 | 315 |
| Line coverage of mutated classes | 82 % | 94 % |
| Scored methods above 10 / 50 | 110 / 12 | 89 / 5 |

The probe's 636 timeouts counted as kills and made it look better than it was.
The highest values of this PR's run, which are real test gaps:

- `HomeCardView.from` 224 (complexity 21, 17 of 22 mutants survive: no test checks its tags)
- `BuildExecutor.execute` 74 (complexity 29)
- `HomeTexts.meta` 66
- `DockerSocketLocator.locate`, `InitCommand.runningJarPath` 56 (no test reaches them)