werkregie/doc/prs/2026-10-14-PR#64-mutation-tests.md

PR#64: Mutation Tests in Each Build, in Full Each Night

WARNING: This document describes only the change applied in this PR. It may already be outdated once the next PR is merged. Historic PR documentation is not maintained along with new PRs; treat it as a snapshot.

Related Links

• PR#59, which brought the other required checks.
• hs.admin.ng, whose split this PR follows: a partial run in each branch build, a full run each night.

Problem

The shared workflow requires mutation tests for Kotlin: werkcrappit measures the Mutation-CRAP of each method, and the gate maxMutationCrap fails the build above the limit. werkregie has no mutation tests yet; they are the last required check it lacks. A nightly run alone finds a weakly tested method only after the merge; each branch build shall find it before.

Non-Goals

• No refactoring of the 67 methods above 15; they follow in PRs of their own.
• No public CodeCharta; it stays an artifact of the nightly build in Werkator.
• No thresholds of PIT itself, e.g. mutationThreshold; werkcrappit's gate decides.

Specification

Feature: mutation tests in each build, in full each night

Scenario#64.01: a branch build judges only the files the branch changes

… so that the mutation tests fit into each branch build.

• Given a branch that changes only the file Summaries.kt
• When Werkator builds the branch
• Then the build judges the Mutation-CRAP of the methods in Summaries.kt only, nested classes included

Verified by

• Manual: pitestPartial mutationCrapCheck with a changed Summaries.kt: 77 mutants, 43 s; PIT selected 8 classes: Summarizer, ChatSummarizer, Summaries and their nested ones.

Scenario#64.02: a weakly tested method in a changed file fails the build

… so that each change leaves the files it touches at the target.

• Given a branch that changes the file Summaries.kt, whose method Summaries.of has a Mutation-CRAP of 16.4
• When Werkator builds the branch
• Then the build fails and names Summaries.of, since 16.4 is above 15

Verified by

• Manual: the same run failed with Summaries.of … has 16.4282 > 15.

Scenario#64.03: a branch without changed code judges nothing

… so that a change of documents or configuration costs no mutation tests.

• Given a branch that changes only documents and the build configuration
• When Werkator builds the branch
• Then the build judges no Mutation-CRAP and passes

Verified by

• Manual: pitestPartial mutationCrapCheck on this branch: pitest and mutationCrapCheck skipped.
• Werkator's build of this PR: the same.

Scenario#64.04: main is judged in full each night

… so that the CodeCharta shows all of main and no method gets worse unnoticed.

• Given main, whose highest Mutation-CRAP is 1806
• When Werkator builds main at night
• Then the build judges all methods and keeps the CodeCharta of all classes

Verified by

• Manual: pitestFull mutationCrapCheck: 187 classes, 3099 mutants, 12 min 54 s with 2 threads; the highest value, 1806, is GiteaSource.pullRequestItem.
• Werkator's build mutation on main.

Solution

• dashboard/build.gradle.kts: the plugins info.solidsoft.pitest with PIT 1.30.0 and Kotest's PIT extension, and de.javagil.werkcrappit 0.26.12 from the Maven registry of git.javagil.de.
• pitestPartial mutates the classes of the Kotlin files the branch changes against the merge base with origin/main, uncommitted changes included; -PmutationBaseline names another ref. Per file it takes the top-level classes and the file class <Name>Kt, each with its nested classes. Without a changed file, pitest and mutationCrapCheck are skipped.
• pitestFull mutates all classes.
• dashboard/gradle/gradle-daemon-jvm.properties, written by updateDaemonJvm: Gradle runs on Temurin 25 from Foojay, since werkcrappit's plugin needs Java 21 and Werkator starts Gradle with the host's Java 17.
• The gate: 15 for pitestPartial, the target for Kotlin; 1806 for pitestFull, a ratchet from today's highest value, lowered with each refactoring.
• .werkator.yml:
  • the build default runs pitestPartial mutationCrapCheck after build; its timeout rises from 10 to 20 min;
  • the nightly build mutation runs pitestFull mutationCrapCodeCharta mutationCrapCheck on main at 04:00 UTC, with the status werkator/mutation.
• doc/agentic-workflow.md, the master copy: each branch build mutates the classes the branch changes, a nightly build all classes; only a branch build made too slow may leave them to the night, and the cause is fixed.
• doc/risk-analysis.md and AGENTS.md follow.
• Version 0.41.0.

PIT's incremental history, which hs.admin.ng uses, does not suit Kotest. Without any change, a second run took 6 min 17 s: it kept only the 520 surviving and timed-out mutants and ran the tests of all 1593 killed ones again. The killing test of a Kotest spec reads e.g. …JournalProposerTest.…JournalProposerTest/one entry per Taiga story…, which the history does not recognize as an unchanged test class.

Open Questions

• Shall the branch build use PIT's incremental history, as hs.admin.ng does? No, chosen by the human after the measurement: git diff against the merge base with main.
• Which limit holds in the branch build and each night? Chosen by the human: 15 for the classes a branch changes; each night a ratchet from today's highest value.

Follow-up PRs

• Refactorings and tests that bring each method of werkregie to a Mutation-CRAP of at most 15. Each of these PRs lowers the limit of the nightly build to the new highest value.