mi/werkator
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

Nightly mutation testing with Mutation-CRAP (v1.28.0) #93

Merged
mi merged 2 commits from mutation-testing into main 2026-10-03 12:00:41 +02:00
Conversation 0 Commits 2 Files Changed 9
+205 -2
mi commented 2026-10-03 11:52:22 +02:00
Owner
Copy Link
Copy Source

The Problem

Werkator's tests say which lines run, not whether a wrong line would be noticed.
Mutation testing answers that: PIT changes the code in small ways and runs the tests against every change, the mutant.
werkcrappit turns PIT's results into a Mutation-CRAP value per method, C² · (1 − MS)³ + C,
so that complex and weakly tested methods stand out.

A run takes hours, so it cannot be part of the build on every push.

Non-Goals

  • A gate: no maxMutationCrap yet; the first reports decide where a limit belongs.
  • Unit tests for BuildExecutor: a follow-up PR (see below).
  • The Testcontainers smoke test under PIT: it needs a container engine per minion and kills nothing in Werkator's code.

The Scenarios

Scenario#93.01: a nightly build measures main

  • Given main has new commits in the last 24 hours
  • When it is 02:00 UTC
  • Then the build mutation runs PIT with every spec but the Testcontainers smoke test
  • And keeps PIT's report, mutation-crap.json/.md and the CodeCharta code city as artifacts
  • And reports its status as werkator/mutation
Verified by
  • ConfigLoaderUnitTest, the trigger of mutation
  • the local full run of 2026-10-03, see the Attachments

Scenario#93.02: only the mutation build gets the host's long limit

  • Given the host allows hours in executor.buildTimeout, for the mutation build
  • Then mutation gets that ceiling
  • And every other build stays at 10 minutes, so a hanging normal build is still stopped soon
Verified by

The Solution

  • build.gradle.kts: the Gradle plugins info.solidsoft.pitest 1.19.0 and de.javagil.werkcrappit 0.15.0, PIT 1.30.0, Kotest's PIT extension.
    werkcrappit 0.15.0 writes the progress at most once a minute, so the build log shows how far the run is.
    • targetTests is every spec but TestcontainersSmokeIntegrationTest.
      A first probe with only *UnitTest and *RestTest left BuildExecutor, GitService and GiteaClient uncovered:
      only their integration specs reach them, and they topped the list with a mutation score of 0.
    • timeoutConstInMillis = 20000: at PIT's default of 4 s, 323 of 787 mutants of the package metrics timed out, because each Kotest run in a fresh minion starts slowly.
    • addJUnitPlatformLauncher = false: otherwise the plugin pins junit-platform-launcher to 1.13.4, against JUnit 6 of Spring Boot, and test discovery fails in every Test task.
    • Incremental analysis with PIT's history in .pitest/, outside build/: the clean command keeps it, and the build worktree is per branch.
      After a full run, a night with few changes re-runs only the changed classes.
  • settings.gradle.kts: the Maven registry of git.javagil.de for the werkcrappit Gradle plugin.
  • .werkator.yml: the definition mutation, nightly at 02:00 on main, only after new commits; builds.default sets timeout: 10m, about 3 times the longest normal build of the last 30 days (220 s), mutation sets it back to empty, the host's ceiling.

Open Questions

  • The host's executor.buildTimeout on mih09 must allow a full run, at least 3h; without it, the first run is stopped and no history is written.
  • Threads: the definition runs PIT with 2 minions, as measured; more may shorten the first run if mih09's load allows it (-PpitestThreads=4).

Additional Changes

  • AGENTS.md names the mutation command; the writing-tests skill says why decision logic belongs where a *UnitTest reaches it.
  • .gitignore: /.pitest/.

Follow-up PRs

  • BuildExecutor: move its pure decisions (status description, timeout and status logic) into small classes with unit tests; its integration spec takes 29 s, paid per mutant.
  • Tests for the tags of HomeCardView.from and for HomeTexts.meta, the highest real gaps of the first report.
  • A maxMutationCrap gate, once the reports show a sensible limit.

Attachments

The local full runs of 2026-10-03

Both locally, 2 threads, werkcrappit with the policy timeouts-killed-v1 (a timeout counts as killed).

Probe: *UnitTest and *RestTest This PR: every spec but Testcontainers
Duration 1 h 40 min 2 h 24 min
Mutants 4368 4368
KILLED / SURVIVED / TIMED_OUT 2372 / 570 / 636 3046 / 928 / 79
NO_COVERAGE 790 315
Line coverage of mutated classes 82 % 94 %
Scored methods above 10 / 50 110 / 12 89 / 5

The probe's 636 timeouts counted as kills and made it look better than it was.
The highest values of this PR's run, which are real test gaps:

  • HomeCardView.from 224 (complexity 21, 17 of 22 mutants survive: no test checks its tags)
  • BuildExecutor.execute 74 (complexity 29)
  • HomeTexts.meta 66
  • DockerSocketLocator.locate, InitCommand.runningJarPath 56 (no test reaches them)

🤖 Generated with Claude Code

## The Problem Werkator's tests say which lines run, not whether a wrong line would be noticed. Mutation testing answers that: PIT changes the code in small ways and runs the tests against every change, the *mutant*. werkcrappit turns PIT's results into a Mutation-CRAP value per method, `C² · (1 − MS)³ + C`, so that complex and weakly tested methods stand out. A run takes hours, so it cannot be part of the build on every push. ## Non-Goals - A gate: no `maxMutationCrap` yet; the first reports decide where a limit belongs. - Unit tests for `BuildExecutor`: a follow-up PR (see below). - The Testcontainers smoke test under PIT: it needs a container engine per minion and kills nothing in Werkator's code. ## The Scenarios #### Scenario#93.01: a nightly build measures `main` - **Given** `main` has new commits in the last 24 hours - **When** it is 02:00 UTC - **Then** the build `mutation` runs PIT with every spec but the Testcontainers smoke test - **And** keeps PIT's report, `mutation-crap.json`/`.md` and the CodeCharta code city as artifacts - **And** reports its status as `werkator/mutation` ##### Verified by - [ConfigLoaderUnitTest](../../src/test/kotlin/de/hoennig/werkator/config/ConfigLoaderUnitTest.kt), the trigger of `mutation` - the local full run of 2026-10-03, see the Attachments #### Scenario#93.02: only the mutation build gets the host's long limit - **Given** the host allows hours in `executor.buildTimeout`, for the mutation build - **Then** `mutation` gets that ceiling - **And** every other build stays at 10 minutes, so a hanging normal build is still stopped soon ##### Verified by - [ConfigLoaderUnitTest](../../src/test/kotlin/de/hoennig/werkator/config/ConfigLoaderUnitTest.kt), test `Werkator's own mutation build takes the host's ceiling…` ## The Solution - `build.gradle.kts`: the Gradle plugins `info.solidsoft.pitest` 1.19.0 and `de.javagil.werkcrappit` 0.15.0, PIT 1.30.0, Kotest's PIT extension. werkcrappit 0.15.0 writes the progress at most once a minute, so the build log shows how far the run is. - `targetTests` is every spec but `TestcontainersSmokeIntegrationTest`. A first probe with only `*UnitTest` and `*RestTest` left `BuildExecutor`, `GitService` and `GiteaClient` uncovered: only their integration specs reach them, and they topped the list with a mutation score of 0. - `timeoutConstInMillis = 20000`: at PIT's default of 4 s, 323 of 787 mutants of the package `metrics` timed out, because each Kotest run in a fresh minion starts slowly. - `addJUnitPlatformLauncher = false`: otherwise the plugin pins `junit-platform-launcher` to 1.13.4, against JUnit 6 of Spring Boot, and test discovery fails in every `Test` task. - Incremental analysis with PIT's history in `.pitest/`, outside `build/`: the clean command keeps it, and the build worktree is per branch. After a full run, a night with few changes re-runs only the changed classes. - `settings.gradle.kts`: the Maven registry of git.javagil.de for the werkcrappit Gradle plugin. - `.werkator.yml`: the definition `mutation`, nightly at 02:00 on `main`, only after new commits; `builds.default` sets `timeout: 10m`, about 3 times the longest normal build of the last 30 days (220 s), `mutation` sets it back to empty, the host's ceiling. ## Open Questions - The host's `executor.buildTimeout` on mih09 must allow a full run, at least 3h; without it, the first run is stopped and no history is written. - Threads: the definition runs PIT with 2 minions, as measured; more may shorten the first run if mih09's load allows it (`-PpitestThreads=4`). ## Additional Changes - AGENTS.md names the mutation command; the writing-tests skill says why decision logic belongs where a `*UnitTest` reaches it. - `.gitignore`: `/.pitest/`. ## Follow-up PRs - `BuildExecutor`: move its pure decisions (status description, timeout and status logic) into small classes with unit tests; its integration spec takes 29 s, paid per mutant. - Tests for the tags of `HomeCardView.from` and for `HomeTexts.meta`, the highest real gaps of the first report. - A `maxMutationCrap` gate, once the reports show a sensible limit. ## Attachments ### The local full runs of 2026-10-03 Both locally, 2 threads, werkcrappit with the policy `timeouts-killed-v1` (a timeout counts as killed). | | Probe: `*UnitTest` and `*RestTest` | This PR: every spec but Testcontainers | |---|---|---| | Duration | 1 h 40 min | 2 h 24 min | | Mutants | 4368 | 4368 | | KILLED / SURVIVED / TIMED_OUT | 2372 / 570 / 636 | 3046 / 928 / 79 | | NO_COVERAGE | 790 | 315 | | Line coverage of mutated classes | 82 % | 94 % | | Scored methods above 10 / 50 | 110 / 12 | 89 / 5 | The probe's 636 timeouts counted as kills and made it look better than it was. The highest values of this PR's run, which are real test gaps: - `HomeCardView.from` 224 (complexity 21, 17 of 22 mutants survive: no test checks its tags) - `BuildExecutor.execute` 74 (complexity 29) - `HomeTexts.meta` 66 - `DockerSocketLocator.locate`, `InitCommand.runningJarPath` 56 (no test reaches them) 🤖 Generated with [Claude Code](https://claude.com/claude-code)
mi added 1 commit 2026-10-03 11:52:22 +02:00
Nightly mutation testing with Mutation-CRAP (v1.28.0) (PR#93)
werkator build succeeded after 02:29
Details
c5175c598f 
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
mi added 1 commit 2026-10-03 11:58:27 +02:00
Normal builds stop after 10 minutes, the mutation build after the host's ceiling (PR#93)
werkator build succeeded after 02:01
Details
af60afbcb8 
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
mi merged commit 9732053d03 into main 2026-10-03 12:00:41 +02:00
mi deleted branch mutation-testing 2026-10-03 12:00:42 +02:00
Sign in to join this conversation.
Reviewers
No Reviewers
Labels
No items
No labels
Milestone
No items
No Milestone
Projects
Clear projects
No projects
Assignees
Clear assignees
ai-reviewer claude-code mi
No Assignees
1 Participants
Notifications
Due Date
No due date set.
Dependencies

No dependencies set.

Reference: mi/werkator#93