miandmhoennig
9732053d03
## The Problem
Werkator's tests say which lines run, not whether a wrong line would be noticed.
Mutation testing answers that: PIT changes the code in small ways and runs the tests against every change, the *mutant*.
werkcrappit turns PIT's results into a Mutation-CRAP value per method, `C² · (1 − MS)³ + C`,
so that complex and weakly tested methods stand out.
A run takes hours, so it cannot be part of the build on every push.
## Non-Goals
- A gate: no `maxMutationCrap` yet; the first reports decide where a limit belongs.
- Unit tests for `BuildExecutor`: a follow-up PR (see below).
- The Testcontainers smoke test under PIT: it needs a container engine per minion and kills nothing in Werkator's code.
## The Scenarios
#### Scenario#93.01: a nightly build measures `main`
- **Given** `main` has new commits in the last 24 hours
- **When** it is 02:00 UTC
- **Then** the build `mutation` runs PIT with every spec but the Testcontainers smoke test
- **And** keeps PIT's report, `mutation-crap.json`/`.md` and the CodeCharta code city as artifacts
- **And** reports its status as `werkator/mutation`
##### Verified by
- [ConfigLoaderUnitTest](../../src/test/kotlin/de/hoennig/werkator/config/ConfigLoaderUnitTest.kt), the trigger of `mutation`
- the local full run of 2026-10-03, see the Attachments
#### Scenario#93.02: only the mutation build gets the host's long limit
- **Given** the host allows hours in `executor.buildTimeout`, for the mutation build
- **Then** `mutation` gets that ceiling
- **And** every other build stays at 10 minutes, so a hanging normal build is still stopped soon
##### Verified by
- [ConfigLoaderUnitTest](../../src/test/kotlin/de/hoennig/werkator/config/ConfigLoaderUnitTest.kt), test `Werkator's own mutation build takes the host's ceiling…`
## The Solution
- `build.gradle.kts`: the Gradle plugins `info.solidsoft.pitest` 1.19.0 and `de.javagil.werkcrappit` 0.15.0, PIT 1.30.0, Kotest's PIT extension.
werkcrappit 0.15.0 writes the progress at most once a minute, so the build log shows how far the run is.
- `targetTests` is every spec but `TestcontainersSmokeIntegrationTest`.
A first probe with only `*UnitTest` and `*RestTest` left `BuildExecutor`, `GitService` and `GiteaClient` uncovered:
only their integration specs reach them, and they topped the list with a mutation score of 0.
- `timeoutConstInMillis = 20000`: at PIT's default of 4 s, 323 of 787 mutants of the package `metrics` timed out, because each Kotest run in a fresh minion starts slowly.
- `addJUnitPlatformLauncher = false`: otherwise the plugin pins `junit-platform-launcher` to 1.13.4, against JUnit 6 of Spring Boot, and test discovery fails in every `Test` task.
- Incremental analysis with PIT's history in `.pitest/`, outside `build/`: the clean command keeps it, and the build worktree is per branch.
After a full run, a night with few changes re-runs only the changed classes.
- `settings.gradle.kts`: the Maven registry of git.javagil.de for the werkcrappit Gradle plugin.
- `.werkator.yml`: the definition `mutation`, nightly at 02:00 on `main`, only after new commits; `builds.default` sets `timeout: 10m`, about 3 times the longest normal build of the last 30 days (220 s), `mutation` sets it back to empty, the host's ceiling.
## Open Questions
- The host's `executor.buildTimeout` on mih09 must allow a full run, at least 3h; without it, the first run is stopped and no history is written.
- Threads: the definition runs PIT with 2 minions, as measured; more may shorten the first run if mih09's load allows it (`-PpitestThreads=4`).
## Additional Changes
- AGENTS.md names the mutation command; the writing-tests skill says why decision logic belongs where a `*UnitTest` reaches it.
- `.gitignore`: `/.pitest/`.
## Follow-up PRs
- `BuildExecutor`: move its pure decisions (status description, timeout and status logic) into small classes with unit tests; its integration spec takes 29 s, paid per mutant.
- Tests for the tags of `HomeCardView.from` and for `HomeTexts.meta`, the highest real gaps of the first report.
- A `maxMutationCrap` gate, once the reports show a sensible limit.
## Attachments
### The local full runs of 2026-10-03
Both locally, 2 threads, werkcrappit with the policy `timeouts-killed-v1` (a timeout counts as killed).
| | Probe: `*UnitTest` and `*RestTest` | This PR: every spec but Testcontainers |
|---|---|---|
| Duration | 1 h 40 min | 2 h 24 min |
| Mutants | 4368 | 4368 |
| KILLED / SURVIVED / TIMED_OUT | 2372 / 570 / 636 | 3046 / 928 / 79 |
| NO_COVERAGE | 790 | 315 |
| Line coverage of mutated classes | 82 % | 94 % |
| Scored methods above 10 / 50 | 110 / 12 | 89 / 5 |
The probe's 636 timeouts counted as kills and made it look better than it was.
The highest values of this PR's run, which are real test gaps:
- `HomeCardView.from` 224 (complexity 21, 17 of 22 mutants survive: no test checks its tags)
- `BuildExecutor.execute` 74 (complexity 29)
- `HomeTexts.meta` 66
- `DockerSocketLocator.locate`, `InitCommand.runningJarPath` 56 (no test reaches them)
🤖 Generated with [Claude Code](https://claude.com/claude-code)
---------
Co-authored-by: mhoennig <michael@hoennig.de>
Reviewed-on: #93
42 lines
652 B
Plaintext
42 lines
652 B
Plaintext
# Compiled class file
|
|
*.class
|
|
|
|
# Log file
|
|
*.log
|
|
|
|
# BlueJ files
|
|
*.ctxt
|
|
|
|
# Mobile Tools for Java (J2ME)
|
|
.mtj.tmp/
|
|
|
|
# Package Files #
|
|
*.jar
|
|
# ... but builds in fresh checkouts (e.g. Werkator worktrees) need the wrapper
|
|
!gradle/wrapper/gradle-wrapper.jar
|
|
*.war
|
|
*.nar
|
|
*.ear
|
|
*.zip
|
|
*.tar.gz
|
|
*.rar
|
|
|
|
# virtual machine crash logs, see http://www.java.com/en/download/help/error_hotspot.xml
|
|
hs_err_pid*
|
|
replay_pid*
|
|
|
|
# Kotlin Gradle plugin data, see https://kotlinlang.org/docs/whatsnew20.html#new-directory-for-kotlin-data-in-gradle-projects
|
|
.kotlin/
|
|
|
|
# Gradle
|
|
.gradle/
|
|
/build/
|
|
|
|
# Other
|
|
/.local/
|
|
/.env
|
|
/.env.*
|
|
|
|
# PIT's incremental history, per worktree
|
|
/.pitest/
|