miandmhoennig
cb24f0fb01
A timed-out mutant counts as killed by default, as in PIT's own mutation score; `strict-v1` stays unchanged and selectable (`werkcrappit { scorePolicy = "strict-v1" }` / `werkcrappit.scorePolicy`). MEMORY_ERROR and RUN_ERROR stay inconclusive.
Why: mutants that turn a loop endless cannot be avoided; under strict-v1 every real project has methods without score and a strict gate that never passes. The hsadmin-ng probe run had 4 such methods.
Deviation from the specification's starting default documented in ADR 0006; every report names its policy.
CI on werkator.javagil.de: green (81 s).
PR-doc: `docs/prs/2026-10-01-PR#9-timeouts-killed-policy.md`
🤖 Generated with [Claude Code](https://claude.com/claude-code)
Co-authored-by: mhoennig <michael@hoennig.de>
Reviewed-on: #9
38 lines
2.0 KiB
Markdown
38 lines
2.0 KiB
Markdown
# ADR 0006: Default Status Policy timeouts-killed-v1
|
|
|
|
- Status: accepted
|
|
- Date: 2026-10-01
|
|
- PR: #9
|
|
|
|
## Context
|
|
|
|
The specification starts with the status policy `strict-v1`, under which one `TIMED_OUT` result makes a method `INCONCLUSIVE` (5.2).
|
|
It allows a later timeout policy under an identifier of its own and forbids changing `strict-v1` unnoticed.
|
|
|
|
The first probe run on hsadmin-ng showed that timeouts are a normal result of mutation testing: a mutant that turns a loop endless makes the tests not finish.
|
|
Under `strict-v1` every such method had no score, and a strict gate failed with an analysis error on every real run.
|
|
Nobody can make these mutants go away.
|
|
|
|
## Decision
|
|
|
|
A second policy `timeouts-killed-v1` is `strict-v1` except that `TIMED_OUT` counts as killed — in the numerator and the denominator.
|
|
It becomes the default; `strict-v1` stays unchanged and selectable.
|
|
|
|
- Gradle plugin: `werkcrappit { scorePolicy = "strict-v1" }`.
|
|
- PIT plugin alone: `pluginConfiguration` key `werkcrappit.scorePolicy`.
|
|
- An unknown policy stops the run.
|
|
|
|
Why killed and not neutral: a timeout shows that the tests reacted to the change, and PIT's own mutation score counts it as detected; werkcrappit's mutation score then equals PIT's for the same counts.
|
|
Why `MEMORY_ERROR` and `RUN_ERROR` stay inconclusive: they may as well come from a broken setup, and counting them as kills would turn a broken setup into good test strength.
|
|
|
|
## Deviation From the Specification
|
|
|
|
The default changes from `strict-v1` to `timeouts-killed-v1`, which section 2.3 calls a starting decision, not a fixed one.
|
|
`strict-v1` itself, the formula and the non-numeric states are unchanged.
|
|
Every report names its policy in `definition.scorePolicyId`, so reports of different policies are never mistaken for each other.
|
|
|
|
## Consequences
|
|
|
|
- A timeout from a genuinely hanging setup counts as a kill; the raw count `TIMED_OUT` stays visible per method.
|
|
- Reports made before 0.3.0 used `strict-v1`; comparing scores across that boundary needs the same policy.
|