werkcrappit/docs/metric.md

# The Metric Mutation-CRAP

Mutation-CRAP rates a JVM method by its complexity and by how many of PIT's mutants its tests kill.
It is an experiment: neither its superiority over other metrics nor any threshold is proven.
This document states the definition of metric version 1, `mutation-crap`, as every report names it in `definition`.
The specification's section 5 is the source; [ADR 0006](adr/0006-default-policy-timeouts-killed.md) adds the default policy.

## Formula

For a method `m` with a score:

```
C(m)  = the cyclomatic complexity JaCoCo Core computes from the original class file, at least 1
K(m)  = mutants the score policy counts as killed
E(m)  = mutants the score policy counts at all: killed, survived and not covered
MS(m) = K(m) / E(m)
mutationCrap(m) = C(m)² × (1 − MS(m))³ + C(m)
```

`MS` is a ratio from 0 to 1, not a percentage.
`C` is JaCoCo's complexity, `B − D + 1`; exception handlers add nothing, and JaCoCo's filters for generated code apply.
The report names the JaCoCo version in `definition.complexityProvider`,
as its filters, and so the values, can change between versions.

| C | killed | survived | not covered | MS | Mutation-CRAP |
| ---: | ---: | ---: | ---: | ---: | ---: |
| 1 | 0 | 1 | 0 | 0 | 2 |
| 10 | 0 | 10 | 0 | 0 | 110 |
| 10 | 5 | 5 | 0 | 0.5 | 22.5 |
| 10 | 8 | 2 | 0 | 0.8 | 10.8 |
| 10 | 10 | 0 | 0 | 1 | 10 |

The value lies between `C` and `C² + C`.
With every mutant killed it equals the complexity, so a method above a limit fails it even with perfect tests.
For a fixed `C`, more kills never raise the value.

The value is compared as an exact fraction:
the counts are integers, so `MutationCrap.exceeds` compares numerator and denominator, and a value exactly at a limit passes.
Rounding is only for display.

## Score Policies

A policy says how each of PIT's final statuses counts.
The report names it in `definition.scorePolicyId`; scores of different policies are not comparable.

| PIT status | `timeouts-killed-v1`, the default | `strict-v1` |
| --- | --- | --- |
| `KILLED` | killed | killed |
| `SURVIVED` | counted, not killed | counted, not killed |
| `NO_COVERAGE` | counted, not killed | counted, not killed |
| `TIMED_OUT` | killed | method `INCONCLUSIVE` |
| `MEMORY_ERROR`, `RUN_ERROR` | method `INCONCLUSIVE` | method `INCONCLUSIVE` |
| `NOT_STARTED`, `STARTED` at the end | method `INCONCLUSIVE` | method `INCONCLUSIVE` |
| `NON_VIABLE`, `EQUIVALENT` | not counted, shown apart | not counted, shown apart |
| any other status | method `INCONCLUSIVE` | method `INCONCLUSIVE` |

One inconclusive result takes the score of the whole method away, even beside many kills:
leaving out a problem result must never improve a score.
The raw counts per status stay in the report.
`timeouts-killed-v1` counts a timeout as a kill, as PIT's own score does; a mutant that turns a loop endless is a normal result.

## States Without a Score

Every method of the analysed classes has one state.
A state without a score has the value `null` in the JSON, never 0, 100 % or NaN, and machine-readable reasons.

| State | Meaning |
| --- | --- |
| `SCORED` | complexity and counted mutants are there |
| `NO_MUTATION_RESULTS` | the method is in the inventory, but PIT made no mutant for it |
| `NO_ELIGIBLE_MUTANTS` | PIT made mutants, but the policy counts none of them |
| `INCONCLUSIVE` | an ambiguous, incomplete, unknown or contradicting result; the gate stops with an analysis error |
| `COMPLEXITY_UNAVAILABLE` | mutants are there, but no complexity or no unique class file |
| `EXCLUDED` | left out on purpose |

The reasons:

- `NO_MUTATION_RESULTS`: `no-mutation-results`;
- `NO_ELIGIBLE_MUTANTS`: `no-eligible-mutants`;
- `INCONCLUSIVE`: `inconclusive-status:<status>`, `incomplete-status:<status>` or `unsupported-status:<status>`;
- `COMPLEXITY_UNAVAILABLE`: `complexity-unavailable`, `class-file-not-found`, `ambiguous-class-file`
  or `method-not-in-inventory`;
- `EXCLUDED`: `jacoco-filtered:<filter>`, or the matching `excludeClasses` or `excludeMethods` rule.

A known complexity stays in the report even without a score.
Abstract and native methods get no invented complexity.
Methods JaCoCo filters as generated, e.g. Kotlin's or Lombok's, are `EXCLUDED` with JaCoCo's filter as reason;
a failed match is never declared "generated".

## Limits of the Value

- **Evidence:** a ratio from one mutant is weaker than the same ratio from many; the metric does not model that.
  The report always shows complexity, mutation score and the number of mutants beside the value.
- **Conditions:** mutators, filters, test selection, bytecode and tool versions change the value
  without any change of the code or its tests; compare values only under the same conditions.
- **Not covered:** the score counts `NO_COVERAGE` as not killed, so it is lower than a score over covered mutants only.
- **No proof:** a low value is neither a proof of correctness nor a replacement for review.
- **Kotlin:** inline and suspend functions have limits of their own, see [ADR 0009](adr/0009-kotlin-jvm.md).

## Limits for the Gate

The gate `mutationCrapCheck` has no default limit; without one it only reports.
No threshold of this metric is proven, so the limit belongs to the project:
start a ratchet at today's highest value, rounded up, and only lower it.
The README's section "Choosing a Limit" gives the reasoning and the values in use,
e.g. 10 in hsadmin-ng, 15 in Werkator, which is written in Kotlin, and 9 in werkcrappit itself.

## Beside the Metric

The report adds per class the cohesion LCOM4 and the coupling Ce and Ca ([RFC 0002](rfcs/0002-cohesion-and-coupling.md)),
and the change risk, which combines them with Mutation-CRAP over the whole class ([ADR 0008](adr/0008-change-risk.md)).
The gate judges these only where a limit is set; it never judges the change risk.