werkcrappit/docs/metric.md

The Metric Mutation-CRAP

Mutation-CRAP rates a JVM method by its complexity and by how many of PIT's mutants its tests kill. It is an experiment: neither its superiority over other metrics nor any threshold is proven. This document states the definition of metric version 1, mutation-crap, as every report names it in definition. The specification's section 5 is the source; ADR 0006 adds the default policy.

Formula

For a method m with a score:

C(m)  = the cyclomatic complexity JaCoCo Core computes from the original class file, at least 1
K(m)  = mutants the score policy counts as killed
E(m)  = mutants the score policy counts at all: killed, survived and not covered
MS(m) = K(m) / E(m)
mutationCrap(m) = C(m)² × (1 − MS(m))³ + C(m)

MS is a ratio from 0 to 1, not a percentage. C is JaCoCo's complexity, B − D + 1; exception handlers add nothing, and JaCoCo's filters for generated code apply. The report names the JaCoCo version in definition.complexityProvider, as its filters, and so the values, can change between versions.

C	killed	survived	not covered	MS	Mutation-CRAP
1	0	1	0	0	2
10	0	10	0	0	110
10	5	5	0	0.5	22.5
10	8	2	0	0.8	10.8
10	10	0	0	1	10

The value lies between C and C² + C. With every mutant killed it equals the complexity, so a method above a limit fails it even with perfect tests. For a fixed C, more kills never raise the value.

The value is compared as an exact fraction: the counts are integers, so MutationCrap.exceeds compares numerator and denominator, and a value exactly at a limit passes. Rounding is only for display.

Score Policies

A policy says how each of PIT's final statuses counts. The report names it in definition.scorePolicyId; scores of different policies are not comparable.

PIT status	timeouts-killed-v1, the default	strict-v1
KILLED	killed	killed
SURVIVED	counted, not killed	counted, not killed
NO_COVERAGE	counted, not killed	counted, not killed
TIMED_OUT	killed	method INCONCLUSIVE
MEMORY_ERROR, RUN_ERROR	method INCONCLUSIVE	method INCONCLUSIVE
NOT_STARTED, STARTED at the end	method INCONCLUSIVE	method INCONCLUSIVE
NON_VIABLE, EQUIVALENT	not counted, shown apart	not counted, shown apart
any other status	method INCONCLUSIVE	method INCONCLUSIVE

One inconclusive result takes the score of the whole method away, even beside many kills: leaving out a problem result must never improve a score. The raw counts per status stay in the report. timeouts-killed-v1 counts a timeout as a kill, as PIT's own score does; a mutant that turns a loop endless is a normal result.

States Without a Score

Every method of the analysed classes has one state. A state without a score has the value null in the JSON, never 0, 100 % or NaN, and machine-readable reasons.

State	Meaning
SCORED	complexity and counted mutants are there
NO_MUTATION_RESULTS	the method is in the inventory, but PIT made no mutant for it
NO_ELIGIBLE_MUTANTS	PIT made mutants, but the policy counts none of them
INCONCLUSIVE	an ambiguous, incomplete, unknown or contradicting result; the gate stops with an analysis error
COMPLEXITY_UNAVAILABLE	mutants are there, but no complexity or no unique class file
EXCLUDED	left out on purpose

The reasons:

• NO_MUTATION_RESULTS: no-mutation-results;
• NO_ELIGIBLE_MUTANTS: no-eligible-mutants;
• INCONCLUSIVE: inconclusive-status:<status>, incomplete-status:<status> or unsupported-status:<status>;
• COMPLEXITY_UNAVAILABLE: complexity-unavailable, class-file-not-found, ambiguous-class-file or method-not-in-inventory;
• EXCLUDED: jacoco-filtered:<filter>, or the matching excludeClasses or excludeMethods rule.

A known complexity stays in the report even without a score. Abstract and native methods get no invented complexity. Methods JaCoCo filters as generated, e.g. Kotlin's or Lombok's, are EXCLUDED with JaCoCo's filter as reason; a failed match is never declared "generated".

Limits of the Value

• Evidence: a ratio from one mutant is weaker than the same ratio from many; the metric does not model that. The report always shows complexity, mutation score and the number of mutants beside the value.
• Conditions: mutators, filters, test selection, bytecode and tool versions change the value without any change of the code or its tests; compare values only under the same conditions.
• Not covered: the score counts NO_COVERAGE as not killed, so it is lower than a score over covered mutants only.
• No proof: a low value is neither a proof of correctness nor a replacement for review.
• Kotlin: inline and suspend functions have limits of their own, see ADR 0009.

Limits for the Gate

The gate mutationCrapCheck has no default limit; without one it only reports. No threshold of this metric is proven, so the limit belongs to the project: start a ratchet at today's highest value, rounded up, and only lower it. The README's section "Choosing a Limit" gives the reasoning and the values in use, e.g. 10 in hsadmin-ng, 15 in Werkator, which is written in Kotlin, and 9 in werkcrappit itself.

Beside the Metric

The report adds per class the cohesion LCOM4 and the coupling Ce and Ca (RFC 0002), and the change risk, which combines them with Mutation-CRAP over the whole class (ADR 0008). The gate judges these only where a limit is set; it never judges the change risk.