Werkcrappit

Mutation-CRAP metric for the JVM, as PIT and Gradle plugins.

The name is a pun: a "crap pit", the pit where the crap code lands, and at the same time the CRAP metric for PIT.

Werkcrappit finds methods that are complex and at the same time weakly tested. Per method it combines the cyclomatic complexity from JaCoCo with the mutation score from PIT (pitest):

mutationCrap(m) = C(m)² × (1 − MS(m))³ + C(m)

This is the classic CRAP metric, with killed mutants instead of covered lines.

• C(m) is the cyclomatic complexity of the method as JaCoCo Core computes it from the original class files.
• MS(m) is the mutation score from PIT: killed mutants divided by killed, survived and uncovered ones. By default a timed-out mutant counts as killed, as in PIT's own score (timeouts-killed-v1); strict-v1 makes such a method inconclusive instead.

A covered line proves only that a test ran it; a killed mutant proves that a test noticed a change. The metric is an experiment: neither its superiority nor any threshold is proven. docs/metric.md defines it exactly: the formula, the score policies, the states without a score and its limits.

Reports come as JSON and Markdown, with an optional build gate mutationCrapCheck. More output formats are to come.

Modules

• werkcrappit-pit — the PIT plugin: a result listener that PIT loads, joins the mutation results with the JaCoCo complexity and writes the reports.
• werkcrappit-gradle-plugin — the Gradle plugin de.javagil.werkcrappit: it wires the PIT plugin into the Gradle PIT plugin and adds the opt-in gate mutationCrapCheck and the code city export mutationCrapCodeCharta.

Status

Version 0.26.3, published to the Maven registry of git.javagil.de (see docs/publishing.md). The plan, with the PRs behind each step, is docs/werkcrappit.werkbaum.

• All eight steps of the specification are done: the metric for Java, its reports, the opt-in gate, Kotlin as the optional goal, the packaging test and the documentation. Step 7: Kotlin/JVM basis methods are tested with Kotlin 2.4.21, with limits for inline and suspend functions (ADR 0009). Step 8: docs/metric.md, the Groovy DSL test, the final report and the example reports are done.
• Beside the steps: the CodeCharta code city (RFC 0001), cohesion and coupling (RFC 0002), the change risk (ADR 0008), and the structure page and matrix (RFC 0003).
• In use: hsadmin-ng with the gate at 10, Werkator in Kotlin at 15, and werkcrappit itself every night at 9.
• The demo is werkprobe: its package werkprobe.shop shows each colour of the charts once, configured in the Groovy DSL. Its latest code city, structure and matrix come from its nightly build.
• werkcrappit's own latest code city shows a project that keeps every value low with ratchets.

Usage (preliminary)

plugins {
    java
    id("info.solidsoft.pitest") version "1.19.0"
    id("de.javagil.werkcrappit") version "0.26.3"
}

pitest {
    junit5PluginVersion = "1.2.3"
    targetClasses = setOf("com.example.*")
}

werkcrappit {
    maxMutationCrap = 30          // no default: without it the gate only reports
    allowEmpty = false            // true for a deliberately empty module
    excludeClasses = listOf("com.example.generated.**")
    excludeMethods = listOf("**#toString")
    scorePolicy = "timeouts-killed-v1"  // the default; "strict-v1" treats timeouts as inconclusive
}

./gradlew pitest writes mutation-crap.json and mutation-crap.md next to PIT's reports. While PIT runs, a line such as werkcrappit: 120 classes, 3104 of about 4368 mutants (71 %), 1:12:05 shows the progress, at most once a minute. The total is an estimate from the previous finished report; PIT does not tell a plugin how many mutants a run has. Beside Mutation-CRAP, the report lists cohesion and coupling per top-level class (RFC 0002): LCOM4, the groups of methods that neither call each other nor share a field, and Ce/Ca, the other classes of the analysed code a class depends on and is depended on by. It combines both with Mutation-CRAP into the change risk of a class (ADR 0008): Mutation-CRAP over all its scored methods, complexity and mutants summed, times 1 + (LCOM4 − 1) / 2 and 1 + Ce / 20. A class whose mutants survive stands out even when no single method does; the gate does not judge the change risk. ./gradlew mutationCrapCheck runs PIT if needed and then judges the report; it is not part of check. 30 above is an example, not a recommendation: no threshold of this metric is proven.

Configure output formats and plugin configuration on the pitest extension, not on the pitest task: the plugin derives the task's values from the extension.

The gate fails when a method's exact score is greater than maxMutationCrap (equal passes), and with an analysis error when the report is missing, unfinished or unreadable, the scope is incomplete, a method is INCONCLUSIVE or has no complexity, or no method could be scored. maxLcom4 and maxEfferentCoupling limit cohesion and outgoing coupling per class (RFC 0002), each optional; the gate judges only the dimensions with a limit, and a class exactly at a limit passes. Used as ratchets, each starts at today's highest value and only goes down.

Choosing a Limit

The limit belongs to the project; no threshold of this metric is proven. Start a ratchet at today's highest value, rounded up, and only lower it.

A higher limit does not only allow more complex methods: it also allows weaker tests for them. With every mutant killed, Mutation-CRAP equals the complexity; the limit bites only where mutants survive. For a method of complexity 10, the mutation score the limit demands drops fast:

Limit	Mutation score needed
10	100 %
16	61 %
30	42 %

A reason to raise the limit is a measurement that counts more than the source holds. Kotlin is one: the compiler adds branches for ?., ?:, when or default arguments, and JaCoCo does not filter all of them. A Kotlin project may set 16 where a Java project sets 10.

No reason to raise it is that nobody reads the code anymore, as in a repository changed by coding agents only. Without a human review, the tests are the only check left, so they need to be stronger, not weaker. The square of the complexity stands for the paths the tests must cover, not for how hard the code is to read; an agent changes these paths as well, often seeing only part of the method. And when an agent cannot fix a failure, a human has to read the code after all.

CodeCharta

./gradlew mutationCrapCodeCharta runs PIT if needed and writes build/reports/codecharta/mutation-crap.cc.json with an index.html beside it. The page links the file; open it in the CodeCharta visualization and map as below.

Served over http(s), the page leads with four buttons that open the city in CodeCharta, …/index.html?file=<URL of the file>&area=rloc&height=complexity&color=…, coloured by Mutation-CRAP, cohesion (lcom4), coupling (efferent_coupling) or change risk (change_risk). The viewer loads the file from that URL in the reader's browser.

Beside the city the export writes four CodeCharta scenarios, mutation-crap.ccscenario, cohesion.ccscenario, coupling.ccscenario and change-risk.ccscenario, one per view with its own colours; the Mutation-CRAP link hands them to the viewer with scenario=…, and the viewer at javagil.de adds them to its scenario list, any other imports them by hand. The link also sets the colour range in hard steps: red from 90 % of maxMutationCrap on, yellow from its half, green below; 5 and 9 without a limit. Red thus starts before the build fails, at the classes to look at next; codeCharta { colorRedFrom = 8 } sets another start, at most maxMutationCrap. With colorRedFrom set, yellow starts at its half, not at half of maxMutationCrap. That keeps yellow when maxMutationCrap is a ratchet far above the goal:

werkcrappit {
    maxMutationCrap = 650        // ratchet: just above today's highest value, lowered with every refactoring
    codeCharta {
        colorRedFrom = 12        // red from 12, yellow from 6
    }
}

codeCharta { colorYellowFrom = 8 } sets where yellow starts, at most where red starts; above it the export fails. That needs the server to answer with Access-Control-Allow-Origin for https://codecharta.com, and the file to be readable without a login.

CodeCharta	Metric	Meaning
Area	rloc	lines of the source file that are neither blank nor comment; annotations count, generated code does not
Height	complexity	sum of the JaCoCo complexities of the class's methods
Color	mutation_crap	the highest Mutation-CRAP of the class's scored methods
Color, cohesion	lcom4	groups of methods that neither call each other nor share a field; a package shows the median
Color, coupling	efferent_coupling, afferent_coupling	other classes of the analysed code a class depends on, or that depend on it; a package shows the sum
Color, change risk	change_risk	Mutation-CRAP over the whole class, raised by its LCOM4 and Ce; yellow from 30, red from 100
Edges	dependencies	the distinct things one class uses of another: the class as a type, each method and field

A building is a top-level class, its nested classes and lambdas included. Edges mark cycles (isCyclic) and dependencies against the levels of their folder (isPointingUpwards); for the released viewer at codecharta.com, which refuses these fields, set codeCharta { levelization = false }. Every class of the compiled main classes is one, also those PIT leaves out, without Mutation-CRAP. The page links https://javagil.de/codecharta/, a CodeCharta with a fix that shows such a class grey and keeps its height; codeCharta { viewerUrl = "…" } links another viewer. The original CodeCharta reads a missing mutation_crap as 0 and shows such a class green; the viewer rule Flatten where mutation_crap < 1 draws exactly these classes flat and grey, as every real value is at least 1. In the viewer it is in the Explorer: the ⋮ beside the search field, Flatten by metric…, mutation_crap, less than, 1. Uploaded by hand, the file is colored by complexity until you choose mutation_crap; the link of the page chooses it. werkcrappit { codeCharta { excludeClasses = listOf("**.generated.**") } } leaves classes out of the city, as excludeClasses does. Packages are districts; a chain of packages without classes of their own becomes one district, such as net.hostsharing.hsadminng. executable_lines (eLOC) counts only the lines with instructions, the logic of a class. A district shows the sum of rloc and complexity, but the median of mutation_crap: the typical class, while an outlier stays a red building. mutation_crap on a class is the maximum of its methods for the picture, not a new metric; a class without a scored method has none, never 0. The city names the analysed commit (git rev-parse HEAD, or codeCharta { commit = "…" }), and with a template every building links its source at that commit, at the method with the highest Mutation-CRAP:

werkcrappit {
    codeCharta {
        sourceLinkTemplate = "https://git.example.org/acme/project/src/commit/{commit}/{path}#L{line}"
    }
}

The export reads the class files of the PIT run once more for the lines and stops when they changed since; see ADR 0007.

Structure

Beside the city the export writes structure.html (RFC 0003): one level diagram per package, the highest level on top, with complexity as the height of a class, LCOM4 as its fill, and every cycle in red; a package cycle along the hierarchy, between a package and the classes of a package above it, is amber. Each diagram also shows the packages outside that use the package and those it uses, with the classes behind a package cycle. A click on a package opens its diagram, the breadcrumb leads back up. The page holds its data and its script and loads nothing, so it works from disk and on a CI server without CORS; index.html links it with the button "Structure".

matrix.html shows the same data as a dependency matrix: it starts with the top package's children, and a click opens a package in place, so several packages can be open at once. Rows and columns follow the levels, so every dependency along them lies right of the diagonal; a red cell is in a cycle. The open packages are in the URL, so a reload and a shared link keep them; "Expand all" is off above 500 classes.

Groovy DSL

The same in the Groovy DSL, as GroovyDslFunctionalTest runs it against the published plugins:

// settings.gradle
pluginManagement {
    repositories {
        maven { url = 'https://git.javagil.de/api/packages/mi/maven' }
        gradlePluginPortal()
    }
}

// build.gradle
plugins {
    id 'java'
    id 'info.solidsoft.pitest' version '1.19.0'
    id 'de.javagil.werkcrappit' version '0.26.8'
}

repositories {
    maven { url = 'https://git.javagil.de/api/packages/mi/maven' }
    mavenCentral()
}

pitest {
    junit5PluginVersion = '1.2.3'
    targetClasses = ['com.example.*']
}

werkcrappit {
    maxMutationCrap = 30            // no default: without it the gate only reports
    excludeMethods = ['**#toString']
    codeCharta {
        viewerUrl = 'https://javagil.de/codecharta/index.html'
    }
}

werkprobe uses this setup for its demo.

Without the Gradle plugin

The PIT plugin alone also works:

dependencies {
    pitest("de.javagil.werkcrappit:werkcrappit-pit:0.26.3")
}

pitest {
    outputFormats = setOf("HTML", "WERKCRAPPIT_MUTATION_CRAP")
    pluginConfiguration.putAll(mapOf("werkcrappit.excludeMethods" to "**#toString"))
}

pluginConfiguration is a MapProperty; the Kotlin DSL needs putAll or set, not =. The options are werkcrappit.analysisContext, werkcrappit.scorePolicy, and werkcrappit.excludeClasses and werkcrappit.excludeMethods, comma-separated; an unknown werkcrappit. option stops the run.

Patterns

** matches any characters, * any except a dot, ? one character except a dot. Class patterns match the binary name (demo.Outer$Inner), method patterns <class pattern>#<method name pattern> the JVM name (<init>). An excluded method stays in the report as EXCLUDED with the rule that matched; PIT's own results are not changed.

License

MIT, see LICENSE.

Runtime Dependencies

The PIT plugin werkcrappit-pit needs these libraries at run time; its POM names them, so Gradle fetches them, and its JAR contains no code of theirs:

Library	Version	License
JaCoCo Core org.jacoco:org.jacoco.core	0.8.15	EPL-2.0
ASM org.ow2.asm:asm, with asm-commons and asm-tree through JaCoCo	9.10.1	BSD-3-Clause

PIT itself is no dependency of the JAR: PIT loads the plugin into its own process. The Gradle plugin de.javagil.werkcrappit depends on werkcrappit-pit and on nothing else at run time; it expects the Gradle PIT plugin info.solidsoft.pitest (Apache-2.0) in the build that applies it.