283 lines
17 KiB
Markdown
283 lines
17 KiB
Markdown
# Werkcrappit
|
||
|
||
Mutation-CRAP metric for the JVM, as PIT and Gradle plugins.
|
||
|
||
The name is a pun: a "crap pit", the pit where the crap code lands, and at the same time the CRAP metric for PIT.
|
||
|
||
Werkcrappit finds methods that are complex *and* at the same time weakly tested.
|
||
Per method it combines the cyclomatic complexity from JaCoCo with the mutation score from PIT (pitest):
|
||
|
||
```
|
||
mutationCrap(m) = C(m)² × (1 − MS(m))³ + C(m)
|
||
```
|
||
|
||
This is the classic CRAP metric, with killed mutants instead of covered lines.
|
||
|
||
- `C(m)` is the cyclomatic complexity of the method as JaCoCo Core computes it from the original class files.
|
||
- `MS(m)` is the mutation score from PIT: killed mutants divided by killed, survived and uncovered ones.
|
||
By default a timed-out mutant counts as killed, as in PIT's own score (`timeouts-killed-v1`); `strict-v1` makes such a method inconclusive instead.
|
||
|
||
A covered line proves only that a test ran it; a killed mutant proves that a test noticed a change.
|
||
The metric is an experiment: neither its superiority nor any threshold is proven.
|
||
[docs/metric.md](docs/metric.md) defines it exactly: the formula, the score policies, the states without a score and its limits.
|
||
|
||
Reports come as JSON and Markdown, with an optional build gate `mutationCrapCheck`.
|
||
More output formats are to come.
|
||
|
||
## Modules
|
||
|
||
- `werkcrappit-pit` — the PIT plugin: a result listener that PIT loads, joins the mutation results with the JaCoCo complexity and writes the reports.
|
||
- `werkcrappit-gradle-plugin` — the Gradle plugin `de.javagil.werkcrappit`: it wires the PIT plugin into the Gradle PIT plugin
|
||
and adds the opt-in gate `mutationCrapCheck` and the code city export `mutationCrapCodeCharta`.
|
||
|
||
## Status
|
||
|
||
Version 0.26.3, published to the Maven registry of git.javagil.de (see [docs/publishing.md](docs/publishing.md)).
|
||
The plan, with the PRs behind each step, is [docs/werkcrappit.werkbaum](docs/werkcrappit.werkbaum).
|
||
|
||
- All eight steps of the specification are done: the metric for Java, its reports, the opt-in gate,
|
||
Kotlin as the optional goal, the packaging test and the documentation.
|
||
Step 7: Kotlin/JVM basis methods are tested with Kotlin 2.4.21, with limits for inline and suspend functions
|
||
([ADR 0009](docs/adr/0009-kotlin-jvm.md)).
|
||
Step 8: [docs/metric.md](docs/metric.md), the Groovy DSL test, the [final report](docs/final-report.md)
|
||
and the [example reports](docs/examples/) are done.
|
||
- Beside the steps: the CodeCharta code city ([RFC 0001](docs/rfcs/0001-codecharta-export.md)),
|
||
cohesion and coupling ([RFC 0002](docs/rfcs/0002-cohesion-and-coupling.md)),
|
||
the change risk ([ADR 0008](docs/adr/0008-change-risk.md)),
|
||
and the structure page and matrix ([RFC 0003](docs/rfcs/0003-structure-view.md)).
|
||
- In use: hsadmin-ng with the gate at 10, Werkator in Kotlin at 15, and werkcrappit itself every night at 9.
|
||
- The demo is [werkprobe](https://git.javagil.de/mi/werkprobe):
|
||
its package `werkprobe.shop` shows each colour of the charts once, configured in the Groovy DSL.
|
||
Its latest [code city][demo-city], [structure][demo-structure] and [matrix][demo-matrix] come from its nightly build.
|
||
- werkcrappit's own latest [code city][own] shows a project that keeps every value low with ratchets.
|
||
|
||
[demo-city]: https://werkator.javagil.de/repos/werkprobe/branches/main_mutation/build/reports/codecharta/index.html
|
||
[demo-structure]: https://werkator.javagil.de/repos/werkprobe/branches/main_mutation/build/reports/codecharta/structure.html
|
||
[demo-matrix]: https://werkator.javagil.de/repos/werkprobe/branches/main_mutation/build/reports/codecharta/matrix.html
|
||
[own]: https://werkator.javagil.de/repos/werkcrappit/branches/main_mutation/werkcrappit-pit/build/reports/codecharta/index.html
|
||
|
||
## Usage (preliminary)
|
||
|
||
```kotlin
|
||
plugins {
|
||
java
|
||
id("info.solidsoft.pitest") version "1.19.0"
|
||
id("de.javagil.werkcrappit") version "0.26.3"
|
||
}
|
||
|
||
pitest {
|
||
junit5PluginVersion = "1.2.3"
|
||
targetClasses = setOf("com.example.*")
|
||
}
|
||
|
||
werkcrappit {
|
||
maxMutationCrap = 30 // no default: without it the gate only reports
|
||
allowEmpty = false // true for a deliberately empty module
|
||
excludeClasses = listOf("com.example.generated.**")
|
||
excludeMethods = listOf("**#toString")
|
||
scorePolicy = "timeouts-killed-v1" // the default; "strict-v1" treats timeouts as inconclusive
|
||
}
|
||
```
|
||
|
||
`./gradlew pitest` writes `mutation-crap.json` and `mutation-crap.md` next to PIT's reports.
|
||
While PIT runs, a line such as `werkcrappit: 120 classes, 3104 of about 4368 mutants (71 %), 1:12:05` shows the progress, at most once a minute.
|
||
The total is an estimate from the previous finished report; PIT does not tell a plugin how many mutants a run has.
|
||
Beside Mutation-CRAP, the report lists cohesion and coupling per top-level class (RFC 0002): LCOM4, the groups of methods that neither call each other nor share a field, and Ce/Ca, the other classes of the analysed code a class depends on and is depended on by.
|
||
It combines both with Mutation-CRAP into the change risk of a class (ADR 0008): Mutation-CRAP over all its scored methods, complexity and mutants summed, times `1 + (LCOM4 − 1) / 2` and `1 + Ce / 20`.
|
||
A class whose mutants survive stands out even when no single method does; the gate does not judge the change risk.
|
||
`./gradlew mutationCrapCheck` runs PIT if needed and then judges the report; it is not part of `check`.
|
||
30 above is an example, not a recommendation: no threshold of this metric is proven.
|
||
|
||
Configure output formats and plugin configuration on the `pitest` extension, not on the `pitest` task: the plugin derives the task's values from the extension.
|
||
|
||
The gate fails when a method's exact score is greater than `maxMutationCrap` (equal passes), and with an analysis error when the report is missing, unfinished or unreadable, the scope is incomplete, a method is `INCONCLUSIVE` or has no complexity, or no method could be scored.
|
||
`maxLcom4` and `maxEfferentCoupling` limit cohesion and outgoing coupling per class (RFC 0002), each optional; the gate judges only the dimensions with a limit, and a class exactly at a limit passes.
|
||
Used as ratchets, each starts at today's highest value and only goes down.
|
||
|
||
### Choosing a Limit
|
||
|
||
The limit belongs to the project; no threshold of this metric is proven.
|
||
Start a ratchet at today's highest value, rounded up, and only lower it.
|
||
|
||
A higher limit does not only allow more complex methods: it also allows weaker tests for them.
|
||
With every mutant killed, Mutation-CRAP equals the complexity; the limit bites only where mutants survive.
|
||
For a method of complexity 10, the mutation score the limit demands drops fast:
|
||
|
||
| Limit | Mutation score needed |
|
||
| ----: | --------------------: |
|
||
| 10 | 100 % |
|
||
| 16 | 61 % |
|
||
| 30 | 42 % |
|
||
|
||
A reason to raise the limit is a measurement that counts more than the source holds.
|
||
Kotlin is one: the compiler adds branches for `?.`, `?:`, `when` or default arguments, and JaCoCo does not filter all of them.
|
||
A Kotlin project may set 16 where a Java project sets 10.
|
||
|
||
No reason to raise it is that nobody reads the code anymore, as in a repository changed by coding agents only.
|
||
Without a human review, the tests are the only check left, so they need to be stronger, not weaker.
|
||
The square of the complexity stands for the paths the tests must cover, not for how hard the code is to read; an agent changes these paths as well, often seeing only part of the method.
|
||
And when an agent cannot fix a failure, a human has to read the code after all.
|
||
|
||
### CodeCharta
|
||
|
||
`./gradlew mutationCrapCodeCharta` runs PIT if needed and writes `build/reports/codecharta/mutation-crap.cc.json` with an `index.html` beside it.
|
||
The page links the file; open it in the [CodeCharta](https://codecharta.com/) visualization and map as below.
|
||
|
||
Served over http(s), the page leads with four buttons that open the city in CodeCharta,
|
||
`…/index.html?file=<URL of the file>&area=rloc&height=complexity&color=…`,
|
||
coloured by Mutation-CRAP, cohesion (`lcom4`), coupling (`efferent_coupling`) or change risk (`change_risk`).
|
||
The viewer loads the file from that URL in the reader's browser.
|
||
|
||
Beside the city the export writes four CodeCharta scenarios, `mutation-crap.ccscenario`, `cohesion.ccscenario`, `coupling.ccscenario` and `change-risk.ccscenario`, one per view with its own colours; the Mutation-CRAP link hands them to the viewer with `scenario=…`, and the viewer at javagil.de adds them to its scenario list, any other imports them by hand.
|
||
The link also sets the colour range in hard steps: red from 90 % of `maxMutationCrap` on, yellow from its half, green below; 5 and 9 without a limit.
|
||
Red thus starts before the build fails, at the classes to look at next; `codeCharta { colorRedFrom = 8 }` sets another start, at most `maxMutationCrap`.
|
||
With `colorRedFrom` set, yellow starts at its half, not at half of `maxMutationCrap`.
|
||
That keeps yellow when `maxMutationCrap` is a ratchet far above the goal:
|
||
|
||
```kotlin
|
||
werkcrappit {
|
||
maxMutationCrap = 650 // ratchet: just above today's highest value, lowered with every refactoring
|
||
codeCharta {
|
||
colorRedFrom = 12 // red from 12, yellow from 6
|
||
}
|
||
}
|
||
```
|
||
|
||
`codeCharta { colorYellowFrom = 8 }` sets where yellow starts, at most where red starts; above it the export fails.
|
||
That needs the server to answer with `Access-Control-Allow-Origin` for `https://codecharta.com`, and the file to be readable without a login.
|
||
|
||
| CodeCharta | Metric | Meaning |
|
||
| ------------------ | ---------------------------------------- | ------------------------------------------------------------------------------------------------------- |
|
||
| Area | `rloc` | lines of the source file that are neither blank nor comment; annotations count, generated code does not |
|
||
| Height | `complexity` | sum of the JaCoCo complexities of the class's methods |
|
||
| Color | `mutation_crap` | the highest Mutation-CRAP of the class's scored methods |
|
||
| Color, cohesion | `lcom4` | groups of methods that neither call each other nor share a field; a package shows the median |
|
||
| Color, coupling | `efferent_coupling`, `afferent_coupling` | other classes of the analysed code a class depends on, or that depend on it; a package shows the sum |
|
||
| Color, change risk | `change_risk` | Mutation-CRAP over the whole class, raised by its LCOM4 and Ce; yellow from 30, red from 100 |
|
||
| Edges | `dependencies` | the distinct things one class uses of another: the class as a type, each method and field |
|
||
|
||
A building is a top-level class, its nested classes and lambdas included.
|
||
Edges mark cycles (`isCyclic`) and dependencies against the levels of their folder (`isPointingUpwards`); for the released viewer at codecharta.com, which refuses these fields, set `codeCharta { levelization = false }`.
|
||
Every class of the compiled main classes is one, also those PIT leaves out, without Mutation-CRAP.
|
||
The page links `https://javagil.de/codecharta/`, a CodeCharta with a fix that shows such a class grey and keeps its height; `codeCharta { viewerUrl = "…" }` links another viewer.
|
||
The original CodeCharta reads a missing `mutation_crap` as 0 and shows such a class green; the viewer rule *Flatten* where `mutation_crap` < 1 draws exactly these classes flat and grey, as every real value is at least 1.
|
||
In the viewer it is in the Explorer: the ⋮ beside the search field, *Flatten by metric…*, `mutation_crap`, *less than*, 1.
|
||
Uploaded by hand, the file is colored by `complexity` until you choose `mutation_crap`; the link of the page chooses it.
|
||
`werkcrappit { codeCharta { excludeClasses = listOf("**.generated.**") } }` leaves classes out of the city, as `excludeClasses` does.
|
||
Packages are districts; a chain of packages without classes of their own becomes one district, such as `net.hostsharing.hsadminng`.
|
||
`executable_lines` (eLOC) counts only the lines with instructions, the logic of a class.
|
||
A district shows the sum of `rloc` and `complexity`, but the median of `mutation_crap`: the typical class, while an outlier stays a red building.
|
||
`mutation_crap` on a class is the maximum of its methods for the picture, not a new metric; a class without a scored method has none, never 0.
|
||
The city names the analysed commit (`git rev-parse HEAD`, or `codeCharta { commit = "…" }`), and with a template every building links its source at that commit, at the method with the highest Mutation-CRAP:
|
||
|
||
```kotlin
|
||
werkcrappit {
|
||
codeCharta {
|
||
sourceLinkTemplate = "https://git.example.org/acme/project/src/commit/{commit}/{path}#L{line}"
|
||
}
|
||
}
|
||
```
|
||
|
||
The export reads the class files of the PIT run once more for the lines and stops when they changed since; see [ADR 0007](docs/adr/0007-codecharta-export.md).
|
||
|
||
### Structure
|
||
|
||
Beside the city the export writes `structure.html` ([RFC 0003](docs/rfcs/0003-structure-view.md)):
|
||
one level diagram per package, the highest level on top,
|
||
with complexity as the height of a class, LCOM4 as its fill, and every cycle in red;
|
||
a package cycle along the hierarchy, between a package and the classes of a package above it, is amber.
|
||
Each diagram also shows the packages outside that use the package and those it uses, with the classes behind a package cycle.
|
||
A click on a package opens its diagram, the breadcrumb leads back up.
|
||
The page holds its data and its script and loads nothing, so it works from disk and on a CI server without CORS;
|
||
`index.html` links it with the button "Structure".
|
||
|
||
`matrix.html` shows the same data as a dependency matrix: it starts with the top package's children,
|
||
and a click opens a package in place, so several packages can be open at once.
|
||
Rows and columns follow the levels, so every dependency along them lies right of the diagonal; a red cell is in a cycle.
|
||
The open packages are in the URL, so a reload and a shared link keep them; "Expand all" is off above 500 classes.
|
||
|
||
### Groovy DSL
|
||
|
||
The same in the Groovy DSL, as `GroovyDslFunctionalTest` runs it against the published plugins:
|
||
|
||
```groovy
|
||
// settings.gradle
|
||
pluginManagement {
|
||
repositories {
|
||
maven { url = 'https://git.javagil.de/api/packages/mi/maven' }
|
||
gradlePluginPortal()
|
||
}
|
||
}
|
||
```
|
||
|
||
```groovy
|
||
// build.gradle
|
||
plugins {
|
||
id 'java'
|
||
id 'info.solidsoft.pitest' version '1.19.0'
|
||
id 'de.javagil.werkcrappit' version '0.26.8'
|
||
}
|
||
|
||
repositories {
|
||
maven { url = 'https://git.javagil.de/api/packages/mi/maven' }
|
||
mavenCentral()
|
||
}
|
||
|
||
pitest {
|
||
junit5PluginVersion = '1.2.3'
|
||
targetClasses = ['com.example.*']
|
||
}
|
||
|
||
werkcrappit {
|
||
maxMutationCrap = 30 // no default: without it the gate only reports
|
||
excludeMethods = ['**#toString']
|
||
codeCharta {
|
||
viewerUrl = 'https://javagil.de/codecharta/index.html'
|
||
}
|
||
}
|
||
```
|
||
|
||
[werkprobe](https://git.javagil.de/mi/werkprobe) uses this setup for its demo.
|
||
|
||
### Without the Gradle plugin
|
||
|
||
The PIT plugin alone also works:
|
||
|
||
```kotlin
|
||
dependencies {
|
||
pitest("de.javagil.werkcrappit:werkcrappit-pit:0.26.3")
|
||
}
|
||
|
||
pitest {
|
||
outputFormats = setOf("HTML", "WERKCRAPPIT_MUTATION_CRAP")
|
||
pluginConfiguration.putAll(mapOf("werkcrappit.excludeMethods" to "**#toString"))
|
||
}
|
||
```
|
||
|
||
`pluginConfiguration` is a `MapProperty`; the Kotlin DSL needs `putAll` or `set`, not `=`.
|
||
The options are `werkcrappit.analysisContext`, `werkcrappit.scorePolicy`, and `werkcrappit.excludeClasses` and `werkcrappit.excludeMethods`, comma-separated; an unknown `werkcrappit.` option stops the run.
|
||
|
||
### Patterns
|
||
|
||
`**` matches any characters, `*` any except a dot, `?` one character except a dot.
|
||
Class patterns match the binary name (`demo.Outer$Inner`), method patterns `<class pattern>#<method name pattern>` the JVM name (`<init>`).
|
||
An excluded method stays in the report as `EXCLUDED` with the rule that matched; PIT's own results are not changed.
|
||
|
||
## License
|
||
|
||
MIT, see [LICENSE](LICENSE).
|
||
|
||
### Runtime Dependencies
|
||
|
||
The PIT plugin `werkcrappit-pit` needs these libraries at run time;
|
||
its POM names them, so Gradle fetches them, and its JAR contains no code of theirs:
|
||
|
||
| Library | Version | License |
|
||
| --- | --- | --- |
|
||
| JaCoCo Core `org.jacoco:org.jacoco.core` | 0.8.15 | EPL-2.0 |
|
||
| ASM `org.ow2.asm:asm`, with `asm-commons` and `asm-tree` through JaCoCo | 9.10.1 | BSD-3-Clause |
|
||
|
||
PIT itself is no dependency of the JAR: PIT loads the plugin into its own process.
|
||
The Gradle plugin `de.javagil.werkcrappit` depends on `werkcrappit-pit` and on nothing else at run time;
|
||
it expects the Gradle PIT plugin `info.solidsoft.pitest` (Apache-2.0) in the build that applies it.
|