Mutation tests in each build, in full each night #64
No Assignees
Notifications
Due Date
No due date set.
Dependencies
No dependencies set.
Reference: mi/werkregie#64
Reference in New Issue
Block a user
Mutation tests with PIT and werkcrappit:
pitestPartialin each build,pitestFullin a nightly build, as in hs.admin.ng.See the PR document under
doc/prs/.🤖 Generated with Claude Code
🟠 werkrevisor ORANGE for
09f59aa, superseded by the result for460e0fa🟠 werkrevisor: ORANGE / FINDINGS
Change kind:
mixed.Reasons
Classifier
5 of 10 units passed all questions (1 PR document(s)).
Usage: model `openrouter/minimax/minimax-m3`, LLM done, tokens counted before the call 18319, calls 1, tokens in 17654 / out 24845 (reasoning included in out), duration 264.0 s, cost 0.0351 USD (reported), classifier `jev-1.13.0` calls 10, tokens in 8289, cost 0.0003 USD, build file judge `jev-1.13.0` calls 4, tokens in 4697, cost 0.0002 USD, phases: path_judge 0.2 s, classifier 0.6 s, context 1.5 s, model_review 261.1 s, config: defaults + host + repository (.werkrevisor.toml at main3 answers ignored because of a confidence below [classifier] min_confidence.
2768a9a).Reviewed head `09f59aabf3a55e7fd4c40b41117708f3b8407614` against base `2768a9af8791af4969392ee3347894f6b653517d`, werkrevisor 0.41.2.
🟡 werkrevisor YELLOW for
460e0fa, superseded by the result ford00b31e🟡 werkrevisor: YELLOW / HUMAN_REVIEW
Change kind:
mixed.Reasons
Findings
Daemon JVM Setup Ineffective at
.werkator.yml:22-23, severity unknown, counted as medium, hint, as a line commentScenario Contradicts Verification at
doc/prs/2026-10-14-PR#64-mutation-tests.md:29-39, severity unknown, counted as medium, hint, as a line commentClassifier
7 of 10 units passed all questions (1 PR document(s)).
Usage: model `openrouter/minimax/minimax-m3`, LLM done, tokens counted before the call 18148, calls 1, tokens in 17490 / out 36198 (reasoning included in out), duration 427.2 s, cost 0.0487 USD (reported), classifier `jev-1.13.0` calls 10, tokens in 8267, cost 0.0003 USD, finding judge `jev-1.13.0` calls 2, tokens in 4379, cost 0.0002 USD, build file judge `jev-1.13.0` calls 4, tokens in 4697, cost 0.0002 USD, phases: path_judge 0.3 s, classifier 0.6 s, context 0.3 s, model_review 425.1 s, finding_judge 0.3 s, config: defaults + host + repository (.werkrevisor.toml at main2 answers ignored because of a confidence below [classifier] min_confidence.
2768a9a).Reviewed head `460e0fabdadfced45a3a0bd4bd126a3cb1f63c2b` against base `2768a9af8791af4969392ee3347894f6b653517d`, werkrevisor 0.41.2.
werkrevisor: YELLOW / HUMAN_REVIEW for
460e0fabdadfced45a3a0bd4bd126a3cb1f63c2b.The result comment of werkrevisor holds the details.
@@ -19,12 +19,15 @@ watcher:builds:# The gradle wrapper is committed, so the build environment needs a JDK# and nothing else; Foojay loads the Temurin 25 of the toolchain (ADR 0001).# Gradle itself runs on it too, by dashboard/gradle/gradle-daemon-jvm.properties:Daemon JVM Setup Ineffective, severity unknown, counted as medium (werkrevisor)
hint; Jev: real 0.82, severity 2.3 of 3
The comment claims Gradle runs on Temurin 25 via dashboard/gradle/gradle-daemon-jvm.properties because the host's Java 17 is too old for werkcrappit. But every Werkator build command in this file uses --no-daemon, which bypasses the daemon JVM entirely: Gradle then runs on the launching JVM, i.e. the host's Java. The PR doc claims the same (lines 89-90). Werkator's nightly "mutation" build reportedly succeeded (Scenario#64.04 in the PR doc), so the host may have Java 25 or werkcrappit may tolerate Java 17, but the comment, the gradle-daemon-jvm.properties file, and the buildCommand's --no-daemon flag do not hang together; either --no-daemon must be dropped (so the daemon actually uses Temurin 25) or the comment and the properties file must be removed.
@@ -0,0 +26,4 @@### Feature: mutation tests in each build, in full each night#### Scenario#64.01: a branch build judges only the classes the branch changesScenario Contradicts Verification, severity unknown, counted as medium (werkrevisor)
hint; Jev: real 0.87, severity 2.0 of 3
Scenario#64.01 states "Given a branch that changes only the class
Summaries" and "Then the build judges the Mutation-CRAP of the methods ofSummariesonly", yet the manual verification reports "8 classes, 77 mutants" for a changed Summaries.kt. The changedClasses logic in build.gradle.kts extracts every top-level declaration plus the file class from each changed Kotlin file, so all classes in the file (eight of them) are mutated, not only Summaries. The scenario's Then clause contradicts both the verification and the implementation.@@ -19,12 +19,15 @@ watcher:builds:# The gradle wrapper is committed, so the build environment needs a JDK# and nothing else; Foojay loads the Temurin 25 of the toolchain (ADR 0001).# Gradle itself runs on it too, by dashboard/gradle/gradle-daemon-jvm.properties:Not a defect, no change. With
--no-daemon, Gradle 9 still honoursgradle/gradle-daemon-jvm.properties: it forks a single-use daemon on the JVM the criteria name.Werkator's log of
09f59aashows it: "To honour the JVM settings for this build a single-use Daemon process will be forked."Before that file, the same build failed on the host's Java 17: "Dependency requires at least JVM runtime version 21. This build uses a Java 17 JVM." (build of
b977c5a).The nightly build
mutationhas not run yet; Scenario#64.04 names it as a verification to come.@@ -0,0 +26,4 @@### Feature: mutation tests in each build, in full each night#### Scenario#64.01: a branch build judges only the files the branch changesFixed in
d00b31e: Scenario#64.01 and #64.02 name the fileSummaries.ktand its three classesSummarizer,ChatSummarizerandSummaries; the verification says that PIT selected these and their nested classes.🟡 werkrevisor YELLOW for
d00b31e, superseded by the result for5db4caa🟡 werkrevisor: YELLOW / HUMAN_REVIEW
Change kind:
mixed.Reasons
Findings
doc/prs/2026-10-14-PR#64-mutation-tests.md:33-35, severity unknown, counted as medium, hint, as a line commentClassifier
7 of 10 units passed all questions (1 PR document(s)).
Usage: model `openrouter/minimax/minimax-m3`, LLM done, tokens counted before the call 18220, calls 1, tokens in 17559 / out 31884 (reasoning included in out), duration 665.6 s, cost 0.0435 USD (reported), classifier `jev-1.13.0` calls 10, tokens in 8305, cost 0.0003 USD, finding judge `jev-1.13.0` calls 1, tokens in 2170, cost 0.0001 USD, build file judge `jev-1.13.0` calls 4, tokens in 4697, cost 0.0002 USD, phases: path_judge 0.2 s, classifier 0.7 s, context 0.5 s, model_review 663.2 s, finding_judge 0.4 s, config: defaults + host + repository (.werkrevisor.toml at main2 answers ignored because of a confidence below [classifier] min_confidence.
2768a9a).Reviewed head `d00b31e3c553e1e013ba72544c24482b293326ca` against base `2768a9af8791af4969392ee3347894f6b653517d`, werkrevisor 0.41.2.
werkrevisor: YELLOW / HUMAN_REVIEW for
d00b31e3c553e1e013ba72544c24482b293326ca.The result comment of werkrevisor holds the details.
@@ -0,0 +30,4 @@… so that the mutation tests fit into each branch build.- **Given** a branch that changes only the file `Summaries.kt`, with the classes `Summarizer`, `ChatSummarizer` and `Summaries`Scenario wording omits nested classes, severity unknown, counted as medium (werkrevisor)
hint; Jev: real 0.84, severity 1.4 of 3
Scenario#64.01's Given names only the three top-level classes in
Summaries.kt(Summarizer, ChatSummarizer, Summaries), and its Then states the build judges "the methods of these three classes only". The implementation (and the same scenario's verification) actually judges eight classes: the three top-level ones and their nested ones. A strict reading of "only" excludes the nested classes and contradicts what the code does. The Given should mention that nested classes are also included, or the Then should say "of these three top-level classes and their nested ones only".@@ -0,0 +30,4 @@… so that the mutation tests fit into each branch build.- **Given** a branch that changes only the file `Summaries.kt`Fixed in
5db4caa: the Then names the methods inSummaries.kt, nested classes included.🟡 werkrevisor: YELLOW / HUMAN_REVIEW
Change kind:
mixed.Reasons
Classifier
8 of 10 units passed all questions (1 PR document(s)).
Usage: model `openrouter/minimax/minimax-m3`, LLM done, tokens counted before the call 18085, calls 1, tokens in 17434 / out 32718 (reasoning included in out), duration 392.2 s, cost 0.0445 USD (reported), classifier `jev-1.13.0` calls 10, tokens in 8279, cost 0.0003 USD, build file judge `jev-1.13.0` calls 4, tokens in 4697, cost 0.0002 USD, phases: path_judge 0.3 s, classifier 0.7 s, context 1.7 s, model_review 389.1 s, config: defaults + host + repository (.werkrevisor.toml at main2 answers ignored because of a confidence below [classifier] min_confidence.
2768a9a).Reviewed head `5db4caaedf4686b0b64493e510677b882f5fac45` against base `2768a9af8791af4969392ee3347894f6b653517d`, werkrevisor 0.41.2.