werkrevisor asks the risk categories that fit werkcrappit #46
No Assignees
Notifications
Due Date
No due date set.
Dependencies
No dependencies set.
Reference: mi/werkcrappit#46
Reference in New Issue
Block a user
See the PR document.
🤖 Generated with Claude Code
🟡 werkrevisor YELLOW for
607b4b4, superseded by the result forc757601🟡 werkrevisor: YELLOW / HUMAN_REVIEW
Change kind:
mixed.Reasons
Findings
docs/prs/2026-10-09-PR#46-risk-categories.md:30-35, severity unknown, counted as medium, hint, as a line commentClassifier
4 of 4 units passed all questions (1 PR document(s)).
Usage: model `openrouter/minimax/minimax-m3`, LLM done, tokens counted before the call 9483, calls 1, tokens in 9179 / out 10106 (reasoning included in out), duration 78.7 s, cost 0.0148 USD (reported), classifier `jev-1.13.0` calls 4, tokens in 3172, cost 0.0001 USD, finding judge `jev-1.13.0` calls 1, tokens in 1508, cost 0.0001 USD, phases: classifier 0.4 s, context 1.5 s, model_review 76.1 s, finding_judge 0.3 s, config: defaults + host + repository (.werkrevisor.toml at mainb640b7d).Reviewed head `607b4b4cb8b581d6737e8309b9e4bc851e663f4f` against base `b640b7d5c6495b07303a10b236675f28230d3ac9`, werkrevisor 0.30.0.
werkrevisor: YELLOW / HUMAN_REVIEW for
607b4b4cb8b581d6737e8309b9e4bc851e663f4f.The result comment of werkrevisor holds the details.
@@ -0,0 +27,4 @@## MeasurementJev judged each changed file of the 30 latest merged PRs (#16 to #45, 159 main files, 82 test files)Unclear threshold claim, severity unknown, counted as medium (werkrevisor)
hint; Jev: real 0.69, severity 1.8 of 3
The PR document claims "No category reached its threshold" for the kept categories (error_handling 0.85, external_integration 0.82, performance/architecture 0.81, concurrency 0.78) without stating what the default thresholds of the
jvm-spring-backendtemplate are. A reader cannot verify the claim without consulting the template's defaults, and the highest observed value (0.85) sits above themetric_integritythreshold of 0.8, so the assumed thresholds matter.Fixed in
c757601: the measurement names the thresholds of the template, 0.95, 0.9 forconcurrencyand 0.85 fordata_correction.Review round of
607b4b4c: the one hint is fixed inc757601. The YELLOW comes from.werkrevisor.toml, which always needs the approval of the human user.🟡 werkrevisor YELLOW for
c757601, superseded by the result ford70a9cc🟡 werkrevisor: YELLOW / HUMAN_REVIEW
Change kind:
mixed.Reasons
Findings
.werkrevisor.toml:36-39, YELLOW, as a line commentClassifier
4 of 4 units passed all questions (1 PR document(s)).
Usage: model `openrouter/minimax/minimax-m3`, LLM done, tokens counted before the call 9507, calls 1, tokens in 9202 / out 20992 (reasoning included in out), duration 97.5 s, cost 0.0279 USD (reported), classifier `jev-1.13.0` calls 4, tokens in 3205, cost 0.0001 USD, finding judge `jev-1.13.0` calls 1, tokens in 1662, cost 0.0001 USD, phases: classifier 0.3 s, context 0.1 s, model_review 96.6 s, finding_judge 0.3 s, config: defaults + host + repository (.werkrevisor.toml at mainb640b7d).Reviewed head `c7576016968715a1602abe95419fb47e79ab356d` against base `b640b7d5c6495b07303a10b236675f28230d3ac9`, werkrevisor 0.30.0.
werkrevisor: YELLOW / HUMAN_REVIEW for
c7576016968715a1602abe95419fb47e79ab356d.The result comment of werkrevisor holds the details.
@@ -33,6 +33,66 @@ secret_scan = true# real key formats still count there: private keys, AWS key ids, `hsak_` and `sk-or-v1-` keys (see werkrevisor's README).secret_scan_exclude = ["**/src/test/**", "**/src/functionalTest/**"][path_judge][high] Possible path judge scope omission (werkrevisor)
YELLOW; Jev: real 0.59, severity 2.9 of 3
The PR document states "Jev asks each category for each changed file" (line 9) and the measurement section states "Jev judged each changed file of the 30 latest merged PRs (#16 to #45, 159 main files, 82 test files)" (line 30). Both imply Jev judged all changed files during the measurement, including the 159 main files and 82 test files. However, the new
[path_judge]section does not setscope = "all". Based on the existing comment in[hard_yellow]("with [path_judge] scope = "all" every other file"), without that option Jev may only judge the paths listed injudged_paths(build/CI/publication files), not all changed files. The newmetric_integritycategory is intended to enforce the AGENTS.md invariants that govern the Mutation-CRAP metric, its inputs and the gate decision; those changes live insrc/main/**andsrc/test/**underwerkcrappit-pitandwerkcrappit-gradle-plugin, not injudged_paths. If my reading of the scope option is correct, the category will not fire on the changes it is meant to catch, so the main protection this PR adds would be largely a no-op, contradicting the measurement claim. The remaining uncertainty is werkrevisor's exact behavior whenscopeis unset; the two header comments in the file contradict each other on this point and should be reconciled, e.g. by addingscope = "all"to[path_judge]or by correcting the PR document.Not a problem: the host config of werkcrappit on the review machine sets
[path_judge] scope = "all", and werkrevisor refusesscopein a repository config, so Jev judges every changed file.d70a9ccmakes the comment at[path_judge]say this.Review round of
c7576016: the one finding is answered;d70a9ccclarifies the comment. The YELLOW comes from.werkrevisor.toml, which always needs the approval of the human user.🟡 werkrevisor YELLOW for
d70a9cc, superseded by the result for6964534🟡 werkrevisor: YELLOW / HUMAN_REVIEW
Change kind:
mixed.Reasons
Findings
[medium] Possible TOML Misplacement at
.werkrevisor.toml:43-95, YELLOW, as a line comment[low] List Inconsistency at
docs/prs/2026-10-09-PR#46-risk-categories.md:11-12, hint, as a line comment[low] Ambiguous Question at
.werkrevisor.toml:92-93, hint, as a line commentClassifier
4 of 4 units passed all questions (1 PR document(s)).
Usage: model `openrouter/minimax/minimax-m3`, LLM done, tokens counted before the call 9552, calls 1, tokens in 9245 / out 20703 (reasoning included in out), duration 127.2 s, cost 0.0267 USD (reported), classifier `jev-1.13.0` calls 4, tokens in 3205, cost 0.0001 USD, finding judge `jev-1.13.0` calls 1, tokens in 2288, cost 0.0001 USD, phases: classifier 0.2 s, context 0.2 s, model_review 126.2 s, finding_judge 0.3 s, config: defaults + host + repository (.werkrevisor.toml at mainb640b7d).Reviewed head `d70a9cc7046957a22478acb96309c5f6458465c6` against base `b640b7d5c6495b07303a10b236675f28230d3ac9`, werkrevisor 0.30.0.
werkrevisor: YELLOW / HUMAN_REVIEW for
d70a9cc7046957a22478acb96309c5f6458465c6.The result comment of werkrevisor holds the details.
@@ -36,0 +40,4 @@risk_template = "jvm-spring-backend"# A Gradle and PIT plugin has no logins, SQL, transactions, migrations, stored records or REST endpoints.[[risk.categories]][medium] Possible TOML Misplacement (werkrevisor)
YELLOW; Jev: real 0.57, severity 2.9 of 3
The 12
[[risk.categories]]blocks (lines 43-95) are defined at the top level rather than nested under the[path_judge]table opened at line 36. The inline comment "Jev asks each changed file the risk categories of this template, as changed below" and the existing[[classifier.scenario_questions]]pattern, which is properly nested as[[classifier.scenario_questions]]under[classifier](line 154), suggest the new blocks should be[[path_judge.risk.categories]]. With the current syntax, TOML creates a top-levelriskarray separate from[path_judge]; if werkrevisor's schema expects the categories to belong to thepath_judgesection, the custommetric_integritycategory and the elevenoffoverrides will not be applied. The discrepancy between the inline comment and the structure should be confirmed against werkrevisor's config schema (PR#63) before merge.Not a problem: werkrevisor reads the categories as the top-level table
[risk] categories, not under[path_judge](werkrevisor README, "Risk categories"). The config of this PR loads with the werkcrappit host config, andrisk.resolvegives the 5 kept categories plusmetric_integrity.@@ -36,0 +89,4 @@[[risk.categories]]name = "metric_integrity"title = "Metric integrity"question = """Does the change alter how the Mutation-CRAP value, its inputs mutation status and complexity, \[low] Ambiguous Question (werkrevisor)
hint
The
metric_integrityquestion reads "Does the change alter how the Mutation-CRAP value, its inputs mutation status and complexity, or the gate decision are computed...". The phrase "its inputs mutation status and complexity" parses ambiguously: it could be "its inputs: mutation status and complexity" (two inputs) or "its input's mutation status and complexity" (one input with two properties). AGENTS.md describes mutation status and complexity as two separate inputs, so the intended reading is the former, but a colon or parenthetical would remove the ambiguity for the judging model.Fixed in
6964534: "its two inputs (mutation status and complexity)". The new wording is measured again; the listener change fell to 0.80, so the threshold is now 0.75 (real files at most 0.66). The PR document has the new values.@@ -0,0 +8,4 @@Since werkrevisor 0.30.0, a repository chooses a template of risk categories in its `.werkrevisor.toml` and tunes it.Jev then asks each category for each changed file; a category above its threshold holds the PR for a human.werkcrappit chooses no template, so Jev asks only whether a change weakens the checks or touches security.The JVM template `jvm-spring-backend` is made for Spring services:[low] List Inconsistency (werkrevisor)
hint
The "The Problem" section (line 12) lists "logins, SQL, transactions, migrations and REST endpoints" as features a Gradle and PIT plugin lacks but omits "stored records", which the config comment at line 42 of
.werkrevisor.tomlcites as the rationale for switching off thedata_correctioncategory. A reader of the PR document cannot infer from the prose alone whydata_correctionis in the off list.Fixed in
6964534: the problem section names stored records too.Review round of
d70a9cc7: two hints fixed in6964534, the TOML placement is correct as it is. The YELLOW comes from.werkrevisor.toml, which always needs the approval of the human user.🟡 werkrevisor: YELLOW / HUMAN_REVIEW
Change kind:
mixed.Reasons
Classifier
4 of 4 units passed all questions (1 PR document(s)).
Usage: model `openrouter/minimax/minimax-m3`, LLM done, tokens counted before the call 9659, calls 1, tokens in 9347 / out 11714 (reasoning included in out), duration 72.8 s, cost 0.0158 USD (reported), classifier `jev-1.13.0` calls 4, tokens in 3305, cost 0.0001 USD, phases: classifier 0.3 s, context 0.2 s, model_review 71.9 s, config: defaults + host + repository (.werkrevisor.toml at mainb640b7d).Reviewed head `6964534d1cf0816f14c87f70db846d9222e44359` against base `b640b7d5c6495b07303a10b236675f28230d3ac9`, werkrevisor 0.30.0.
claude-code referenced this pull request from mi/werkprobe2026-10-09 07:48:24 +02:00