werkrevisor/docs/prs/2026-10-06-PR#27-judged-paths.md

PR#27: Jev judges changes of build files instead of a fixed YELLOW

WARNING: This document describes only the change applied in this PR. It may already be outdated once the next PR is merged. Historic PR documentation is not maintained along with new PRs; treat it as a snapshot.

The Problem

Each change of a path in [hard_yellow] paths gives YELLOW, so a human has to approve the PR. For build files this is mostly noise: the PRs #24, #25 and #26 of werkrevisor got YELLOW only for the version number in server/build.gradle.kts or for buildInfo().

A real risk in a build file looks different: tests switched off, failures ignored, a threshold lowered, or a script downloaded from an unknown host. Such a change deserves more than YELLOW, but the fixed rule cannot tell it from a version number.

Some paths must stay a fixed YELLOW, though: hsadmin-ng wants a human for each change of its security, CI and build configuration.

Non-Goals

• No change of [hard_yellow] paths: these paths keep their fixed YELLOW, without Jev.
• No judgment of the built-in hard paths, e.g. .werkrevisor.toml: a repository config change always needs a human.
• No judgment by the LLM: Jev answers two fixed questions, and the thresholds come from the measurement below.

The Scenarios

Background

• Given [hard_yellow] judged_paths holds server/*.gradle.kts
  • and [path_judge] judge = "jev"

Feature: Jev judges a change of a judged path

Scenario#27.01: Jev's answers decide the color of a judged path

So that a routine build change needs no human, and a weakened check gets more attention than YELLOW.

• Given a PR changes server/build.gradle.kts by <change>
• When werkrevisor reviews the PR
  • and Jev answers <weakens> for "weakens checks" and <risk> for the build risk
• Then the path gives <color>

change	weakens	risk	color
a new version number	0.02	0.00	GREEN, as a hint
buildInfo() added	0.03	1.27	GREEN, as a hint
a major upgrade of the toolchain	0.15	1.90	YELLOW
ignoreFailures = true for the tests	0.97	2.76	ORANGE
a script from an unknown host run before the compile step	0.13	3.00	ORANGE

Verified by

• test_pathjudge: test_jev_answers_decide_the_verdict

Scenario#27.02: a path that Jev could not judge stays YELLOW

So that an outage of Jev never makes a build change GREEN.

• Given a PR changes server/build.gradle.kts
• When werkrevisor reviews the PR <case>
• Then the path gives YELLOW, with the reason "build files not judged by Jev"

case
and Jev fails
and the diff holds a possible secret, which must not reach Jev
with --no-llm
and the PR deletes the file
and the file has no text diff

Verified by

• test_pathjudge: test_unjudged_paths_give_yellow
• test_pathjudge: test_deleted_or_empty_judged_files_are_not_judged

Scenario#27.06: a renamed judged file gives YELLOW without Jev

So that moving a build file out of its place needs a human; the diff of its content does not show the move.

• Given a PR renames server/build.gradle.kts
• When werkrevisor reviews the PR
• Then the file gives YELLOW as a risky path
  • and Jev is not asked

Verified by

• test_pathjudge: test_a_renamed_judged_file_is_a_risky_path

Feature: hard paths stay hard

Scenario#27.03: a path in both lists stays a fixed YELLOW

So that a judged path of the repository config cannot soften a hard path of the host config.

• Given the host config lists build.gradle* in [hard_yellow] paths
  • and the repository config lists build.gradle.kts in [hard_yellow] judged_paths
• When werkrevisor reviews a PR that changes build.gradle.kts
• Then the path gives YELLOW as a risky path
  • and Jev is not asked

Verified by

• test_pathjudge: test_hard_paths_win

Scenario#27.04: without the judge, a judged path gives YELLOW

So that a repository that does not use Jev keeps today's behavior.

• Given [path_judge] judge = "never"
• When werkrevisor reviews a PR that changes server/build.gradle.kts
• Then the path gives YELLOW as a risky path

Verified by

• test_pathjudge: test_never_judge_gives_yellow

Feature: the host decides whether a repository may switch the judge on

Scenario#27.05: the repository config switches the judge on only if the host allows it

So that hsadmin-ng can keep every listed path a fixed YELLOW, whatever its repository config says.

• Given the host config sets [path_judge] repository_may_enable = <allowed>
• When the repository config sets [path_judge] judge = "jev"
• Then <result>

allowed	result
true	Jev judges the judged paths
false	the repository config is refused as invalid

Verified by

• test_pathjudge: test_repository_may_enable_the_judge_only_if_the_host_allows_it

The Solution

• [hard_yellow] judged_paths lists paths that Jev judges; the host config and the repository config each add paths. A path that matches [hard_yellow] paths or a built-in hard path stays a fixed YELLOW.
• [path_judge] sets judge (never or jev) and the thresholds; the repository config may set only judge, and jev only if the host sets repository_may_enable = true, the default.
• Jev gets the path and the diff hunks of each judged file, with their context lines, and answers two questions: weakens_checks (probability) and build_risk (score 0 to 3). ORANGE if weakens_checks ≥ 0.8 or build_risk ≥ 2.8; GREEN as a hint if weakens_checks < 0.3 and build_risk ≤ 1.5; YELLOW otherwise.
• The usage record holds path_judgments with the values and the color per file.
• werkrevisor moves server/*.gradle.kts, which covers build.gradle.kts and settings.gradle.kts, to judged_paths and switches the judge on; server/gradle/** and server/gradlew* stay hard, since they decide which Gradle runs.
• hsadmin-ng's host config sets repository_may_enable = false.
• werkrevisor reads .werkrevisor.toml at the base commit, so its own judged paths take effect with the first PR after the merge; the installed werkrevisor must know the new settings by then, so 0.19.0 is deployed right after the merge.
• werkrevisor 0.19.0.

Additional Changes

• The README section "How a run works" starts with a Mermaid diagram of how the workflow code, Jev and the LLM work together.

Measurement

On 2026-10-06, Jev answered the two questions for 41 diffs of build files, for 0.0022 USD in all, with the state that werkrevisor sends:

• 24 changes of build.gradle.kts in hsadmin-ng and 3 of server/build.gradle.kts in werkrevisor;
• 8 constructed harmful changes: tests disabled, failures ignored, tests excluded, the test task dropped from check, coverage and mutation thresholds lowered, a script downloaded and run, an insecure repository added;
• 4 constructed harmless changes: a dependency, a module, an extra check, a raised threshold;
• 2 harmful changes with a comment that calls them harmless.

group	GREEN	YELLOW	ORANGE
constructed harmful, also with a misleading comment	0	0	10
constructed harmless	3	1	0
real changes	12	15	0

The constructed harmless change that stays YELLOW raises a mutation threshold (weakens_checks 0.41). The real changes that get GREEN are version updates, new tasks and buildInfo(); YELLOW go, among others, the conversion to Kotlin DSL, the upgrade to JDK 25 and Spring Boot 4, and the change that bases the coverage verification on unit tests only (weakens_checks 0.49). Without the context lines, Jev rated enabled = false in a test task only 0.76 for weakens_checks, so it got YELLOW.

Prerequisite PRs

• PR#17: Jev judges DB changesets; this PR uses the same pattern.