werkrevisor/docs/prs/2026-10-06-PR#27-judged-paths.md

# PR#27: Jev judges changes of build files instead of a fixed YELLOW

> **WARNING:** This document describes only the change applied in this PR.
> It may already be outdated once the next PR is merged.
> Historic PR documentation is not maintained along with new PRs; treat it as a snapshot.

## The Problem

Each change of a path in `[hard_yellow] paths` gives YELLOW, so a human has to approve the PR.
For build files this is mostly noise:
the PRs #24, #25 and #26 of werkrevisor got YELLOW only for the version number in `server/build.gradle.kts`
or for `buildInfo()`.

A real risk in a build file looks different: tests switched off, failures ignored, a threshold lowered,
or a script downloaded from an unknown host.
Such a change deserves more than YELLOW, but the fixed rule cannot tell it from a version number.

Some paths must stay a fixed YELLOW, though:
hsadmin-ng wants a human for each change of its security, CI and build configuration.

## Non-Goals

- No change of `[hard_yellow] paths`: these paths keep their fixed YELLOW, without Jev.
- No judgment of the built-in hard paths, e.g. `.werkrevisor.toml`: a repository config change always needs a human.
- No judgment by the LLM: Jev answers two fixed questions, and the thresholds come from the measurement below.

## The Scenarios

### Background

- **Given** `[hard_yellow] judged_paths` holds `server/*.gradle.kts`
  - **and** `[path_judge] judge = "jev"`

### Feature: Jev judges a change of a judged path

#### Scenario#27.01: Jev's answers decide the color of a judged path

So that a routine build change needs no human, and a weakened check gets more attention than YELLOW.

- **Given** a PR changes `server/build.gradle.kts` by `<change>`
- **When** werkrevisor reviews the PR
  - **and** Jev answers `<weakens>` for "weakens checks" and `<risk>` for the build risk
- **Then** the path gives `<color>`

| change | weakens | risk | color |
|---|---|---|---|
| a new version number | 0.02 | 0.00 | GREEN, as a hint |
| `buildInfo()` added | 0.03 | 1.27 | GREEN, as a hint |
| a major upgrade of the toolchain | 0.15 | 1.90 | YELLOW |
| `ignoreFailures = true` for the tests | 0.97 | 2.76 | ORANGE |
| a script from an unknown host run before the compile step | 0.13 | 3.00 | ORANGE |

##### Verified by

- [test_pathjudge: test_jev_answers_decide_the_verdict](../../tests/test_pathjudge.py)

#### Scenario#27.02: a path that Jev could not judge stays YELLOW

So that an outage of Jev never makes a build change GREEN.

- **Given** a PR changes `server/build.gradle.kts`
- **When** werkrevisor reviews the PR `<case>`
- **Then** the path gives YELLOW, with the reason "build files not judged by Jev"

| case |
|---|
| and Jev fails |
| and the diff holds a possible secret, which must not reach Jev |
| with `--no-llm` |
| and the PR deletes the file |
| and the file has no text diff |

##### Verified by

- [test_pathjudge: test_unjudged_paths_give_yellow](../../tests/test_pathjudge.py)
- [test_pathjudge: test_deleted_or_empty_judged_files_are_not_judged](../../tests/test_pathjudge.py)

#### Scenario#27.06: a renamed judged file gives YELLOW without Jev

So that moving a build file out of its place needs a human; the diff of its content does not show the move.

- **Given** a PR renames `server/build.gradle.kts`
- **When** werkrevisor reviews the PR
- **Then** the file gives YELLOW as a risky path
  - **and** Jev is not asked

##### Verified by

- [test_pathjudge: test_a_renamed_judged_file_is_a_risky_path](../../tests/test_pathjudge.py)

### Feature: hard paths stay hard

#### Scenario#27.03: a path in both lists stays a fixed YELLOW

So that a judged path of the repository config cannot soften a hard path of the host config.

- **Given** the host config lists `build.gradle*` in `[hard_yellow] paths`
  - **and** the repository config lists `build.gradle.kts` in `[hard_yellow] judged_paths`
- **When** werkrevisor reviews a PR that changes `build.gradle.kts`
- **Then** the path gives YELLOW as a risky path
  - **and** Jev is not asked

##### Verified by

- [test_pathjudge: test_hard_paths_win](../../tests/test_pathjudge.py)

#### Scenario#27.04: without the judge, a judged path gives YELLOW

So that a repository that does not use Jev keeps today's behavior.

- **Given** `[path_judge] judge = "never"`
- **When** werkrevisor reviews a PR that changes `server/build.gradle.kts`
- **Then** the path gives YELLOW as a risky path

##### Verified by

- [test_pathjudge: test_never_judge_gives_yellow](../../tests/test_pathjudge.py)

### Feature: the host decides whether a repository may switch the judge on

#### Scenario#27.05: the repository config switches the judge on only if the host allows it

So that hsadmin-ng can keep every listed path a fixed YELLOW, whatever its repository config says.

- **Given** the host config sets `[path_judge] repository_may_enable = <allowed>`
- **When** the repository config sets `[path_judge] judge = "jev"`
- **Then** `<result>`

| allowed | result |
|---|---|
| true | Jev judges the judged paths |
| false | the repository config is refused as invalid |

##### Verified by

- [test_pathjudge: test_repository_may_enable_the_judge_only_if_the_host_allows_it](../../tests/test_pathjudge.py)

## The Solution

- `[hard_yellow] judged_paths` lists paths that Jev judges; the host config and the repository config each add paths.
  A path that matches `[hard_yellow] paths` or a built-in hard path stays a fixed YELLOW.
- `[path_judge]` sets `judge` (`never` or `jev`) and the thresholds;
  the repository config may set only `judge`, and `jev` only if the host sets `repository_may_enable = true`, the default.
- Jev gets the path and the diff hunks of each judged file, with their context lines, and answers two questions:
  `weakens_checks` (probability) and `build_risk` (score 0 to 3).
  ORANGE if `weakens_checks` ≥ 0.8 or `build_risk` ≥ 2.8;
  GREEN as a hint if `weakens_checks` < 0.3 and `build_risk` ≤ 1.5; YELLOW otherwise.
- The usage record holds `path_judgments` with the values and the color per file.
- werkrevisor moves `server/*.gradle.kts`, which covers `build.gradle.kts` and `settings.gradle.kts`, to `judged_paths`
  and switches the judge on;
  `server/gradle/**` and `server/gradlew*` stay hard, since they decide which Gradle runs.
- hsadmin-ng's host config sets `repository_may_enable = false`.
- werkrevisor reads `.werkrevisor.toml` at the base commit,
  so its own judged paths take effect with the first PR after the merge; the installed werkrevisor must know the new settings by then, so 0.19.0 is deployed right after the merge.
- werkrevisor 0.19.0.

## Additional Changes

- The README section "How a run works" starts with a Mermaid diagram of how the workflow code, Jev and the LLM work together.

## Measurement

On 2026-10-06, Jev answered the two questions for 41 diffs of build files, for 0.0022 USD in all,
with the state that werkrevisor sends:

- 24 changes of `build.gradle.kts` in hsadmin-ng and 3 of `server/build.gradle.kts` in werkrevisor;
- 8 constructed harmful changes: tests disabled, failures ignored, tests excluded, the test task dropped from `check`,
  coverage and mutation thresholds lowered, a script downloaded and run, an insecure repository added;
- 4 constructed harmless changes: a dependency, a module, an extra check, a raised threshold;
- 2 harmful changes with a comment that calls them harmless.

| group | GREEN | YELLOW | ORANGE |
|---|---|---|---|
| constructed harmful, also with a misleading comment | 0 | 0 | 10 |
| constructed harmless | 3 | 1 | 0 |
| real changes | 12 | 15 | 0 |

The constructed harmless change that stays YELLOW raises a mutation threshold (`weakens_checks` 0.41).
The real changes that get GREEN are version updates, new tasks and `buildInfo()`;
YELLOW go, among others, the conversion to Kotlin DSL, the upgrade to JDK 25 and Spring Boot 4,
and the change that bases the coverage verification on unit tests only (`weakens_checks` 0.49).
Without the context lines, Jev rated `enabled = false` in a test task only 0.76 for `weakens_checks`, so it got YELLOW.

## Prerequisite PRs

- PR#17: Jev judges DB changesets; this PR uses the same pattern.