werkrevisor/docs/prs/2026-10-06-PR#39-jev-triage.md

# PR#39: Jev decides whether a PR needs a human, the LLM review is optional

> **WARNING:** This document describes only the change applied in this PR.
> It may already be outdated once the next PR is merged.
> Historic PR documentation is not maintained along with new PRs; treat it as a snapshot.

## The Problem

On git.javagil.de, the last review before the merge was GREEN for 6 of 38 merged PRs.
The others waited for a human, mostly for reasons that tell nothing about harm:

| reason | PRs |
|---|---|
| a hard path changed, e.g. `config/**`, `werkrevisor/config.py` or a Gradle file | 27 |
| test changes that are not purely additive | 10 |
| findings of the model | 10 |
| a file reached the model only as diff or excerpt | 9 |

The review is a second look by one other developer, who wants to know whether something was built in
that can do real harm and is not covered by tests, or that causes a severe security problem.
Agreed changes, e.g. renamed API properties, need no human.

On the 25 merged hs.hsadmin.ng PRs #281 to #316, Jev separated the PRs that need a human from the others
with two questions per changed file: does the change touch security, and does it weaken the checks of the project.
A third question, whether a mistake would do harm the tests do not reveal, did not separate them.

Finding defects is secondary: on 4 heads with 10 known findings, Sonnet 5.5, MiniMax M3 and Mistral Large 4
found 1 to 3 findings each, each a different one.

## Non-Goals

- No change for built-in hard paths, e.g. `.werkrevisor.toml`: a repository config may replace limits and change kinds.
- No change for deterministic RED and YELLOW rules: open questions in PR documents, lost or disabled tests, secrets, size.
- No new model: an operator chooses the model and pays for it, e.g. GPT-6 Astra for hsadmin-ng.

## The Scenarios

### Feature: Jev judges every changed file

#### Scenario#39.01: a harmless change of a judged path needs no human

So that a version number in a Gradle file does not wait for a review.

- **Given** `[path_judge] scope = "all"`
- **When** a PR changes a judged path, and Jev rates it as weakening no check, touching no security and of low risk
- **Then** the judged path gives no YELLOW

##### Verified by

- [test_pathjudge: test_a_harmless_judged_path_needs_no_human](../../tests/test_pathjudge.py)

#### Scenario#39.02: a change that touches security needs a human

So that a change of an access rule always gets a second look.

- **Given** `[path_judge] scope = "all"`
- **When** Jev rates a changed main-code file as touching security
- **Then** the verdict is YELLOW, and the reason names the file and the value

##### Verified by

- [test_pathjudge: test_a_change_that_touches_security_needs_a_human](../../tests/test_pathjudge.py)
- [test_pathjudge: test_a_deleted_file_is_judged_on_its_removed_lines](../../tests/test_pathjudge.py)

#### Scenario#39.03: a change that weakens the checks gets ORANGE

So that a disabled test task or a cached test result never merges unseen.

- **Given** `[path_judge] scope = "all"`
- **When** Jev rates a changed file as weakening the checks
- **Then** the verdict is ORANGE

##### Verified by

- [test_pathjudge: test_a_change_that_weakens_the_checks_gets_orange](../../tests/test_pathjudge.py)

### Feature: the operator chooses whether the model decides or advises

#### Scenario#39.04: when the model only advises, its doubts are hints

So that a cheap model may find small things without holding a PR.

- **Given** `[path_judge] scope = "all"`, `[engine] model_role = "advises"` and Jev judged every changed file
- **When** the model reports `<reason>`
- **Then** the reason is a hint, not YELLOW

| reason |
|---|
| a finding that Jev doubts |
| a high risk |
| a file that reached the model only as diff or excerpt |
| a PR too large for the model, which then gets no call |

##### Verified by

- [test_pathjudge: test_doubts_of_the_model_are_hints_when_jev_judged_all_files](../../tests/test_pathjudge.py)
- [test_pathjudge: test_a_pr_too_large_for_the_model_is_a_hint_when_the_model_only_advises](../../tests/test_pathjudge.py)

#### Scenario#39.05: without a model, Jev's judgment is enough for GREEN

So that an operator can switch the LLM review off.

- **Given** `[engine] mode = "classifier"` and `[path_judge] scope = "all"`
- **When** Jev judged every changed file and found nothing
- **Then** the verdict is GREEN without a model call

##### Verified by

- [test_pathjudge: test_without_a_model_jev_is_enough_for_green](../../tests/test_pathjudge.py)

#### Scenario#39.06: by default, the model decides as before

So that an operator who pays for a frontier model, e.g. GPT-6 Astra, keeps its findings as gate.

- **Given** `[path_judge] scope = "all"` and no `[engine] model_role`
- **When** the model reports a finding that Jev doubts
- **Then** the verdict is YELLOW, as before this PR

##### Verified by

- [test_pathjudge: test_by_default_the_model_decides_as_before](../../tests/test_pathjudge.py)

### Feature: the repository config decides which paths are hard

#### Scenario#39.07: the repository config replaces the hard and judged paths of the host

So that a repository keeps only the few paths that change the review or the agentic development hard.

- **Given** the host config lists `tools/**` as hard path
- **When** the `.werkrevisor.toml` of the repository lists `[hard_yellow] paths = ["**/*.pem"]`
- **Then** a change of `tools/deploy.sh` is no hard path
  - **and** a change of `.werkrevisor.toml` still needs a human

##### Verified by

- [test_config_layers: test_the_repository_replaces_the_hard_and_judged_paths](../../tests/test_config_layers.py)

#### Scenario#39.08: the DB schema paths are configurable

So that a repository whose `*.sql` files hold no schema does not have them judged as changesets.

- **Given** the `.werkrevisor.toml` lists `[db_schema] paths = ["src/main/resources/db/**"]`
- **When** a PR changes `tools/report.sql`
- **Then** werkrevisor treats `tools/report.sql` as an ordinary file

##### Verified by

- [test_dbschema: test_the_db_schema_paths_are_configurable](../../tests/test_dbschema.py)

#### Scenario#39.09: the repository config comes from the default branch

So that no PR, also not one stacked on another branch, changes the rules of its own review.

- **Given** `.werkrevisor.toml` sets `[size] max_files = 11` on the default branch and `99` on another branch
- **When** werkrevisor reviews a PR into the other branch
- **Then** the limit of 11 files applies

##### Verified by

- [test_config_layers: test_the_repository_config_comes_from_the_default_branch](../../tests/test_config_layers.py)

## The Solution

- `[path_judge]` gets `scope`: `"judged_paths"`, the default, or `"all"`, every changed file outside the docs,
  the mechanical files and the hard paths.
- With `scope = "all"`, Jev answers a third question, `security`, beside `weakens_checks` and `build_risk`;
  `build_risk` counts only for judged paths, since it did not separate main code.
- A new threshold in `[path_judge]`, `yellow_security_min` (0.93); `orange_weakens_min` (0.8) also applies to other files.
- With `scope = "all"`, a main-code file gets the questions `weakens_checks` and `security`, a test file only `weakens_checks`,
  since a test that names a role tests the access rule instead of changing it.
- `[engine] model_role`, host only: `"decides"`, the default and the behavior before this PR,
  or `"advises"`: with Jev's judgment of every file, model reasons below ORANGE become hints, except a security concern.
  With `[engine] mode = "classifier"`, there is no model, and Jev's judgment of all files suffices for GREEN.
- The repository config replaces `[hard_yellow] paths` and `judged_paths` of the host instead of adding to them,
  and `[db_schema] paths` replaces the DB schema paths, which were built in; only `.werkrevisor.toml` stays built in.
- werkrevisor reads `.werkrevisor.toml` at the head of the default branch, no longer at the base commit of the PR.
- `.werkrevisor.toml` and the example configs of werkator and werkrevisor keep only secrets hard,
  and for werkator its isolation paths;
  Jev judges the build, image and CI files as judged paths and every other file through `scope = "all"`.
  `config/werkprobe.toml` stays unchanged, since the tests build on it;
  the example configs of werkator and werkrevisor set `model_role = "advises"`, `config/hsadmin-ng.toml` keeps `"decides"`.
  The machine configs of the three repositories on mih09 get the same change at the deployment.
- `docs/anleitung-dev.hostsharing.net.md` names the new switches for hsadmin-ng, which keeps `"decides"` until the team decides.
- werkrevisor 0.21.0.

## Measurement

On 2026-10-06, the code of this PR reviewed the 37 merged PRs of git.javagil.de with a final werkrevisor result
in `[engine] mode = "classifier"`, without a model, with the configs of this PR; Jev cost less than 0.01 USD.
The gate reasons self-approval and "never marked ready" are left out, since they belong to the gate, not to the triage.

| result | before, as reviewed | with this PR |
|---|---|---|
| GREEN | 6 | 20 |
| YELLOW or ORANGE | 31 | 17 |

Of the 17, 13 changed `.werkrevisor.toml`, 8 of them only that;
3 had a scenario that Jev rated as unclear, 3 a change that touches security according to Jev,
2 lost tests, 2 a binary file Jev cannot read, 2 a risky build change, and #9 weakened its tests (ORANGE).
Among the GREEN, #25 made the statistics public, which the user had asked for;
Jev rated its files up to 0.90 for security, below the threshold of 0.93.
The user decided that `.werkrevisor.toml` stays hard.

## Follow-up PRs

- A review specification in one place, as a DSL in the style of `RbacSpec` in hsadmin-ng:
  under which conditions which workflow runs, which questions Jev answers and which values give which verdict.
  The switches of this PR, `[path_judge] scope`, `[engine] mode` and `[engine] model_role`, are its first building blocks.

## Prerequisite PRs

- PR#38: the guide for dev.hostsharing.net names the state of werkrevisor 0.20.6, merged.