werkrevisor/docs/prs/2026-10-06-PR#39-jev-triage.md

PR#39: Jev decides whether a PR needs a human, the LLM review is optional

WARNING: This document describes only the change applied in this PR. It may already be outdated once the next PR is merged. Historic PR documentation is not maintained along with new PRs; treat it as a snapshot.

The Problem

On git.javagil.de, the last review before the merge was GREEN for 6 of 38 merged PRs. The others waited for a human, mostly for reasons that tell nothing about harm:

reason	PRs
a hard path changed, e.g. config/**, werkrevisor/config.py or a Gradle file	27
test changes that are not purely additive	10
findings of the model	10
a file reached the model only as diff or excerpt	9

The review is a second look by one other developer, who wants to know whether something was built in that can do real harm and is not covered by tests, or that causes a severe security problem. Agreed changes, e.g. renamed API properties, need no human.

On the 25 merged hs.hsadmin.ng PRs #281 to #316, Jev separated the PRs that need a human from the others with two questions per changed file: does the change touch security, and does it weaken the checks of the project. A third question, whether a mistake would do harm the tests do not reveal, did not separate them.

Finding defects is secondary: on 4 heads with 10 known findings, Sonnet 5.5, MiniMax M3 and Mistral Large 4 found 1 to 3 findings each, each a different one.

Non-Goals

• No change for built-in hard paths, e.g. .werkrevisor.toml: a repository config may replace limits and change kinds.
• No change for deterministic RED and YELLOW rules: open questions in PR documents, lost or disabled tests, secrets, size.
• No new model: an operator chooses the model and pays for it, e.g. GPT-6 Astra for hsadmin-ng.

The Scenarios

Feature: Jev judges every changed file

Scenario#39.01: a harmless change of a judged path needs no human

So that a version number in a Gradle file does not wait for a review.

• Given [path_judge] scope = "all"
• When a PR changes a judged path, and Jev rates it as weakening no check, touching no security and of low risk
• Then the judged path gives no YELLOW

Verified by

• test_pathjudge: test_a_harmless_judged_path_needs_no_human

Scenario#39.02: a change that touches security needs a human

So that a change of an access rule always gets a second look.

• Given [path_judge] scope = "all"
• When Jev rates a changed main-code file as touching security
• Then the verdict is YELLOW, and the reason names the file and the value

Verified by

• test_pathjudge: test_a_change_that_touches_security_needs_a_human
• test_pathjudge: test_a_deleted_file_is_judged_on_its_removed_lines

Scenario#39.03: a change that weakens the checks gets ORANGE

So that a disabled test task or a cached test result never merges unseen.

• Given [path_judge] scope = "all"
• When Jev rates a changed file as weakening the checks
• Then the verdict is ORANGE

Verified by

• test_pathjudge: test_a_change_that_weakens_the_checks_gets_orange

Feature: the operator chooses whether the model decides or advises

Scenario#39.04: when the model only advises, its doubts are hints

So that a cheap model may find small things without holding a PR.

• Given [path_judge] scope = "all", [engine] model_role = "advises" and Jev judged every changed file
• When the model reports <reason>
• Then the reason is a hint, not YELLOW

reason
a finding that Jev doubts
a high risk
a file that reached the model only as diff or excerpt
a PR too large for the model, which then gets no call

Verified by

• test_pathjudge: test_doubts_of_the_model_are_hints_when_jev_judged_all_files
• test_pathjudge: test_a_pr_too_large_for_the_model_is_a_hint_when_the_model_only_advises

Scenario#39.05: without a model, Jev's judgment is enough for GREEN

So that an operator can switch the LLM review off.

• Given [engine] mode = "classifier" and [path_judge] scope = "all"
• When Jev judged every changed file and found nothing
• Then the verdict is GREEN without a model call

Verified by

• test_pathjudge: test_without_a_model_jev_is_enough_for_green

Scenario#39.06: by default, the model decides as before

So that an operator who pays for a frontier model, e.g. GPT-6 Astra, keeps its findings as gate.

• Given [path_judge] scope = "all" and no [engine] model_role
• When the model reports a finding that Jev doubts
• Then the verdict is YELLOW, as before this PR

Verified by

• test_pathjudge: test_by_default_the_model_decides_as_before

Feature: the repository config decides which paths are hard

Scenario#39.07: the repository config replaces the hard and judged paths of the host

So that a repository keeps only the few paths that change the review or the agentic development hard.

• Given the host config lists tools/** as hard path
• When the .werkrevisor.toml of the repository lists [hard_yellow] paths = ["**/*.pem"]
• Then a change of tools/deploy.sh is no hard path
  • and a change of .werkrevisor.toml still needs a human

Verified by

• test_config_layers: test_the_repository_replaces_the_hard_and_judged_paths

Scenario#39.08: the DB schema paths are configurable

So that a repository whose *.sql files hold no schema does not have them judged as changesets.

• Given the .werkrevisor.toml lists [db_schema] paths = ["src/main/resources/db/**"]
• When a PR changes tools/report.sql
• Then werkrevisor treats tools/report.sql as an ordinary file

Verified by

• test_dbschema: test_the_db_schema_paths_are_configurable

Scenario#39.09: the repository config comes from the default branch

So that no PR, also not one stacked on another branch, changes the rules of its own review.

• Given .werkrevisor.toml sets [size] max_files = 11 on the default branch and 99 on another branch
• When werkrevisor reviews a PR into the other branch
• Then the limit of 11 files applies

Verified by

• test_config_layers: test_the_repository_config_comes_from_the_default_branch

The Solution

• [path_judge] gets scope: "judged_paths", the default, or "all", every changed file outside the docs, the mechanical files and the hard paths.
• With scope = "all", Jev answers a third question, security, beside weakens_checks and build_risk; build_risk counts only for judged paths, since it did not separate main code.
• A new threshold in [path_judge], yellow_security_min (0.93); orange_weakens_min (0.8) also applies to other files.
• With scope = "all", a main-code file gets the questions weakens_checks and security, a test file only weakens_checks, since a test that names a role tests the access rule instead of changing it.
• [engine] model_role, host only: "decides", the default and the behavior before this PR, or "advises": with Jev's judgment of every file, model reasons below ORANGE become hints, except a security concern. With [engine] mode = "classifier", there is no model, and Jev's judgment of all files suffices for GREEN.
• The repository config replaces [hard_yellow] paths and judged_paths of the host instead of adding to them, and [db_schema] paths replaces the DB schema paths, which were built in; only .werkrevisor.toml stays built in.
• werkrevisor reads .werkrevisor.toml at the head of the default branch, no longer at the base commit of the PR.
• .werkrevisor.toml and the example configs of werkator and werkrevisor keep only secrets hard, and for werkator its isolation paths; Jev judges the build, image and CI files as judged paths and every other file through scope = "all". config/werkprobe.toml stays unchanged, since the tests build on it; the example configs of werkator and werkrevisor set model_role = "advises", config/hsadmin-ng.toml keeps "decides". The machine configs of the three repositories on mih09 get the same change at the deployment.
• docs/anleitung-dev.hostsharing.net.md names the new switches for hsadmin-ng, which keeps "decides" until the team decides.
• werkrevisor 0.21.0.

Measurement

On 2026-10-06, the code of this PR reviewed the 37 merged PRs of git.javagil.de with a final werkrevisor result in [engine] mode = "classifier", without a model, with the configs of this PR; Jev cost less than 0.01 USD. The gate reasons self-approval and "never marked ready" are left out, since they belong to the gate, not to the triage.

result	before, as reviewed	with this PR
GREEN	6	20
YELLOW or ORANGE	31	17

Of the 17, 13 changed .werkrevisor.toml, 8 of them only that; 3 had a scenario that Jev rated as unclear, 3 a change that touches security according to Jev, 2 lost tests, 2 a binary file Jev cannot read, 2 a risky build change, and #9 weakened its tests (ORANGE). Among the GREEN, #25 made the statistics public, which the user had asked for; Jev rated its files up to 0.90 for security, below the threshold of 0.93. The user decided that .werkrevisor.toml stays hard.

Follow-up PRs

• A review specification in one place, as a DSL in the style of RbacSpec in hsadmin-ng: under which conditions which workflow runs, which questions Jev answers and which values give which verdict. The switches of this PR, [path_judge] scope, [engine] mode and [engine] model_role, are its first building blocks.

Prerequisite PRs

• PR#38: the guide for dev.hostsharing.net names the state of werkrevisor 0.20.6, merged.