werkrevisor/docs/prs/2026-10-08-PR#46-risk-categories.md

PR#46: Jev judges each changed file by the risk categories of a template

WARNING: This document describes only the change applied in this PR. It may already be outdated once the next PR is merged. Historic PR documentation is not maintained along with new PRs; treat it as a snapshot.

The Problem

werkrevisor shall separate the PRs that need a human from the others, so that the others merge without a human; finding defects is secondary. With [path_judge] scope = "all", Jev asks two questions per changed file: does it weaken the checks, and does it touch security. On the 25 merged hs.hsadmin.ng PRs #281 to #316, these found 7 of the 9 PRs that needed a human, 8 in the run of this PR.

Two questions can miss risks outside security, e.g. a changed transaction boundary, a migration of existing data, or a request to a third party in a frontend. The reason "touches security" also does not tell the developer what to check or how to lower the risk. And which risks matter depends on the ecosystem of the repository: a Spring Boot backend has other ones than a React frontend.

The design of 2026-10-08 defines 15 risk categories for JVM backends and 15 for TypeScript frontends. Later each repository keeps its categories in its .werkrevisor.toml; until then werkrevisor keeps them as templates.

Non-Goals

• No change summary of an economy model and no comparison of the PR document with it; that is the next PR.
• No PR-wide classification over all files, and no RED from Jev.
• No categories in .werkrevisor.toml yet; the host config selects the template.
• No grouping of related changes across files: a unit is still one file with its diff hunks.
• No change of the questions for judged build paths, of weakens_checks and security, or of the PR document questions.
• No change of a deployed host config: the reviewed repositories on git.javagil.de are Python, and hs.hsadmin.ng waits.

The Scenarios

Feature: Jev judges each changed file by the risk categories of a template

Background

• Given [path_judge] judge = "jev" and scope = "all"
  • and [path_judge] risk_template names a template of werkrevisor

Scenario#46.01: a likely risk gives YELLOW and names the category

So that the developer knows what a human checks, and can revert a change that the task does not need.

• Given a category with action = "yellow"
• When Jev rates a changed file in this category at or above its yellow_min
• Then the review is YELLOW
  • and the reason names the category, the file and the value
  • and the reason says to revert the change if the task does not need it

Verified by

• test_risk: test_a_likely_risk_gives_yellow_and_names_the_category

Scenario#46.02: a quality category gives only a hint

So that categories like performance or accessibility inform the author without holding the merge for a human.

• Given a category with action = "hint"
• When Jev rates a changed file in this category at or above its yellow_min
• Then the category is a hint with the file and the value
  • and it alone keeps the review GREEN

Verified by

• test_risk: test_a_quality_category_gives_only_a_hint

Scenario#46.03: a very likely risk gives ORANGE

So that the author lowers the risk with further commits before a human spends time on the PR.

• Given a category with action = "yellow" and an orange_min
• When Jev rates a changed file in this category at or above its orange_min
• Then the review is ORANGE
  • and the reason names the category, the file and the value

Verified by

• test_risk: test_a_very_likely_risk_gives_orange

Scenario#46.04: a test file gets only the categories for tests

So that a test that names a role does not count as a change of the access rule.

• Given a category without test in its kinds
• When a PR changes a test file
• Then Jev does not answer this category for the test file

Verified by

• test_risk: test_a_test_file_gets_only_the_categories_for_tests

Scenario#46.05: the question about security stays besides the categories

So that the triage finds at least the PRs it found before: the narrower categories alone missed 2 critical PRs.

• When Jev rates a main-code file at or above yellow_security_min for security
• Then the review is YELLOW with the reason files_touch_security, as before

Verified by

• test_risk: test_the_security_question_stays_besides_the_categories

Scenario#46.06: without a template, the two measured questions stay

So that the repositories without a fitting template, e.g. Python ones, keep their calibrated triage.

• Given no [path_judge] risk_template
• When a PR changes a main code file
• Then Jev answers weakens_checks and security, as before

Verified by

• test_risk: test_without_a_template_the_two_measured_questions_stay

Scenario#46.07: an unknown template is a config error

So that a typo in the config does not silently disable the categories.

• Given [path_judge] risk_template = "no-such-template"
• When werkrevisor loads the config
• Then the run ends with a config error that names the known templates

Verified by

• test_risk: test_an_unknown_template_is_a_config_error

Scenario#46.08: the usage record logs each category value

So that the thresholds can be calibrated on real PRs.

• Given Jev judged a changed file by the categories of a template
• When werkrevisor writes the usage record
• Then the record holds the value of each category for this file

Verified by

• test_risk: test_the_usage_record_logs_each_category_value

The Solution

• A new host-only key [path_judge] risk_template; it needs scope = "all".
• The templates are TOML files in werkrevisor/risk/: jvm-spring-backend and typescript-frontend, 15 categories each.
• Each category has name, title, a noul question, kinds (main, test, default main), yellow_min, an optional orange_min, and action (yellow or hint).
• The categories come on top of weakens_checks and security; the reason risk_<category> names category, files and values.
• REST API compatibility, performance and architecture, and in the frontend also accessibility, start with action = "hint": agreed changes, e.g. a renamed API property, need no human.
• All categories start with yellow_min = 0.95 and, unless a hint, orange_min = 0.99.
• Jev answers all questions of a file in one request, so the cost depends on the diff size, not on the number of categories.
• werkrevisor 0.22.0.

Measurement

On 2026-10-08, Jev judged each changed file outside the docs of the 25 merged hs.hsadmin.ng PRs #281 to #316, with the questions of this PR; a file below src/test/ counted as a test. The labels are those of 2026-10-06: 9 PRs need a human, #313 is borderline, 15 do not. Each run cost about 0.03 USD for 650,000 input tokens.

questions	critical found	others YELLOW or worse
weakens_checks and the categories, without security	6 of 9	0 of 15
weakens_checks and security	8 of 9	1 of 15 (#302)
with the categories from 0.93	8 of 9	3 of 15
with the categories from 0.95	8 of 9	1 of 15

Without security, #281 and #288 were lost: authorization reached only 0.92 and 0.58 there. From 0.95 on, the categories find no further PR, but they name the risk: #283 authentication and concurrency, #289 also authorization, #306 authentication and authorization, and #310 authorization and schema migration. They also mark the borderline #313 for error handling. #282 stays missed; its highest category was concurrency with 0.82. #302 reached 0.93 for security in this run, 0.92 on 2026-10-06. No category reached 0.99, so orange_min is not calibrated yet.

Follow-up PRs

• An economy model summarizes the diff without the PR document; Jev compares the PR document with this summary. Its contradictions are the first concrete issues.
• A PR-wide classification over the consolidated signals; ORANGE for a likely high risk or a concrete issue, RED for a likely high risk and a concrete issue in the same file or category.
• The categories move into .werkrevisor.toml of each repository.

Prerequisite PRs

• PR#45: werkrevisor 0.21.5, merged.