93 lines
3.8 KiB
Markdown
93 lines
3.8 KiB
Markdown
# PR#53: a high risk and a confirmed finding of the same category give RED
|
|
|
|
> **WARNING:** This document describes only the change applied in this PR.
|
|
> It may already be outdated once the next PR is merged.
|
|
> Historic PR documentation is not maintained along with new PRs; treat it as a snapshot.
|
|
|
|
## The Problem
|
|
|
|
The agreed verdict rule gives RED for a very likely high risk and a concrete issue in the same file or category.
|
|
Since PR#51, werkrevisor gives RED for both in the same file.
|
|
A finding often lies in another file than the risk it belongs to,
|
|
e.g. a missing ownership check in a controller, while Jev rates the repository for object-level authorization.
|
|
Such a PR stays ORANGE, and a human can approve it.
|
|
|
|
## Non-Goals
|
|
|
|
- No limit of the correction rounds after RED yet.
|
|
- No category for findings in test code: there, Jev assigned 3 of 4 findings to a wrong category.
|
|
- No category for a contradiction of the PR document.
|
|
|
|
## The Scenarios
|
|
|
|
### Feature: a high risk and a confirmed finding of the same category give RED
|
|
|
|
#### Background
|
|
|
|
- **Given** Jev judges every changed file with the risk categories of a template
|
|
- **and** Jev judges the findings of the model, and also assigns each to a category of the template
|
|
|
|
#### Scenario#53.01: a very likely risk and a confirmed finding of the same category give RED
|
|
|
|
So that the author lowers the risk with a new commit before a human spends time on the PR.
|
|
|
|
- **Given** Jev rates a file at or above `orange_min` of a risk category
|
|
- **When** the model reports a finding in another main-code file, Jev confirms it,
|
|
and assigns it to the same category with a confidence of at least 0.7
|
|
- **Then** the review is RED
|
|
- **and** the reason names the category, the risky file and the finding
|
|
|
|
##### Verified by
|
|
|
|
- [test_risk_with_issue: test_a_very_likely_risk_and_a_confirmed_finding_of_the_same_category_give_red](../../tests/test_risk_with_issue.py)
|
|
|
|
#### Scenario#53.02: an uncertain category gives no RED
|
|
|
|
So that a vague assignment does not block the PR.
|
|
|
|
- **When** Jev assigns the confirmed finding to the same category with a confidence below 0.7
|
|
- **Then** the review is ORANGE
|
|
|
|
##### Verified by
|
|
|
|
- [test_risk_with_issue: test_an_uncertain_category_gives_no_red](../../tests/test_risk_with_issue.py)
|
|
|
|
#### Scenario#53.03: a finding in test code gives no RED by its category
|
|
|
|
So that a finding about a test does not count as the risk of the code under test.
|
|
|
|
- **When** the confirmed finding lies in a test file, and Jev assigns it to the same category
|
|
- **Then** the review is ORANGE
|
|
|
|
##### Verified by
|
|
|
|
- [test_risk_with_issue: test_a_finding_in_test_code_gives_no_red_by_its_category](../../tests/test_risk_with_issue.py)
|
|
|
|
## The Solution
|
|
|
|
- With a risk template, the finding judge asks one more question: a Choice over the categories of the template and "none".
|
|
- The finding keeps the category and its confidence; the usage record logs both.
|
|
- The reason `risk_with_issue` also covers a confirmed finding in main code whose category is very likely risky in any file.
|
|
- The README explains the rule; docs/werkrevisor.werkbaum marks the node as merged.
|
|
- werkrevisor 0.25.0.
|
|
|
|
## Measurement
|
|
|
|
On 2026-10-08 Jev assigned 32 findings of the model from earlier reviews of hs.hsadmin.ng to the 15 categories
|
|
of `jvm-spring-backend` or "none"; the expected categories were set by hand before, with one or two accepted per finding.
|
|
The state held title, location and text of the finding, without the code excerpt.
|
|
|
|
| findings | correct |
|
|
|---|---|
|
|
| all | 28 of 32 |
|
|
| with a risk category | 11 of 12 |
|
|
| confidence from 0.7 on | 23 of 24 |
|
|
| confidence from 0.7 on, outside test code | 21 of 21 |
|
|
|
|
3 of the 4 wrong ones were about test code, e.g. a report index written by a shutdown hook, assigned to concurrency.
|
|
One person set the expected categories, and 32 findings are a small sample.
|
|
|
|
## Prerequisite PRs
|
|
|
|
- PR#52: the plan as a Werkbaum, merged.
|