claude-code
fcab52151f
werkator build succeeded after 00:25
werkrevisor 0.12.0: Jev answers with low confidence no longer give YELLOW. Calibration on 69 hsadmin-ng PR documents: 58 of 74 YELLOW answers came only from low confidence, not from a problem value; the 16 threshold hits were justified in spot checks. Thresholds stay. Details: the PR document under `docs/prs/`. 🤖 Generated with [Claude Code](https://claude.com/claude-code) Reviewed-on: #10 Co-authored-by: claude-code <claude-code@javagil.de>
141 lines
6.6 KiB
Markdown
141 lines
6.6 KiB
Markdown
# PR#10: Jev answers with low confidence do not count for the verdict
|
|
|
|
> **WARNING:** This document describes only the change applied in this PR.
|
|
> It may already be outdated once the next PR is merged.
|
|
> Historic PR documentation is not maintained along with new PRs; treat it as a snapshot.
|
|
|
|
## The Problem
|
|
|
|
In werkrevisor 0.11.0, a choice or score answer below `[classifier] min_confidence` gives YELLOW.
|
|
|
|
`tools/jev-eval` with `config/hsadmin-ng.toml` measured the 69 hsadmin-ng PR documents of 2026, one run per document:
|
|
281 units, 682 answers, cost 0.023 USD.
|
|
74 answers were YELLOW, and no answer reached `red_above`.
|
|
|
|
58 of the 74 YELLOW answers were YELLOW only because of a low confidence (`min_confidence = 0.3`).
|
|
Their problem values were typically 0.24 to 0.45, their confidence 0.00 to 0.29.
|
|
Most of them came from short sections like "Release Note" or "Folge-PRs".
|
|
Such an answer tells nothing about the unit, but it stops the PR for a human review.
|
|
|
|
16 answers were YELLOW because a problem value was above `yellow_above`, spread over 8 of 69 documents.
|
|
Spot checks showed that these findings were justified,
|
|
e.g. Scenario#270.03 with more than one aspect, a PR#247 scenario with implementation details,
|
|
and Scenario#306.00 with a vague Then.
|
|
|
|
The problem values per question:
|
|
|
|
| Question | Median | p90 | Maximum |
|
|
|---|---|---|---|
|
|
| `hard_to_understand` | 0.07 | 0.22 | 0.51 |
|
|
| `slop` | 0.05 | 0.18 | 0.62 |
|
|
| `hard_to_read` | 0.20 | 0.48 | 0.63 |
|
|
| `more_than_one_aspect` | 0.20 | 0.51 | 0.82 |
|
|
| `implementation_details` | 0.17 | 0.61 | 0.72 |
|
|
| `given_when_then_misused` | 0.10 | 0.35 | 0.50 |
|
|
| `vague_then` | 0.06 | 0.14 | 0.77 |
|
|
|
|
## Non-Goals
|
|
|
|
- No change of `yellow_above` or `red_above`: the 16 findings above `yellow_above` were justified.
|
|
- No change of `min_confidence = 0.3` in the example configs.
|
|
- No change for noul answers: they have no confidence.
|
|
The example configs ask `more_than_one_aspect`, `implementation_details`, `given_when_then_misused` and `vague_then` as noul,
|
|
so the change applies to `hard_to_read`, `hard_to_understand` and `slop`.
|
|
|
|
## The Scenarios
|
|
|
|
### Feature: answers with low confidence do not count for the verdict
|
|
|
|
#### Background
|
|
|
|
- The config is `config/werkprobe.toml` with `[classifier] min_confidence = 0.3`.
|
|
- The PR changes only a PR document.
|
|
|
|
#### Scenario#10.01: an answer with low confidence gives no reason and no YELLOW
|
|
|
|
So that an uncertain answer does not stop a PR for a human review.
|
|
|
|
- **Given** the PR document has 2 scenarios and 4 sections
|
|
- **and** Jev answers `hard_to_read` for each of the 2 scenarios with the problem value 0.9 and the confidence 0.1
|
|
- **and** all other answers are below `yellow_above` with enough confidence
|
|
- **When** werkrevisor reviews the PR in mode `classifier`
|
|
- **Then** the verdict is GREEN
|
|
- **and** there is no reason with `hard_to_read`
|
|
- **and** the comment says `2 answers ignored because of a confidence below [classifier] min_confidence.`
|
|
- **and** the usage record has `classifier_ignored = 2`
|
|
|
|
##### Verified by
|
|
|
|
- [test_classifier.ClassifierFlowTest.test_low_confidence_gives_no_reason_and_no_yellow](../../tests/test_classifier.py)
|
|
- [test_classifier.JudgeUnitsTest.test_low_confidence_answers_are_ignored](../../tests/test_classifier.py)
|
|
|
|
#### Scenario#10.02: in mode combined the LLM gets an answer with low confidence as an uncertain hint
|
|
|
|
So that the LLM can still check the unit, while it knows that the classifier was not sure.
|
|
|
|
- **Given** Jev answers `hard_to_read` for Scenario#900.01 with the problem value 0.7 and the confidence 0.1
|
|
- **When** werkrevisor reviews the PR in mode `combined`
|
|
- **Then** the hint list for the LLM contains
|
|
`` `doc/PR/2026-10-01-PR#900-mandate-notice.md` line 20, Scenario#900.01: hard to read, 0.70 (uncertain, confidence 0.10) ``
|
|
- **and** the hint list says that an uncertain answer does not count for the verdict
|
|
- **and** the verdict is GREEN
|
|
|
|
##### Verified by
|
|
|
|
- [test_classifier.ClassifierFlowTest.test_combined_passes_low_confidence_answers_as_uncertain_hints](../../tests/test_classifier.py)
|
|
|
|
#### Scenario#10.03: jev-eval shows an answer with low confidence as IGNORED
|
|
|
|
So that a calibration run shows the ignored answers and counts them apart from the verdicts.
|
|
|
|
- **Given** Jev answers `hard_to_read` for Scenario#901.02 with the problem value 0.9 and the confidence 0.1
|
|
- **When** `tools/jev-eval` judges the PR document
|
|
- **Then** the row of this answer ends with `IGNORED (low confidence 0.10 < [classifier] min_confidence = 0.3)`
|
|
- **and** the totals count it as `IGNORED 1`, not as GREEN or YELLOW
|
|
|
|
##### Verified by
|
|
|
|
- [test_classifier.JevEvalTest.test_eval_shows_low_confidence_answers_as_ignored](../../tests/test_classifier.py)
|
|
|
|
#### Scenario#10.04: the thresholds apply as before to answers with enough confidence
|
|
|
|
So that a clear finding still gives YELLOW or ORANGE.
|
|
|
|
- **Given** a score question with `yellow_above = 0.5` and `red_above = 0.8`
|
|
- **When** Jev answers with the problem value `<problem>` and the confidence `<confidence>`
|
|
- **Then** the verdict of the answer is `<verdict>`
|
|
|
|
| problem | confidence | verdict |
|
|
|---|---|---|
|
|
| 0.90 | 0.90 | ORANGE `0.90 > red_above = 0.8` |
|
|
| 0.60 | 0.90 | YELLOW `0.60 > yellow_above = 0.5` |
|
|
| 0.50 | 0.90 | GREEN |
|
|
| 0.90 | 0.20 | ignored `low confidence 0.20 < [classifier] min_confidence = 0.3` |
|
|
| 0.81 | none (noul) | ORANGE `0.81 > red_above = 0.8` |
|
|
|
|
##### Verified by
|
|
|
|
- [test_classifier.MappingTest.test_rate](../../tests/test_classifier.py)
|
|
|
|
## The Solution
|
|
|
|
- `rate` returns no verdict for a choice or score answer below `[classifier] min_confidence`.
|
|
The row stays GREEN for the verdict and gets the mark `ignored` with the note of the confidence.
|
|
- The hint list in mode `combined` contains the flagged and the ignored answers, as before 0.12.0;
|
|
an ignored answer shows `uncertain, confidence x` instead of a verdict, and the note of the list explains it.
|
|
- The comment table counts the ignored answers; the usage record has the new field `classifier_ignored`.
|
|
- `tools/jev-eval` shows `IGNORED (low confidence …)` and the totals have `IGNORED n` next to GREEN, YELLOW and ORANGE.
|
|
- The README, the comments in the example configs, `docs/konzept-jev-classifier.md`
|
|
and `docs/betrieb-git.javagil.de.md` describe the change; werkrevisor 0.12.0.
|
|
|
|
## Open Questions
|
|
|
|
- [x] Which answers with low confidence go into the hint list?
|
|
All of them, as before 0.12.0, when they were YELLOW; the hint marks them as uncertain.
|
|
|
|
## Additional Changes
|
|
|
|
- `tools/jev-eval` no longer cuts the verdict column at 60 characters,
|
|
so the note shows the full config key and value.
|
|
- `FakeJev` in the tests accepts a confidence per unit and question.
|