werkrevisor/docs/prs/2026-10-05-PR#10-jev-low-confidence-ignored.md

PR#10: Jev answers with low confidence do not count for the verdict

WARNING: This document describes only the change applied in this PR. It may already be outdated once the next PR is merged. Historic PR documentation is not maintained along with new PRs; treat it as a snapshot.

The Problem

In werkrevisor 0.11.0, a choice or score answer below [classifier] min_confidence gives YELLOW.

tools/jev-eval with config/hsadmin-ng.toml measured the 69 hsadmin-ng PR documents of 2026, one run per document: 281 units, 682 answers, cost 0.023 USD. 74 answers were YELLOW, and no answer reached red_above.

58 of the 74 YELLOW answers were YELLOW only because of a low confidence (min_confidence = 0.3). Their problem values were typically 0.24 to 0.45, their confidence 0.00 to 0.29. Most of them came from short sections like "Release Note" or "Folge-PRs". Such an answer tells nothing about the unit, but it stops the PR for a human review.

16 answers were YELLOW because a problem value was above yellow_above, spread over 8 of 69 documents. Spot checks showed that these findings were justified, e.g. Scenario#270.03 with more than one aspect, a PR#247 scenario with implementation details, and Scenario#306.00 with a vague Then.

The problem values per question:

Question	Median	p90	Maximum
hard_to_understand	0.07	0.22	0.51
slop	0.05	0.18	0.62
hard_to_read	0.20	0.48	0.63
more_than_one_aspect	0.20	0.51	0.82
implementation_details	0.17	0.61	0.72
given_when_then_misused	0.10	0.35	0.50
vague_then	0.06	0.14	0.77

Non-Goals

• No change of yellow_above or red_above: the 16 findings above yellow_above were justified.
• No change of min_confidence = 0.3 in the example configs.
• No change for noul answers: they have no confidence. The example configs ask more_than_one_aspect, implementation_details, given_when_then_misused and vague_then as noul, so the change applies to hard_to_read, hard_to_understand and slop.

The Scenarios

Feature: answers with low confidence do not count for the verdict

Background

• The config is config/werkprobe.toml with [classifier] min_confidence = 0.3.
• The PR changes only a PR document.

Scenario#10.01: an answer with low confidence gives no reason and no YELLOW

So that an uncertain answer does not stop a PR for a human review.

• Given the PR document has 2 scenarios and 4 sections
  • and Jev answers hard_to_read for each of the 2 scenarios with the problem value 0.9 and the confidence 0.1
  • and all other answers are below yellow_above with enough confidence
• When werkrevisor reviews the PR in mode classifier
• Then the verdict is GREEN
  • and there is no reason with hard_to_read
  • and the comment says 2 answers ignored because of a confidence below [classifier] min_confidence.
  • and the usage record has classifier_ignored = 2

Verified by

• test_classifier.ClassifierFlowTest.test_low_confidence_gives_no_reason_and_no_yellow
• test_classifier.JudgeUnitsTest.test_low_confidence_answers_are_ignored

Scenario#10.02: in mode combined the LLM gets an answer with low confidence as an uncertain hint

So that the LLM can still check the unit, while it knows that the classifier was not sure.

• Given Jev answers hard_to_read for Scenario#900.01 with the problem value 0.7 and the confidence 0.1
• When werkrevisor reviews the PR in mode combined
• Then the hint list for the LLM contains `doc/PR/2026-10-01-PR#900-mandate-notice.md` line 20, Scenario#900.01: hard to read, 0.70 (uncertain, confidence 0.10)
  • and the hint list says that an uncertain answer does not count for the verdict
  • and the verdict is GREEN

Verified by

• test_classifier.ClassifierFlowTest.test_combined_passes_low_confidence_answers_as_uncertain_hints

Scenario#10.03: jev-eval shows an answer with low confidence as IGNORED

So that a calibration run shows the ignored answers and counts them apart from the verdicts.

• Given Jev answers hard_to_read for Scenario#901.02 with the problem value 0.9 and the confidence 0.1
• When tools/jev-eval judges the PR document
• Then the row of this answer ends with IGNORED (low confidence 0.10 < [classifier] min_confidence = 0.3)
  • and the totals count it as IGNORED 1, not as GREEN or YELLOW

Verified by

• test_classifier.JevEvalTest.test_eval_shows_low_confidence_answers_as_ignored

Scenario#10.04: the thresholds apply as before to answers with enough confidence

So that a clear finding still gives YELLOW or ORANGE.

• Given a score question with yellow_above = 0.5 and red_above = 0.8
• When Jev answers with the problem value <problem> and the confidence <confidence>
• Then the verdict of the answer is <verdict>

problem	confidence	verdict
0.90	0.90	ORANGE 0.90 > red_above = 0.8
0.60	0.90	YELLOW 0.60 > yellow_above = 0.5
0.50	0.90	GREEN
0.90	0.20	ignored low confidence 0.20 < [classifier] min_confidence = 0.3
0.81	none (noul)	ORANGE 0.81 > red_above = 0.8

Verified by

• test_classifier.MappingTest.test_rate

The Solution

• rate returns no verdict for a choice or score answer below [classifier] min_confidence. The row stays GREEN for the verdict and gets the mark ignored with the note of the confidence.
• The hint list in mode combined contains the flagged and the ignored answers, as before 0.12.0; an ignored answer shows uncertain, confidence x instead of a verdict, and the note of the list explains it.
• The comment table counts the ignored answers; the usage record has the new field classifier_ignored.
• tools/jev-eval shows IGNORED (low confidence …) and the totals have IGNORED n next to GREEN, YELLOW and ORANGE.
• The README, the comments in the example configs, docs/konzept-jev-classifier.md and docs/betrieb-git.javagil.de.md describe the change; werkrevisor 0.12.0.

Open Questions

• Which answers with low confidence go into the hint list? All of them, as before 0.12.0, when they were YELLOW; the hint marks them as uncertain.

Additional Changes

• tools/jev-eval no longer cuts the verdict column at 60 characters, so the note shows the full config key and value.
• FakeJev in the tests accepts a confidence per unit and question.