werkrevisor 0.12.0: Jev answers with low confidence are ignored #10

2026-10-05T18:49:28+02:00

claude-code commented

2026-10-05 18:49:28 +02:00

werkrevisor 0.12.0: Jev answers with low confidence no longer give YELLOW.

Calibration on 69 hsadmin-ng PR documents: 58 of 74 YELLOW answers came only from low confidence, not from a problem value; the 16 threshold hits were justified in spot checks. Thresholds stay.

Details: the PR document under docs/prs/.

🤖 Generated with Claude Code

werkrevisor 0.12.0: Jev answers with low confidence no longer give YELLOW. Calibration on 69 hsadmin-ng PR documents: 58 of 74 YELLOW answers came only from low confidence, not from a problem value; the 16 threshold hits were justified in spot checks. Thresholds stay. Details: the PR document under `docs/prs/`. 🤖 Generated with [Claude Code](https://claude.com/claude-code)

claude-code added 3 commits 2026-10-05 18:49:28 +02:00

docs: PR#10 document for ignoring Jev answers with low confidence 2de23be050

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

Jev answers with low confidence no longer give YELLOW 2ac37e656e

An answer below [classifier] min_confidence is ignored for the verdict.
In mode combined the LLM still gets it as a hint, marked as uncertain.
jev-eval shows it as IGNORED and counts it separately; the comment and the usage record count it too.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

docs: werkrevisor 0.12.0 ignores Jev answers with low confidence, calibration result

werkator build succeeded after 00:23

Details

ai-review/assessment AI findings; a developer approval after checking them unlocks the merge; review requested from mi

Details

ai-review/gate build succeeded after 00:32

Details

dc162d28d1

README, example config comments, the Jev concept and the operations doc describe the change.
The PR#10 document has the calibration numbers and the solution.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

claude-code marked the pull request as ready for review 2026-10-05 18:49:29 +02:00

ai-reviewer commented

2026-10-05 18:50:09 +02:00

🟠 werkrevisor: ORANGE / FINDINGS

Change kind: mixed.

Reasons

YELLOW test changes are not purely additive: tests/fakes.py:212 changes an existing test line, tests/fakes.py:214 changes an existing test line, tests/fakes.py:230 changes an existing test line and 7 more
YELLOW change kind 'mixed' (main code, tests, other files, docs); AI-approvable are only: docs only, tests only added, docs and tests, refactoring with unchanged tests, feature with purely additive tests, harmless files only
YELLOW risky paths changed: .werkrevisor.toml (built in), config/hsadmin-ng.toml (config/), config/werkprobe.toml (config/)
ORANGE model reports 2 key issues ([review] red_min_findings = 2)
YELLOW model recommends merge with caution

Findings

Wrong test reference at docs/prs/2026-10-05-PR#10-jev-low-confidence-ignored.md:115-118

Scenario#10.03 lists test_classifier.JevEvalTest.test_eval_shows_low_confidence_answers_as_ignored as its test, and the PR document points to tests/test_classifier.py for it. The test exists there as JevEvalTest, so the reference is valid. However, Scenario#10.04 claims test_spec_problem_is_orange_without_llm_finding verifies the threshold table. That test only checks a noul ORANGE case and does not cover the table rows for YELLOW, GREEN or ignored. The ignored row is covered only by test_rate. This is a minor traceability inaccuracy.
Doc contradicts code at docs/prs/2026-10-05-PR#10-jev-low-confidence-ignored.md:58-64

Scenario#10.01 says the comment shows 2 answers ignored ..., but the Background sets min_confidence for config/werkprobe.toml and the scenario says "for all scenarios". Jev's hard_to_read is a scenario question, and the fixture has 4 scenarios, yet the test and the doc expect 2 ignored answers (the test fixture only judges 2 scenarios with this question). The count of 2 is not derivable from the scenario text, so the stated outcome looks inconsistent with "for all scenarios". The test in test_low_confidence_answers_are_ignored expects 6 ignored answers for hard_to_read plus slop across the fixture, which suggests a different scenario count than 2 in the flow test.

_{Usage: model `openrouter/anthropic/claude-sonnet-5.5`, LLM done, tokens counted before the call 49841, calls 1, tokens in 65792 / out 735 (reasoning included in out), duration 28.6 s, cost 0.1389 USD (reported), config: defaults + host + repository (.werkrevisor.toml at base 488b149).

Reviewed head `dc162d28d135b4a350545e2fa6f06d59d2e9cf2b` against base `488b1498a594cfadc0327d3ffd8a77ad3f9d4f1c`, werkrevisor 0.11.0.}

## 🟠 werkrevisor: ORANGE / FINDINGS Change kind: `mixed`. ### Reasons - **YELLOW** test changes are not purely additive: tests/fakes.py:212 changes an existing test line, tests/fakes.py:214 changes an existing test line, tests/fakes.py:230 changes an existing test line and 7 more - **YELLOW** change kind 'mixed' (main code, tests, other files, docs); AI-approvable are only: docs only, tests only added, docs and tests, refactoring with unchanged tests, feature with purely additive tests, harmless files only - **YELLOW** risky paths changed: .werkrevisor.toml (built in), config/hsadmin-ng.toml (config/**), config/werkprobe.toml (config/**) - **ORANGE** model reports 2 key issues ([review] red_min_findings = 2) - **YELLOW** model recommends merge with caution ### Findings 1. **Wrong test reference** at `docs/prs/2026-10-05-PR#10-jev-low-confidence-ignored.md:115-118` Scenario#10.03 lists `test_classifier.JevEvalTest.test_eval_shows_low_confidence_answers_as_ignored` as its test, and the PR document points to `tests/test_classifier.py` for it. The test exists there as `JevEvalTest`, so the reference is valid. However, Scenario#10.04 claims `test_spec_problem_is_orange_without_llm_finding` verifies the threshold table. That test only checks a noul ORANGE case and does not cover the table rows for YELLOW, GREEN or ignored. The ignored row is covered only by `test_rate`. This is a minor traceability inaccuracy. 2. **Doc contradicts code** at `docs/prs/2026-10-05-PR#10-jev-low-confidence-ignored.md:58-64` Scenario#10.01 says the comment shows `2 answers ignored ...`, but the Background sets `min_confidence` for `config/werkprobe.toml` and the scenario says "for all scenarios". Jev's `hard_to_read` is a scenario question, and the fixture has 4 scenarios, yet the test and the doc expect 2 ignored answers (the test fixture only judges 2 scenarios with this question). The count of 2 is not derivable from the scenario text, so the stated outcome looks inconsistent with "for all scenarios". The test in `test_low_confidence_answers_are_ignored` expects 6 ignored answers for `hard_to_read` plus `slop` across the fixture, which suggests a different scenario count than 2 in the flow test. Usage: model `openrouter/anthropic/claude-sonnet-5.5`, LLM done, tokens counted before the call 49841, calls 1, tokens in 65792 / out 735 (reasoning included in out), duration 28.6 s, cost 0.1389 USD (reported), config: defaults + host + repository (.werkrevisor.toml at base 488b149). Reviewed head `dc162d28d135b4a350545e2fa6f06d59d2e9cf2b` against base `488b1498a594cfadc0327d3ffd8a77ad3f9d4f1c`, werkrevisor 0.11.0.

ai-reviewer requested review from mi 2026-10-05 18:50:09 +02:00

claude-code added 1 commit 2026-10-05 18:51:33 +02:00

review: PR#10 scenarios state their data and link the tests that verify them

werkator build succeeded after 00:05

Details

ai-review/assessment human review required; a human approval unlocks the merge; a human review request is pending

Details

ai-review/gate build succeeded after 00:11

Details

ffa9813e32

Scenario#10.01 names the 2 scenarios and 4 sections, so the count of 2 ignored answers follows; the test checks this shape.
Scenario#10.04 links only test_rate, which covers every row of the table.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

ai-reviewer commented

2026-10-05 18:51:52 +02:00

🟡 werkrevisor: YELLOW / HUMAN_REVIEW

Change kind: mixed.

Reasons

YELLOW test changes are not purely additive: tests/fakes.py:212 changes an existing test line, tests/fakes.py:214 changes an existing test line, tests/fakes.py:230 changes an existing test line and 7 more
YELLOW change kind 'mixed' (main code, tests, other files, docs); AI-approvable are only: docs only, tests only added, docs and tests, refactoring with unchanged tests, feature with purely additive tests, harmless files only
YELLOW risky paths changed: .werkrevisor.toml (built in), config/hsadmin-ng.toml (config/), config/werkprobe.toml (config/)
YELLOW model recommends merge with caution
YELLOW model reports 1 key issue(s)

Findings

Wrong test reference at docs/prs/2026-10-05-PR#10-jev-low-confidence-ignored.md:67-70

Scenario#10.03 lists test_classifier.JevEvalTest.test_eval_shows_low_confidence_answers_as_ignored under "Verified by". The test is in the JevEvalTest class, which is in tests/test_classifier.py, so the path is plausible. However, the "Verified by" for Scenario#10.04 lists only MappingTest.test_rate, which does not check the noul case through a flow. This is minor. The more concrete problem is that Scenario#10.01 says "the 2 scenarios ... hard_to_read" produce 2 answers ignored, but the test also ignores slop-free sections only because hard_to_read is a scenario question. This matches, so no contradiction. The only open uncertainty is that the PR text says the 58 low-confidence YELLOW answers were "typically" 0.24 to 0.45, and the README claims they "named no real problem" while the PR document also reports that the problem values were up to 0.45; this is a soft claim and not a code contradiction.

_{Usage: model `openrouter/anthropic/claude-sonnet-5.5`, LLM done, tokens counted before the call 49975, calls 1, tokens in 65946 / out 522 (reasoning included in out), duration 10.0 s, cost 0.1371 USD (reported), config: defaults + host + repository (.werkrevisor.toml at base 488b149).

Reviewed head `ffa9813e32f3947d6569a31998cf946d205e3423` against base `488b1498a594cfadc0327d3ffd8a77ad3f9d4f1c`, werkrevisor 0.11.0.}

## 🟡 werkrevisor: YELLOW / HUMAN_REVIEW Change kind: `mixed`. ### Reasons - **YELLOW** test changes are not purely additive: tests/fakes.py:212 changes an existing test line, tests/fakes.py:214 changes an existing test line, tests/fakes.py:230 changes an existing test line and 7 more - **YELLOW** change kind 'mixed' (main code, tests, other files, docs); AI-approvable are only: docs only, tests only added, docs and tests, refactoring with unchanged tests, feature with purely additive tests, harmless files only - **YELLOW** risky paths changed: .werkrevisor.toml (built in), config/hsadmin-ng.toml (config/**), config/werkprobe.toml (config/**) - **YELLOW** model recommends merge with caution - **YELLOW** model reports 1 key issue(s) ### Findings 1. **Wrong test reference** at `docs/prs/2026-10-05-PR#10-jev-low-confidence-ignored.md:67-70` Scenario#10.03 lists `test_classifier.JevEvalTest.test_eval_shows_low_confidence_answers_as_ignored` under "Verified by". The test is in the `JevEvalTest` class, which is in `tests/test_classifier.py`, so the path is plausible. However, the "Verified by" for Scenario#10.04 lists only `MappingTest.test_rate`, which does not check the noul case through a flow. This is minor. The more concrete problem is that Scenario#10.01 says "the 2 scenarios ... `hard_to_read`" produce `2 answers ignored`, but the test also ignores `slop`-free sections only because `hard_to_read` is a scenario question. This matches, so no contradiction. The only open uncertainty is that the PR text says the 58 low-confidence YELLOW answers were "typically" 0.24 to 0.45, and the README claims they "named no real problem" while the PR document also reports that the problem values were up to 0.45; this is a soft claim and not a code contradiction. Usage: model `openrouter/anthropic/claude-sonnet-5.5`, LLM done, tokens counted before the call 49975, calls 1, tokens in 65946 / out 522 (reasoning included in out), duration 10.0 s, cost 0.1371 USD (reported), config: defaults + host + repository (.werkrevisor.toml at base 488b149). Reviewed head `ffa9813e32f3947d6569a31998cf946d205e3423` against base `488b1498a594cfadc0327d3ffd8a77ad3f9d4f1c`, werkrevisor 0.11.0.

mi merged commit fcab52151f into main

2026-10-05 18:56:39 +02:00

mi deleted branch claude-code/jev-low-confidence-ignored

2026-10-05 18:56:39 +02:00

Sign in to join this conversation.

2 Participants

Notifications

Due Date

No due date set.

Dependencies

No dependencies set.

Reference: mi/werkrevisor#10