ai-review/assessment AI findings; a developer approval after checking them unlocks the merge; review requested from mi; merge scheduled for when the checks succeed
YELLOW 1 changed file was given to the AI only as diff, not in full, so it could not check it against the rest of its file (all full files together would exceed [review] max_full_files_chars = 160000, and no excerpt fits the limits either): werkrevisor/rules.py
YELLOW 1 finding(s) possibly severe, but Jev doubts that they are real: werkrevisor/rules.py:258-260
YELLOW model rates the risk as high
YELLOW model reports a security concern
hint writing rules violated in the changed docs: docs/prs/2026-10-06-PR#39-jev-triage.md:119: line longer than 128 characters (135)
hint 1 build change(s) of low risk according to Jev: statistics/build.gradle.kts (weakens checks 0.02, risk 0.00)
hint 2 changed files were given to the AI as excerpt of the selected lines and 5 lines around each changed line, not in full (all full files together would exceed [review] max_full_files_chars = 160000): werkrevisor/config.py, werkrevisor/report.py
hint model recommends changes before merge
Findings
[high] Advisory downgrade hides unreviewed files at werkrevisor/cli.py:377-389, ORANGE, as a line comment
[high] triage_complete true when Jev judged nothing at werkrevisor/cli.py:307-312, ORANGE, as a line comment
[medium] Deleted main-code files skip the security question at werkrevisor/rules.py:258-260, YELLOW, as a line comment
Classifier
10 of 10 units passed all questions (1 PR document(s)).
4 answers ignored because of a confidence below [classifier] min_confidence.
Usage: model `openrouter/anthropic/claude-sonnet-5.5`, LLM done, tokens counted before the call 97981, calls 1, tokens in 96881 / out 1029 (reasoning included in out), duration 32.8 s, cost 0.2041 USD (reported), classifier `jev-1.13.0` calls 10, tokens in 8756, cost 0.0004 USD, finding judge `jev-1.13.0` calls 3, tokens in 5107, cost 0.0002 USD, excerpt selection `openrouter/openai/gpt-6-luna` calls 3, tokens in 25519 / out 796, cost 0.0036 USD (reported), build file judge `jev-1.13.0` calls 1, tokens in 560, cost 0.0000 USD, config: defaults + host + repository (.werkrevisor.toml at base c83d016).
Reviewed head `a57d442c74b08265b64dcfe49ee3d1cb69c55def` against base `c83d01690888bb1f8da62cc8bc4b3710308135bf`, werkrevisor 0.20.6.
<!-- werkrevisor:a57d442c74b08265b64dcfe49ee3d1cb69c55def -->
<!-- werkrevisor-verdict:ORANGE -->
<!-- werkrevisor-folded -->
<details><summary>🟠 werkrevisor ORANGE for <code>a57d442</code>, superseded by <a href="http://git.javagil.de/mi/werkrevisor/pulls/39#issuecomment-1663">the result for <code>d29946c</code></a></summary>
## 🟠 werkrevisor: ORANGE / FINDINGS
Change kind: `mixed`.
### Reasons
- **ORANGE** 2 finding(s) that Jev rates as probably real and severe: werkrevisor/cli.py:377-389, werkrevisor/cli.py:307-312
- **YELLOW** risky paths changed: .werkrevisor.toml (built in), werkrevisor/config.py (werkrevisor/config.py)
- **YELLOW** 1 changed file was given to the AI only as diff, not in full, so it could not check it against the rest of its file (all full files together would exceed [review] max_full_files_chars = 160000, and no excerpt fits the limits either): werkrevisor/rules.py
- **YELLOW** 1 finding(s) possibly severe, but Jev doubts that they are real: werkrevisor/rules.py:258-260
- **YELLOW** model rates the risk as high
- **YELLOW** model reports a security concern
- **hint** writing rules violated in the changed docs: docs/prs/2026-10-06-PR#39-jev-triage.md:119: line longer than 128 characters (135)
- **hint** 1 build change(s) of low risk according to Jev: statistics/build.gradle.kts (weakens checks 0.02, risk 0.00)
- **hint** 2 changed files were given to the AI as excerpt of the selected lines and 5 lines around each changed line, not in full (all full files together would exceed [review] max_full_files_chars = 160000): werkrevisor/config.py, werkrevisor/report.py
- **hint** model recommends changes before merge
### Findings
1. **[high] Advisory downgrade hides unreviewed files** at `werkrevisor/cli.py:377-389`, ORANGE, as a line comment
2. **[high] triage_complete true when Jev judged nothing** at `werkrevisor/cli.py:307-312`, ORANGE, as a line comment
3. **[medium] Deleted main-code files skip the security question** at `werkrevisor/rules.py:258-260`, YELLOW, as a line comment
### Classifier
10 of 10 units passed all questions (1 PR document(s)).
4 answers ignored because of a confidence below [classifier] min_confidence.
<sub>
Usage: model `openrouter/anthropic/claude-sonnet-5.5`, LLM done, tokens counted before the call 97981, calls 1, tokens in 96881 / out 1029 (reasoning included in out), duration 32.8 s, cost 0.2041 USD (reported), classifier `jev-1.13.0` calls 10, tokens in 8756, cost 0.0004 USD, finding judge `jev-1.13.0` calls 3, tokens in 5107, cost 0.0002 USD, excerpt selection `openrouter/openai/gpt-6-luna` calls 3, tokens in 25519 / out 796, cost 0.0036 USD (reported), build file judge `jev-1.13.0` calls 1, tokens in 560, cost 0.0000 USD, config: defaults + host + repository (.werkrevisor.toml at base c83d016).<br>
Reviewed head `a57d442c74b08265b64dcfe49ee3d1cb69c55def` against base `c83d01690888bb1f8da62cc8bc4b3710308135bf`, werkrevisor 0.20.6.
</sub>
</details>
werkrevisor: ORANGE / FINDINGS for a57d442c74b08265b64dcfe49ee3d1cb69c55def.
The result comment of werkrevisor holds the details.
werkrevisor: ORANGE / FINDINGS for `a57d442c74b08265b64dcfe49ee3d1cb69c55def`.
The result comment of werkrevisor holds the details.
<!-- werkrevisor-review:a57d442c74b08265b64dcfe49ee3d1cb69c55def -->
[medium] triage_complete without a model result (werkrevisor)
hint; Jev: real 0.37, severity 2.2 of 3
When there are no judged files, triage_complete is set True, and _final_verdict then skips the no_model_review check. In combined or llm mode, if the model review is skipped or fails (for example classifier_failed, or a budget stop that adds a YELLOW reason that is not in ADVISORY_CODES), YELLOW still holds. Reasons like no_llm or invalid_answer are not advisory, so that case is fine. However, a PR with only docs and scope = "all" in combined mode where the model is skipped because of rules.skip_llm_reason can reach GREEN without any review of the PR documents beyond the classifier. This is probably intended, but it is not covered by a test.
Updated by the review of d29946cc6b75a634686e24c234310722ccb433de.
**[medium] triage_complete without a model result** (werkrevisor)
hint; Jev: real 0.37, severity 2.2 of 3
When there are no judged files, `triage_complete` is set True, and `_final_verdict` then skips the `no_model_review` check. In `combined` or `llm` mode, if the model review is skipped or fails (for example `classifier_failed`, or a budget stop that adds a YELLOW reason that is not in `ADVISORY_CODES`), YELLOW still holds. Reasons like `no_llm` or `invalid_answer` are not advisory, so that case is fine. However, a PR with only docs and `scope = "all"` in `combined` mode where the model is skipped because of `rules.skip_llm_reason` can reach GREEN without any review of the PR documents beyond the classifier. This is probably intended, but it is not covered by a test.
<sub>Updated by the review of `d29946cc6b75a634686e24c234310722ccb433de`.</sub>
Not changed: without a judged file, the changed files are docs, mechanical files, hard paths or DB schema files. Hard paths still give risky_paths, schema files are judged by [db_schema], and mechanical files are explained by deterministic rules; a docs-only PR is harmless. A secret in the diff keeps triage_complete false. Left open for the human reviewer.
Not changed: without a judged file, the changed files are docs, mechanical files, hard paths or DB schema files. Hard paths still give `risky_paths`, schema files are judged by `[db_schema]`, and mechanical files are explained by deterministic rules; a docs-only PR is harmless. A secret in the diff keeps `triage_complete` false. Left open for the human reviewer.
With triage_complete, _advisory turns every YELLOW finding_unconfirmed and finding_not_judged reason into GREEN. The docs say "a finding that Jev confirms stays", but a finding Jev could not judge (finding_not_judged) is also downgraded. A real bug the model found, but Jev never evaluated because of a Jev error on the finding unit, no longer holds the PR. Jev only judged the files, not the findings, so this goes beyond the documented behavior.
Updated by the review of d29946cc6b75a634686e24c234310722ccb433de.
**[high] Advisory downgrade hides unconfirmed findings** (werkrevisor)
ORANGE; Jev: real 0.72, severity 3.0 of 3
With `triage_complete`, `_advisory` turns every YELLOW `finding_unconfirmed` and `finding_not_judged` reason into GREEN. The docs say "a finding that Jev confirms stays", but a finding Jev could not judge (`finding_not_judged`) is also downgraded. A real bug the model found, but Jev never evaluated because of a Jev error on the finding unit, no longer holds the PR. Jev only judged the files, not the findings, so this goes beyond the documented behavior.
<sub>Updated by the review of `d29946cc6b75a634686e24c234310722ccb433de`.</sub>
Not changed, intended: with Jev judging every changed file, the model only advises, also when it could not run because the prompt is too large; that is the same decision as [engine] mode = "classifier". The deterministic size limits size_lines and size_files stay YELLOW, and a file Jev cannot read, e.g. a binary one, makes the judgment incomplete, so nothing is downgraded then. Left open for the human reviewer.
Not changed, intended: with Jev judging every changed file, the model only advises, also when it could not run because the prompt is too large; that is the same decision as `[engine] mode = "classifier"`. The deterministic size limits `size_lines` and `size_files` stay YELLOW, and a file Jev cannot read, e.g. a binary one, makes the judgment incomplete, so nothing is downgraded then. Left open for the human reviewer.
[medium] Deleted main-code files skip the security question (werkrevisor)
YELLOW; Jev: real 0.47, severity 3.0 of 3
In collect, a deleted non-judged file is judged on its diff. Removing an authorization class or an access-control file is a security-relevant change, and Jev gets only removed lines. The weakens_checks and security answers on a pure deletion diff are not calibrated by the measurement. Deleting a test file also passes as GREEN if Jev rates low, though the existing tests_weakened rule is separate. This is an uncertain concern, since the measurement set may have included deletions.
**[medium] Deleted main-code files skip the security question** (werkrevisor)
YELLOW; Jev: real 0.47, severity 3.0 of 3
In `collect`, a deleted non-judged file is judged on its diff. Removing an authorization class or an access-control file is a security-relevant change, and Jev gets only removed lines. The `weakens_checks` and `security` answers on a pure deletion diff are not calibrated by the measurement. Deleting a test file also passes as GREEN if Jev rates low, though the existing `tests_weakened` rule is separate. This is an uncertain concern, since the measurement set may have included deletions.
Not changed: Jev does get a deleted main-code file, with its removed lines, and asks security and weakens_checks (only a deleted judged path is unjudgeable). A deleted test file gives tests_weakened deterministically. Right is that pure deletions were not a separate case of the measurement. Left open for the human reviewer.
Not changed: Jev does get a deleted main-code file, with its removed lines, and asks `security` and `weakens_checks` (only a deleted judged path is unjudgeable). A deleted test file gives `tests_weakened` deterministically. Right is that pure deletions were not a separate case of the measurement. Left open for the human reviewer.
ai-reviewer
scheduled this pull request to auto merge when all checks succeed 2026-10-07 06:04:04 +02:00
ai-reviewer
requested review from mi 2026-10-07 06:04:04 +02:00
ORANGE cli.py:377-389 (advisory downgrade) and cli.py:307-312 (triage complete without units): intended, reasons in the threads.
YELLOW rules.py:258-260 (deleted files): Jev judges them on their removed lines; reasons in the thread.
YELLOW risky paths .werkrevisor.toml and werkrevisor/config.py: this PR changes the review rules, so it needs the human review.
YELLOW full-file context of rules.py, model risk high and security concern: the PR deliberately loosens the gate; the measurement in the PR doc shows the effect on 37 merged PRs.
hint writing rule, PR doc line 119: wrapped in d29946c.
Answer to the review of `a57d442` (ORANGE):
- ORANGE `cli.py:377-389` (advisory downgrade) and `cli.py:307-312` (triage complete without units): intended, reasons in the threads.
- YELLOW `rules.py:258-260` (deleted files): Jev judges them on their removed lines; reasons in the thread.
- YELLOW risky paths `.werkrevisor.toml` and `werkrevisor/config.py`: this PR changes the review rules, so it needs the human review.
- YELLOW full-file context of `rules.py`, model risk high and security concern: the PR deliberately loosens the gate; the measurement in the PR doc shows the effect on 37 merged PRs.
- hint writing rule, PR doc line 119: wrapped in d29946c.
ai-review/assessment AI findings; a developer approval after checking them unlocks the merge; a human review request is pending; merge already scheduled
YELLOW 2 changed files were given to the AI only as diff, not in full, so it could not check them against the rest of their file (all full files together would exceed [review] max_full_files_chars = 160000, and no excerpt fits the limits either): werkrevisor/report.py, werkrevisor/rules.py
hint 1 build change(s) of low risk according to Jev: statistics/build.gradle.kts (weakens checks 0.02, risk 0.00)
hint 1 changed file was given to the AI as excerpt of the selected lines and 5 lines around each changed line, not in full (all full files together would exceed [review] max_full_files_chars = 160000): werkrevisor/config.py
hint 2 finding(s) of low severity or rated minor by Jev: werkrevisor/cli.py:306-312, docs/prs/2026-10-06-PR#39-jev-triage.md:76-95
hint model recommends merge with caution
Findings
[high] Advisory downgrade hides unconfirmed findings at werkrevisor/cli.py:377-389, ORANGE, still open in an earlier line comment
[medium] triage_complete without a model result at werkrevisor/cli.py:306-312, hint, still open in an earlier line comment
[low] Document and code disagree on the model hint at docs/prs/2026-10-06-PR#39-jev-triage.md:76-95, hint, as a line comment
Classifier
10 of 10 units passed all questions (1 PR document(s)).
3 answers ignored because of a confidence below [classifier] min_confidence.
Usage: model `openrouter/anthropic/claude-sonnet-5.5`, LLM done, tokens counted before the call 98932, calls 1, tokens in 97785 / out 896 (reasoning included in out), duration 31.2 s, cost 0.2045 USD (reported), classifier `jev-1.13.0` calls 10, tokens in 8757, cost 0.0004 USD, finding judge `jev-1.13.0` calls 2, tokens in 3472, cost 0.0001 USD, excerpt selection `openrouter/openai/gpt-6-luna` calls 3, tokens in 25519 / out 762, cost 0.0006 USD (reported), build file judge `jev-1.13.0` calls 1, tokens in 560, cost 0.0000 USD, config: defaults + host + repository (.werkrevisor.toml at base c83d016).
Reviewed head `d29946cc6b75a634686e24c234310722ccb433de` against base `c83d01690888bb1f8da62cc8bc4b3710308135bf`, werkrevisor 0.20.6.
<!-- werkrevisor:d29946cc6b75a634686e24c234310722ccb433de -->
<!-- werkrevisor-verdict:ORANGE -->
<!-- werkrevisor-folded -->
<details><summary>🟠 werkrevisor ORANGE for <code>d29946c</code>, superseded by <a href="http://git.javagil.de/mi/werkrevisor/pulls/39#issuecomment-1667">the result for <code>f22a3d1</code></a></summary>
## 🟠 werkrevisor: ORANGE / FINDINGS
Change kind: `mixed`.
### Reasons
- **ORANGE** 1 finding(s) that Jev rates as probably real and severe: werkrevisor/cli.py:377-389
- **YELLOW** risky paths changed: .werkrevisor.toml (built in), werkrevisor/config.py (werkrevisor/config.py)
- **YELLOW** 2 changed files were given to the AI only as diff, not in full, so it could not check them against the rest of their file (all full files together would exceed [review] max_full_files_chars = 160000, and no excerpt fits the limits either): werkrevisor/report.py, werkrevisor/rules.py
- **hint** 1 build change(s) of low risk according to Jev: statistics/build.gradle.kts (weakens checks 0.02, risk 0.00)
- **hint** 1 changed file was given to the AI as excerpt of the selected lines and 5 lines around each changed line, not in full (all full files together would exceed [review] max_full_files_chars = 160000): werkrevisor/config.py
- **hint** 2 finding(s) of low severity or rated minor by Jev: werkrevisor/cli.py:306-312, docs/prs/2026-10-06-PR#39-jev-triage.md:76-95
- **hint** model recommends merge with caution
### Findings
1. **[high] Advisory downgrade hides unconfirmed findings** at `werkrevisor/cli.py:377-389`, ORANGE, still open in [an earlier line comment](http://git.javagil.de/mi/werkrevisor/pulls/39#issuecomment-1652)
2. **[medium] triage_complete without a model result** at `werkrevisor/cli.py:306-312`, hint, still open in [an earlier line comment](http://git.javagil.de/mi/werkrevisor/pulls/39#issuecomment-1653)
3. **[low] Document and code disagree on the model hint** at `docs/prs/2026-10-06-PR#39-jev-triage.md:76-95`, hint, as a line comment
### Classifier
10 of 10 units passed all questions (1 PR document(s)).
3 answers ignored because of a confidence below [classifier] min_confidence.
<sub>
Usage: model `openrouter/anthropic/claude-sonnet-5.5`, LLM done, tokens counted before the call 98932, calls 1, tokens in 97785 / out 896 (reasoning included in out), duration 31.2 s, cost 0.2045 USD (reported), classifier `jev-1.13.0` calls 10, tokens in 8757, cost 0.0004 USD, finding judge `jev-1.13.0` calls 2, tokens in 3472, cost 0.0001 USD, excerpt selection `openrouter/openai/gpt-6-luna` calls 3, tokens in 25519 / out 762, cost 0.0006 USD (reported), build file judge `jev-1.13.0` calls 1, tokens in 560, cost 0.0000 USD, config: defaults + host + repository (.werkrevisor.toml at base c83d016).<br>
Reviewed head `d29946cc6b75a634686e24c234310722ccb433de` against base `c83d01690888bb1f8da62cc8bc4b3710308135bf`, werkrevisor 0.20.6.
</sub>
</details>
werkrevisor: ORANGE / FINDINGS for d29946cc6b75a634686e24c234310722ccb433de.
The result comment of werkrevisor holds the details.
werkrevisor: ORANGE / FINDINGS for `d29946cc6b75a634686e24c234310722ccb433de`.
The result comment of werkrevisor holds the details.
<!-- werkrevisor-review:d29946cc6b75a634686e24c234310722ccb433de -->
[low] Document and code disagree on the model hint (werkrevisor)
hint
The document says [path_judge] scope = "all" makes "a file that reached the model only as diff or excerpt" a hint. The code downgrades full_files_incomplete, too_large_tokens, too_large_cost and unreviewable_files too, and the document mentions only some of them. The PR doc also says .werkrevisor.toml stays hard "since the user decided", but the document does not state that the code enforces it via the built-in hard paths.
**[low] Document and code disagree on the model hint** (werkrevisor)
hint
The document says `[path_judge] scope = "all"` makes "a file that reached the model only as diff or excerpt" a hint. The code downgrades `full_files_incomplete`, `too_large_tokens`, `too_large_cost` and `unreviewable_files` too, and the document mentions only some of them. The PR doc also says `.werkrevisor.toml` stays hard "since the user decided", but the document does not state that the code enforces it via the built-in hard paths.
ai-review/assessment AI findings; a developer approval after checking them unlocks the merge; a human review request is pending; merge already scheduled
ORANGE 3 finding(s) that Jev rates as probably real and severe: werkrevisor/cli.py:905-912, werkrevisor/cli.py:382-394, werkrevisor/pathjudge.py:91-106
YELLOW 15 changed files were given to the AI only as diff, not in full, so it could not check them against the rest of their file (the prompt with full files or excerpts exceeded the input limit): .werkrevisor.toml, config/werkator.toml, config/werkrevisor.toml and 12 more
hint writing rules violated in the changed docs: README.md:955: paragraph with 6 sentences, more than 5
hint 1 build change(s) of low risk according to Jev: statistics/build.gradle.kts (weakens checks 0.03, risk 0.00)
hint the prompt exceeded the input limit, so the AI got the diff only
hint model recommends merge with caution
Findings
[medium] Config read from default branch breaks the PR-head/base contract and has no fallback at werkrevisor/cli.py:905-912, ORANGE, as a line comment
[medium] Advisory downgrade can hide model YELLOW reasons whenever triage is complete at werkrevisor/cli.py:382-394, ORANGE, as a line comment
[medium] Deleted non-judged file is judged from its diff only at werkrevisor/pathjudge.py:91-106, ORANGE, as a line comment
14 of 15 units passed all questions (1 PR document(s)).
5 answers ignored because of a confidence below [classifier] min_confidence.
Usage: model `openrouter/anthropic/claude-sonnet-5.5`, LLM done, tokens counted before the call 49745, calls 1, tokens in 50634 / out 994 (reasoning included in out), duration 46.4 s, cost 0.1112 USD (reported), classifier `jev-1.13.0` calls 15, tokens in 13419, cost 0.0006 USD, finding judge `jev-1.13.0` calls 3, tokens in 5094, cost 0.0002 USD, excerpt selection `openrouter/openai/gpt-6-luna` calls 4, tokens in 44443 / out 1628, cost 0.0064 USD (reported), discarded after the context reduction, build file judge `jev-1.13.0` calls 1, tokens in 560, cost 0.0000 USD, config: defaults + host + repository (.werkrevisor.toml at base c83d016).
Reviewed head `f22a3d1b2c7f643b1ab610dc47257e5a8f034163` against base `c83d01690888bb1f8da62cc8bc4b3710308135bf`, werkrevisor 0.20.6.
<!-- werkrevisor:f22a3d1b2c7f643b1ab610dc47257e5a8f034163 -->
<!-- werkrevisor-verdict:ORANGE -->
<!-- werkrevisor-folded -->
<details><summary>🟠 werkrevisor ORANGE for <code>f22a3d1</code>, superseded by <a href="http://git.javagil.de/mi/werkrevisor/pulls/39#issuecomment-1677">the result for <code>2f16713</code></a></summary>
## 🟠 werkrevisor: ORANGE / FINDINGS
Change kind: `mixed`.
### Reasons
- **ORANGE** 3 finding(s) that Jev rates as probably real and severe: werkrevisor/cli.py:905-912, werkrevisor/cli.py:382-394, werkrevisor/pathjudge.py:91-106
- **YELLOW** risky paths changed: .werkrevisor.toml (built in), werkrevisor/config.py (werkrevisor/config.py)
- **YELLOW** spec: docs/prs/2026-10-06-PR#39-jev-triage.md Scenario#39.09 given_when_then_misused (0.64 > yellow_above = 0.6)
- **YELLOW** 15 changed files were given to the AI only as diff, not in full, so it could not check them against the rest of their file (the prompt with full files or excerpts exceeded the input limit): .werkrevisor.toml, config/werkator.toml, config/werkrevisor.toml and 12 more
- **hint** writing rules violated in the changed docs: README.md:955: paragraph with 6 sentences, more than 5
- **hint** 1 build change(s) of low risk according to Jev: statistics/build.gradle.kts (weakens checks 0.03, risk 0.00)
- **hint** the prompt exceeded the input limit, so the AI got the diff only
- **hint** model recommends merge with caution
### Findings
1. **[medium] Config read from default branch breaks the PR-head/base contract and has no fallback** at `werkrevisor/cli.py:905-912`, ORANGE, as a line comment
2. **[medium] Advisory downgrade can hide model YELLOW reasons whenever triage is complete** at `werkrevisor/cli.py:382-394`, ORANGE, as a line comment
3. **[medium] Deleted non-judged file is judged from its diff only** at `werkrevisor/pathjudge.py:91-106`, ORANGE, as a line comment
### Classifier
| Unit | Question | Problem | Confidence | Verdict |
|---|---|---|---|---|
| docs/prs/2026-10-06-PR#39-jev-triage.md Scenario#39.09 | given_when_then_misused | 0.64 | – | YELLOW |
14 of 15 units passed all questions (1 PR document(s)).
5 answers ignored because of a confidence below [classifier] min_confidence.
<sub>
Usage: model `openrouter/anthropic/claude-sonnet-5.5`, LLM done, tokens counted before the call 49745, calls 1, tokens in 50634 / out 994 (reasoning included in out), duration 46.4 s, cost 0.1112 USD (reported), classifier `jev-1.13.0` calls 15, tokens in 13419, cost 0.0006 USD, finding judge `jev-1.13.0` calls 3, tokens in 5094, cost 0.0002 USD, excerpt selection `openrouter/openai/gpt-6-luna` calls 4, tokens in 44443 / out 1628, cost 0.0064 USD (reported), discarded after the context reduction, build file judge `jev-1.13.0` calls 1, tokens in 560, cost 0.0000 USD, config: defaults + host + repository (.werkrevisor.toml at base c83d016).<br>
Reviewed head `f22a3d1b2c7f643b1ab610dc47257e5a8f034163` against base `c83d01690888bb1f8da62cc8bc4b3710308135bf`, werkrevisor 0.20.6.
</sub>
</details>
werkrevisor: ORANGE / FINDINGS for f22a3d1b2c7f643b1ab610dc47257e5a8f034163.
The result comment of werkrevisor holds the details.
werkrevisor: ORANGE / FINDINGS for `f22a3d1b2c7f643b1ab610dc47257e5a8f034163`.
The result comment of werkrevisor holds the details.
<!-- werkrevisor-review:f22a3d1b2c7f643b1ab610dc47257e5a8f034163 -->
[medium] Advisory downgrade also hides a failed or unreadable model review (werkrevisor)
YELLOW; Jev: real 0.52, severity 2.7 of 3
ADVISORY_CODES includes full_files_incomplete, unreviewable_files and too_large_*, which are turned GREEN when triage_complete is set. In _final_verdict, triage_complete also bypasses the no_model_review check. With model_role = "decides" and scope = "all", _jev_decides is false and nothing changes. With advises, a model call that failed (llm_status not done) still gives GREEN, so a model outage can no longer hold any PR. This looks intended, but there is no test for a failed model call combined with complete triage.
Updated by the review of de1a4098e466b4fb934b323633f419bdbda34668.
**[medium] Advisory downgrade also hides a failed or unreadable model review** (werkrevisor)
YELLOW; Jev: real 0.52, severity 2.7 of 3
`ADVISORY_CODES` includes `full_files_incomplete`, `unreviewable_files` and `too_large_*`, which are turned GREEN when `triage_complete` is set. In `_final_verdict`, `triage_complete` also bypasses the `no_model_review` check. With `model_role = "decides"` and `scope = "all"`, `_jev_decides` is false and nothing changes. With `advises`, a model call that failed (`llm_status` not done) still gives GREEN, so a model outage can no longer hold any PR. This looks intended, but there is no test for a failed model call combined with complete triage.
<sub>Updated by the review of `de1a4098e466b4fb934b323633f419bdbda34668`.</sub>
Intended, now documented and tested in 2f16713: only with [engine] model_role = "advises" (or without a model in classifier mode) and Jev's judgment of every file does a PR too large for the model become a hint; Scenario#39.04 names it, and test_a_pr_too_large_for_the_model_is_a_hint_when_the_model_only_advises shows both roles. The deterministic size limits stay YELLOW. Left open for the human reviewer.
Intended, now documented and tested in 2f16713: only with `[engine] model_role = "advises"` (or without a model in classifier mode) and Jev's judgment of every file does a PR too large for the model become a hint; Scenario#39.04 names it, and `test_a_pr_too_large_for_the_model_is_a_hint_when_the_model_only_advises` shows both roles. The deterministic size limits stay YELLOW. Left open for the human reviewer.
[medium] Config read from default branch breaks the PR-head/base contract and has no fallback (werkrevisor)
ORANGE; Jev: real 0.80, severity 3.0 of 3
main now calls default_branch_head(gitea) before any other step and passes branch or "default branch" as ref. If default_branch is missing from the repo response, sha is empty and read_repository_config raises "the default branch commit is unknown" as an ERROR for every PR. This is intended to be strict, but the error message says "the default branch commit is unknown", and the exception text in default_branch_head only covers exceptions, not an empty name or sha. Also, ref in the message is default branch, so the source line reads "at default branch ddddddd", which is inconsistent with the test expectation "at main ddddddd" only when the name is empty. No test covers the empty-name or empty-sha case.
**[medium] Config read from default branch breaks the PR-head/base contract and has no fallback** (werkrevisor)
ORANGE; Jev: real 0.80, severity 3.0 of 3
`main` now calls `default_branch_head(gitea)` before any other step and passes `branch or "default branch"` as ref. If `default_branch` is missing from the repo response, `sha` is empty and `read_repository_config` raises "the default branch commit is unknown" as an ERROR for every PR. This is intended to be strict, but the error message says "the default branch commit is unknown", and the exception text in `default_branch_head` only covers exceptions, not an empty name or sha. Also, `ref` in the message is `default branch`, so the source line reads "at default branch ddddddd", which is inconsistent with the test expectation "at main ddddddd" only when the name is empty. No test covers the empty-name or empty-sha case.
Fixed in 2f16713: an empty name or commit of the default branch is a configuration error that names its cause ("Gitea names no default branch" or "no commit of main"), covered by test_the_repository_config_comes_from_the_default_branch; the ref is always the branch name now.
Fixed in 2f16713: an empty name or commit of the default branch is a configuration error that names its cause ("Gitea names no default branch" or "no commit of main"), covered by `test_the_repository_config_comes_from_the_default_branch`; the ref is always the branch name now.
[medium] Deleted non-judged file is judged from its diff only (werkrevisor)
ORANGE; Jev: real 0.64, severity 2.9 of 3
With scope = "all", a deleted main-code file is no longer listed as unjudgeable but judged from its removal diff. Deleting a security rule or a whole test file may then be rated low, and the security question only sees removed lines. The deleted file's absence cannot be seen by Jev as "check removed" with certainty, and no test covers the deleted-file path with scope = "all".
**[medium] Deleted non-judged file is judged from its diff only** (werkrevisor)
ORANGE; Jev: real 0.64, severity 2.9 of 3
With `scope = "all"`, a deleted main-code file is no longer listed as unjudgeable but judged from its removal diff. Deleting a security rule or a whole test file may then be rated low, and the `security` question only sees removed lines. The deleted file's absence cannot be seen by Jev as "check removed" with certainty, and no test covers the deleted-file path with `scope = "all"`.
Tested in 2f16713: test_a_deleted_file_is_judged_on_its_removed_lines shows that Jev gets the removed lines and its security answer counts, and that a deleted test file gives tests_weakened deterministically, whatever Jev answers.
Tested in 2f16713: `test_a_deleted_file_is_judged_on_its_removed_lines` shows that Jev gets the removed lines and its security answer counts, and that a deleted test file gives `tests_weakened` deterministically, whatever Jev answers.
ORANGE cli.py:905-912 (default branch without name or commit): fixed in 2f16713, resolved.
ORANGE cli.py:382-394 (advisory downgrade): intended with model_role = "advises", now in Scenario#39.04 and tested; left open.
ORANGE pathjudge.py:91-106 (deleted files): tested in 2f16713, resolved.
YELLOW spec Scenario#39.09: reworded by its effect in 2f16713.
hint writing rule README.md:955: paragraph split in 2f16713.
YELLOW risky paths .werkrevisor.toml, werkrevisor/config.py: this PR changes the review rules and needs the human review.
Answer to the review of `f22a3d1` (ORANGE):
- ORANGE `cli.py:905-912` (default branch without name or commit): fixed in 2f16713, resolved.
- ORANGE `cli.py:382-394` (advisory downgrade): intended with `model_role = "advises"`, now in Scenario#39.04 and tested; left open.
- ORANGE `pathjudge.py:91-106` (deleted files): tested in 2f16713, resolved.
- YELLOW spec Scenario#39.09: reworded by its effect in 2f16713.
- hint writing rule `README.md:955`: paragraph split in 2f16713.
- YELLOW risky paths `.werkrevisor.toml`, `werkrevisor/config.py`: this PR changes the review rules and needs the human review.
ai-review/assessment AI findings; a developer approval after checking them unlocks the merge; a human review request is pending; merge already scheduled
YELLOW 14 changed files were given to the AI only as diff, not in full, so it could not check them against the rest of their file (the prompt with full files or excerpts exceeded the input limit): .werkrevisor.toml, config/werkator.toml, config/werkrevisor.toml and 11 more
hint 1 build change(s) of low risk according to Jev: statistics/build.gradle.kts (weakens checks 0.02, risk 0.00)
hint the prompt exceeded the input limit, so the AI got the diff only
hint 2 finding(s) of low severity or rated minor by Jev: tests/test_config_budget_redact.py:61-73, werkrevisor/cli.py:382-394
hint model recommends changes before merge
Findings
[high] Judged-path verdict ignores security for kind JUDGED at werkrevisor/pathjudge.py:147-161, ORANGE, as a line comment
[high] Config is read from the default branch, not the PR base, with a silent trust change at werkrevisor/cli.py:905-915, ORANGE, as a line comment
[high] Test weakened for werkator host config at tests/test_config_budget_redact.py:61-73, hint, as a line comment
1 more finding(s) not shown.
Classifier
15 of 15 units passed all questions (1 PR document(s)).
5 answers ignored because of a confidence below [classifier] min_confidence.
Usage: model `openrouter/anthropic/claude-sonnet-5.5`, LLM done, tokens counted before the call 50940, calls 1, tokens in 51946 / out 1174 (reasoning included in out), duration 46.1 s, cost 0.1156 USD (reported), classifier `jev-1.13.0` calls 15, tokens in 13466, cost 0.0006 USD, finding judge `jev-1.13.0` calls 4, tokens in 7184, cost 0.0003 USD, excerpt selection `openrouter/openai/gpt-6-luna` calls 4, tokens in 44557 / out 1502, cost 0.0032 USD (reported), discarded after the context reduction, build file judge `jev-1.13.0` calls 1, tokens in 560, cost 0.0000 USD, config: defaults + host + repository (.werkrevisor.toml at base c83d016).
Reviewed head `2f1671309478ff165a947b8faacc328fe42eef89` against base `c83d01690888bb1f8da62cc8bc4b3710308135bf`, werkrevisor 0.20.6.
<!-- werkrevisor:2f1671309478ff165a947b8faacc328fe42eef89 -->
<!-- werkrevisor-verdict:ORANGE -->
<!-- werkrevisor-folded -->
<details><summary>🟠 werkrevisor ORANGE for <code>2f16713</code>, superseded by <a href="http://git.javagil.de/mi/werkrevisor/pulls/39#issuecomment-1687">the result for <code>49dfbcc</code></a></summary>
## 🟠 werkrevisor: ORANGE / FINDINGS
Change kind: `mixed`.
### Reasons
- **ORANGE** 2 finding(s) that Jev rates as probably real and severe: werkrevisor/pathjudge.py:147-161, werkrevisor/cli.py:905-915
- **YELLOW** risky paths changed: .werkrevisor.toml (built in), werkrevisor/config.py (werkrevisor/config.py)
- **YELLOW** 14 changed files were given to the AI only as diff, not in full, so it could not check them against the rest of their file (the prompt with full files or excerpts exceeded the input limit): .werkrevisor.toml, config/werkator.toml, config/werkrevisor.toml and 11 more
- **hint** 1 build change(s) of low risk according to Jev: statistics/build.gradle.kts (weakens checks 0.02, risk 0.00)
- **hint** the prompt exceeded the input limit, so the AI got the diff only
- **hint** 2 finding(s) of low severity or rated minor by Jev: tests/test_config_budget_redact.py:61-73, werkrevisor/cli.py:382-394
- **hint** model recommends changes before merge
### Findings
1. **[high] Judged-path verdict ignores security for kind JUDGED** at `werkrevisor/pathjudge.py:147-161`, ORANGE, as a line comment
2. **[high] Config is read from the default branch, not the PR base, with a silent trust change** at `werkrevisor/cli.py:905-915`, ORANGE, as a line comment
3. **[high] Test weakened for werkator host config** at `tests/test_config_budget_redact.py:61-73`, hint, as a line comment
1 more finding(s) not shown.
### Classifier
15 of 15 units passed all questions (1 PR document(s)).
5 answers ignored because of a confidence below [classifier] min_confidence.
<sub>
Usage: model `openrouter/anthropic/claude-sonnet-5.5`, LLM done, tokens counted before the call 50940, calls 1, tokens in 51946 / out 1174 (reasoning included in out), duration 46.1 s, cost 0.1156 USD (reported), classifier `jev-1.13.0` calls 15, tokens in 13466, cost 0.0006 USD, finding judge `jev-1.13.0` calls 4, tokens in 7184, cost 0.0003 USD, excerpt selection `openrouter/openai/gpt-6-luna` calls 4, tokens in 44557 / out 1502, cost 0.0032 USD (reported), discarded after the context reduction, build file judge `jev-1.13.0` calls 1, tokens in 560, cost 0.0000 USD, config: defaults + host + repository (.werkrevisor.toml at base c83d016).<br>
Reviewed head `2f1671309478ff165a947b8faacc328fe42eef89` against base `c83d01690888bb1f8da62cc8bc4b3710308135bf`, werkrevisor 0.20.6.
</sub>
</details>
werkrevisor: ORANGE / FINDINGS for 2f1671309478ff165a947b8faacc328fe42eef89.
The result comment of werkrevisor holds the details.
werkrevisor: ORANGE / FINDINGS for `2f1671309478ff165a947b8faacc328fe42eef89`.
The result comment of werkrevisor holds the details.
<!-- werkrevisor-review:2f1671309478ff165a947b8faacc328fe42eef89 -->
[high] Test weakened for werkator host config (werkrevisor)
hint; Jev: real 0.38, severity 1.9 of 3
The old test asserted that packaging/werkator.service, build.gradle.kts, .werkator.yml and ConfigLoader.kt all matched hard_yellow.paths. The new test moves them to judged_paths and asserts that ConfigLoader.kt and UiFormats.kt match neither. This is a deliberate config change. It still removes the hard-path protection for config and auth code from the test, and those files now depend on Jev's judgment alone.
**[high] Test weakened for werkator host config** (werkrevisor)
hint; Jev: real 0.38, severity 1.9 of 3
The old test asserted that `packaging/werkator.service`, `build.gradle.kts`, `.werkator.yml` and `ConfigLoader.kt` all matched `hard_yellow.paths`. The new test moves them to `judged_paths` and asserts that `ConfigLoader.kt` and `UiFormats.kt` match neither. This is a deliberate config change. It still removes the hard-path protection for config and auth code from the test, and those files now depend on Jev's judgment alone.
Not changed, deliberate: the user wants only the few paths that change the review or the agentic development hard; werkator keeps its sandbox, pinned config, trust gate and control token hard, and Jev's security question covers config and auth code (threshold 0.93, measured on 25 hsadmin-ng PRs). Left open for the human reviewer.
Not changed, deliberate: the user wants only the few paths that change the review or the agentic development hard; werkator keeps its sandbox, pinned config, trust gate and control token hard, and Jev's security question covers config and auth code (threshold 0.93, measured on 25 hsadmin-ng PRs). Left open for the human reviewer.
[high] Config is read from the default branch, not the PR base, with a silent trust change (werkrevisor)
ORANGE; Jev: real 0.67, severity 3.0 of 3
main() now calls default_branch_head(gitea) for every run. If the Gitea API call fails or the default branch has no commit, the run ends with a config error (exit 30). Rules for a PR whose base is a stacked branch now come from main, which may lack keys the base branch depends on. This is documented as intended, but the unsupported error path makes every review fail when get_repo is temporarily unavailable. There is no fallback, and the failure message mentions only a config problem.
**[high] Config is read from the default branch, not the PR base, with a silent trust change** (werkrevisor)
ORANGE; Jev: real 0.67, severity 3.0 of 3
`main()` now calls `default_branch_head(gitea)` for every run. If the Gitea API call fails or the default branch has no commit, the run ends with a config error (exit 30). Rules for a PR whose base is a stacked branch now come from `main`, which may lack keys the base branch depends on. This is documented as intended, but the unsupported error path makes every review fail when `get_repo` is temporarily unavailable. There is no fallback, and the failure message mentions only a config problem.
Not changed: a failed read of .werkrevisor.toml was a configuration error before this PR too, so an unavailable Gitea API never reviewed a PR; the message now names the default branch. Rules from main are the intent, as the user decided: a stacked branch's config is no merged state. Left open for the human reviewer.
Not changed: a failed read of `.werkrevisor.toml` was a configuration error before this PR too, so an unavailable Gitea API never reviewed a PR; the message now names the default branch. Rules from `main` are the intent, as the user decided: a stacked branch's config is no merged state. Left open for the human reviewer.
[high] Judged-path verdict ignores security for kind JUDGED (werkrevisor)
ORANGE; Jev: real 0.70, severity 3.0 of 3
In verdict(), weakens_checks is read first, but for a judged path build_risk is read only after the security check. The documented flow is fine. However judge() validates answers against QUESTIONS_BY_KIND[unit.kind], and the JUDGED kind includes security. So a judged path with high security gives YELLOW even when build_risk is harmless. This contradicts test_a_harmless_judged_path_needs_no_human, which does not set security. It is more serious that judged_files of a judged path are always sent the security question even when scope = "judged_paths". That changes existing behavior and cost, and the PR doc says scope default is unchanged.
**[high] Judged-path verdict ignores security for kind JUDGED** (werkrevisor)
ORANGE; Jev: real 0.70, severity 3.0 of 3
In `verdict()`, `weakens_checks` is read first, but for a judged path `build_risk` is read only after the security check. The documented flow is fine. However `judge()` validates answers against `QUESTIONS_BY_KIND[unit.kind]`, and the JUDGED kind includes `security`. So a judged path with high security gives YELLOW even when `build_risk` is harmless. This contradicts `test_a_harmless_judged_path_needs_no_human`, which does not set security. It is more serious that `judged_files` of a judged path are always sent the `security` question even when `scope = "judged_paths"`. That changes existing behavior and cost, and the PR doc says `scope` default is unchanged.
Fixed in 49dfbcc: a judged path gets security only with scope = "all"; with the default it keeps the two calibrated questions, which test_jev_answers_decide_the_verdict now asserts. With scope = "all", a build file that touches security gives YELLOW on purpose.
Fixed in 49dfbcc: a judged path gets `security` only with `scope = "all"`; with the default it keeps the two calibrated questions, which `test_jev_answers_decide_the_verdict` now asserts. With `scope = "all"`, a build file that touches security gives YELLOW on purpose.
ORANGE pathjudge.py:147-161 (security question also with the default scope): fixed in 49dfbcc, resolved.
ORANGE cli.py:905-915 (default branch, no fallback): intended; a failed config read was an error before too. Left open.
hint test_config_budget_redact.py:61-73 (werkator hard paths): deliberate config change. Left open.
YELLOW risky paths and diff-only files: this PR changes the review rules and needs the human review.
Answer to the review of `2f16713` (ORANGE):
- ORANGE `pathjudge.py:147-161` (security question also with the default scope): fixed in 49dfbcc, resolved.
- ORANGE `cli.py:905-915` (default branch, no fallback): intended; a failed config read was an error before too. Left open.
- hint `test_config_budget_redact.py:61-73` (werkator hard paths): deliberate config change. Left open.
- YELLOW risky paths and diff-only files: this PR changes the review rules and needs the human review.
YELLOW 14 changed files were given to the AI only as diff, not in full, so it could not check them against the rest of their file (the prompt with full files or excerpts exceeded the input limit): .werkrevisor.toml, config/werkator.toml, config/werkrevisor.toml and 11 more
YELLOW 1 finding(s) possibly severe, but Jev doubts that they are real: werkrevisor/cli.py:382-394
YELLOW model rates the risk as high
YELLOW model reports a security concern
hint 1 build change(s) of low risk according to Jev: statistics/build.gradle.kts (weakens checks 0.02, risk 0.00)
hint the prompt exceeded the input limit, so the AI got the diff only
hint 2 finding(s) of low severity or rated minor by Jev: docs/prs/2026-10-06-PR#39-jev-triage.md:9, werkrevisor/config.py:523-524
hint model recommends changes before merge
Findings
[high] Model advisory downgrade can hide findings when Jev only judged some units at werkrevisor/cli.py:382-394, YELLOW, still open in an earlier line comment
[medium] PR doc names a test that does not exist at docs/prs/2026-10-06-PR#39-jev-triage.md:9, hint, as a line comment
[medium] model_role is not enforced as host-only at werkrevisor/config.py:523-524, hint, as a line comment
Classifier
15 of 15 units passed all questions (1 PR document(s)).
4 answers ignored because of a confidence below [classifier] min_confidence.
Usage: model `openrouter/anthropic/claude-sonnet-5.5`, LLM done, tokens counted before the call 51450, calls 1, tokens in 52482 / out 1139 (reasoning included in out), duration 43.8 s, cost 0.1164 USD (reported), classifier `jev-1.13.0` calls 15, tokens in 13474, cost 0.0006 USD, finding judge `jev-1.13.0` calls 3, tokens in 4950, cost 0.0002 USD, excerpt selection `openrouter/openai/gpt-6-luna` calls 4, tokens in 44614 / out 1717, cost 0.0033 USD (reported), discarded after the context reduction, build file judge `jev-1.13.0` calls 1, tokens in 560, cost 0.0000 USD, config: defaults + host + repository (.werkrevisor.toml at base c83d016).
Reviewed head `49dfbcc1e7323f6daf8e19f8391b76021a878c3d` against base `c83d01690888bb1f8da62cc8bc4b3710308135bf`, werkrevisor 0.20.6.
<!-- werkrevisor:49dfbcc1e7323f6daf8e19f8391b76021a878c3d -->
<!-- werkrevisor-verdict:YELLOW -->
<!-- werkrevisor-folded -->
<details><summary>🟡 werkrevisor YELLOW for <code>49dfbcc</code>, superseded by <a href="http://git.javagil.de/mi/werkrevisor/pulls/39#issuecomment-1730">the result for <code>5d7fb3e</code></a></summary>
## 🟡 werkrevisor: YELLOW / HUMAN_REVIEW
Change kind: `mixed`.
### Reasons
- **YELLOW** risky paths changed: .werkrevisor.toml (built in), werkrevisor/config.py (werkrevisor/config.py)
- **YELLOW** 14 changed files were given to the AI only as diff, not in full, so it could not check them against the rest of their file (the prompt with full files or excerpts exceeded the input limit): .werkrevisor.toml, config/werkator.toml, config/werkrevisor.toml and 11 more
- **YELLOW** 1 finding(s) possibly severe, but Jev doubts that they are real: werkrevisor/cli.py:382-394
- **YELLOW** model rates the risk as high
- **YELLOW** model reports a security concern
- **hint** 1 build change(s) of low risk according to Jev: statistics/build.gradle.kts (weakens checks 0.02, risk 0.00)
- **hint** the prompt exceeded the input limit, so the AI got the diff only
- **hint** 2 finding(s) of low severity or rated minor by Jev: docs/prs/2026-10-06-PR#39-jev-triage.md:9, werkrevisor/config.py:523-524
- **hint** model recommends changes before merge
### Findings
1. **[high] Model advisory downgrade can hide findings when Jev only judged some units** at `werkrevisor/cli.py:382-394`, YELLOW, still open in [an earlier line comment](http://git.javagil.de/mi/werkrevisor/pulls/39#issuecomment-1669)
2. **[medium] PR doc names a test that does not exist** at `docs/prs/2026-10-06-PR#39-jev-triage.md:9`, hint, as a line comment
3. **[medium] `model_role` is not enforced as host-only** at `werkrevisor/config.py:523-524`, hint, as a line comment
### Classifier
15 of 15 units passed all questions (1 PR document(s)).
4 answers ignored because of a confidence below [classifier] min_confidence.
<sub>
Usage: model `openrouter/anthropic/claude-sonnet-5.5`, LLM done, tokens counted before the call 51450, calls 1, tokens in 52482 / out 1139 (reasoning included in out), duration 43.8 s, cost 0.1164 USD (reported), classifier `jev-1.13.0` calls 15, tokens in 13474, cost 0.0006 USD, finding judge `jev-1.13.0` calls 3, tokens in 4950, cost 0.0002 USD, excerpt selection `openrouter/openai/gpt-6-luna` calls 4, tokens in 44614 / out 1717, cost 0.0033 USD (reported), discarded after the context reduction, build file judge `jev-1.13.0` calls 1, tokens in 560, cost 0.0000 USD, config: defaults + host + repository (.werkrevisor.toml at base c83d016).<br>
Reviewed head `49dfbcc1e7323f6daf8e19f8391b76021a878c3d` against base `c83d01690888bb1f8da62cc8bc4b3710308135bf`, werkrevisor 0.20.6.
</sub>
</details>
werkrevisor: YELLOW / HUMAN_REVIEW for 49dfbcc1e7323f6daf8e19f8391b76021a878c3d.
The result comment of werkrevisor holds the details.
werkrevisor: YELLOW / HUMAN_REVIEW for `49dfbcc1e7323f6daf8e19f8391b76021a878c3d`.
The result comment of werkrevisor holds the details.
<!-- werkrevisor-review:49dfbcc1e7323f6daf8e19f8391b76021a878c3d -->
[medium] PR doc names a test that does not exist (werkrevisor)
hint; Jev: real 0.77, severity 1.5 of 3
Scenarios#39.01, 39.04 and 39.06 list their tests as test_pathjudge: test_a_harmless_judged_path_needs_no_human and similar. The PR document points Scenario#39.09 at test_config_layers: test_the_repository_config_comes_from_the_default_branch; the other cited names appear in the diff, so check them against the tests. The document also says "PR#38: ... names the state of werkrevisor 0.20.6", while the doc file name uses the id PR#39 and the PR title and branch say 39 only in the doc. The measurement section says Jev judged "37 merged PRs" but the problem section says 38, so the document is inconsistent about the sample.
**[medium] PR doc names a test that does not exist** (werkrevisor)
hint; Jev: real 0.77, severity 1.5 of 3
Scenarios#39.01, 39.04 and 39.06 list their tests as `test_pathjudge: test_a_harmless_judged_path_needs_no_human` and similar. The PR document points Scenario#39.09 at `test_config_layers: test_the_repository_config_comes_from_the_default_branch`; the other cited names appear in the diff, so check them against the tests. The document also says "PR#38: ... names the state of werkrevisor 0.20.6", while the doc file name uses the id `PR#39` and the PR title and branch say 39 only in the doc. The measurement section says Jev judged "37 merged PRs" but the problem section says 38, so the document is inconsistent about the sample.
[medium] model_role is not enforced as host-only (werkrevisor)
hint; Jev: real 0.43, severity 2.2 of 3
The README and the PR doc say [engine] model_role is host only. REPOSITORY_KEYS lists no engine section, so a repository key is likely rejected as unknown. No test in the diff covers model_role validation or the repository rejection, e.g. model_role = "maybe".
**[medium] `model_role` is not enforced as host-only** (werkrevisor)
hint; Jev: real 0.43, severity 2.2 of 3
The README and the PR doc say `[engine] model_role` is host only. `REPOSITORY_KEYS` lists no `engine` section, so a repository key is likely rejected as unknown. No test in the diff covers `model_role` validation or the repository rejection, e.g. `model_role = "maybe"`.
YELLOW 14 changed files were given to the AI only as diff, not in full, so it could not check them against the rest of their file (the prompt with full files or excerpts exceeded the input limit): .werkrevisor.toml, config/werkator.toml, config/werkrevisor.toml and 11 more
YELLOW 3 finding(s) possibly severe, but Jev doubts that they are real: werkrevisor/cli.py:312-317, werkrevisor/pathjudge.py:154-173, werkrevisor/config.py:592-596
YELLOW model reports a security concern
hint writing rules violated in the changed docs: docs/anleitung-dev.hostsharing.net.md:91: paragraph with 7 sentences, more than 5
hint 1 build change(s) of low risk according to Jev: statistics/build.gradle.kts (weakens checks 0.03, risk 0.00)
hint the prompt exceeded the input limit, so the AI got the diff only
hint model recommends merge with caution
Findings
[medium] Advisory downgrade ignores model_role at werkrevisor/cli.py:312-317, YELLOW, as a line comment
[medium] Security question not asked for judged paths unless with_security, but verdict reads it from unrelated state at werkrevisor/pathjudge.py:154-173, YELLOW, as a line comment
[medium] A repository can replace all hard paths including secret patterns at werkrevisor/config.py:592-596, YELLOW, as a line comment
Classifier
15 of 15 units passed all questions (1 PR document(s)).
4 answers ignored because of a confidence below [classifier] min_confidence.
Usage: model `openrouter/anthropic/claude-sonnet-5.5`, LLM done, tokens counted before the call 53393, calls 1, tokens in 54795 / out 1093 (reasoning included in out), duration 41.4 s, cost 0.1205 USD (reported), classifier `jev-1.13.0` calls 15, tokens in 13511, cost 0.0006 USD, finding judge `jev-1.13.0` calls 3, tokens in 5216, cost 0.0002 USD, excerpt selection `openrouter/openai/gpt-6-luna` calls 4, tokens in 44614 / out 1421, cost 0.0012 USD (reported), discarded after the context reduction, build file judge `jev-1.13.0` calls 1, tokens in 560, cost 0.0000 USD, config: defaults + host + repository (.werkrevisor.toml at base 9bc9d60).
Reviewed head `5d7fb3eaab8f8d483a5e54ed8c23dd5627853bf5` against base `9bc9d60fd6cf4b7c8399a6851f286910b6499964`, werkrevisor 0.20.6.
<!-- werkrevisor:5d7fb3eaab8f8d483a5e54ed8c23dd5627853bf5 -->
<!-- werkrevisor-verdict:YELLOW -->
<!-- werkrevisor-folded -->
<details><summary>🟡 werkrevisor YELLOW for <code>5d7fb3e</code>, superseded by <a href="http://git.javagil.de/mi/werkrevisor/pulls/39#issuecomment-1740">the result for <code>de1a409</code></a></summary>
## 🟡 werkrevisor: YELLOW / HUMAN_REVIEW
Change kind: `mixed`.
### Reasons
- **YELLOW** risky paths changed: .werkrevisor.toml (built in), werkrevisor/config.py (werkrevisor/config.py)
- **YELLOW** 14 changed files were given to the AI only as diff, not in full, so it could not check them against the rest of their file (the prompt with full files or excerpts exceeded the input limit): .werkrevisor.toml, config/werkator.toml, config/werkrevisor.toml and 11 more
- **YELLOW** 3 finding(s) possibly severe, but Jev doubts that they are real: werkrevisor/cli.py:312-317, werkrevisor/pathjudge.py:154-173, werkrevisor/config.py:592-596
- **YELLOW** model reports a security concern
- **hint** writing rules violated in the changed docs: docs/anleitung-dev.hostsharing.net.md:91: paragraph with 7 sentences, more than 5
- **hint** 1 build change(s) of low risk according to Jev: statistics/build.gradle.kts (weakens checks 0.03, risk 0.00)
- **hint** the prompt exceeded the input limit, so the AI got the diff only
- **hint** model recommends merge with caution
### Findings
1. **[medium] Advisory downgrade ignores `model_role`** at `werkrevisor/cli.py:312-317`, YELLOW, as a line comment
2. **[medium] Security question not asked for judged paths unless `with_security`, but `verdict` reads it from unrelated state** at `werkrevisor/pathjudge.py:154-173`, YELLOW, as a line comment
3. **[medium] A repository can replace all hard paths including secret patterns** at `werkrevisor/config.py:592-596`, YELLOW, as a line comment
### Classifier
15 of 15 units passed all questions (1 PR document(s)).
4 answers ignored because of a confidence below [classifier] min_confidence.
<sub>
Usage: model `openrouter/anthropic/claude-sonnet-5.5`, LLM done, tokens counted before the call 53393, calls 1, tokens in 54795 / out 1093 (reasoning included in out), duration 41.4 s, cost 0.1205 USD (reported), classifier `jev-1.13.0` calls 15, tokens in 13511, cost 0.0006 USD, finding judge `jev-1.13.0` calls 3, tokens in 5216, cost 0.0002 USD, excerpt selection `openrouter/openai/gpt-6-luna` calls 4, tokens in 44614 / out 1421, cost 0.0012 USD (reported), discarded after the context reduction, build file judge `jev-1.13.0` calls 1, tokens in 560, cost 0.0000 USD, config: defaults + host + repository (.werkrevisor.toml at base 9bc9d60).<br>
Reviewed head `5d7fb3eaab8f8d483a5e54ed8c23dd5627853bf5` against base `9bc9d60fd6cf4b7c8399a6851f286910b6499964`, werkrevisor 0.20.6.
</sub>
</details>
werkrevisor: YELLOW / HUMAN_REVIEW for 5d7fb3eaab8f8d483a5e54ed8c23dd5627853bf5.
The result comment of werkrevisor holds the details.
werkrevisor: YELLOW / HUMAN_REVIEW for `5d7fb3eaab8f8d483a5e54ed8c23dd5627853bf5`.
The result comment of werkrevisor holds the details.
<!-- werkrevisor-review:5d7fb3eaab8f8d483a5e54ed8c23dd5627853bf5 -->
_advisory turns reasons like finding_unconfirmed, model_high_risk and too_large_tokens into GREEN whenever result.triage_complete is true. triage_complete is computed via _jev_decides, which only checks model_role == "advises" when mode != "classifier". This looks consistent for the two paths shown. However, the early-return branch in _judge_paths (no judged files) sets triage_complete with use_llm but never checks that Jev actually judged anything, e.g. a PR touching only files that are neither docs nor mechanical would already have judged files. A PR with only docs files has nothing for Jev to miss, but the model's doubts about the PR documents then also become hints. Uncertain: confirm that docs-only PRs should have model findings downgraded.
**[medium] Advisory downgrade ignores `model_role`** (werkrevisor)
YELLOW; Jev: real 0.47, severity 2.6 of 3
`_advisory` turns reasons like `finding_unconfirmed`, `model_high_risk` and `too_large_tokens` into GREEN whenever `result.triage_complete` is true. `triage_complete` is computed via `_jev_decides`, which only checks `model_role == "advises"` when `mode != "classifier"`. This looks consistent for the two paths shown. However, the early-return branch in `_judge_paths` (no judged files) sets `triage_complete` with `use_llm` but never checks that Jev actually judged anything, e.g. a PR touching only files that are neither docs nor mechanical would already have judged files. A PR with only docs files has nothing for Jev to miss, but the model's doubts about the PR documents then also become hints. Uncertain: confirm that docs-only PRs should have model findings downgraded.
Not changed, intended: a docs-only PR has nothing Jev could miss, its PR documents are judged by the classifier, and with model_role = "advises" the model's doubts are hints by design; with the default "decides" they stay YELLOW.
Not changed, intended: a docs-only PR has nothing Jev could miss, its PR documents are judged by the classifier, and with `model_role = "advises"` the model's doubts are hints by design; with the default `"decides"` they stay YELLOW.
[medium] A repository can replace all hard paths including secret patterns (werkrevisor)
YELLOW; Jev: real 0.59, severity 3.0 of 3
hard_yellow.paths changed from _ADD to _REPLACE in the repository layer. A repository config on the default branch can now drop host-defined hard paths such as the isolation paths in config/werkator.toml (werkdock/**, **/*Sandbox*). The host cannot pin them. Only .werkrevisor.toml stays built in. The PR doc states this is intended, but a host operator has no way to keep isolation paths hard against a repository config. Consider a host-only must_keep list.
**[medium] A repository can replace all hard paths including secret patterns** (werkrevisor)
YELLOW; Jev: real 0.59, severity 3.0 of 3
`hard_yellow.paths` changed from `_ADD` to `_REPLACE` in the repository layer. A repository config on the default branch can now drop host-defined hard paths such as the isolation paths in `config/werkator.toml` (`werkdock/**`, `**/*Sandbox*`). The host cannot pin them. Only `.werkrevisor.toml` stays built in. The PR doc states this is intended, but a host operator has no way to keep isolation paths hard against a repository config. Consider a host-only `must_keep` list.
Not changed, as the user decided: the repository config, read from the default branch and changed only with a human approval, names its few hard paths itself. A host-only pin of paths, e.g. werkator's isolation paths, fits the planned review DSL (Follow-up PRs in the PR doc). Left open for the human reviewer.
Not changed, as the user decided: the repository config, read from the default branch and changed only with a human approval, names its few hard paths itself. A host-only pin of paths, e.g. werkator's isolation paths, fits the planned review DSL (Follow-up PRs in the PR doc). Left open for the human reviewer.
[medium] Security question not asked for judged paths unless with_security, but verdict reads it from unrelated state (werkrevisor)
YELLOW; Jev: real 0.59, severity 2.7 of 3
verdict for a JUDGED kind uses values.get("security", 0.0) and checks it against yellow_security_min before the build-risk logic. With scope = "all", a judged path that Jev rates high on security becomes YELLOW even if build_risk is low. That is intended. But reasons collects judged paths under by_verdict and the YELLOW text from _judged_text is then reported as a build change, not as a security touch. The operator sees a build-risk reason for a security-triggered verdict, which hides the actual cause.
**[medium] Security question not asked for judged paths unless `with_security`, but `verdict` reads it from unrelated state** (werkrevisor)
YELLOW; Jev: real 0.59, severity 2.7 of 3
`verdict` for a `JUDGED` kind uses `values.get("security", 0.0)` and checks it against `yellow_security_min` before the build-risk logic. With `scope = "all"`, a judged path that Jev rates high on security becomes YELLOW even if `build_risk` is low. That is intended. But `reasons` collects judged paths under `by_verdict` and the YELLOW text from `_judged_text` is then reported as a build change, not as a security touch. The operator sees a build-risk reason for a security-triggered verdict, which hides the actual cause.
Fixed in de1a409: a judged path that only Jev's security answer holds is listed under files_touch_security, covered by test_a_build_file_that_touches_security_is_named_as_such.
Fixed in de1a409: a judged path that only Jev's security answer holds is listed under `files_touch_security`, covered by `test_a_build_file_that_touches_security_is_named_as_such`.
YELLOW cli.py:312-317 (docs-only PRs and advisory): intended; resolved.
YELLOW pathjudge.py:154-173 (security-held build file named as build risk): fixed in de1a409, resolved.
YELLOW config.py:592-596 (host cannot pin hard paths): as decided; a host pin belongs to the review DSL. Left open.
hint writing rule, guide line 91: paragraph split in de1a409.
YELLOW risky paths and the model's security concern: this PR changes the review rules and needs the human review.
Answer to the review of `5d7fb3e` (YELLOW):
- YELLOW `cli.py:312-317` (docs-only PRs and advisory): intended; resolved.
- YELLOW `pathjudge.py:154-173` (security-held build file named as build risk): fixed in de1a409, resolved.
- YELLOW `config.py:592-596` (host cannot pin hard paths): as decided; a host pin belongs to the review DSL. Left open.
- hint writing rule, guide line 91: paragraph split in de1a409.
- YELLOW risky paths and the model's security concern: this PR changes the review rules and needs the human review.
YELLOW 14 changed files were given to the AI only as diff, not in full, so it could not check them against the rest of their file (the prompt with full files or excerpts exceeded the input limit): .werkrevisor.toml, config/werkator.toml, config/werkrevisor.toml and 11 more
YELLOW 2 finding(s) possibly severe, but Jev doubts that they are real: werkrevisor/cli.py:382-401, werkrevisor/pathjudge.py:255-257
YELLOW model rates the risk as high
hint 1 build change(s) of low risk according to Jev: statistics/build.gradle.kts (weakens checks 0.02, risk 0.00)
hint the prompt exceeded the input limit, so the AI got the diff only
hint 1 finding(s) of low severity or rated minor by Jev: docs/prs/2026-10-06-PR#39-jev-triage.md:150-160
hint model recommends changes before merge
hint Gitea refused the line comments (POST /repos/mi/werkrevisor/pulls/39/reviews failed with HTTP 500); the findings are in this comment
Findings
[medium] Advisory downgrade also hides a failed or unreadable model review at werkrevisor/cli.py:382-401, YELLOW, still open in an earlier line comment
[medium] A renamed or deleted file in judge-all scope is judged on an incomplete picture at werkrevisor/pathjudge.py:255-257
YELLOW; Jev: real 0.37, severity 3.0 of 3
In rules.py only f.path is checked via cats_by_file[f.path], and a file with previous_path (a rename) in scope "all" is added to judged_files as main. The security question then sees only the new path's diff, so a rename that moves an auth file out of its place is judged on the diff of content alone. The existing comment for judged paths ("A rename can move a build file out of its place") does not apply to the new branch, so a pure rename of a security-relevant file may get GREEN. No test covers renames in scope "all".
[medium] PR document cites a test that does not exist at docs/prs/2026-10-06-PR#39-jev-triage.md:150-160
hint; Jev: real 0.83, severity 2.0 of 3
Scenario#39.09 lists test_dbschema: test_the_db_schema_paths_are_configurable for 39.08, which lives in tests/test_dbschema.py. The actual file is tests/test_dbschema.py, but the diff shows the test in tests/test_dbschema.py only as a class DbSchemaPathsTest; the link targets look fine. The real mismatch: the PR document claims the Scenario 39.09 test test_the_repository_config_comes_from_the_default_branch verifies "a PR into the other branch", but the test only calls default_branch_head and resolve_config directly and never runs a PR whose base is another branch. The scenario is therefore only partly verified.
Classifier
15 of 15 units passed all questions (1 PR document(s)).
5 answers ignored because of a confidence below [classifier] min_confidence.
Usage: model `openrouter/anthropic/claude-sonnet-5.5`, LLM done, tokens counted before the call 53841, calls 1, tokens in 55253 / out 975 (reasoning included in out), duration 42.7 s, cost 0.1203 USD (reported), classifier `jev-1.13.0` calls 15, tokens in 13511, cost 0.0006 USD, finding judge `jev-1.13.0` calls 3, tokens in 4625, cost 0.0002 USD, excerpt selection `openrouter/openai/gpt-6-luna` calls 5, tokens in 50848 / out 1702, cost 0.0021 USD (reported), discarded after the context reduction, build file judge `jev-1.13.0` calls 1, tokens in 560, cost 0.0000 USD, config: defaults + host + repository (.werkrevisor.toml at base 9bc9d60).
Reviewed head `de1a4098e466b4fb934b323633f419bdbda34668` against base `9bc9d60fd6cf4b7c8399a6851f286910b6499964`, werkrevisor 0.20.6.
<!-- werkrevisor:de1a4098e466b4fb934b323633f419bdbda34668 -->
<!-- werkrevisor-verdict:YELLOW -->
## 🟡 werkrevisor: YELLOW / HUMAN_REVIEW
Change kind: `mixed`.
### Reasons
- **YELLOW** risky paths changed: .werkrevisor.toml (built in), werkrevisor/config.py (werkrevisor/config.py)
- **YELLOW** 14 changed files were given to the AI only as diff, not in full, so it could not check them against the rest of their file (the prompt with full files or excerpts exceeded the input limit): .werkrevisor.toml, config/werkator.toml, config/werkrevisor.toml and 11 more
- **YELLOW** 2 finding(s) possibly severe, but Jev doubts that they are real: werkrevisor/cli.py:382-401, werkrevisor/pathjudge.py:255-257
- **YELLOW** model rates the risk as high
- **hint** 1 build change(s) of low risk according to Jev: statistics/build.gradle.kts (weakens checks 0.02, risk 0.00)
- **hint** the prompt exceeded the input limit, so the AI got the diff only
- **hint** 1 finding(s) of low severity or rated minor by Jev: docs/prs/2026-10-06-PR#39-jev-triage.md:150-160
- **hint** model recommends changes before merge
- **hint** Gitea refused the line comments (POST /repos/mi/werkrevisor/pulls/39/reviews failed with HTTP 500); the findings are in this comment
### Findings
1. **[medium] Advisory downgrade also hides a failed or unreadable model review** at `werkrevisor/cli.py:382-401`, YELLOW, still open in [an earlier line comment](http://git.javagil.de/mi/werkrevisor/pulls/39#issuecomment-1669)
2. **[medium] A renamed or deleted file in judge-all scope is judged on an incomplete picture** at `werkrevisor/pathjudge.py:255-257`
YELLOW; Jev: real 0.37, severity 3.0 of 3
In `rules.py` only `f.path` is checked via `cats_by_file[f.path]`, and a file with `previous_path` (a rename) in scope "all" is added to `judged_files` as `main`. The security question then sees only the new path's diff, so a rename that moves an auth file out of its place is judged on the diff of content alone. The existing comment for judged paths ("A rename can move a build file out of its place") does not apply to the new branch, so a pure rename of a security-relevant file may get GREEN. No test covers renames in scope "all".
3. **[medium] PR document cites a test that does not exist** at `docs/prs/2026-10-06-PR#39-jev-triage.md:150-160`
hint; Jev: real 0.83, severity 2.0 of 3
Scenario#39.09 lists `test_dbschema: test_the_db_schema_paths_are_configurable` for 39.08, which lives in `tests/test_dbschema.py`. The actual file is `tests/test_dbschema.py`, but the diff shows the test in `tests/test_dbschema.py` only as a class `DbSchemaPathsTest`; the link targets look fine. The real mismatch: the PR document claims the Scenario 39.09 test `test_the_repository_config_comes_from_the_default_branch` verifies "a PR into the other branch", but the test only calls `default_branch_head` and `resolve_config` directly and never runs a PR whose base is another branch. The scenario is therefore only partly verified.
### Classifier
15 of 15 units passed all questions (1 PR document(s)).
5 answers ignored because of a confidence below [classifier] min_confidence.
<sub>
Usage: model `openrouter/anthropic/claude-sonnet-5.5`, LLM done, tokens counted before the call 53841, calls 1, tokens in 55253 / out 975 (reasoning included in out), duration 42.7 s, cost 0.1203 USD (reported), classifier `jev-1.13.0` calls 15, tokens in 13511, cost 0.0006 USD, finding judge `jev-1.13.0` calls 3, tokens in 4625, cost 0.0002 USD, excerpt selection `openrouter/openai/gpt-6-luna` calls 5, tokens in 50848 / out 1702, cost 0.0021 USD (reported), discarded after the context reduction, build file judge `jev-1.13.0` calls 1, tokens in 560, cost 0.0000 USD, config: defaults + host + repository (.werkrevisor.toml at base 9bc9d60).<br>
Reviewed head `de1a4098e466b4fb934b323633f419bdbda34668` against base `9bc9d60fd6cf4b7c8399a6851f286910b6499964`, werkrevisor 0.20.6.
</sub>
Blocking a user prevents them from interacting with repositories, such as opening or commenting on pull requests or issues. Learn more about blocking a user.
See the PR doc.
🤖 Generated with Claude Code
🟠 werkrevisor ORANGE for
a57d442, superseded by the result ford29946c🟠 werkrevisor: ORANGE / FINDINGS
Change kind:
mixed.Reasons
Findings
[high] Advisory downgrade hides unreviewed files at
werkrevisor/cli.py:377-389, ORANGE, as a line comment[high] triage_complete true when Jev judged nothing at
werkrevisor/cli.py:307-312, ORANGE, as a line comment[medium] Deleted main-code files skip the security question at
werkrevisor/rules.py:258-260, YELLOW, as a line commentClassifier
10 of 10 units passed all questions (1 PR document(s)).
Usage: model `openrouter/anthropic/claude-sonnet-5.5`, LLM done, tokens counted before the call 97981, calls 1, tokens in 96881 / out 1029 (reasoning included in out), duration 32.8 s, cost 0.2041 USD (reported), classifier `jev-1.13.0` calls 10, tokens in 8756, cost 0.0004 USD, finding judge `jev-1.13.0` calls 3, tokens in 5107, cost 0.0002 USD, excerpt selection `openrouter/openai/gpt-6-luna` calls 3, tokens in 25519 / out 796, cost 0.0036 USD (reported), build file judge `jev-1.13.0` calls 1, tokens in 560, cost 0.0000 USD, config: defaults + host + repository (.werkrevisor.toml at base4 answers ignored because of a confidence below [classifier] min_confidence.
c83d016).Reviewed head `a57d442c74b08265b64dcfe49ee3d1cb69c55def` against base `c83d01690888bb1f8da62cc8bc4b3710308135bf`, werkrevisor 0.20.6.
werkrevisor: ORANGE / FINDINGS for
a57d442c74b08265b64dcfe49ee3d1cb69c55def.The result comment of werkrevisor holds the details.
@@ -305,8 +305,12 @@ def _excerpt_selector(config, deps, use_llm, result):def _judge_paths(config, deps, use_llm, result, rules, diffs):if not rules.judged_files:[medium] triage_complete without a model result (werkrevisor)
hint; Jev: real 0.37, severity 2.2 of 3
When there are no judged files,
triage_completeis set True, and_final_verdictthen skips theno_model_reviewcheck. Incombinedorllmmode, if the model review is skipped or fails (for exampleclassifier_failed, or a budget stop that adds a YELLOW reason that is not inADVISORY_CODES), YELLOW still holds. Reasons likeno_llmorinvalid_answerare not advisory, so that case is fine. However, a PR with only docs andscope = "all"incombinedmode where the model is skipped because ofrules.skip_llm_reasoncan reach GREEN without any review of the PR documents beyond the classifier. This is probably intended, but it is not covered by a test.Updated by the review of
d29946cc6b75a634686e24c234310722ccb433de.Not changed: without a judged file, the changed files are docs, mechanical files, hard paths or DB schema files. Hard paths still give
risky_paths, schema files are judged by[db_schema], and mechanical files are explained by deterministic rules; a docs-only PR is harmless. A secret in the diff keepstriage_completefalse. Left open for the human reviewer.@@ -369,10 +374,26 @@ def _coverage(files, run, deps, config, excluded):return coverage_reasons(files, run, deps.is_reviewable, config.model.max_output_tokens, excluded)# Reasons that only say the model doubts or could not see enough; Jev's judgment of every file covers them.[high] Advisory downgrade hides unconfirmed findings (werkrevisor)
ORANGE; Jev: real 0.72, severity 3.0 of 3
With
triage_complete,_advisoryturns every YELLOWfinding_unconfirmedandfinding_not_judgedreason into GREEN. The docs say "a finding that Jev confirms stays", but a finding Jev could not judge (finding_not_judged) is also downgraded. A real bug the model found, but Jev never evaluated because of a Jev error on the finding unit, no longer holds the PR. Jev only judged the files, not the findings, so this goes beyond the documented behavior.Updated by the review of
d29946cc6b75a634686e24c234310722ccb433de.Not changed, intended: with Jev judging every changed file, the model only advises, also when it could not run because the prompt is too large; that is the same decision as
[engine] mode = "classifier". The deterministic size limitssize_linesandsize_filesstay YELLOW, and a file Jev cannot read, e.g. a binary one, makes the judgment incomplete, so nothing is downgraded then. Left open for the human reviewer.@@ -252,6 +255,9 @@ def evaluate_rules(files, diffs, config):judged_files.append(f)elif judged:risky.append(f"{judged[0][0]} ({judged[0][1]})")elif judge_all and cats_by_file[f.path] != "docs" and f.path not in mechanical:[medium] Deleted main-code files skip the security question (werkrevisor)
YELLOW; Jev: real 0.47, severity 3.0 of 3
In
collect, a deleted non-judged file is judged on its diff. Removing an authorization class or an access-control file is a security-relevant change, and Jev gets only removed lines. Theweakens_checksandsecurityanswers on a pure deletion diff are not calibrated by the measurement. Deleting a test file also passes as GREEN if Jev rates low, though the existingtests_weakenedrule is separate. This is an uncertain concern, since the measurement set may have included deletions.Not changed: Jev does get a deleted main-code file, with its removed lines, and asks
securityandweakens_checks(only a deleted judged path is unjudgeable). A deleted test file givestests_weakeneddeterministically. Right is that pure deletions were not a separate case of the measurement. Left open for the human reviewer.Answer to the review of
a57d442(ORANGE):cli.py:377-389(advisory downgrade) andcli.py:307-312(triage complete without units): intended, reasons in the threads.rules.py:258-260(deleted files): Jev judges them on their removed lines; reasons in the thread..werkrevisor.tomlandwerkrevisor/config.py: this PR changes the review rules, so it needs the human review.rules.py, model risk high and security concern: the PR deliberately loosens the gate; the measurement in the PR doc shows the effect on 37 merged PRs.d29946c.🟠 werkrevisor ORANGE for
d29946c, superseded by the result forf22a3d1🟠 werkrevisor: ORANGE / FINDINGS
Change kind:
mixed.Reasons
Findings
[high] Advisory downgrade hides unconfirmed findings at
werkrevisor/cli.py:377-389, ORANGE, still open in an earlier line comment[medium] triage_complete without a model result at
werkrevisor/cli.py:306-312, hint, still open in an earlier line comment[low] Document and code disagree on the model hint at
docs/prs/2026-10-06-PR#39-jev-triage.md:76-95, hint, as a line commentClassifier
10 of 10 units passed all questions (1 PR document(s)).
Usage: model `openrouter/anthropic/claude-sonnet-5.5`, LLM done, tokens counted before the call 98932, calls 1, tokens in 97785 / out 896 (reasoning included in out), duration 31.2 s, cost 0.2045 USD (reported), classifier `jev-1.13.0` calls 10, tokens in 8757, cost 0.0004 USD, finding judge `jev-1.13.0` calls 2, tokens in 3472, cost 0.0001 USD, excerpt selection `openrouter/openai/gpt-6-luna` calls 3, tokens in 25519 / out 762, cost 0.0006 USD (reported), build file judge `jev-1.13.0` calls 1, tokens in 560, cost 0.0000 USD, config: defaults + host + repository (.werkrevisor.toml at base3 answers ignored because of a confidence below [classifier] min_confidence.
c83d016).Reviewed head `d29946cc6b75a634686e24c234310722ccb433de` against base `c83d01690888bb1f8da62cc8bc4b3710308135bf`, werkrevisor 0.20.6.
werkrevisor: ORANGE / FINDINGS for
d29946cc6b75a634686e24c234310722ccb433de.The result comment of werkrevisor holds the details.
@@ -0,0 +73,4 @@- [test_pathjudge: test_a_change_that_weakens_the_checks_gets_orange](../../tests/test_pathjudge.py)### Feature: the model only advises[low] Document and code disagree on the model hint (werkrevisor)
hint
The document says
[path_judge] scope = "all"makes "a file that reached the model only as diff or excerpt" a hint. The code downgradesfull_files_incomplete,too_large_tokens,too_large_costandunreviewable_filestoo, and the document mentions only some of them. The PR doc also says.werkrevisor.tomlstays hard "since the user decided", but the document does not state that the code enforces it via the built-in hard paths.🟠 werkrevisor ORANGE for
f22a3d1, superseded by the result for2f16713🟠 werkrevisor: ORANGE / FINDINGS
Change kind:
mixed.Reasons
Findings
[medium] Config read from default branch breaks the PR-head/base contract and has no fallback at
werkrevisor/cli.py:905-912, ORANGE, as a line comment[medium] Advisory downgrade can hide model YELLOW reasons whenever triage is complete at
werkrevisor/cli.py:382-394, ORANGE, as a line comment[medium] Deleted non-judged file is judged from its diff only at
werkrevisor/pathjudge.py:91-106, ORANGE, as a line commentClassifier
14 of 15 units passed all questions (1 PR document(s)).
Usage: model `openrouter/anthropic/claude-sonnet-5.5`, LLM done, tokens counted before the call 49745, calls 1, tokens in 50634 / out 994 (reasoning included in out), duration 46.4 s, cost 0.1112 USD (reported), classifier `jev-1.13.0` calls 15, tokens in 13419, cost 0.0006 USD, finding judge `jev-1.13.0` calls 3, tokens in 5094, cost 0.0002 USD, excerpt selection `openrouter/openai/gpt-6-luna` calls 4, tokens in 44443 / out 1628, cost 0.0064 USD (reported), discarded after the context reduction, build file judge `jev-1.13.0` calls 1, tokens in 560, cost 0.0000 USD, config: defaults + host + repository (.werkrevisor.toml at base5 answers ignored because of a confidence below [classifier] min_confidence.
c83d016).Reviewed head `f22a3d1b2c7f643b1ab610dc47257e5a8f034163` against base `c83d01690888bb1f8da62cc8bc4b3710308135bf`, werkrevisor 0.20.6.
werkrevisor: ORANGE / FINDINGS for
f22a3d1b2c7f643b1ab610dc47257e5a8f034163.The result comment of werkrevisor holds the details.
@@ -369,10 +379,26 @@ def _coverage(files, run, deps, config, excluded):return coverage_reasons(files, run, deps.is_reviewable, config.model.max_output_tokens, excluded)# Reasons that only say the model doubts or could not see enough; Jev's judgment of every file covers them.[medium] Advisory downgrade also hides a failed or unreadable model review (werkrevisor)
YELLOW; Jev: real 0.52, severity 2.7 of 3
ADVISORY_CODESincludesfull_files_incomplete,unreviewable_filesandtoo_large_*, which are turned GREEN whentriage_completeis set. In_final_verdict,triage_completealso bypasses theno_model_reviewcheck. Withmodel_role = "decides"andscope = "all",_jev_decidesis false and nothing changes. Withadvises, a model call that failed (llm_statusnot done) still gives GREEN, so a model outage can no longer hold any PR. This looks intended, but there is no test for a failed model call combined with complete triage.Updated by the review of
de1a4098e466b4fb934b323633f419bdbda34668.Intended, now documented and tested in
2f16713: only with[engine] model_role = "advises"(or without a model in classifier mode) and Jev's judgment of every file does a PR too large for the model become a hint; Scenario#39.04 names it, andtest_a_pr_too_large_for_the_model_is_a_hint_when_the_model_only_advisesshows both roles. The deterministic size limits stay YELLOW. Left open for the human reviewer.@@ -877,0 +902,4 @@return config, describe_sources(sha if repo_data is not None else "", ref)def default_branch_head(gitea):[medium] Config read from default branch breaks the PR-head/base contract and has no fallback (werkrevisor)
ORANGE; Jev: real 0.80, severity 3.0 of 3
mainnow callsdefault_branch_head(gitea)before any other step and passesbranch or "default branch"as ref. Ifdefault_branchis missing from the repo response,shais empty andread_repository_configraises "the default branch commit is unknown" as an ERROR for every PR. This is intended to be strict, but the error message says "the default branch commit is unknown", and the exception text indefault_branch_headonly covers exceptions, not an empty name or sha. Also,refin the message isdefault branch, so the source line reads "at default branch ddddddd", which is inconsistent with the test expectation "at main ddddddd" only when the name is empty. No test covers the empty-name or empty-sha case.Fixed in
2f16713: an empty name or commit of the default branch is a configuration error that names its cause ("Gitea names no default branch" or "no commit of main"), covered bytest_the_repository_config_comes_from_the_default_branch; the ref is always the branch name now.@@ -64,3 +90,2 @@def collect(files, diffs):"""Returns (units, unjudgeable files); an empty state would look harmless to Jev."""def collect(files, diffs, kinds=None):[medium] Deleted non-judged file is judged from its diff only (werkrevisor)
ORANGE; Jev: real 0.64, severity 2.9 of 3
With
scope = "all", a deleted main-code file is no longer listed as unjudgeable but judged from its removal diff. Deleting a security rule or a whole test file may then be rated low, and thesecurityquestion only sees removed lines. The deleted file's absence cannot be seen by Jev as "check removed" with certainty, and no test covers the deleted-file path withscope = "all".Tested in
2f16713:test_a_deleted_file_is_judged_on_its_removed_linesshows that Jev gets the removed lines and its security answer counts, and that a deleted test file givestests_weakeneddeterministically, whatever Jev answers.Answer to the review of
f22a3d1(ORANGE):cli.py:905-912(default branch without name or commit): fixed in2f16713, resolved.cli.py:382-394(advisory downgrade): intended withmodel_role = "advises", now in Scenario#39.04 and tested; left open.pathjudge.py:91-106(deleted files): tested in2f16713, resolved.2f16713.README.md:955: paragraph split in2f16713..werkrevisor.toml,werkrevisor/config.py: this PR changes the review rules and needs the human review.🟠 werkrevisor ORANGE for
2f16713, superseded by the result for49dfbcc🟠 werkrevisor: ORANGE / FINDINGS
Change kind:
mixed.Reasons
Findings
[high] Judged-path verdict ignores security for kind JUDGED at
werkrevisor/pathjudge.py:147-161, ORANGE, as a line comment[high] Config is read from the default branch, not the PR base, with a silent trust change at
werkrevisor/cli.py:905-915, ORANGE, as a line comment[high] Test weakened for werkator host config at
tests/test_config_budget_redact.py:61-73, hint, as a line comment1 more finding(s) not shown.
Classifier
15 of 15 units passed all questions (1 PR document(s)).
Usage: model `openrouter/anthropic/claude-sonnet-5.5`, LLM done, tokens counted before the call 50940, calls 1, tokens in 51946 / out 1174 (reasoning included in out), duration 46.1 s, cost 0.1156 USD (reported), classifier `jev-1.13.0` calls 15, tokens in 13466, cost 0.0006 USD, finding judge `jev-1.13.0` calls 4, tokens in 7184, cost 0.0003 USD, excerpt selection `openrouter/openai/gpt-6-luna` calls 4, tokens in 44557 / out 1502, cost 0.0032 USD (reported), discarded after the context reduction, build file judge `jev-1.13.0` calls 1, tokens in 560, cost 0.0000 USD, config: defaults + host + repository (.werkrevisor.toml at base5 answers ignored because of a confidence below [classifier] min_confidence.
c83d016).Reviewed head `2f1671309478ff165a947b8faacc328fe42eef89` against base `c83d01690888bb1f8da62cc8bc4b3710308135bf`, werkrevisor 0.20.6.
werkrevisor: ORANGE / FINDINGS for
2f1671309478ff165a947b8faacc328fe42eef89.The result comment of werkrevisor holds the details.
@@ -61,3 +61,1 @@for path in ("packaging/werkator.service", "werkdock/build-rootfs", "build.gradle.kts", ".werkator.yml","src/main/kotlin/de/hoennig/werkator/config/ConfigLoader.kt","src/main/kotlin/de/hoennig/werkator/build/PullRequestTargets.kt"):# The isolation of the builds stays hard, Jev judges the build and CI files and, through scope "all", the rest.[high] Test weakened for werkator host config (werkrevisor)
hint; Jev: real 0.38, severity 1.9 of 3
The old test asserted that
packaging/werkator.service,build.gradle.kts,.werkator.ymlandConfigLoader.ktall matchedhard_yellow.paths. The new test moves them tojudged_pathsand asserts thatConfigLoader.ktandUiFormats.ktmatch neither. This is a deliberate config change. It still removes the hard-path protection for config and auth code from the test, and those files now depend on Jev's judgment alone.Not changed, deliberate: the user wants only the few paths that change the review or the agentic development hard; werkator keeps its sandbox, pinned config, trust gate and control token hard, and Jev's security question covers config and auth code (threshold 0.93, measured on 25 hsadmin-ng PRs). Left open for the human reviewer.
@@ -877,0 +902,4 @@return config, describe_sources(sha if repo_data is not None else "", ref)def default_branch_head(gitea):[high] Config is read from the default branch, not the PR base, with a silent trust change (werkrevisor)
ORANGE; Jev: real 0.67, severity 3.0 of 3
main()now callsdefault_branch_head(gitea)for every run. If the Gitea API call fails or the default branch has no commit, the run ends with a config error (exit 30). Rules for a PR whose base is a stacked branch now come frommain, which may lack keys the base branch depends on. This is documented as intended, but the unsupported error path makes every review fail whenget_repois temporarily unavailable. There is no fallback, and the failure message mentions only a config problem.Not changed: a failed read of
.werkrevisor.tomlwas a configuration error before this PR too, so an unavailable Gitea API never reviewed a PR; the message now names the default branch. Rules frommainare the intent, as the user decided: a stacked branch's config is no merged state. Left open for the human reviewer.@@ -115,3 +147,1 @@def verdict(values, config):weakens, risk = valuesif weakens >= config.orange_weakens_min or risk >= config.orange_risk_min:def verdict(values, config, kind=JUDGED):[high] Judged-path verdict ignores security for kind JUDGED (werkrevisor)
ORANGE; Jev: real 0.70, severity 3.0 of 3
In
verdict(),weakens_checksis read first, but for a judged pathbuild_riskis read only after the security check. The documented flow is fine. Howeverjudge()validates answers againstQUESTIONS_BY_KIND[unit.kind], and the JUDGED kind includessecurity. So a judged path with high security gives YELLOW even whenbuild_riskis harmless. This contradictstest_a_harmless_judged_path_needs_no_human, which does not set security. It is more serious thatjudged_filesof a judged path are always sent thesecurityquestion even whenscope = "judged_paths". That changes existing behavior and cost, and the PR doc saysscopedefault is unchanged.Fixed in
49dfbcc: a judged path getssecurityonly withscope = "all"; with the default it keeps the two calibrated questions, whichtest_jev_answers_decide_the_verdictnow asserts. Withscope = "all", a build file that touches security gives YELLOW on purpose.Answer to the review of
2f16713(ORANGE):pathjudge.py:147-161(security question also with the default scope): fixed in49dfbcc, resolved.cli.py:905-915(default branch, no fallback): intended; a failed config read was an error before too. Left open.test_config_budget_redact.py:61-73(werkator hard paths): deliberate config change. Left open.🟡 werkrevisor YELLOW for
49dfbcc, superseded by the result for5d7fb3e🟡 werkrevisor: YELLOW / HUMAN_REVIEW
Change kind:
mixed.Reasons
Findings
[high] Model advisory downgrade can hide findings when Jev only judged some units at
werkrevisor/cli.py:382-394, YELLOW, still open in an earlier line comment[medium] PR doc names a test that does not exist at
docs/prs/2026-10-06-PR#39-jev-triage.md:9, hint, as a line comment[medium]
model_roleis not enforced as host-only atwerkrevisor/config.py:523-524, hint, as a line commentClassifier
15 of 15 units passed all questions (1 PR document(s)).
Usage: model `openrouter/anthropic/claude-sonnet-5.5`, LLM done, tokens counted before the call 51450, calls 1, tokens in 52482 / out 1139 (reasoning included in out), duration 43.8 s, cost 0.1164 USD (reported), classifier `jev-1.13.0` calls 15, tokens in 13474, cost 0.0006 USD, finding judge `jev-1.13.0` calls 3, tokens in 4950, cost 0.0002 USD, excerpt selection `openrouter/openai/gpt-6-luna` calls 4, tokens in 44614 / out 1717, cost 0.0033 USD (reported), discarded after the context reduction, build file judge `jev-1.13.0` calls 1, tokens in 560, cost 0.0000 USD, config: defaults + host + repository (.werkrevisor.toml at base4 answers ignored because of a confidence below [classifier] min_confidence.
c83d016).Reviewed head `49dfbcc1e7323f6daf8e19f8391b76021a878c3d` against base `c83d01690888bb1f8da62cc8bc4b3710308135bf`, werkrevisor 0.20.6.
werkrevisor: YELLOW / HUMAN_REVIEW for
49dfbcc1e7323f6daf8e19f8391b76021a878c3d.The result comment of werkrevisor holds the details.
@@ -0,0 +6,4 @@## The ProblemOn git.javagil.de, the last review before the merge was GREEN for 6 of 38 merged PRs.[medium] PR doc names a test that does not exist (werkrevisor)
hint; Jev: real 0.77, severity 1.5 of 3
Scenarios#39.01, 39.04 and 39.06 list their tests as
test_pathjudge: test_a_harmless_judged_path_needs_no_humanand similar. The PR document points Scenario#39.09 attest_config_layers: test_the_repository_config_comes_from_the_default_branch; the other cited names appear in the diff, so check them against the tests. The document also says "PR#38: ... names the state of werkrevisor 0.20.6", while the doc file name uses the idPR#39and the PR title and branch say 39 only in the doc. The measurement section says Jev judged "37 merged PRs" but the problem section says 38, so the document is inconsistent about the sample.@@ -508,9 +520,13 @@ def _validate(config):raise ConfigError(f"[engine] mode must be one of {', '.join(ENGINE_MODES)}")if config.engine.mode != "llm" and config.classifier is None:raise ConfigError(f"[engine] mode = '{config.engine.mode}' needs a [classifier] section")if config.engine.model_role not in MODEL_ROLES:[medium]
model_roleis not enforced as host-only (werkrevisor)hint; Jev: real 0.43, severity 2.2 of 3
The README and the PR doc say
[engine] model_roleis host only.REPOSITORY_KEYSlists noenginesection, so a repository key is likely rejected as unknown. No test in the diff coversmodel_rolevalidation or the repository rejection, e.g.model_role = "maybe".🟡 werkrevisor YELLOW for
5d7fb3e, superseded by the result forde1a409🟡 werkrevisor: YELLOW / HUMAN_REVIEW
Change kind:
mixed.Reasons
Findings
[medium] Advisory downgrade ignores
model_roleatwerkrevisor/cli.py:312-317, YELLOW, as a line comment[medium] Security question not asked for judged paths unless
with_security, butverdictreads it from unrelated state atwerkrevisor/pathjudge.py:154-173, YELLOW, as a line comment[medium] A repository can replace all hard paths including secret patterns at
werkrevisor/config.py:592-596, YELLOW, as a line commentClassifier
15 of 15 units passed all questions (1 PR document(s)).
Usage: model `openrouter/anthropic/claude-sonnet-5.5`, LLM done, tokens counted before the call 53393, calls 1, tokens in 54795 / out 1093 (reasoning included in out), duration 41.4 s, cost 0.1205 USD (reported), classifier `jev-1.13.0` calls 15, tokens in 13511, cost 0.0006 USD, finding judge `jev-1.13.0` calls 3, tokens in 5216, cost 0.0002 USD, excerpt selection `openrouter/openai/gpt-6-luna` calls 4, tokens in 44614 / out 1421, cost 0.0012 USD (reported), discarded after the context reduction, build file judge `jev-1.13.0` calls 1, tokens in 560, cost 0.0000 USD, config: defaults + host + repository (.werkrevisor.toml at base4 answers ignored because of a confidence below [classifier] min_confidence.
9bc9d60).Reviewed head `5d7fb3eaab8f8d483a5e54ed8c23dd5627853bf5` against base `9bc9d60fd6cf4b7c8399a6851f286910b6499964`, werkrevisor 0.20.6.
werkrevisor: YELLOW / HUMAN_REVIEW for
5d7fb3eaab8f8d483a5e54ed8c23dd5627853bf5.The result comment of werkrevisor holds the details.
@@ -306,1 +309,4 @@and (config.engine.mode == "classifier" or config.engine.model_role == "advises"))def _judge_paths(config, deps, use_llm, result, rules, diffs):[medium] Advisory downgrade ignores
model_role(werkrevisor)YELLOW; Jev: real 0.47, severity 2.6 of 3
_advisoryturns reasons likefinding_unconfirmed,model_high_riskandtoo_large_tokensinto GREEN wheneverresult.triage_completeis true.triage_completeis computed via_jev_decides, which only checksmodel_role == "advises"whenmode != "classifier". This looks consistent for the two paths shown. However, the early-return branch in_judge_paths(no judged files) setstriage_completewithuse_llmbut never checks that Jev actually judged anything, e.g. a PR touching only files that are neither docs nor mechanical would already have judged files. A PR with only docs files has nothing for Jev to miss, but the model's doubts about the PR documents then also become hints. Uncertain: confirm that docs-only PRs should have model findings downgraded.Not changed, intended: a docs-only PR has nothing Jev could miss, its PR documents are judged by the classifier, and with
model_role = "advises"the model's doubts are hints by design; with the default"decides"they stay YELLOW.@@ -569,3 +590,3 @@"categories": {key: _REPLACE for key in ("docs", "tests", "main", "api_spec", "harmless","auto_approvable")},"hard_yellow": {"paths": _ADD, "judged_paths": _ADD, "secret_scan": _ONLY_ON, "secret_scan_exclude": _REPLACE},# The repository names its few hard paths itself; a change of its config always needs a human, and it is read[medium] A repository can replace all hard paths including secret patterns (werkrevisor)
YELLOW; Jev: real 0.59, severity 3.0 of 3
hard_yellow.pathschanged from_ADDto_REPLACEin the repository layer. A repository config on the default branch can now drop host-defined hard paths such as the isolation paths inconfig/werkator.toml(werkdock/**,**/*Sandbox*). The host cannot pin them. Only.werkrevisor.tomlstays built in. The PR doc states this is intended, but a host operator has no way to keep isolation paths hard against a repository config. Consider a host-onlymust_keeplist.Not changed, as the user decided: the repository config, read from the default branch and changed only with a human approval, names its few hard paths itself. A host-only pin of paths, e.g. werkator's isolation paths, fits the planned review DSL (Follow-up PRs in the PR doc). Left open for the human reviewer.
@@ -115,3 +154,1 @@def verdict(values, config):weakens, risk = valuesif weakens >= config.orange_weakens_min or risk >= config.orange_risk_min:def verdict(values, config, kind=JUDGED):[medium] Security question not asked for judged paths unless
with_security, butverdictreads it from unrelated state (werkrevisor)YELLOW; Jev: real 0.59, severity 2.7 of 3
verdictfor aJUDGEDkind usesvalues.get("security", 0.0)and checks it againstyellow_security_minbefore the build-risk logic. Withscope = "all", a judged path that Jev rates high on security becomes YELLOW even ifbuild_riskis low. That is intended. Butreasonscollects judged paths underby_verdictand the YELLOW text from_judged_textis then reported as a build change, not as a security touch. The operator sees a build-risk reason for a security-triggered verdict, which hides the actual cause.Fixed in
de1a409: a judged path that only Jev's security answer holds is listed underfiles_touch_security, covered bytest_a_build_file_that_touches_security_is_named_as_such.Answer to the review of
5d7fb3e(YELLOW):cli.py:312-317(docs-only PRs and advisory): intended; resolved.pathjudge.py:154-173(security-held build file named as build risk): fixed inde1a409, resolved.config.py:592-596(host cannot pin hard paths): as decided; a host pin belongs to the review DSL. Left open.de1a409.🟡 werkrevisor: YELLOW / HUMAN_REVIEW
Change kind:
mixed.Reasons
Findings
[medium] Advisory downgrade also hides a failed or unreadable model review at
werkrevisor/cli.py:382-401, YELLOW, still open in an earlier line comment[medium] A renamed or deleted file in judge-all scope is judged on an incomplete picture at
werkrevisor/pathjudge.py:255-257YELLOW; Jev: real 0.37, severity 3.0 of 3
In
rules.pyonlyf.pathis checked viacats_by_file[f.path], and a file withprevious_path(a rename) in scope "all" is added tojudged_filesasmain. The security question then sees only the new path's diff, so a rename that moves an auth file out of its place is judged on the diff of content alone. The existing comment for judged paths ("A rename can move a build file out of its place") does not apply to the new branch, so a pure rename of a security-relevant file may get GREEN. No test covers renames in scope "all".[medium] PR document cites a test that does not exist at
docs/prs/2026-10-06-PR#39-jev-triage.md:150-160hint; Jev: real 0.83, severity 2.0 of 3
Scenario#39.09 lists
test_dbschema: test_the_db_schema_paths_are_configurablefor 39.08, which lives intests/test_dbschema.py. The actual file istests/test_dbschema.py, but the diff shows the test intests/test_dbschema.pyonly as a classDbSchemaPathsTest; the link targets look fine. The real mismatch: the PR document claims the Scenario 39.09 testtest_the_repository_config_comes_from_the_default_branchverifies "a PR into the other branch", but the test only callsdefault_branch_headandresolve_configdirectly and never runs a PR whose base is another branch. The scenario is therefore only partly verified.Classifier
15 of 15 units passed all questions (1 PR document(s)).
Usage: model `openrouter/anthropic/claude-sonnet-5.5`, LLM done, tokens counted before the call 53841, calls 1, tokens in 55253 / out 975 (reasoning included in out), duration 42.7 s, cost 0.1203 USD (reported), classifier `jev-1.13.0` calls 15, tokens in 13511, cost 0.0006 USD, finding judge `jev-1.13.0` calls 3, tokens in 4625, cost 0.0002 USD, excerpt selection `openrouter/openai/gpt-6-luna` calls 5, tokens in 50848 / out 1702, cost 0.0021 USD (reported), discarded after the context reduction, build file judge `jev-1.13.0` calls 1, tokens in 560, cost 0.0000 USD, config: defaults + host + repository (.werkrevisor.toml at base5 answers ignored because of a confidence below [classifier] min_confidence.
9bc9d60).Reviewed head `de1a4098e466b4fb934b323633f419bdbda34668` against base `9bc9d60fd6cf4b7c8399a6851f286910b6499964`, werkrevisor 0.20.6.