werkrevisor 0.20.2: a reused line comment gets the text of the new review #33

Merged
ai-reviewer merged 15 commits from claude-code/reused-comment-gets-new-text into main 2026-10-06 16:50:30 +02:00
Collaborator

See the PR doc.

🤖 Generated with Claude Code

See [the PR doc](docs/prs/2026-10-06-PR%2333-reused-line-comment-gets-new-text.md). 🤖 Generated with [Claude Code](https://claude.com/claude-code)
claude-code added 1 commit 2026-10-06 15:46:54 +02:00
docs: PR#33 doc, a reused line comment gets the text of the new review
werkator build succeeded after 00:38
werkator/server build succeeded after 00:44
27d8ba4e59
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
mi added 1 commit 2026-10-06 15:49:10 +02:00
werkrevisor 0.20.2: a reused line comment gets the text of the new review
werkator build succeeded after 00:14
ai-review/assessment human review required; a human approval unlocks the merge; review requested from mi; merge scheduled for when the checks succeed
ai-review/gate build succeeded after 00:20
werkator/server build succeeded after 00:34
a0bffc1408
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
claude-code marked the pull request as ready for review 2026-10-06 15:49:14 +02:00
Collaborator
🟡 werkrevisor YELLOW for a0bffc1, superseded by the result for 704fd0b

🟡 werkrevisor: YELLOW / HUMAN_REVIEW

Change kind: mixed.

Reasons

  • YELLOW 1 finding(s) possibly severe, but Jev doubts that they are real: werkrevisor/cli.py:207-215
  • hint 1 build change(s) of low risk according to Jev: server/build.gradle.kts (weakens checks 0.02, risk 0.00)
  • hint 1 finding(s) of low severity or rated minor by Jev: werkrevisor/cli.py:460-473
  • hint model recommends merge with caution

Findings

  1. [medium] Update note regex only matches a lowercase hex head SHA at werkrevisor/cli.py:207-215, YELLOW, as a line comment

  2. [medium] Edited line comment may be re-edited every round after redaction or &lt;!-- escaping at werkrevisor/cli.py:460-473, hint, as a line comment

Classifier

7 of 7 units passed all questions (1 PR document(s)).
2 answers ignored because of a confidence below [classifier] min_confidence.

Usage: model `openrouter/anthropic/claude-sonnet-5.5`, LLM done, tokens counted before the call 58753, calls 1, tokens in 57327 / out 527 (reasoning included in out), duration 16.1 s, cost 0.1199 USD (reported), classifier `jev-1.13.0` calls 7, tokens in 5573, cost 0.0002 USD, finding judge `jev-1.13.0` calls 2, tokens in 3358, cost 0.0001 USD, build file judge `jev-1.13.0` calls 1, tokens in 560, cost 0.0000 USD, config: defaults + host + repository (.werkrevisor.toml at base dcdcf48).
Reviewed head `a0bffc1408b48421502a1b88110e3c342b77736e` against base `dcdcf48bed8f9fc8a7752b3fddce300c185e8358`, werkrevisor 0.20.1.
<!-- werkrevisor:a0bffc1408b48421502a1b88110e3c342b77736e --> <!-- werkrevisor-verdict:YELLOW --> <!-- werkrevisor-folded --> <details><summary>🟡 werkrevisor YELLOW for <code>a0bffc1</code>, superseded by <a href="http://git.javagil.de/mi/werkrevisor/pulls/33#issuecomment-1409">the result for <code>704fd0b</code></a></summary> ## 🟡 werkrevisor: YELLOW / HUMAN_REVIEW Change kind: `mixed`. ### Reasons - **YELLOW** 1 finding(s) possibly severe, but Jev doubts that they are real: werkrevisor/cli.py:207-215 - **hint** 1 build change(s) of low risk according to Jev: server/build.gradle.kts (weakens checks 0.02, risk 0.00) - **hint** 1 finding(s) of low severity or rated minor by Jev: werkrevisor/cli.py:460-473 - **hint** model recommends merge with caution ### Findings 1. **[medium] Update note regex only matches a lowercase hex head SHA** at `werkrevisor/cli.py:207-215`, YELLOW, as a line comment 2. **[medium] Edited line comment may be re-edited every round after redaction or `&lt;!--` escaping** at `werkrevisor/cli.py:460-473`, hint, as a line comment ### Classifier 7 of 7 units passed all questions (1 PR document(s)). 2 answers ignored because of a confidence below [classifier] min_confidence. <sub> Usage: model `openrouter/anthropic/claude-sonnet-5.5`, LLM done, tokens counted before the call 58753, calls 1, tokens in 57327 / out 527 (reasoning included in out), duration 16.1 s, cost 0.1199 USD (reported), classifier `jev-1.13.0` calls 7, tokens in 5573, cost 0.0002 USD, finding judge `jev-1.13.0` calls 2, tokens in 3358, cost 0.0001 USD, build file judge `jev-1.13.0` calls 1, tokens in 560, cost 0.0000 USD, config: defaults + host + repository (.werkrevisor.toml at base dcdcf48).<br> Reviewed head `a0bffc1408b48421502a1b88110e3c342b77736e` against base `dcdcf48bed8f9fc8a7752b3fddce300c185e8358`, werkrevisor 0.20.1. </sub> </details>
ai-reviewer left a comment
Collaborator

werkrevisor: YELLOW / HUMAN_REVIEW for a0bffc1408b48421502a1b88110e3c342b77736e.
The result comment of werkrevisor holds the details.

werkrevisor: YELLOW / HUMAN_REVIEW for `a0bffc1408b48421502a1b88110e3c342b77736e`. The result comment of werkrevisor holds the details. <!-- werkrevisor-review:a0bffc1408b48421502a1b88110e3c342b77736e -->
@@ -203,5 +203,5 @@
stop = e
break
full = _reduced_context(reductions.pop(0), full, files, fetch, pr, config, deps, rules, diffs, result)
prompt_section = full.section + hints
extra_chars = (len(instructions) + len(pr_data.get("title") or "") + len(pr_data.get("body") or "")
Collaborator

[medium] Update note regex only matches a lowercase hex head SHA (werkrevisor)

YELLOW; Jev: real 0.18, severity 2.6 of 3

_UPDATE_NOTE ends with $, which matches before a trailing newline but not before other trailing whitespace. If Gitea returns the body with trailing whitespace or CRLF line endings, the note is not stripped. The comparison then fails and the comment is edited again each round, and the note is stacked on a body that already holds one. This is uncertain because it depends on Gitea's body normalization.

**[medium] Update note regex only matches a lowercase hex head SHA** (werkrevisor) YELLOW; Jev: real 0.18, severity 2.6 of 3 `_UPDATE_NOTE` ends with `$`, which matches before a trailing newline but not before other trailing whitespace. If Gitea returns the body with trailing whitespace or CRLF line endings, the note is not stripped. The comparison then fails and the comment is edited again each round, and the note is stacked on a body that already holds one. This is uncertain because it depends on Gitea's body normalization.
Author
Collaborator

Fixed in 704fd0b: _UPDATE_NOTE accepts any blanks and CRLF around the note and upper-case hex; the comparison normalizes line endings. Stacking cannot happen anyway: an edit replaces the whole body by the new text with exactly one note.

Fixed in 704fd0b: `_UPDATE_NOTE` accepts any blanks and CRLF around the note and upper-case hex; the comparison normalizes line endings. Stacking cannot happen anyway: an edit replaces the whole body by the new text with exactly one note.
claude-code marked this conversation as resolved
@@ -453,6 +457,22 @@ def _plan_line_comments(gitea, pr, result, bot_login):
return comments
def _update_line_comment(gitea, pr, earlier, text, redactor, updated):
Collaborator

[medium] Edited line comment may be re-edited every round after redaction or &lt;!-- escaping (werkrevisor)

hint; Jev: real 0.80, severity 2.0 of 3

The no-change check compares without_update_note(earlier["body"]) with redactor(text). Gitea may return a body that differs from what was posted, for example after normalization of line endings. In that case the comment is edited and marked as edited on every round, which the PR document says it avoids. The test fake stores the body verbatim, so this path is not covered. The effect is limited to extra edits, not data loss.

**[medium] Edited line comment may be re-edited every round after redaction or `&lt;!--` escaping** (werkrevisor) hint; Jev: real 0.80, severity 2.0 of 3 The no-change check compares `without_update_note(earlier["body"])` with `redactor(text)`. Gitea may return a body that differs from what was posted, for example after normalization of line endings. In that case the comment is edited and marked as edited on every round, which the PR document says it avoids. The test fake stores the body verbatim, so this path is not covered. The effect is limited to extra edits, not data loss.
Author
Collaborator

Fixed in 704fd0b: the no-change check now ignores line endings and trailing blanks (report.same_line_comment_text), so a body that Gitea normalized is no change; the test test_a_reused_line_comment_with_the_same_text_stays_untouched covers a body with CRLF and trailing blanks (red without the fix). Redaction and the &lt;!-- escaping are not affected: both are deterministic, the earlier comment was posted through the same redactor and line_comment.

Fixed in 704fd0b: the no-change check now ignores line endings and trailing blanks (`report.same_line_comment_text`), so a body that Gitea normalized is no change; the test `test_a_reused_line_comment_with_the_same_text_stays_untouched` covers a body with CRLF and trailing blanks (red without the fix). Redaction and the `&lt;!--` escaping are not affected: both are deterministic, the earlier comment was posted through the same redactor and `line_comment`.
claude-code marked this conversation as resolved
ai-reviewer scheduled this pull request to auto merge when all checks succeed 2026-10-06 15:49:32 +02:00
ai-reviewer requested review from mi 2026-10-06 15:49:32 +02:00
mi added 1 commit 2026-10-06 16:04:21 +02:00
review: line endings and trailing blanks from Gitea are no change of a reused line comment
werkator build succeeded after 00:09
ai-review/assessment human review required; a human approval unlocks the merge; a human review request is pending; merge already scheduled
ai-review/gate build succeeded after 00:16
werkator/server build succeeded after 00:28
704fd0bbc0
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Author
Collaborator

Answer to the review of a0bffc1 (YELLOW):

  • YELLOW werkrevisor/cli.py:207-215 (CRLF/trailing whitespace defeats the note regex): fixed in 704fd0b, thread resolved.
  • hint werkrevisor/cli.py:460-473 (re-edit every round after normalization): fixed in 704fd0b, thread resolved.
  • hint build change server/build.gradle.kts: only the version bump to 0.20.2.
  • hint merge with caution: no further action.
Answer to the review of `a0bffc1` (YELLOW): - YELLOW `werkrevisor/cli.py:207-215` (CRLF/trailing whitespace defeats the note regex): fixed in 704fd0b, thread resolved. - hint `werkrevisor/cli.py:460-473` (re-edit every round after normalization): fixed in 704fd0b, thread resolved. - hint build change `server/build.gradle.kts`: only the version bump to 0.20.2. - hint merge with caution: no further action.
Collaborator
🟡 werkrevisor YELLOW for 704fd0b, superseded by the result for c2043de

🟡 werkrevisor: YELLOW / HUMAN_REVIEW

Change kind: mixed.

Reasons

  • YELLOW spec: docs/prs/2026-10-06-PR#33-reused-line-comment-gets-new-text.md Scenario#33.01 given_when_then_misused (0.63 > yellow_above = 0.6)
  • YELLOW 1 finding(s) possibly severe, but Jev doubts that they are real: werkrevisor/cli.py:460-473
  • hint 1 build change(s) of low risk according to Jev: server/build.gradle.kts (weakens checks 0.02, risk 0.00)
  • hint 2 finding(s) of low severity or rated minor by Jev: werkrevisor/cli.py:715-716, werkrevisor/report.py:207-218
  • hint model recommends merge with caution

Findings

  1. [medium] Edit via issue-comment endpoint for line comments at werkrevisor/cli.py:460-473, YELLOW, still open in an earlier line comment

  2. [medium] Line comment is edited before the result comment is posted at werkrevisor/cli.py:715-716, hint, as a line comment

  3. [low] Update note regex is not anchored to bot-written text at werkrevisor/report.py:207-218, hint, as a line comment

Classifier

Unit Question Problem Confidence Verdict
docs/prs/2026-10-06-PR#33-reused-line-comment-gets-new-text.md Scenario#33.01 given_when_then_misused 0.63 – YELLOW

6 of 7 units passed all questions (1 PR document(s)).
2 answers ignored because of a confidence below [classifier] min_confidence.

Usage: model `openrouter/anthropic/claude-sonnet-5.5`, LLM done, tokens counted before the call 59279, calls 1, tokens in 57864 / out 758 (reasoning included in out), duration 13.5 s, cost 0.1233 USD (reported), classifier `jev-1.13.0` calls 7, tokens in 5593, cost 0.0002 USD, finding judge `jev-1.13.0` calls 2, tokens in 3208, cost 0.0001 USD, build file judge `jev-1.13.0` calls 1, tokens in 560, cost 0.0000 USD, config: defaults + host + repository (.werkrevisor.toml at base dcdcf48).
Reviewed head `704fd0bbc0531f5d7274bf18c0a5796e58533107` against base `dcdcf48bed8f9fc8a7752b3fddce300c185e8358`, werkrevisor 0.20.1.
<!-- werkrevisor:704fd0bbc0531f5d7274bf18c0a5796e58533107 --> <!-- werkrevisor-verdict:YELLOW --> <!-- werkrevisor-folded --> <details><summary>🟡 werkrevisor YELLOW for <code>704fd0b</code>, superseded by <a href="http://git.javagil.de/mi/werkrevisor/pulls/33#issuecomment-1417">the result for <code>c2043de</code></a></summary> ## 🟡 werkrevisor: YELLOW / HUMAN_REVIEW Change kind: `mixed`. ### Reasons - **YELLOW** spec: docs/prs/2026-10-06-PR#33-reused-line-comment-gets-new-text.md Scenario#33.01 given_when_then_misused (0.63 > yellow_above = 0.6) - **YELLOW** 1 finding(s) possibly severe, but Jev doubts that they are real: werkrevisor/cli.py:460-473 - **hint** 1 build change(s) of low risk according to Jev: server/build.gradle.kts (weakens checks 0.02, risk 0.00) - **hint** 2 finding(s) of low severity or rated minor by Jev: werkrevisor/cli.py:715-716, werkrevisor/report.py:207-218 - **hint** model recommends merge with caution ### Findings 1. **[medium] Edit via issue-comment endpoint for line comments** at `werkrevisor/cli.py:460-473`, YELLOW, still open in [an earlier line comment](http://git.javagil.de/mi/werkrevisor/pulls/33#issuecomment-1406) 2. **[medium] Line comment is edited before the result comment is posted** at `werkrevisor/cli.py:715-716`, hint, as a line comment 3. **[low] Update note regex is not anchored to bot-written text** at `werkrevisor/report.py:207-218`, hint, as a line comment ### Classifier | Unit | Question | Problem | Confidence | Verdict | |---|---|---|---|---| | docs/prs/2026-10-06-PR#33-reused-line-comment-gets-new-text.md Scenario#33.01 | given_when_then_misused | 0.63 | – | YELLOW | 6 of 7 units passed all questions (1 PR document(s)). 2 answers ignored because of a confidence below [classifier] min_confidence. <sub> Usage: model `openrouter/anthropic/claude-sonnet-5.5`, LLM done, tokens counted before the call 59279, calls 1, tokens in 57864 / out 758 (reasoning included in out), duration 13.5 s, cost 0.1233 USD (reported), classifier `jev-1.13.0` calls 7, tokens in 5593, cost 0.0002 USD, finding judge `jev-1.13.0` calls 2, tokens in 3208, cost 0.0001 USD, build file judge `jev-1.13.0` calls 1, tokens in 560, cost 0.0000 USD, config: defaults + host + repository (.werkrevisor.toml at base dcdcf48).<br> Reviewed head `704fd0bbc0531f5d7274bf18c0a5796e58533107` against base `dcdcf48bed8f9fc8a7752b3fddce300c185e8358`, werkrevisor 0.20.1. </sub> </details>
ai-reviewer left a comment
Collaborator

werkrevisor: YELLOW / HUMAN_REVIEW for 704fd0bbc0531f5d7274bf18c0a5796e58533107.
The result comment of werkrevisor holds the details.

werkrevisor: YELLOW / HUMAN_REVIEW for `704fd0bbc0531f5d7274bf18c0a5796e58533107`. The result comment of werkrevisor holds the details. <!-- werkrevisor-review:704fd0bbc0531f5d7274bf18c0a5796e58533107 -->
@@ -693,3 +713,3 @@
"""The result comment comes first: if anything fails later, no bot approval stands without its comment."""
fresh = existing is None and result.verdict is not Verdict.ERROR
comments = _plan_line_comments(gitea, pr, result, bot_login) if fresh else []
comments = _plan_line_comments(gitea, pr, result, bot_login, redactor) if fresh else []
Collaborator

[medium] Line comment is edited before the result comment is posted (werkrevisor)

hint; Jev: real 0.71, severity 2.1 of 3

_plan_line_comments now writes to Gitea (edits earlier line comments) before _post_comment runs. If the later steps fail, for example the result comment post or a superseded head, the earlier line comment already holds text from a review of a head that never got a result. The old behaviour had no write in the planning phase, so the "result comment comes first" docstring of _publish no longer holds for these edits. The impact is limited to wording and the update note naming the head.

**[medium] Line comment is edited before the result comment is posted** (werkrevisor) hint; Jev: real 0.71, severity 2.1 of 3 `_plan_line_comments` now writes to Gitea (edits earlier line comments) before `_post_comment` runs. If the later steps fail, for example the result comment post or a superseded head, the earlier line comment already holds text from a review of a head that never got a result. The old behaviour had no write in the planning phase, so the "result comment comes first" docstring of `_publish` no longer holds for these edits. The impact is limited to wording and the update note naming the head.
Author
Collaborator

Not changed, on purpose. The edit only brings the wording of the open finding up to date and names the head that wrote it; if the result comment fails afterwards, the next run of that head or of a later head writes the result and edits the text again. The docstring of _publish is about the approval, which still never stands without its result comment. Moving the edits behind the result comment would lose the fallback: a failed edit puts the new text into the result comment, so the result comment must know the outcome of the edit first. Left open for the human reviewer.

Not changed, on purpose. The edit only brings the wording of the open finding up to date and names the head that wrote it; if the result comment fails afterwards, the next run of that head or of a later head writes the result and edits the text again. The docstring of `_publish` is about the approval, which still never stands without its result comment. Moving the edits behind the result comment would lose the fallback: a failed edit puts the new text into the result comment, so the result comment must know the outcome of the edit first. Left open for the human reviewer.
claude-code marked this conversation as resolved
@@ -195,6 +204,20 @@ def line_comment(f, v):
return body.replace("<!--", "&lt;!--")
_UPDATE_NOTE = re.compile(r"\s*<sub>Updated by the review of `[0-9a-fA-F]+`\.</sub>\s*$")
Collaborator

[low] Update note regex is not anchored to bot-written text (werkrevisor)

hint

same_line_comment_text strips a trailing <sub>Updated by ...</sub> from the earlier body. This is harmless for bot comments, but a model finding ending with that literal text would also be stripped before comparison, so a real change would not be detected. This needs deliberate model output and has low impact.

**[low] Update note regex is not anchored to bot-written text** (werkrevisor) hint `same_line_comment_text` strips a trailing `<sub>Updated by ...</sub>` from the earlier body. This is harmless for bot comments, but a model finding ending with that literal text would also be stripped before comparison, so a real change would not be detected. This needs deliberate model output and has low impact.
Author
Collaborator

Not changed: only a model text that ends with exactly this note line, with a hex SHA, is affected, and the only consequence is one skipped edit of wording; the finding itself stays open and listed. Left open for the human reviewer.

Not changed: only a model text that ends with exactly this note line, with a hex SHA, is affected, and the only consequence is one skipped edit of wording; the finding itself stays open and listed. Left open for the human reviewer.
claude-code marked this conversation as resolved
mi added 1 commit 2026-10-06 16:07:31 +02:00
review: a reply in the thread of a bot line comment is neither reused nor an open hint
werkator build succeeded after 00:11
ai-review/assessment human review required; a human approval unlocks the merge; a human review request is pending; merge already scheduled
ai-review/gate build succeeded after 00:19
werkator/server build succeeded after 00:30
c2043deb92
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Author
Collaborator

Answer to the review of 704fd0b (YELLOW):

  • YELLOW spec Scenario#33.01 given_when_then_misused: the When step held state; fixed in c2043de, the different wording is now a Given.
  • YELLOW werkrevisor/cli.py:460-473 (edit via the issue-comment endpoint): PATCH /issues/comments/{id} edits review comments too on Gitea 1.27; it was checked on git.javagil.de before this PR.
    This finding also showed a real, older bug: werkrevisor 0.20.1 reused my reply 1406 as the "earlier line comment", because Gitea lists a reply among the comments of the review that started the thread; the reply also counted as an open hint, and with this PR the bot would have tried to edit it.
    Fixed in c2043de: only comments written by the bot count, with Scenario#33.04 and red/green tests.
  • hint werkrevisor/cli.py:715-716 (edit before the result comment): not changed, reasons in the thread.
  • hint werkrevisor/report.py:207-218 (note regex not tied to bot text): not changed, reasons in the thread.
  • hint build change server/build.gradle.kts: only the version bump.

The reviews run with the deployed 0.20.1, so this PR still shows the old behaviour: the reused comment kept its text.

Answer to the review of `704fd0b` (YELLOW): - YELLOW spec Scenario#33.01 `given_when_then_misused`: the When step held state; fixed in c2043de, the different wording is now a Given. - YELLOW `werkrevisor/cli.py:460-473` (edit via the issue-comment endpoint): `PATCH /issues/comments/{id}` edits review comments too on Gitea 1.27; it was checked on git.javagil.de before this PR. This finding also showed a real, older bug: werkrevisor 0.20.1 reused my reply 1406 as the "earlier line comment", because Gitea lists a reply among the comments of the review that started the thread; the reply also counted as an open hint, and with this PR the bot would have tried to edit it. Fixed in c2043de: only comments written by the bot count, with Scenario#33.04 and red/green tests. - hint `werkrevisor/cli.py:715-716` (edit before the result comment): not changed, reasons in the thread. - hint `werkrevisor/report.py:207-218` (note regex not tied to bot text): not changed, reasons in the thread. - hint build change `server/build.gradle.kts`: only the version bump. The reviews run with the deployed 0.20.1, so this PR still shows the old behaviour: the reused comment kept its text.
Collaborator
🟡 werkrevisor YELLOW for c2043de, superseded by the result for f572e65

🟡 werkrevisor: YELLOW / HUMAN_REVIEW

Change kind: mixed.

Reasons

  • YELLOW spec: docs/prs/2026-10-06-PR#33-reused-line-comment-gets-new-text.md Scenario#33.01 given_when_then_misused (0.71 > yellow_above = 0.6)
  • hint 1 build change(s) of low risk according to Jev: server/build.gradle.kts (weakens checks 0.02, risk 0.00)
  • hint 2 finding(s) of low severity or rated minor by Jev: werkrevisor/cli.py:426-430, docs/prs/2026-10-06-PR#33-reused-line-comment-gets-new-text.md:69-82
  • hint model recommends merge with caution

Findings

  1. [medium] Redacted text compared against the unredacted body, and the held-hint count includes reused comments at werkrevisor/cli.py:426-430, hint, as a line comment

  2. [low] Scenario 33.04 names a test file that does not contain one of its tests at docs/prs/2026-10-06-PR#33-reused-line-comment-gets-new-text.md:69-82, hint, as a line comment

Classifier

Unit Question Problem Confidence Verdict
docs/prs/2026-10-06-PR#33-reused-line-comment-gets-new-text.md Scenario#33.01 given_when_then_misused 0.71 – YELLOW

7 of 8 units passed all questions (1 PR document(s)).
2 answers ignored because of a confidence below [classifier] min_confidence.

Usage: model `openrouter/anthropic/claude-sonnet-5.5`, LLM done, tokens counted before the call 65440, calls 1, tokens in 63836 / out 678 (reasoning included in out), duration 16.0 s, cost 0.1345 USD (reported), classifier `jev-1.13.0` calls 8, tokens in 6647, cost 0.0003 USD, finding judge `jev-1.13.0` calls 1, tokens in 1688, cost 0.0001 USD, build file judge `jev-1.13.0` calls 1, tokens in 560, cost 0.0000 USD, config: defaults + host + repository (.werkrevisor.toml at base dcdcf48).
Reviewed head `c2043deb92eb15ed8d59ff8607a1366b064a340b` against base `dcdcf48bed8f9fc8a7752b3fddce300c185e8358`, werkrevisor 0.20.1.
<!-- werkrevisor:c2043deb92eb15ed8d59ff8607a1366b064a340b --> <!-- werkrevisor-verdict:YELLOW --> <!-- werkrevisor-folded --> <details><summary>🟡 werkrevisor YELLOW for <code>c2043de</code>, superseded by <a href="http://git.javagil.de/mi/werkrevisor/pulls/33#issuecomment-1425">the result for <code>f572e65</code></a></summary> ## 🟡 werkrevisor: YELLOW / HUMAN_REVIEW Change kind: `mixed`. ### Reasons - **YELLOW** spec: docs/prs/2026-10-06-PR#33-reused-line-comment-gets-new-text.md Scenario#33.01 given_when_then_misused (0.71 > yellow_above = 0.6) - **hint** 1 build change(s) of low risk according to Jev: server/build.gradle.kts (weakens checks 0.02, risk 0.00) - **hint** 2 finding(s) of low severity or rated minor by Jev: werkrevisor/cli.py:426-430, docs/prs/2026-10-06-PR#33-reused-line-comment-gets-new-text.md:69-82 - **hint** model recommends merge with caution ### Findings 1. **[medium] Redacted text compared against the unredacted body, and the held-hint count includes reused comments** at `werkrevisor/cli.py:426-430`, hint, as a line comment 2. **[low] Scenario 33.04 names a test file that does not contain one of its tests** at `docs/prs/2026-10-06-PR#33-reused-line-comment-gets-new-text.md:69-82`, hint, as a line comment ### Classifier | Unit | Question | Problem | Confidence | Verdict | |---|---|---|---|---| | docs/prs/2026-10-06-PR#33-reused-line-comment-gets-new-text.md Scenario#33.01 | given_when_then_misused | 0.71 | – | YELLOW | 7 of 8 units passed all questions (1 PR document(s)). 2 answers ignored because of a confidence below [classifier] min_confidence. <sub> Usage: model `openrouter/anthropic/claude-sonnet-5.5`, LLM done, tokens counted before the call 65440, calls 1, tokens in 63836 / out 678 (reasoning included in out), duration 16.0 s, cost 0.1345 USD (reported), classifier `jev-1.13.0` calls 8, tokens in 6647, cost 0.0003 USD, finding judge `jev-1.13.0` calls 1, tokens in 1688, cost 0.0001 USD, build file judge `jev-1.13.0` calls 1, tokens in 560, cost 0.0000 USD, config: defaults + host + repository (.werkrevisor.toml at base dcdcf48).<br> Reviewed head `c2043deb92eb15ed8d59ff8607a1366b064a340b` against base `dcdcf48bed8f9fc8a7752b3fddce300c185e8358`, werkrevisor 0.20.1. </sub> </details>
ai-reviewer left a comment
Collaborator

werkrevisor: YELLOW / HUMAN_REVIEW for c2043deb92eb15ed8d59ff8607a1366b064a340b.
The result comment of werkrevisor holds the details.

werkrevisor: YELLOW / HUMAN_REVIEW for `c2043deb92eb15ed8d59ff8607a1366b064a340b`. The result comment of werkrevisor holds the details. <!-- werkrevisor-review:c2043deb92eb15ed8d59ff8607a1366b064a340b -->
@@ -0,0 +66,4 @@
### Feature: only comments of the bot count as its line comments
#### Scenario#33.04: a reply in the thread of a bot line comment is neither reused nor a hint
Collaborator

[low] Scenario 33.04 names a test file that does not contain one of its tests (werkrevisor)

hint

Both entries under "Verified by" for Scenario#33.04 are listed, but the merge-hold test is in tests/test_hints_hold_merge.py, and the scenario's Given says the bot comment is "resolved" while only one test sets it up this way. The link text is consistent with the code. However, the scenario title claims a reply is "neither reused nor a hint", and the two tests each cover only one half. This is acceptable, but the Given/When/Then mixes both halves in a single scenario, which makes it hard to read.

**[low] Scenario 33.04 names a test file that does not contain one of its tests** (werkrevisor) hint Both entries under "Verified by" for Scenario#33.04 are listed, but the merge-hold test is in `tests/test_hints_hold_merge.py`, and the scenario's Given says the bot comment is "resolved" while only one test sets it up this way. The link text is consistent with the code. However, the scenario title claims a reply is "neither reused nor a hint", and the two tests each cover only one half. This is acceptable, but the Given/When/Then mixes both halves in a single scenario, which makes it hard to read.
Author
Collaborator

Fixed in f572e65: the reply case is split into Scenario#33.04 (not reused) and Scenario#33.05 (no hold), one test each.

Fixed in f572e65: the reply case is split into Scenario#33.04 (not reused) and Scenario#33.05 (no hold), one test each.
claude-code marked this conversation as resolved
@@ -425,3 +425,2 @@
for c in gitea.list_review_comments(pr.number, review["id"]) or ():
if not c.get("resolver") and (c.get("position") or 0) > 0:
found.append({"path": c.get("path"), "line": c["position"], "url": c.get("html_url") or ""})
# A reply of a human is listed in the review that started the thread, but is no hint of the bot.
Collaborator

[medium] Redacted text compared against the unredacted body, and the held-hint count includes reused comments (werkrevisor)

hint; Jev: real 0.51, severity 2.1 of 3

_update_line_comment redacts text before comparing it with earlier["body"]. Redacted text is also what gets posted, so a secret in the model text is stored as [REDACTED] and the comparison against the stored body stays consistent. This part is fine. A real problem is that a reply from the bot itself in the same thread (author equals bot_login) is still treated as a reusable line comment, because the filter checks only the author, not whether the comment starts a thread. A bot reply would then be edited and counted as a hint. This is an edge case, because the bot does not reply to threads in the code shown. The uncertainty is whether the bot ever writes replies.

**[medium] Redacted text compared against the unredacted body, and the held-hint count includes reused comments** (werkrevisor) hint; Jev: real 0.51, severity 2.1 of 3 `_update_line_comment` redacts `text` before comparing it with `earlier["body"]`. Redacted text is also what gets posted, so a secret in the model text is stored as `[REDACTED]` and the comparison against the stored body stays consistent. This part is fine. A real problem is that a reply from the bot itself in the same thread (author equals `bot_login`) is still treated as a reusable line comment, because the filter checks only the author, not whether the comment starts a thread. A bot reply would then be edited and counted as a hint. This is an edge case, because the bot does not reply to threads in the code shown. The uncertainty is whether the bot ever writes replies.
Author
Collaborator

Not changed: werkrevisor never replies in a thread; it posts line comments only as part of its own review. Should it ever reply, its reply would be bot text in an open thread, and reusing or counting it would be right. Left open for the human reviewer.

Not changed: werkrevisor never replies in a thread; it posts line comments only as part of its own review. Should it ever reply, its reply would be bot text in an open thread, and reusing or counting it would be right. Left open for the human reviewer.
claude-code marked this conversation as resolved
mi added 1 commit 2026-10-06 16:08:46 +02:00
review: Scenario#33.01 with concrete texts, the reply case split into Scenario#33.04 and #33.05
werkator/server build succeeded after 00:29
werkator build succeeded after 00:08
ai-review/assessment AI review passed; approved by werkrevisor; merge scheduled for when the checks succeed
ai-review/gate build succeeded after 00:01
f572e656d1
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Author
Collaborator

Answer to the review of c2043de (YELLOW):

  • YELLOW spec Scenario#33.01 given_when_then_misused: reworded in f572e65 with concrete texts: the Given holds the earlier comment, the When the new review, the Then the result.
  • hint werkrevisor/cli.py:426-430 (a bot reply would count): not changed, the bot never replies; reasons in the thread.
  • hint PR doc Scenario#33.04 mixes two halves: fixed in f572e65, split into #33.04 and #33.05, thread resolved.
  • hint build change server/build.gradle.kts: only the version bump.
Answer to the review of `c2043de` (YELLOW): - YELLOW spec Scenario#33.01 `given_when_then_misused`: reworded in f572e65 with concrete texts: the Given holds the earlier comment, the When the new review, the Then the result. - hint `werkrevisor/cli.py:426-430` (a bot reply would count): not changed, the bot never replies; reasons in the thread. - hint PR doc Scenario#33.04 mixes two halves: fixed in f572e65, split into #33.04 and #33.05, thread resolved. - hint build change `server/build.gradle.kts`: only the version bump.
Collaborator
🟢 werkrevisor GREEN for f572e65, superseded by the result for dfd5f69

🟢 werkrevisor: GREEN / PASS

Gate: approved by werkrevisor; merge waits for 10 open hint(s): reply to each line comment and resolve it, then restart the review or push a commit.

Change kind: mixed.

Reasons

  • hint 1 build change(s) of low risk according to Jev: server/build.gradle.kts (weakens checks 0.02, risk 0.00)
  • hint 2 finding(s) of low severity or rated minor by Jev: werkrevisor/cli.py:453-454, werkrevisor/cli.py:462-475
  • hint model recommends merge with caution

Findings

  1. [medium] Reused comment is edited while publishing can still fail at werkrevisor/cli.py:453-454, hint, as a line comment

  2. [medium] Line comment body with a redacted text and the update note differ in comparison at werkrevisor/cli.py:462-475, hint, still open in an earlier line comment

Classifier

9 of 9 units passed all questions (1 PR document(s)).
2 answers ignored because of a confidence below [classifier] min_confidence.

Usage: model `openrouter/anthropic/claude-sonnet-5.5`, LLM done, tokens counted before the call 65479, calls 1, tokens in 63872 / out 637 (reasoning included in out), duration 12.9 s, cost 0.1341 USD (reported), classifier `jev-1.13.0` calls 9, tokens in 7526, cost 0.0003 USD, finding judge `jev-1.13.0` calls 2, tokens in 3291, cost 0.0001 USD, build file judge `jev-1.13.0` calls 1, tokens in 560, cost 0.0000 USD, config: defaults + host + repository (.werkrevisor.toml at base dcdcf48).
Reviewed head `f572e656d1ae88e70a2b0d1a36a054371e42692e` against base `dcdcf48bed8f9fc8a7752b3fddce300c185e8358`, werkrevisor 0.20.1.
<!-- werkrevisor:f572e656d1ae88e70a2b0d1a36a054371e42692e --> <!-- werkrevisor-verdict:GREEN --> <!-- werkrevisor-folded --> <details><summary>🟢 werkrevisor GREEN for <code>f572e65</code>, superseded by <a href="http://git.javagil.de/mi/werkrevisor/pulls/33#issuecomment-1435">the result for <code>dfd5f69</code></a></summary> ## 🟢 werkrevisor: GREEN / PASS Gate: approved by werkrevisor; merge waits for 10 open hint(s): reply to each line comment and resolve it, then restart the review or push a commit. Change kind: `mixed`. ### Reasons - **hint** 1 build change(s) of low risk according to Jev: server/build.gradle.kts (weakens checks 0.02, risk 0.00) - **hint** 2 finding(s) of low severity or rated minor by Jev: werkrevisor/cli.py:453-454, werkrevisor/cli.py:462-475 - **hint** model recommends merge with caution ### Findings 1. **[medium] Reused comment is edited while publishing can still fail** at `werkrevisor/cli.py:453-454`, hint, as a line comment 2. **[medium] Line comment body with a redacted text and the update note differ in comparison** at `werkrevisor/cli.py:462-475`, hint, still open in [an earlier line comment](http://git.javagil.de/mi/werkrevisor/pulls/33#issuecomment-1406) ### Classifier 9 of 9 units passed all questions (1 PR document(s)). 2 answers ignored because of a confidence below [classifier] min_confidence. <sub> Usage: model `openrouter/anthropic/claude-sonnet-5.5`, LLM done, tokens counted before the call 65479, calls 1, tokens in 63872 / out 637 (reasoning included in out), duration 12.9 s, cost 0.1341 USD (reported), classifier `jev-1.13.0` calls 9, tokens in 7526, cost 0.0003 USD, finding judge `jev-1.13.0` calls 2, tokens in 3291, cost 0.0001 USD, build file judge `jev-1.13.0` calls 1, tokens in 560, cost 0.0000 USD, config: defaults + host + repository (.werkrevisor.toml at base dcdcf48).<br> Reviewed head `f572e656d1ae88e70a2b0d1a36a054371e42692e` against base `dcdcf48bed8f9fc8a7752b3fddce300c185e8358`, werkrevisor 0.20.1. </sub> </details>
ai-reviewer approved these changes 2026-10-06 16:09:00 +02:00
Dismissed
ai-reviewer left a comment
Collaborator

werkrevisor: GREEN / PASS for f572e656d1ae88e70a2b0d1a36a054371e42692e.
The result comment of werkrevisor holds the details.

werkrevisor: GREEN / PASS for `f572e656d1ae88e70a2b0d1a36a054371e42692e`. The result comment of werkrevisor holds the details. <!-- werkrevisor-review:f572e656d1ae88e70a2b0d1a36a054371e42692e -->
@@ -446,6 +450,8 @@ def _plan_line_comments(gitea, pr, result, bot_login):
and first - LINE_SLACK <= c["line"] <= last + LINE_SLACK), None)
if earlier is not None:
result.reused[i] = earlier["url"]
if not _update_line_comment(gitea, pr, earlier, report.line_comment(f, v), redactor, updated):
Collaborator

[medium] Reused comment is edited while publishing can still fail (werkrevisor)

hint; Jev: real 0.76, severity 2.4 of 3

_plan_line_comments edits the earlier line comment before the result comment is posted and before the review is submitted. If _post_comment raises afterwards, the run ends in ERROR but the earlier comment already carries the new text and the "Updated by" note. A rerun on the same head then finds an equal text, so it makes no edit and does not report the finding in the result comment. The side effect also breaks the docstring contract of _publish that the result comment comes first. The impact is limited: the edited line comment keeps the finding visible.

**[medium] Reused comment is edited while publishing can still fail** (werkrevisor) hint; Jev: real 0.76, severity 2.4 of 3 `_plan_line_comments` edits the earlier line comment before the result comment is posted and before the review is submitted. If `_post_comment` raises afterwards, the run ends in ERROR but the earlier comment already carries the new text and the "Updated by" note. A rerun on the same head then finds an equal text, so it makes no edit and does not report the finding in the result comment. The side effect also breaks the docstring contract of `_publish` that the result comment comes first. The impact is limited: the edited line comment keeps the finding visible.
Author
Collaborator

Not changed: the same point as the thread at werkrevisor/cli.py:715; the result comment must know the outcome of the edit to carry the text after a failed edit, and the next review writes the wording again. Left open for the human reviewer.

Not changed: the same point as the thread at `werkrevisor/cli.py:715`; the result comment must know the outcome of the edit to carry the text after a failed edit, and the next review writes the wording again. Left open for the human reviewer.
claude-code marked this conversation as resolved
ai-reviewer canceled auto merging this pull request when all checks succeed 2026-10-06 16:09:00 +02:00
Author
Collaborator

Answer to the review of f572e65 (GREEN, merge held for 10 open hints):

  • hint werkrevisor/cli.py:453-454 (edit while publishing can fail): not changed, as in the thread at cli.py:715.
  • hint werkrevisor/cli.py:462-475 (redacted text and update note in the comparison): its text is lost, since 0.20.1 reused my reply 1406 again, which is the bug this PR fixes. From the title: the comparison strips the note from the stored body and compares it with the redacted new text; the posted body is exactly the redacted text plus the note, so both sides match. Not changed.
  • 6 of the 10 open hints are my replies, counted by the deployed 0.20.1; with 0.20.2 only the 4 open bot comments count.
Answer to the review of `f572e65` (GREEN, merge held for 10 open hints): - hint `werkrevisor/cli.py:453-454` (edit while publishing can fail): not changed, as in the thread at `cli.py:715`. - hint `werkrevisor/cli.py:462-475` (redacted text and update note in the comparison): its text is lost, since 0.20.1 reused my reply 1406 again, which is the bug this PR fixes. From the title: the comparison strips the note from the stored body and compares it with the redacted new text; the posted body is exactly the redacted text plus the note, so both sides match. Not changed. - 6 of the 10 open hints are my replies, counted by the deployed 0.20.1; with 0.20.2 only the 4 open bot comments count.
ai-reviewer scheduled this pull request to auto merge when all checks succeed 2026-10-06 16:21:41 +02:00
Collaborator

werkrevisor, review restarted for f572e656d1ae88e70a2b0d1a36a054371e42692e: approved by werkrevisor; merge scheduled for when the checks succeed.

werkrevisor, review restarted for `f572e656d1ae88e70a2b0d1a36a054371e42692e`: approved by werkrevisor; merge scheduled for when the checks succeed.
mi added 1 commit 2026-10-06 16:24:13 +02:00
review: a GREEN review that schedules the merge withdraws the review request of an earlier round
werkator build succeeded after 00:09
ai-review/assessment AI findings; a developer approval after checking them unlocks the merge; a human review request is pending; merge already scheduled
ai-review/gate build succeeded after 00:17
werkator/server build succeeded after 00:28
dfd5f6972d
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
mi dismissed ai-reviewer's review 2026-10-06 16:24:13 +02:00
Reason:

New commits pushed, approval review dismissed automatically according to repository settings

Collaborator
🟠 werkrevisor ORANGE for dfd5f69, superseded by the result for 08ecc13

🟠 werkrevisor: ORANGE / FINDINGS

Change kind: mixed.

Reasons

  • ORANGE 1 finding(s) that Jev rates as probably real and severe: werkrevisor/cli.py:530-533
  • YELLOW risky paths changed: werkrevisor/gitea.py (werkrevisor/gitea.py)
  • YELLOW spec: docs/prs/2026-10-06-PR#33-reused-line-comment-gets-new-text.md Scenario#33.06 given_when_then_misused (0.62 > yellow_above = 0.6)
  • hint 1 build change(s) of low risk according to Jev: server/build.gradle.kts (weakens checks 0.02, risk 0.00)
  • hint 2 finding(s) of low severity or rated minor by Jev: werkrevisor/cli.py:557-561, docs/prs/2026-10-06-PR#33-reused-line-comment-gets-new-text.md:95-103
  • hint model recommends merge with caution

Findings

  1. [medium] Withdrawal of review requests has no test with the automerge flag off or a matching case at werkrevisor/cli.py:530-533, ORANGE, as a line comment

  2. [medium] Held-merge hint count ignores finding text carried in the result comment at werkrevisor/cli.py:557-561, hint, as a line comment

  3. [low] Scenario#33.06 misuses Given/When/Then at docs/prs/2026-10-06-PR#33-reused-line-comment-gets-new-text.md:95-103, hint, as a line comment

Classifier

Unit Question Problem Confidence Verdict
docs/prs/2026-10-06-PR#33-reused-line-comment-gets-new-text.md Scenario#33.06 given_when_then_misused 0.62 – YELLOW

9 of 10 units passed all questions (1 PR document(s)).
4 answers ignored because of a confidence below [classifier] min_confidence.

Usage: model `openrouter/anthropic/claude-sonnet-5.5`, LLM done, tokens counted before the call 73240, calls 1, tokens in 71417 / out 749 (reasoning included in out), duration 14.0 s, cost 0.1503 USD (reported), classifier `jev-1.13.0` calls 10, tokens in 8574, cost 0.0004 USD, finding judge `jev-1.13.0` calls 2, tokens in 3425, cost 0.0001 USD, build file judge `jev-1.13.0` calls 1, tokens in 560, cost 0.0000 USD, config: defaults + host + repository (.werkrevisor.toml at base dcdcf48).
Reviewed head `dfd5f6972d230c671e518537f57f7575dab88251` against base `dcdcf48bed8f9fc8a7752b3fddce300c185e8358`, werkrevisor 0.20.1.
<!-- werkrevisor:dfd5f6972d230c671e518537f57f7575dab88251 --> <!-- werkrevisor-verdict:ORANGE --> <!-- werkrevisor-folded --> <details><summary>🟠 werkrevisor ORANGE for <code>dfd5f69</code>, superseded by <a href="http://git.javagil.de/mi/werkrevisor/pulls/33#issuecomment-1445">the result for <code>08ecc13</code></a></summary> ## 🟠 werkrevisor: ORANGE / FINDINGS Change kind: `mixed`. ### Reasons - **ORANGE** 1 finding(s) that Jev rates as probably real and severe: werkrevisor/cli.py:530-533 - **YELLOW** risky paths changed: werkrevisor/gitea.py (werkrevisor/gitea.py) - **YELLOW** spec: docs/prs/2026-10-06-PR#33-reused-line-comment-gets-new-text.md Scenario#33.06 given_when_then_misused (0.62 > yellow_above = 0.6) - **hint** 1 build change(s) of low risk according to Jev: server/build.gradle.kts (weakens checks 0.02, risk 0.00) - **hint** 2 finding(s) of low severity or rated minor by Jev: werkrevisor/cli.py:557-561, docs/prs/2026-10-06-PR#33-reused-line-comment-gets-new-text.md:95-103 - **hint** model recommends merge with caution ### Findings 1. **[medium] Withdrawal of review requests has no test with the automerge flag off or a matching case** at `werkrevisor/cli.py:530-533`, ORANGE, as a line comment 2. **[medium] Held-merge hint count ignores finding text carried in the result comment** at `werkrevisor/cli.py:557-561`, hint, as a line comment 3. **[low] Scenario#33.06 misuses Given/When/Then** at `docs/prs/2026-10-06-PR#33-reused-line-comment-gets-new-text.md:95-103`, hint, as a line comment ### Classifier | Unit | Question | Problem | Confidence | Verdict | |---|---|---|---|---| | docs/prs/2026-10-06-PR#33-reused-line-comment-gets-new-text.md Scenario#33.06 | given_when_then_misused | 0.62 | – | YELLOW | 9 of 10 units passed all questions (1 PR document(s)). 4 answers ignored because of a confidence below [classifier] min_confidence. <sub> Usage: model `openrouter/anthropic/claude-sonnet-5.5`, LLM done, tokens counted before the call 73240, calls 1, tokens in 71417 / out 749 (reasoning included in out), duration 14.0 s, cost 0.1503 USD (reported), classifier `jev-1.13.0` calls 10, tokens in 8574, cost 0.0004 USD, finding judge `jev-1.13.0` calls 2, tokens in 3425, cost 0.0001 USD, build file judge `jev-1.13.0` calls 1, tokens in 560, cost 0.0000 USD, config: defaults + host + repository (.werkrevisor.toml at base dcdcf48).<br> Reviewed head `dfd5f6972d230c671e518537f57f7575dab88251` against base `dcdcf48bed8f9fc8a7752b3fddce300c185e8358`, werkrevisor 0.20.1. </sub> </details>
ai-reviewer left a comment
Collaborator

werkrevisor: ORANGE / FINDINGS for dfd5f6972d230c671e518537f57f7575dab88251.
The result comment of werkrevisor holds the details.

werkrevisor: ORANGE / FINDINGS for `dfd5f6972d230c671e518537f57f7575dab88251`. The result comment of werkrevisor holds the details. <!-- werkrevisor-review:dfd5f6972d230c671e518537f57f7575dab88251 -->
@@ -0,0 +92,4 @@
### Feature: a GREEN review releases its review request
#### Scenario#33.06: a scheduled merge withdraws the review request of an earlier round
Collaborator

[low] Scenario#33.06 misuses Given/When/Then (werkrevisor)

hint

The scenario has two "Given" lines: an automerge setting and a historic YELLOW round. The "So that" line also states a mechanism rather than a goal. The classifier flagged this scenario. Rephrase it so each step has one clear role.

**[low] Scenario#33.06 misuses Given/When/Then** (werkrevisor) hint The scenario has two "Given" lines: an automerge setting and a historic YELLOW round. The "So that" line also states a mechanism rather than a goal. The classifier flagged this scenario. Rephrase it so each step has one clear role.
Author
Collaborator

Fixed in 08ecc13: one Given (the open request), one When (the GREEN review schedules the merge), one Then; the So-that states the goal.

Fixed in 08ecc13: one Given (the open request), one When (the GREEN review schedules the merge), one Then; the So-that states the goal.
claude-code marked this conversation as resolved
@@ -506,3 +528,3 @@
exit_code = 0 if cancelled else Verdict.YELLOW.exit_code
return GateDecision(exit_code, "approved by werkrevisor" + note, held=True)
return GateDecision(0, "approved by werkrevisor" + _schedule_merge(config, pr, gitea, redactor, base_ref))
scheduled = _schedule_merge(config, pr, gitea, redactor, base_ref)
Collaborator

[medium] Withdrawal of review requests has no test with the automerge flag off or a matching case (werkrevisor)

ORANGE; Jev: real 0.87, severity 3.0 of 3

_withdraw_review_request withdraws requests when the merge is scheduled, but the withdrawal runs even when _schedule_merge returned "merge not scheduled" (for example when branch protection is missing or the merge call failed). The review request is then withdrawn without a scheduled merge, which drops the human's pending review and leaves the PR unlocked by neither path. Withdraw only when the scheduling note starts with "; merge scheduled" or "; merge already scheduled".

**[medium] Withdrawal of review requests has no test with the automerge flag off or a matching case** (werkrevisor) ORANGE; Jev: real 0.87, severity 3.0 of 3 `_withdraw_review_request` withdraws requests when the merge is scheduled, but the withdrawal runs even when `_schedule_merge` returned "merge not scheduled" (for example when branch protection is missing or the merge call failed). The review request is then withdrawn without a scheduled merge, which drops the human's pending review and leaves the PR unlocked by neither path. Withdraw only when the scheduling note starts with "; merge scheduled" or "; merge already scheduled".
Author
Collaborator

Fixed in 08ecc13: the request is withdrawn only after merge scheduled or merge already scheduled; test_a_merge_not_scheduled_keeps_the_review_request covers a missing branch protection (red without the fix).

Fixed in 08ecc13: the request is withdrawn only after `merge scheduled` or `merge already scheduled`; `test_a_merge_not_scheduled_keeps_the_review_request` covers a missing branch protection (red without the fix).
claude-code marked this conversation as resolved
@@ -553,5 +553,5 @@
if not config.gate.automerge:
return ""
try:
# Without these rules Gitea would merge a YELLOW or ORANGE PR as soon as the checks pass, unapproved.
branch = gitea.get_branch(base_ref) or {}
Collaborator

[medium] Held-merge hint count ignores finding text carried in the result comment (werkrevisor)

hint; Jev: real 0.75, severity 2.3 of 3

_open_hints excludes every index in result.reused from in_comment_only. A reused finding whose edit failed (reused_not_updated) now has its full text only in the result comment, yet it counts as neither in-comment nor attached. It is still counted through the open line comment, so the count is not lost, but the gate note says "reply to each line comment" while the new text sits elsewhere. No test covers the hold together with a failed edit.

**[medium] Held-merge hint count ignores finding text carried in the result comment** (werkrevisor) hint; Jev: real 0.75, severity 2.3 of 3 `_open_hints` excludes every index in `result.reused` from `in_comment_only`. A reused finding whose edit failed (`reused_not_updated`) now has its full text only in the result comment, yet it counts as neither in-comment nor attached. It is still counted through the open line comment, so the count is not lost, but the gate note says "reply to each line comment" while the new text sits elsewhere. No test covers the hold together with a failed edit.
Author
Collaborator

Not changed: the count stays right, as the finding says, since the open line comment counts. The author still replies to that line comment and resolves it; the result comment only carries the newer wording. Left open for the human reviewer.

Not changed: the count stays right, as the finding says, since the open line comment counts. The author still replies to that line comment and resolves it; the result comment only carries the newer wording. Left open for the human reviewer.
mi added 1 commit 2026-10-06 16:26:34 +02:00
review: the review request is withdrawn only after a scheduled merge; Scenario#33.06 with one role per step
werkator build succeeded after 00:09
ai-review/assessment AI findings; a developer approval after checking them unlocks the merge; a human review request is pending; merge already scheduled
ai-review/gate build succeeded after 00:23
werkator/server build succeeded after 00:28
08ecc13cbb
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Author
Collaborator

Answer to the review of dfd5f69 (ORANGE):

  • ORANGE werkrevisor/cli.py:530-533 (withdrawal without a scheduled merge): fixed in 08ecc13, thread resolved.
  • YELLOW risky path werkrevisor/gitea.py: one new call, withdraw_review_request (DELETE /pulls/{n}/requested_reviewers); needs the human review.
  • YELLOW spec Scenario#33.06 given_when_then_misused: reworded in 08ecc13, thread resolved.
  • hint werkrevisor/cli.py:557-561 (hint count and text in the result comment): not changed, reasons in the thread.
  • hint build change server/build.gradle.kts: only the version bump.
Answer to the review of `dfd5f69` (ORANGE): - ORANGE `werkrevisor/cli.py:530-533` (withdrawal without a scheduled merge): fixed in 08ecc13, thread resolved. - YELLOW risky path `werkrevisor/gitea.py`: one new call, `withdraw_review_request` (`DELETE /pulls/{n}/requested_reviewers`); needs the human review. - YELLOW spec Scenario#33.06 `given_when_then_misused`: reworded in 08ecc13, thread resolved. - hint `werkrevisor/cli.py:557-561` (hint count and text in the result comment): not changed, reasons in the thread. - hint build change `server/build.gradle.kts`: only the version bump.
Collaborator
🟠 werkrevisor ORANGE for 08ecc13, superseded by the result for 3d0fbbc

🟠 werkrevisor: ORANGE / FINDINGS

Change kind: mixed.

Reasons

  • ORANGE 1 finding(s) that Jev rates as probably real and severe: werkrevisor/cli.py:532-536
  • YELLOW risky paths changed: werkrevisor/gitea.py (werkrevisor/gitea.py)
  • hint 1 build change(s) of low risk according to Jev: server/build.gradle.kts (weakens checks 0.02, risk 0.00)
  • hint 1 finding(s) of low severity or rated minor by Jev: docs/prs/2026-10-06-PR#33-reused-line-comment-gets-new-text.md:118-125
  • hint model recommends merge with caution

Findings

  1. [medium] Review request withdrawn although the merge may not follow at werkrevisor/cli.py:532-536, ORANGE, still open in an earlier line comment

  2. [low] Document names a reproduction detail instead of the change at docs/prs/2026-10-06-PR#33-reused-line-comment-gets-new-text.md:118-125, hint, as a line comment

Classifier

10 of 10 units passed all questions (1 PR document(s)).
3 answers ignored because of a confidence below [classifier] min_confidence.

Usage: model `openrouter/anthropic/claude-sonnet-5.5`, LLM done, tokens counted before the call 74368, calls 1, tokens in 72692 / out 597 (reasoning included in out), duration 19.6 s, cost 0.1514 USD (reported), classifier `jev-1.13.0` calls 10, tokens in 8522, cost 0.0004 USD, finding judge `jev-1.13.0` calls 1, tokens in 1761, cost 0.0001 USD, build file judge `jev-1.13.0` calls 1, tokens in 560, cost 0.0000 USD, config: defaults + host + repository (.werkrevisor.toml at base dcdcf48).
Reviewed head `08ecc13cbbc31f5c85f41ab55f222749c62a453a` against base `dcdcf48bed8f9fc8a7752b3fddce300c185e8358`, werkrevisor 0.20.1.
<!-- werkrevisor:08ecc13cbbc31f5c85f41ab55f222749c62a453a --> <!-- werkrevisor-verdict:ORANGE --> <!-- werkrevisor-folded --> <details><summary>🟠 werkrevisor ORANGE for <code>08ecc13</code>, superseded by <a href="http://git.javagil.de/mi/werkrevisor/pulls/33#issuecomment-1456">the result for <code>3d0fbbc</code></a></summary> ## 🟠 werkrevisor: ORANGE / FINDINGS Change kind: `mixed`. ### Reasons - **ORANGE** 1 finding(s) that Jev rates as probably real and severe: werkrevisor/cli.py:532-536 - **YELLOW** risky paths changed: werkrevisor/gitea.py (werkrevisor/gitea.py) - **hint** 1 build change(s) of low risk according to Jev: server/build.gradle.kts (weakens checks 0.02, risk 0.00) - **hint** 1 finding(s) of low severity or rated minor by Jev: docs/prs/2026-10-06-PR#33-reused-line-comment-gets-new-text.md:118-125 - **hint** model recommends merge with caution ### Findings 1. **[medium] Review request withdrawn although the merge may not follow** at `werkrevisor/cli.py:532-536`, ORANGE, still open in [an earlier line comment](http://git.javagil.de/mi/werkrevisor/pulls/33#issuecomment-1440) 2. **[low] Document names a reproduction detail instead of the change** at `docs/prs/2026-10-06-PR#33-reused-line-comment-gets-new-text.md:118-125`, hint, as a line comment ### Classifier 10 of 10 units passed all questions (1 PR document(s)). 3 answers ignored because of a confidence below [classifier] min_confidence. <sub> Usage: model `openrouter/anthropic/claude-sonnet-5.5`, LLM done, tokens counted before the call 74368, calls 1, tokens in 72692 / out 597 (reasoning included in out), duration 19.6 s, cost 0.1514 USD (reported), classifier `jev-1.13.0` calls 10, tokens in 8522, cost 0.0004 USD, finding judge `jev-1.13.0` calls 1, tokens in 1761, cost 0.0001 USD, build file judge `jev-1.13.0` calls 1, tokens in 560, cost 0.0000 USD, config: defaults + host + repository (.werkrevisor.toml at base dcdcf48).<br> Reviewed head `08ecc13cbbc31f5c85f41ab55f222749c62a453a` against base `dcdcf48bed8f9fc8a7752b3fddce300c185e8358`, werkrevisor 0.20.1. </sub> </details>
ai-reviewer left a comment
Collaborator

werkrevisor: ORANGE / FINDINGS for 08ecc13cbbc31f5c85f41ab55f222749c62a453a.
The result comment of werkrevisor holds the details.

werkrevisor: ORANGE / FINDINGS for `08ecc13cbbc31f5c85f41ab55f222749c62a453a`. The result comment of werkrevisor holds the details. <!-- werkrevisor-review:08ecc13cbbc31f5c85f41ab55f222749c62a453a -->
@@ -0,0 +115,4 @@
without a change of the text, ignoring that line, line endings and trailing blanks, it does not edit,
since Gitea may return a body normalized otherwise than posted.
- A failed edit, or a second finding at the same comment, puts the full text into the result comment.
- Only comments written by the bot count as its line comments:
Collaborator

[low] Document names a reproduction detail instead of the change (werkrevisor)

hint

The Solution bullets narrate the history of this PR ("0.20.1 reused a reply of claude-code in this PR", "the request of the YELLOW rounds blocked the merge scheduled by the GREEN review of f572e65"). This is incident narration that does not describe the change and goes stale. Scenario#33.03 also covers only two causes in a table but gives one When line, which makes the scenario harder to read.

**[low] Document names a reproduction detail instead of the change** (werkrevisor) hint The Solution bullets narrate the history of this PR ("0.20.1 reused a reply of claude-code in this PR", "the request of the YELLOW rounds blocked the merge scheduled by the GREEN review of f572e65"). This is incident narration that does not describe the change and goes stale. Scenario#33.03 also covers only two causes in a table but gives one `When` line, which makes the scenario harder to read.
Author
Collaborator

Fixed in 308b44e: the two defects found in the reviews of this PR moved from the solution into the problem. Scenario#33.03 keeps its table: both causes lead to the same outcome, which the template allows as data-driven scenario.

Fixed in 308b44e: the two defects found in the reviews of this PR moved from the solution into the problem. Scenario#33.03 keeps its table: both causes lead to the same outcome, which the template allows as data-driven scenario.
claude-code marked this conversation as resolved
Author
Collaborator

Answer to the review of 08ecc13 (ORANGE):

  • ORANGE werkrevisor/cli.py:532-536 (request withdrawn although the merge may not follow): its text is lost again, since 0.20.1 reused my reply 1440, the bug this PR fixes. From the title: not changed. The request is withdrawn only once Gitea accepted the scheduled merge. If a check fails later, the PR stays unmerged; the author pushes a fix, and a review that is not GREEN requests the human review again. A GREEN head needs no human review, since the approval of the bot is the lock.
  • YELLOW risky path werkrevisor/gitea.py: one new call, withdraw_review_request; needs the human review.
  • hint PR doc narrates incidents in the solution: fixed in 308b44e, thread resolved.
  • hint build change server/build.gradle.kts: only the version bump.
Answer to the review of `08ecc13` (ORANGE): - ORANGE `werkrevisor/cli.py:532-536` (request withdrawn although the merge may not follow): its text is lost again, since 0.20.1 reused my reply 1440, the bug this PR fixes. From the title: not changed. The request is withdrawn only once Gitea accepted the scheduled merge. If a check fails later, the PR stays unmerged; the author pushes a fix, and a review that is not GREEN requests the human review again. A GREEN head needs no human review, since the approval of the bot is the lock. - YELLOW risky path `werkrevisor/gitea.py`: one new call, `withdraw_review_request`; needs the human review. - hint PR doc narrates incidents in the solution: fixed in 308b44e, thread resolved. - hint build change `server/build.gradle.kts`: only the version bump.
mi added 1 commit 2026-10-06 16:28:14 +02:00
review: the PR#33 doc names the defects found in its reviews in the problem, not in the solution
werkator build succeeded after 00:09
ai-review/assessment review not completed, human action needed
ai-review/gate build failed after 00:15
werkator/server build succeeded after 00:26
308b44ee1d
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Collaborator
🟠 werkrevisor ORANGE for 308b44e, superseded by the result for 3d0fbbc

🟠 werkrevisor: ORANGE / FINDINGS

Change kind: mixed.

Reasons

  • ORANGE 1 finding(s) that Jev rates as probably real and severe: werkrevisor/cli.py:579-590
  • YELLOW risky paths changed: werkrevisor/gitea.py (werkrevisor/gitea.py)
  • hint 1 build change(s) of low risk according to Jev: server/build.gradle.kts (weakens checks 0.02, risk 0.00)
  • hint 1 finding(s) of low severity or rated minor by Jev: tests/test_line_comments.py:464-477
  • hint model recommends merge with caution

Findings

  1. [medium] Withdrawal only uses requested_reviewers of a stale PR snapshot at werkrevisor/cli.py:579-590

    ORANGE; Jev: real 0.67, severity 3.0 of 3

    _withdraw_review_request reads requested_reviewers from pr_data, which _assert_current fetched before the publish step. For an approval YELLOW round the review request may also exist only as a REQUEST_REVIEW review, which _review_request_pending treats as a pending request. Such a request is not withdrawn and the merge stays blocked. The snapshot may also be stale if the request was changed during the run. Uncertain: whether Gitea reports every pending request in requested_reviewers.

  2. [medium] Reuse of a later finding after an update failure is untested at tests/test_line_comments.py:464-477

    hint; Jev: real 0.54, severity 2.2 of 3

    The updated set marks a comment as updated before the edit is tried. If the edit fails for the first finding, a second finding at the same comment gets reused_not_updated through the updated check, which is correct. A failed first edit is never retried by the second finding, so only one of two failed findings gets the update attempt. The tests cover each case separately but not the combination.

Classifier

10 of 10 units passed all questions (1 PR document(s)).
3 answers ignored because of a confidence below [classifier] min_confidence.

Usage: model `openrouter/anthropic/claude-sonnet-5.5`, LLM done, tokens counted before the call 74388, calls 1, tokens in 72715 / out 554 (reasoning included in out), duration 12.2 s, cost 0.1510 USD (reported), classifier `jev-1.13.0` calls 10, tokens in 8528, cost 0.0004 USD, finding judge `jev-1.13.0` calls 2, tokens in 2292, cost 0.0001 USD, build file judge `jev-1.13.0` calls 1, tokens in 560, cost 0.0000 USD, config: defaults + host + repository (.werkrevisor.toml at base dcdcf48).
Reviewed head `308b44ee1d022a01f3dbcdaf7ae5174012d2ee26` against base `dcdcf48bed8f9fc8a7752b3fddce300c185e8358`, werkrevisor 0.20.1.
<!-- werkrevisor:308b44ee1d022a01f3dbcdaf7ae5174012d2ee26 --> <!-- werkrevisor-verdict:ORANGE --> <!-- werkrevisor-folded --> <details><summary>🟠 werkrevisor ORANGE for <code>308b44e</code>, superseded by <a href="http://git.javagil.de/mi/werkrevisor/pulls/33#issuecomment-1456">the result for <code>3d0fbbc</code></a></summary> ## 🟠 werkrevisor: ORANGE / FINDINGS Change kind: `mixed`. ### Reasons - **ORANGE** 1 finding(s) that Jev rates as probably real and severe: werkrevisor/cli.py:579-590 - **YELLOW** risky paths changed: werkrevisor/gitea.py (werkrevisor/gitea.py) - **hint** 1 build change(s) of low risk according to Jev: server/build.gradle.kts (weakens checks 0.02, risk 0.00) - **hint** 1 finding(s) of low severity or rated minor by Jev: tests/test_line_comments.py:464-477 - **hint** model recommends merge with caution ### Findings 1. **[medium] Withdrawal only uses `requested_reviewers` of a stale PR snapshot** at `werkrevisor/cli.py:579-590` ORANGE; Jev: real 0.67, severity 3.0 of 3 `_withdraw_review_request` reads `requested_reviewers` from `pr_data`, which `_assert_current` fetched before the publish step. For an `approval` YELLOW round the review request may also exist only as a `REQUEST_REVIEW` review, which `_review_request_pending` treats as a pending request. Such a request is not withdrawn and the merge stays blocked. The snapshot may also be stale if the request was changed during the run. Uncertain: whether Gitea reports every pending request in `requested_reviewers`. 2. **[medium] Reuse of a later finding after an update failure is untested** at `tests/test_line_comments.py:464-477` hint; Jev: real 0.54, severity 2.2 of 3 The `updated` set marks a comment as updated before the edit is tried. If the edit fails for the first finding, a second finding at the same comment gets `reused_not_updated` through the `updated` check, which is correct. A failed first edit is never retried by the second finding, so only one of two failed findings gets the update attempt. The tests cover each case separately but not the combination. ### Classifier 10 of 10 units passed all questions (1 PR document(s)). 3 answers ignored because of a confidence below [classifier] min_confidence. <sub> Usage: model `openrouter/anthropic/claude-sonnet-5.5`, LLM done, tokens counted before the call 74388, calls 1, tokens in 72715 / out 554 (reasoning included in out), duration 12.2 s, cost 0.1510 USD (reported), classifier `jev-1.13.0` calls 10, tokens in 8528, cost 0.0004 USD, finding judge `jev-1.13.0` calls 2, tokens in 2292, cost 0.0001 USD, build file judge `jev-1.13.0` calls 1, tokens in 560, cost 0.0000 USD, config: defaults + host + repository (.werkrevisor.toml at base dcdcf48).<br> Reviewed head `308b44ee1d022a01f3dbcdaf7ae5174012d2ee26` against base `dcdcf48bed8f9fc8a7752b3fddce300c185e8358`, werkrevisor 0.20.1. </sub> </details>
Collaborator
⚠️ werkrevisor ERROR for 308b44e, superseded by the result for 3d0fbbc

⚠️ werkrevisor: ERROR / review not completed, human action needed

Change kind: mixed.

Reasons

  • ERROR GiteaError: POST /repos/mi/werkrevisor/pulls/33/reviews failed with HTTP 500
  • ORANGE 1 finding(s) that Jev rates as probably real and severe: werkrevisor/cli.py:579-590
  • YELLOW risky paths changed: werkrevisor/gitea.py (werkrevisor/gitea.py)
  • hint 1 build change(s) of low risk according to Jev: server/build.gradle.kts (weakens checks 0.02, risk 0.00)
  • hint 1 finding(s) of low severity or rated minor by Jev: tests/test_line_comments.py:464-477
  • hint model recommends merge with caution

Findings

  1. [medium] Withdrawal only uses requested_reviewers of a stale PR snapshot at werkrevisor/cli.py:579-590

    ORANGE; Jev: real 0.67, severity 3.0 of 3

    _withdraw_review_request reads requested_reviewers from pr_data, which _assert_current fetched before the publish step. For an approval YELLOW round the review request may also exist only as a REQUEST_REVIEW review, which _review_request_pending treats as a pending request. Such a request is not withdrawn and the merge stays blocked. The snapshot may also be stale if the request was changed during the run. Uncertain: whether Gitea reports every pending request in requested_reviewers.

  2. [medium] Reuse of a later finding after an update failure is untested at tests/test_line_comments.py:464-477

    hint; Jev: real 0.54, severity 2.2 of 3

    The updated set marks a comment as updated before the edit is tried. If the edit fails for the first finding, a second finding at the same comment gets reused_not_updated through the updated check, which is correct. A failed first edit is never retried by the second finding, so only one of two failed findings gets the update attempt. The tests cover each case separately but not the combination.

Classifier

10 of 10 units passed all questions (1 PR document(s)).
3 answers ignored because of a confidence below [classifier] min_confidence.

Usage: model `openrouter/anthropic/claude-sonnet-5.5`, LLM done, tokens counted before the call 74388, calls 1, tokens in 72715 / out 554 (reasoning included in out), duration 12.2 s, cost 0.1510 USD (reported), classifier `jev-1.13.0` calls 10, tokens in 8528, cost 0.0004 USD, finding judge `jev-1.13.0` calls 2, tokens in 2292, cost 0.0001 USD, build file judge `jev-1.13.0` calls 1, tokens in 560, cost 0.0000 USD, config: defaults + host + repository (.werkrevisor.toml at base dcdcf48).
Reviewed head `308b44ee1d022a01f3dbcdaf7ae5174012d2ee26` against base `dcdcf48bed8f9fc8a7752b3fddce300c185e8358`, werkrevisor 0.20.1.
<!-- werkrevisor-error:308b44ee1d022a01f3dbcdaf7ae5174012d2ee26 --> <!-- werkrevisor-folded --> <details><summary>⚠️ werkrevisor ERROR for <code>308b44e</code>, superseded by <a href="http://git.javagil.de/mi/werkrevisor/pulls/33#issuecomment-1456">the result for <code>3d0fbbc</code></a></summary> ## ⚠️ werkrevisor: ERROR / review not completed, human action needed Change kind: `mixed`. ### Reasons - **ERROR** GiteaError: POST /repos/mi/werkrevisor/pulls/33/reviews failed with HTTP 500 - **ORANGE** 1 finding(s) that Jev rates as probably real and severe: werkrevisor/cli.py:579-590 - **YELLOW** risky paths changed: werkrevisor/gitea.py (werkrevisor/gitea.py) - **hint** 1 build change(s) of low risk according to Jev: server/build.gradle.kts (weakens checks 0.02, risk 0.00) - **hint** 1 finding(s) of low severity or rated minor by Jev: tests/test_line_comments.py:464-477 - **hint** model recommends merge with caution ### Findings 1. **[medium] Withdrawal only uses `requested_reviewers` of a stale PR snapshot** at `werkrevisor/cli.py:579-590` ORANGE; Jev: real 0.67, severity 3.0 of 3 `_withdraw_review_request` reads `requested_reviewers` from `pr_data`, which `_assert_current` fetched before the publish step. For an `approval` YELLOW round the review request may also exist only as a `REQUEST_REVIEW` review, which `_review_request_pending` treats as a pending request. Such a request is not withdrawn and the merge stays blocked. The snapshot may also be stale if the request was changed during the run. Uncertain: whether Gitea reports every pending request in `requested_reviewers`. 2. **[medium] Reuse of a later finding after an update failure is untested** at `tests/test_line_comments.py:464-477` hint; Jev: real 0.54, severity 2.2 of 3 The `updated` set marks a comment as updated before the edit is tried. If the edit fails for the first finding, a second finding at the same comment gets `reused_not_updated` through the `updated` check, which is correct. A failed first edit is never retried by the second finding, so only one of two failed findings gets the update attempt. The tests cover each case separately but not the combination. ### Classifier 10 of 10 units passed all questions (1 PR document(s)). 3 answers ignored because of a confidence below [classifier] min_confidence. <sub> Usage: model `openrouter/anthropic/claude-sonnet-5.5`, LLM done, tokens counted before the call 74388, calls 1, tokens in 72715 / out 554 (reasoning included in out), duration 12.2 s, cost 0.1510 USD (reported), classifier `jev-1.13.0` calls 10, tokens in 8528, cost 0.0004 USD, finding judge `jev-1.13.0` calls 2, tokens in 2292, cost 0.0001 USD, build file judge `jev-1.13.0` calls 1, tokens in 560, cost 0.0000 USD, config: defaults + host + repository (.werkrevisor.toml at base dcdcf48).<br> Reviewed head `308b44ee1d022a01f3dbcdaf7ae5174012d2ee26` against base `dcdcf48bed8f9fc8a7752b3fddce300c185e8358`, werkrevisor 0.20.1. </sub> </details>
Author
Collaborator

Answer to the review of 308b44e (ERROR):

  • ERROR POST /pulls/33/reviews HTTP 500: the finding at tests/test_line_comments.py:464-477 lies beyond the end of the file (224 lines), and Gitea answers such a line comment with 500, which 0.20.1 does not treat as refusal. Fixed in 3d0fbbc: 500 counts like 400 and 422, Scenario#33.07, red/green.
  • ORANGE werkrevisor/cli.py:579-590 (withdrawal reads only requested_reviewers): fixed in 3d0fbbc, a request listed only as REQUEST_REVIEW review is withdrawn too, the same sources as _review_request_pending. A request added during the run can only come from this run, which adds none at GREEN.
  • hint tests/test_line_comments.py:464-477 (a failed first edit is not retried by a second finding): not changed; one edit per comment and round is intended, and the second finding's text goes into the result comment.
  • YELLOW risky path werkrevisor/gitea.py: needs the human review.

The findings of this round have no line comments, since the review failed; this comment answers them.

Answer to the review of `308b44e` (ERROR): - ERROR `POST /pulls/33/reviews` HTTP 500: the finding at `tests/test_line_comments.py:464-477` lies beyond the end of the file (224 lines), and Gitea answers such a line comment with 500, which 0.20.1 does not treat as refusal. Fixed in 3d0fbbc: 500 counts like 400 and 422, Scenario#33.07, red/green. - ORANGE `werkrevisor/cli.py:579-590` (withdrawal reads only `requested_reviewers`): fixed in 3d0fbbc, a request listed only as `REQUEST_REVIEW` review is withdrawn too, the same sources as `_review_request_pending`. A request added during the run can only come from this run, which adds none at GREEN. - hint `tests/test_line_comments.py:464-477` (a failed first edit is not retried by a second finding): not changed; one edit per comment and round is intended, and the second finding's text goes into the result comment. - YELLOW risky path `werkrevisor/gitea.py`: needs the human review. The findings of this round have no line comments, since the review failed; this comment answers them.
mi added 1 commit 2026-10-06 16:30:59 +02:00
review: HTTP 500 on line comments counts as refusal; a request listed only as review is withdrawn too
werkator build succeeded after 00:10
ai-review/assessment human review required; a human approval unlocks the merge; a human review request is pending; merge already scheduled
ai-review/gate build succeeded after 00:18
werkator/server build succeeded after 00:28
3d0fbbc2aa
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Collaborator
🟡 werkrevisor YELLOW for 3d0fbbc, superseded by the result for a356441

🟡 werkrevisor: YELLOW / HUMAN_REVIEW

Change kind: mixed.

Reasons

  • YELLOW risky paths changed: werkrevisor/gitea.py (werkrevisor/gitea.py)
  • YELLOW spec: docs/prs/2026-10-06-PR#33-reused-line-comment-gets-new-text.md Scenario#33.07 given_when_then_misused (0.71 > yellow_above = 0.6)
  • YELLOW 2 finding(s) possibly severe, but Jev doubts that they are real: werkrevisor/cli.py:464-477, werkrevisor/cli.py:499-500
  • hint 1 build change(s) of low risk according to Jev: server/build.gradle.kts (weakens checks 0.02, risk 0.00)
  • hint 1 finding(s) of low severity or rated minor by Jev: tests/test_line_comments.py:161-169
  • hint model recommends changes before merge

Findings

  1. [high] Line-comment edit may fail on real Gitea at werkrevisor/cli.py:464-477, YELLOW, as a line comment

  2. [medium] HTTP 500 treated as a line-comment refusal at werkrevisor/cli.py:499-500, YELLOW, as a line comment

  3. [medium] Reply filter has no test with an open reply to an open bot comment at tests/test_line_comments.py:161-169, hint, as a line comment

Classifier

Unit Question Problem Confidence Verdict
docs/prs/2026-10-06-PR#33-reused-line-comment-gets-new-text.md Scenario#33.07 given_when_then_misused 0.71 – YELLOW

10 of 11 units passed all questions (1 PR document(s)).
5 answers ignored because of a confidence below [classifier] min_confidence.

Usage: model `openrouter/anthropic/claude-sonnet-5.5`, LLM done, tokens counted before the call 78993, calls 1, tokens in 77260 / out 715 (reasoning included in out), duration 14.8 s, cost 0.1617 USD (reported), classifier `jev-1.13.0` calls 11, tokens in 9572, cost 0.0004 USD, finding judge `jev-1.13.0` calls 3, tokens in 5174, cost 0.0002 USD, build file judge `jev-1.13.0` calls 1, tokens in 560, cost 0.0000 USD, config: defaults + host + repository (.werkrevisor.toml at base dcdcf48).
Reviewed head `3d0fbbc2aaef555958127831bc0974ea77a798a1` against base `dcdcf48bed8f9fc8a7752b3fddce300c185e8358`, werkrevisor 0.20.1.
<!-- werkrevisor:3d0fbbc2aaef555958127831bc0974ea77a798a1 --> <!-- werkrevisor-verdict:YELLOW --> <!-- werkrevisor-folded --> <details><summary>🟡 werkrevisor YELLOW for <code>3d0fbbc</code>, superseded by <a href="http://git.javagil.de/mi/werkrevisor/pulls/33#issuecomment-1467">the result for <code>a356441</code></a></summary> ## 🟡 werkrevisor: YELLOW / HUMAN_REVIEW Change kind: `mixed`. ### Reasons - **YELLOW** risky paths changed: werkrevisor/gitea.py (werkrevisor/gitea.py) - **YELLOW** spec: docs/prs/2026-10-06-PR#33-reused-line-comment-gets-new-text.md Scenario#33.07 given_when_then_misused (0.71 > yellow_above = 0.6) - **YELLOW** 2 finding(s) possibly severe, but Jev doubts that they are real: werkrevisor/cli.py:464-477, werkrevisor/cli.py:499-500 - **hint** 1 build change(s) of low risk according to Jev: server/build.gradle.kts (weakens checks 0.02, risk 0.00) - **hint** 1 finding(s) of low severity or rated minor by Jev: tests/test_line_comments.py:161-169 - **hint** model recommends changes before merge ### Findings 1. **[high] Line-comment edit may fail on real Gitea** at `werkrevisor/cli.py:464-477`, YELLOW, as a line comment 2. **[medium] HTTP 500 treated as a line-comment refusal** at `werkrevisor/cli.py:499-500`, YELLOW, as a line comment 3. **[medium] Reply filter has no test with an open reply to an open bot comment** at `tests/test_line_comments.py:161-169`, hint, as a line comment ### Classifier | Unit | Question | Problem | Confidence | Verdict | |---|---|---|---|---| | docs/prs/2026-10-06-PR#33-reused-line-comment-gets-new-text.md Scenario#33.07 | given_when_then_misused | 0.71 | – | YELLOW | 10 of 11 units passed all questions (1 PR document(s)). 5 answers ignored because of a confidence below [classifier] min_confidence. <sub> Usage: model `openrouter/anthropic/claude-sonnet-5.5`, LLM done, tokens counted before the call 78993, calls 1, tokens in 77260 / out 715 (reasoning included in out), duration 14.8 s, cost 0.1617 USD (reported), classifier `jev-1.13.0` calls 11, tokens in 9572, cost 0.0004 USD, finding judge `jev-1.13.0` calls 3, tokens in 5174, cost 0.0002 USD, build file judge `jev-1.13.0` calls 1, tokens in 560, cost 0.0000 USD, config: defaults + host + repository (.werkrevisor.toml at base dcdcf48).<br> Reviewed head `3d0fbbc2aaef555958127831bc0974ea77a798a1` against base `dcdcf48bed8f9fc8a7752b3fddce300c185e8358`, werkrevisor 0.20.1. </sub> </details>
ai-reviewer left a comment
Collaborator

werkrevisor: YELLOW / HUMAN_REVIEW for 3d0fbbc2aaef555958127831bc0974ea77a798a1.
The result comment of werkrevisor holds the details.

werkrevisor: YELLOW / HUMAN_REVIEW for `3d0fbbc2aaef555958127831bc0974ea77a798a1`. The result comment of werkrevisor holds the details. <!-- werkrevisor-review:3d0fbbc2aaef555958127831bc0974ea77a798a1 -->
@@ -104,0 +158,4 @@
self.assertEqual(result.count("still open in [an earlier line comment](https://x/77)"), 2)
self.assertEqual(result.count("Breaks X."), 1)
def test_a_reply_in_the_thread_of_a_bot_review_is_not_reused(self):
Collaborator

[medium] Reply filter has no test with an open reply to an open bot comment (werkrevisor)

hint; Jev: real 0.71, severity 2.0 of 3

The new author filter in _open_bot_line_comments is tested only for a resolved bot comment with a human reply. There is no test that a human reply to an open bot comment is not counted as an extra hint or reuse target, which changes _open_hints counts.

**[medium] Reply filter has no test with an open reply to an open bot comment** (werkrevisor) hint; Jev: real 0.71, severity 2.0 of 3 The new author filter in `_open_bot_line_comments` is tested only for a resolved bot comment with a human reply. There is no test that a human reply to an open bot comment is not counted as an extra hint or reuse target, which changes `_open_hints` counts.
Author
Collaborator

Fixed in a356441: test_an_open_reply_to_an_open_hint_is_no_second_hint, red without the author filter.

Fixed in a356441: `test_an_open_reply_to_an_open_hint_is_no_second_hint`, red without the author filter.
claude-code marked this conversation as resolved
@@ -453,6 +461,22 @@ def _plan_line_comments(gitea, pr, result, bot_login):
return comments
def _update_line_comment(gitea, pr, earlier, text, redactor, updated):
Collaborator

[high] Line-comment edit may fail on real Gitea (werkrevisor)

YELLOW; Jev: real 0.45, severity 2.7 of 3

_update_line_comment edits a pull-review line comment via PATCH /issues/comments/{id}. The fake was changed to accept this for line comments ("Gitea edits issue comments and line comments under the same path"), which hides the question whether real Gitea honors it. The id comes from the review comments listing. If real Gitea rejects the id on that path, the code only falls back to the result comment, so the feature silently does nothing. This remains uncertain, since the real API was not checked here.

**[high] Line-comment edit may fail on real Gitea** (werkrevisor) YELLOW; Jev: real 0.45, severity 2.7 of 3 `_update_line_comment` edits a pull-review line comment via `PATCH /issues/comments/{id}`. The fake was changed to accept this for line comments ("Gitea edits issue comments and line comments under the same path"), which hides the question whether real Gitea honors it. The id comes from the review comments listing. If real Gitea rejects the id on that path, the code only falls back to the result comment, so the feature silently does nothing. This remains uncertain, since the real API was not checked here.
Author
Collaborator

Checked on git.javagil.de just now: PATCH /issues/comments/1406 with the unchanged body of line comment 1406 answered HTTP 200 with the same id, so Gitea 1.27 edits line comments under this path. Not changed.

Checked on git.javagil.de just now: `PATCH /issues/comments/1406` with the unchanged body of line comment 1406 answered HTTP 200 with the same id, so Gitea 1.27 edits line comments under this path. Not changed.
claude-code marked this conversation as resolved
@@ -473,3 +497,3 @@
return False
except GiteaError as e:
if not comments or e.status not in (400, 422):
# Gitea 1.27 answers a line beyond the end of the file with HTTP 500.
Collaborator

[medium] HTTP 500 treated as a line-comment refusal (werkrevisor)

YELLOW; Jev: real 0.59, severity 2.9 of 3

_publish_review now treats any HTTP 500 from create_review as a refusal of line comments and falls back to a review without them. A genuine transient server error then silently drops the line comments and produces a review that is carried in the result comment. For an APPROVED event the approval is retried once without comments. That is acceptable, but a transient 500 can no longer surface as ERROR. This is likely intended; the broader behavior change is worth confirming.

**[medium] HTTP 500 treated as a line-comment refusal** (werkrevisor) YELLOW; Jev: real 0.59, severity 2.9 of 3 `_publish_review` now treats any HTTP 500 from `create_review` as a refusal of line comments and falls back to a review without them. A genuine transient server error then silently drops the line comments and produces a review that is carried in the result comment. For an APPROVED event the approval is retried once without comments. That is acceptable, but a transient 500 can no longer surface as ERROR. This is likely intended; the broader behavior change is worth confirming.
Author
Collaborator

Not changed, intended: a transient 500 still gets a second POST without line comments; if that fails too, the run ends with ERROR. Otherwise the findings are in the result comment, so nothing is lost. A line beyond the file is a model error that would otherwise end every rerun of that head with ERROR. Left open for the human reviewer.

Not changed, intended: a transient 500 still gets a second POST without line comments; if that fails too, the run ends with ERROR. Otherwise the findings are in the result comment, so nothing is lost. A line beyond the file is a model error that would otherwise end every rerun of that head with ERROR. Left open for the human reviewer.
@@ -548,6 +576,20 @@ def _hold_merge(pr, gitea, redactor, count):
return note, True
def _withdraw_review_request(config, pr, gitea, redactor, pr_data):
Collaborator

[medium] Withdrawal only uses requested_reviewers of a stale PR snapshot (werkrevisor)

ORANGE; Jev: real 0.67, severity 3.0 of 3

_withdraw_review_request reads requested_reviewers from pr_data, which _assert_current fetched before the publish step. For an approval YELLOW round the review request may also exist only as a REQUEST_REVIEW review, which _review_request_pending treats as a pending request. Such a request is not withdrawn and the merge stays blocked. The snapshot may also be stale if the request was changed during the run. Uncertain: whether Gitea reports every pending request in requested_reviewers.

**[medium] Withdrawal only uses `requested_reviewers` of a stale PR snapshot** (werkrevisor) ORANGE; Jev: real 0.67, severity 3.0 of 3 `_withdraw_review_request` reads `requested_reviewers` from `pr_data`, which `_assert_current` fetched before the publish step. For an `approval` YELLOW round the review request may also exist only as a `REQUEST_REVIEW` review, which `_review_request_pending` treats as a pending request. Such a request is not withdrawn and the merge stays blocked. The snapshot may also be stale if the request was changed during the run. Uncertain: whether Gitea reports every pending request in `requested_reviewers`.
Author
Collaborator

Fixed in 3d0fbbc: requests listed only as REQUEST_REVIEW review are withdrawn too.

Fixed in 3d0fbbc: requests listed only as `REQUEST_REVIEW` review are withdrawn too.
claude-code marked this conversation as resolved
mi added 1 commit 2026-10-06 16:32:54 +02:00
review: an open reply to an open hint is no second hint; Scenario#33.07 with one role per step
werkator build succeeded after 00:09
ai-review/assessment human review required; a human approval unlocks the merge; a human review request is pending; merge already scheduled
ai-review/gate build succeeded after 00:16
werkator/server build succeeded after 00:27
a356441634
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Author
Collaborator

Answer to the review of 3d0fbbc (YELLOW):

  • YELLOW werkrevisor/cli.py:464-477 (edit of line comments on real Gitea): verified with an unchanged edit of line comment 1406, HTTP 200; resolved.
  • YELLOW werkrevisor/cli.py:499-500 (HTTP 500 as refusal): intended, reasons in the thread, left open.
  • YELLOW spec Scenario#33.07: reworded in a356441.
  • hint tests/test_line_comments.py:161-169 (open reply to an open hint): test added in a356441, resolved.
  • YELLOW risky path werkrevisor/gitea.py: needs the human review.
Answer to the review of `3d0fbbc` (YELLOW): - YELLOW `werkrevisor/cli.py:464-477` (edit of line comments on real Gitea): verified with an unchanged edit of line comment 1406, HTTP 200; resolved. - YELLOW `werkrevisor/cli.py:499-500` (HTTP 500 as refusal): intended, reasons in the thread, left open. - YELLOW spec Scenario#33.07: reworded in a356441. - hint `tests/test_line_comments.py:161-169` (open reply to an open hint): test added in a356441, resolved. - YELLOW risky path `werkrevisor/gitea.py`: needs the human review.
Collaborator
🟡 werkrevisor YELLOW for a356441, superseded by the result for eeb7934

🟡 werkrevisor: YELLOW / HUMAN_REVIEW

Change kind: mixed.

Reasons

  • YELLOW risky paths changed: werkrevisor/gitea.py (werkrevisor/gitea.py)
  • YELLOW spec: docs/prs/2026-10-06-PR#33-reused-line-comment-gets-new-text.md Scenario#33.07 given_when_then_misused (0.66 > yellow_above = 0.6)
  • hint 1 build change(s) of low risk according to Jev: server/build.gradle.kts (weakens checks 0.02, risk 0.00)
  • hint 3 finding(s) of low severity or rated minor by Jev: werkrevisor/cli.py:455-456, docs/prs/2026-10-06-PR#33-reused-line-comment-gets-new-text.md:119-126, werkrevisor/cli.py:499-501
  • hint model recommends merge with caution

Findings

  1. [medium] Line comment edited before the result comment is posted at werkrevisor/cli.py:455-456, hint, as a line comment

  2. [low] Scenario#33.07 contradicts the code on the cause at docs/prs/2026-10-06-PR#33-reused-line-comment-gets-new-text.md:119-126, hint, still open in an earlier line comment

  3. [low] Any HTTP 500 is now treated as a refusal of line comments at werkrevisor/cli.py:499-501, hint, still open in an earlier line comment

Classifier

Unit Question Problem Confidence Verdict
docs/prs/2026-10-06-PR#33-reused-line-comment-gets-new-text.md Scenario#33.07 given_when_then_misused 0.66 – YELLOW

10 of 11 units passed all questions (1 PR document(s)).
4 answers ignored because of a confidence below [classifier] min_confidence.

Usage: model `openrouter/anthropic/claude-sonnet-5.5`, LLM done, tokens counted before the call 79357, calls 1, tokens in 77645 / out 786 (reasoning included in out), duration 13.5 s, cost 0.1631 USD (reported), classifier `jev-1.13.0` calls 11, tokens in 9588, cost 0.0004 USD, finding judge `jev-1.13.0` calls 1, tokens in 1493, cost 0.0001 USD, build file judge `jev-1.13.0` calls 1, tokens in 560, cost 0.0000 USD, config: defaults + host + repository (.werkrevisor.toml at base dcdcf48).
Reviewed head `a356441634415fc4556e0bc14416ee084559b6e2` against base `dcdcf48bed8f9fc8a7752b3fddce300c185e8358`, werkrevisor 0.20.1.
<!-- werkrevisor:a356441634415fc4556e0bc14416ee084559b6e2 --> <!-- werkrevisor-verdict:YELLOW --> <!-- werkrevisor-folded --> <details><summary>🟡 werkrevisor YELLOW for <code>a356441</code>, superseded by <a href="http://git.javagil.de/mi/werkrevisor/pulls/33#issuecomment-1473">the result for <code>eeb7934</code></a></summary> ## 🟡 werkrevisor: YELLOW / HUMAN_REVIEW Change kind: `mixed`. ### Reasons - **YELLOW** risky paths changed: werkrevisor/gitea.py (werkrevisor/gitea.py) - **YELLOW** spec: docs/prs/2026-10-06-PR#33-reused-line-comment-gets-new-text.md Scenario#33.07 given_when_then_misused (0.66 > yellow_above = 0.6) - **hint** 1 build change(s) of low risk according to Jev: server/build.gradle.kts (weakens checks 0.02, risk 0.00) - **hint** 3 finding(s) of low severity or rated minor by Jev: werkrevisor/cli.py:455-456, docs/prs/2026-10-06-PR#33-reused-line-comment-gets-new-text.md:119-126, werkrevisor/cli.py:499-501 - **hint** model recommends merge with caution ### Findings 1. **[medium] Line comment edited before the result comment is posted** at `werkrevisor/cli.py:455-456`, hint, as a line comment 2. **[low] Scenario#33.07 contradicts the code on the cause** at `docs/prs/2026-10-06-PR#33-reused-line-comment-gets-new-text.md:119-126`, hint, still open in [an earlier line comment](http://git.javagil.de/mi/werkrevisor/pulls/33#issuecomment-1448) 3. **[low] Any HTTP 500 is now treated as a refusal of line comments** at `werkrevisor/cli.py:499-501`, hint, still open in [an earlier line comment](http://git.javagil.de/mi/werkrevisor/pulls/33#issuecomment-1458) ### Classifier | Unit | Question | Problem | Confidence | Verdict | |---|---|---|---|---| | docs/prs/2026-10-06-PR#33-reused-line-comment-gets-new-text.md Scenario#33.07 | given_when_then_misused | 0.66 | – | YELLOW | 10 of 11 units passed all questions (1 PR document(s)). 4 answers ignored because of a confidence below [classifier] min_confidence. <sub> Usage: model `openrouter/anthropic/claude-sonnet-5.5`, LLM done, tokens counted before the call 79357, calls 1, tokens in 77645 / out 786 (reasoning included in out), duration 13.5 s, cost 0.1631 USD (reported), classifier `jev-1.13.0` calls 11, tokens in 9588, cost 0.0004 USD, finding judge `jev-1.13.0` calls 1, tokens in 1493, cost 0.0001 USD, build file judge `jev-1.13.0` calls 1, tokens in 560, cost 0.0000 USD, config: defaults + host + repository (.werkrevisor.toml at base dcdcf48).<br> Reviewed head `a356441634415fc4556e0bc14416ee084559b6e2` against base `dcdcf48bed8f9fc8a7752b3fddce300c185e8358`, werkrevisor 0.20.1. </sub> </details>
ai-reviewer left a comment
Collaborator

werkrevisor: YELLOW / HUMAN_REVIEW for a356441634415fc4556e0bc14416ee084559b6e2.
The result comment of werkrevisor holds the details.

werkrevisor: YELLOW / HUMAN_REVIEW for `a356441634415fc4556e0bc14416ee084559b6e2`. The result comment of werkrevisor holds the details. <!-- werkrevisor-review:a356441634415fc4556e0bc14416ee084559b6e2 -->
@@ -446,6 +452,8 @@ def _plan_line_comments(gitea, pr, result, bot_login):
and first - LINE_SLACK <= c["line"] <= last + LINE_SLACK), None)
if earlier is not None:
result.reused[i] = earlier["url"]
if not _update_line_comment(gitea, pr, earlier, report.line_comment(f, v), redactor, updated):
Collaborator

[medium] Line comment edited before the result comment is posted (werkrevisor)

hint; Jev: real 0.71, severity 2.2 of 3

_plan_line_comments now edits earlier line comments while planning, before _post_comment and _publish_review run. If the result comment post or the review submission then fails, the earlier line comment already carries the new text and the "Updated by the review of <head>" note, but no result comment for that head exists. This contradicts the documented order "result comment first". The edit also runs on a path that was side-effect free before. Impact is limited because a rerun finds the text unchanged and does not edit again.

**[medium] Line comment edited before the result comment is posted** (werkrevisor) hint; Jev: real 0.71, severity 2.2 of 3 `_plan_line_comments` now edits earlier line comments while planning, before `_post_comment` and `_publish_review` run. If the result comment post or the review submission then fails, the earlier line comment already carries the new text and the "Updated by the review of `<head>`" note, but no result comment for that head exists. This contradicts the documented order "result comment first". The edit also runs on a path that was side-effect free before. Impact is limited because a rerun finds the text unchanged and does not edit again.
Author
Collaborator

Not changed: the same point as the threads at cli.py:715 and cli.py:453; the result comment must know whether the edit failed, so the edit comes first. Left open for the human reviewer.

Not changed: the same point as the threads at `cli.py:715` and `cli.py:453`; the result comment must know whether the edit failed, so the edit comes first. Left open for the human reviewer.
Author
Collaborator

Answer to the review of a356441 (YELLOW):

  • YELLOW spec Scenario#33.07 and hint "contradicts the code on the cause": reworded in eeb7934; the scenario starts from the finding beyond the file, and the Then names the outcome.
  • hint werkrevisor/cli.py:455-456 (edit before the result comment): repeated finding, not changed, see its thread.
  • hint werkrevisor/cli.py:499-501 (any HTTP 500 as refusal): repeated finding, answered in its thread 1458.
  • YELLOW risky path werkrevisor/gitea.py: needs the human review.
Answer to the review of `a356441` (YELLOW): - YELLOW spec Scenario#33.07 and hint "contradicts the code on the cause": reworded in eeb7934; the scenario starts from the finding beyond the file, and the Then names the outcome. - hint `werkrevisor/cli.py:455-456` (edit before the result comment): repeated finding, not changed, see its thread. - hint `werkrevisor/cli.py:499-501` (any HTTP 500 as refusal): repeated finding, answered in its thread 1458. - YELLOW risky path `werkrevisor/gitea.py`: needs the human review.
mi added 1 commit 2026-10-06 16:34:29 +02:00
review: Scenario#33.07 starts from the finding, not from the answer of Gitea
ai-review/assessment AI findings; a developer approval after checking them unlocks the merge; a human review request is pending; merge already scheduled
werkator build succeeded after 00:08
ai-review/gate build succeeded after 00:16
werkator/server build succeeded after 00:28
eeb79347d9
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Collaborator
🟠 werkrevisor ORANGE for eeb7934, superseded by the result for 0fcbb0f

🟠 werkrevisor: ORANGE / FINDINGS

Change kind: mixed.

Reasons

  • ORANGE 1 finding(s) that Jev rates as probably real and severe: werkrevisor/cli.py:430-432
  • YELLOW risky paths changed: werkrevisor/gitea.py (werkrevisor/gitea.py)
  • YELLOW spec: docs/prs/2026-10-06-PR#33-reused-line-comment-gets-new-text.md Scenario#33.07 given_when_then_misused (0.84 > yellow_above = 0.6)
  • hint 1 build change(s) of low risk according to Jev: server/build.gradle.kts (weakens checks 0.02, risk 0.00)
  • hint 2 finding(s) of low severity or rated minor by Jev: docs/prs/2026-10-06-PR#33-reused-line-comment-gets-new-text.md:119-126, werkrevisor/cli.py:455-456
  • hint model recommends merge with caution

Findings

  1. [medium] Reuse also matches a bot comment that is a reply, and edit bypasses the resolver check at werkrevisor/cli.py:430-432, ORANGE, as a line comment

  2. [medium] Scenario#33.07 contradicts the code and tests at docs/prs/2026-10-06-PR#33-reused-line-comment-gets-new-text.md:119-126, hint, still open in an earlier line comment

  3. [medium] Update of a reused line comment happens before the review is published at werkrevisor/cli.py:455-456, hint, still open in an earlier line comment

Classifier

Unit Question Problem Confidence Verdict
docs/prs/2026-10-06-PR#33-reused-line-comment-gets-new-text.md Scenario#33.07 given_when_then_misused 0.84 – YELLOW

10 of 11 units passed all questions (1 PR document(s)).
4 answers ignored because of a confidence below [classifier] min_confidence.

Usage: model `openrouter/anthropic/claude-sonnet-5.5`, LLM done, tokens counted before the call 79351, calls 1, tokens in 77634 / out 786 (reasoning included in out), duration 13.5 s, cost 0.1631 USD (reported), classifier `jev-1.13.0` calls 11, tokens in 9572, cost 0.0004 USD, finding judge `jev-1.13.0` calls 3, tokens in 4777, cost 0.0002 USD, build file judge `jev-1.13.0` calls 1, tokens in 560, cost 0.0000 USD, config: defaults + host + repository (.werkrevisor.toml at base dcdcf48).
Reviewed head `eeb79347d9f517e1f223a796e382124d42fbac55` against base `dcdcf48bed8f9fc8a7752b3fddce300c185e8358`, werkrevisor 0.20.1.
<!-- werkrevisor:eeb79347d9f517e1f223a796e382124d42fbac55 --> <!-- werkrevisor-verdict:ORANGE --> <!-- werkrevisor-folded --> <details><summary>🟠 werkrevisor ORANGE for <code>eeb7934</code>, superseded by <a href="http://git.javagil.de/mi/werkrevisor/pulls/33#issuecomment-1479">the result for <code>0fcbb0f</code></a></summary> ## 🟠 werkrevisor: ORANGE / FINDINGS Change kind: `mixed`. ### Reasons - **ORANGE** 1 finding(s) that Jev rates as probably real and severe: werkrevisor/cli.py:430-432 - **YELLOW** risky paths changed: werkrevisor/gitea.py (werkrevisor/gitea.py) - **YELLOW** spec: docs/prs/2026-10-06-PR#33-reused-line-comment-gets-new-text.md Scenario#33.07 given_when_then_misused (0.84 > yellow_above = 0.6) - **hint** 1 build change(s) of low risk according to Jev: server/build.gradle.kts (weakens checks 0.02, risk 0.00) - **hint** 2 finding(s) of low severity or rated minor by Jev: docs/prs/2026-10-06-PR#33-reused-line-comment-gets-new-text.md:119-126, werkrevisor/cli.py:455-456 - **hint** model recommends merge with caution ### Findings 1. **[medium] Reuse also matches a bot comment that is a reply, and edit bypasses the resolver check** at `werkrevisor/cli.py:430-432`, ORANGE, as a line comment 2. **[medium] Scenario#33.07 contradicts the code and tests** at `docs/prs/2026-10-06-PR#33-reused-line-comment-gets-new-text.md:119-126`, hint, still open in [an earlier line comment](http://git.javagil.de/mi/werkrevisor/pulls/33#issuecomment-1448) 3. **[medium] Update of a reused line comment happens before the review is published** at `werkrevisor/cli.py:455-456`, hint, still open in [an earlier line comment](http://git.javagil.de/mi/werkrevisor/pulls/33#issuecomment-1468) ### Classifier | Unit | Question | Problem | Confidence | Verdict | |---|---|---|---|---| | docs/prs/2026-10-06-PR#33-reused-line-comment-gets-new-text.md Scenario#33.07 | given_when_then_misused | 0.84 | – | YELLOW | 10 of 11 units passed all questions (1 PR document(s)). 4 answers ignored because of a confidence below [classifier] min_confidence. <sub> Usage: model `openrouter/anthropic/claude-sonnet-5.5`, LLM done, tokens counted before the call 79351, calls 1, tokens in 77634 / out 786 (reasoning included in out), duration 13.5 s, cost 0.1631 USD (reported), classifier `jev-1.13.0` calls 11, tokens in 9572, cost 0.0004 USD, finding judge `jev-1.13.0` calls 3, tokens in 4777, cost 0.0002 USD, build file judge `jev-1.13.0` calls 1, tokens in 560, cost 0.0000 USD, config: defaults + host + repository (.werkrevisor.toml at base dcdcf48).<br> Reviewed head `eeb79347d9f517e1f223a796e382124d42fbac55` against base `dcdcf48bed8f9fc8a7752b3fddce300c185e8358`, werkrevisor 0.20.1. </sub> </details>
ai-reviewer left a comment
Collaborator

werkrevisor: ORANGE / FINDINGS for eeb79347d9f517e1f223a796e382124d42fbac55.
The result comment of werkrevisor holds the details.

werkrevisor: ORANGE / FINDINGS for `eeb79347d9f517e1f223a796e382124d42fbac55`. The result comment of werkrevisor holds the details. <!-- werkrevisor-review:eeb79347d9f517e1f223a796e382124d42fbac55 -->
@@ -427,1 +428,3 @@
found.append({"path": c.get("path"), "line": c["position"], "url": c.get("html_url") or ""})
# A reply of a human is listed in the review that started the thread, but is no hint of the bot.
author = (c.get("user") or {}).get("login") or ""
if author == bot_login and not c.get("resolver") and (c.get("position") or 0) > 0:
Collaborator

[medium] Reuse also matches a bot comment that is a reply, and edit bypasses the resolver check (werkrevisor)

ORANGE; Jev: real 0.68, severity 2.6 of 3

_open_bot_line_comments now accepts every comment authored by the bot with an empty resolver. A bot reply inside a thread started by a human would be treated as a hint and edited. Such a comment is not necessarily a bot line comment of a finding. This is uncertain, since bots do not reply today, but the new edit makes the case destructive, and no test covers it.

**[medium] Reuse also matches a bot comment that is a reply, and edit bypasses the resolver check** (werkrevisor) ORANGE; Jev: real 0.68, severity 2.6 of 3 `_open_bot_line_comments` now accepts every comment authored by the bot with an empty `resolver`. A bot reply inside a thread started by a human would be treated as a hint and edited. Such a comment is not necessarily a bot line comment of a finding. This is uncertain, since bots do not reply today, but the new edit makes the case destructive, and no test covers it.
Author
Collaborator

Not changed. _open_bot_line_comments reads only the comments of reviews submitted by the bot, so a bot reply in a thread started by a human, which Gitea lists under the human's review, is never seen. A bot reply in its own thread does not exist: werkrevisor never calls the reply endpoint. Should it ever do so, that reply is bot text in an open thread of the bot. Left open for the human reviewer.

Not changed. `_open_bot_line_comments` reads only the comments of reviews submitted by the bot, so a bot reply in a thread started by a human, which Gitea lists under the human's review, is never seen. A bot reply in its own thread does not exist: werkrevisor never calls the reply endpoint. Should it ever do so, that reply is bot text in an open thread of the bot. Left open for the human reviewer.
Author
Collaborator

Answer to the review of eeb7934 (ORANGE):

  • ORANGE werkrevisor/cli.py:430-432 (a bot reply would be reused and edited): not changed, reasons in the thread; only bot reviews are read, and the bot writes no replies.
  • YELLOW spec Scenario#33.07 and hint "contradicts the code and tests": fixed in 0fcbb0f; the When is now the HTTP 500 of Gitea, as in the test, and the line beyond the file moved into the So-that as the known cause.
  • hint werkrevisor/cli.py:455-456 (edit before publishing): repeated, see thread 1468.
  • YELLOW risky path werkrevisor/gitea.py: needs the human review.
Answer to the review of `eeb7934` (ORANGE): - ORANGE `werkrevisor/cli.py:430-432` (a bot reply would be reused and edited): not changed, reasons in the thread; only bot reviews are read, and the bot writes no replies. - YELLOW spec Scenario#33.07 and hint "contradicts the code and tests": fixed in 0fcbb0f; the When is now the HTTP 500 of Gitea, as in the test, and the line beyond the file moved into the So-that as the known cause. - hint `werkrevisor/cli.py:455-456` (edit before publishing): repeated, see thread 1468. - YELLOW risky path `werkrevisor/gitea.py`: needs the human review.
mi added 1 commit 2026-10-06 16:35:52 +02:00
review: Scenario#33.07 names the answer of Gitea as event, as its test does
ai-review/assessment AI findings; a developer approval after checking them unlocks the merge; a human review request is pending; merge already scheduled
werkator build succeeded after 00:10
ai-review/gate build succeeded after 00:17
werkator/server build succeeded after 00:28
0fcbb0f971
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Collaborator
🟠 werkrevisor ORANGE for 0fcbb0f, superseded by the result for c3ad49e

🟠 werkrevisor: ORANGE / FINDINGS

Change kind: mixed.

Reasons

  • ORANGE 1 finding(s) that Jev rates as probably real and severe: werkrevisor/cli.py:499-504
  • YELLOW risky paths changed: werkrevisor/gitea.py (werkrevisor/gitea.py)
  • YELLOW spec: docs/prs/2026-10-06-PR#33-reused-line-comment-gets-new-text.md Scenario#33.07 given_when_then_misused (0.70 > yellow_above = 0.6)
  • YELLOW 1 finding(s) possibly severe, but Jev doubts that they are real: werkrevisor/cli.py:533-537
  • hint 1 build change(s) of low risk according to Jev: server/build.gradle.kts (weakens checks 0.02, risk 0.00)
  • hint 1 finding(s) of low severity or rated minor by Jev: docs/prs/2026-10-06-PR#33-reused-line-comment-gets-new-text.md:119-127
  • hint model recommends merge with caution

Findings

  1. [medium] Edited reused comment is not kept in sync across runs when a 500 is refused at werkrevisor/cli.py:499-504, ORANGE, still open in an earlier line comment

  2. [medium] Withdrawal skips reviewers whose login differs only in case from the request at werkrevisor/cli.py:533-537, YELLOW, as a line comment

  3. [low] Scenario 33.07 contradicts the code for the COMMENT event at docs/prs/2026-10-06-PR#33-reused-line-comment-gets-new-text.md:119-127, hint, still open in an earlier line comment

Classifier

Unit Question Problem Confidence Verdict
docs/prs/2026-10-06-PR#33-reused-line-comment-gets-new-text.md Scenario#33.07 given_when_then_misused 0.70 – YELLOW

10 of 11 units passed all questions (1 PR document(s)).
3 answers ignored because of a confidence below [classifier] min_confidence.

Usage: model `openrouter/anthropic/claude-sonnet-5.5`, LLM done, tokens counted before the call 79309, calls 1, tokens in 77601 / out 795 (reasoning included in out), duration 15.1 s, cost 0.1632 USD (reported), classifier `jev-1.13.0` calls 11, tokens in 9590, cost 0.0004 USD, finding judge `jev-1.13.0` calls 2, tokens in 3422, cost 0.0001 USD, build file judge `jev-1.13.0` calls 1, tokens in 560, cost 0.0000 USD, config: defaults + host + repository (.werkrevisor.toml at base dcdcf48).
Reviewed head `0fcbb0f971e0e63780d1c8eeddf8ffd5ed7e631d` against base `dcdcf48bed8f9fc8a7752b3fddce300c185e8358`, werkrevisor 0.20.1.
<!-- werkrevisor:0fcbb0f971e0e63780d1c8eeddf8ffd5ed7e631d --> <!-- werkrevisor-verdict:ORANGE --> <!-- werkrevisor-folded --> <details><summary>🟠 werkrevisor ORANGE for <code>0fcbb0f</code>, superseded by <a href="http://git.javagil.de/mi/werkrevisor/pulls/33#issuecomment-1485">the result for <code>c3ad49e</code></a></summary> ## 🟠 werkrevisor: ORANGE / FINDINGS Change kind: `mixed`. ### Reasons - **ORANGE** 1 finding(s) that Jev rates as probably real and severe: werkrevisor/cli.py:499-504 - **YELLOW** risky paths changed: werkrevisor/gitea.py (werkrevisor/gitea.py) - **YELLOW** spec: docs/prs/2026-10-06-PR#33-reused-line-comment-gets-new-text.md Scenario#33.07 given_when_then_misused (0.70 > yellow_above = 0.6) - **YELLOW** 1 finding(s) possibly severe, but Jev doubts that they are real: werkrevisor/cli.py:533-537 - **hint** 1 build change(s) of low risk according to Jev: server/build.gradle.kts (weakens checks 0.02, risk 0.00) - **hint** 1 finding(s) of low severity or rated minor by Jev: docs/prs/2026-10-06-PR#33-reused-line-comment-gets-new-text.md:119-127 - **hint** model recommends merge with caution ### Findings 1. **[medium] Edited reused comment is not kept in sync across runs when a 500 is refused** at `werkrevisor/cli.py:499-504`, ORANGE, still open in [an earlier line comment](http://git.javagil.de/mi/werkrevisor/pulls/33#issuecomment-1458) 2. **[medium] Withdrawal skips reviewers whose login differs only in case from the request** at `werkrevisor/cli.py:533-537`, YELLOW, as a line comment 3. **[low] Scenario 33.07 contradicts the code for the COMMENT event** at `docs/prs/2026-10-06-PR#33-reused-line-comment-gets-new-text.md:119-127`, hint, still open in [an earlier line comment](http://git.javagil.de/mi/werkrevisor/pulls/33#issuecomment-1448) ### Classifier | Unit | Question | Problem | Confidence | Verdict | |---|---|---|---|---| | docs/prs/2026-10-06-PR#33-reused-line-comment-gets-new-text.md Scenario#33.07 | given_when_then_misused | 0.70 | – | YELLOW | 10 of 11 units passed all questions (1 PR document(s)). 3 answers ignored because of a confidence below [classifier] min_confidence. <sub> Usage: model `openrouter/anthropic/claude-sonnet-5.5`, LLM done, tokens counted before the call 79309, calls 1, tokens in 77601 / out 795 (reasoning included in out), duration 15.1 s, cost 0.1632 USD (reported), classifier `jev-1.13.0` calls 11, tokens in 9590, cost 0.0004 USD, finding judge `jev-1.13.0` calls 2, tokens in 3422, cost 0.0001 USD, build file judge `jev-1.13.0` calls 1, tokens in 560, cost 0.0000 USD, config: defaults + host + repository (.werkrevisor.toml at base dcdcf48).<br> Reviewed head `0fcbb0f971e0e63780d1c8eeddf8ffd5ed7e631d` against base `dcdcf48bed8f9fc8a7752b3fddce300c185e8358`, werkrevisor 0.20.1. </sub> </details>
ai-reviewer left a comment
Collaborator

werkrevisor: ORANGE / FINDINGS for 0fcbb0f971e0e63780d1c8eeddf8ffd5ed7e631d.
The result comment of werkrevisor holds the details.

werkrevisor: ORANGE / FINDINGS for `0fcbb0f971e0e63780d1c8eeddf8ffd5ed7e631d`. The result comment of werkrevisor holds the details. <!-- werkrevisor-review:0fcbb0f971e0e63780d1c8eeddf8ffd5ed7e631d -->
@@ -506,3 +531,3 @@
exit_code = 0 if cancelled else Verdict.YELLOW.exit_code
return GateDecision(exit_code, "approved by werkrevisor" + note, held=True)
return GateDecision(0, "approved by werkrevisor" + _schedule_merge(config, pr, gitea, redactor, base_ref))
scheduled = _schedule_merge(config, pr, gitea, redactor, base_ref)
Collaborator

[medium] Withdrawal skips reviewers whose login differs only in case from the request (werkrevisor)

YELLOW; Jev: real 0.54, severity 2.7 of 3

_withdraw_review_request passes the logins as Gitea returned them, which is correct. However, the withdrawal is only attempted when the merge note equals exactly MERGE_SCHEDULED or MERGE_ALREADY_SCHEDULED; the comparison of rendered note strings is fragile, since _schedule_merge returns redactor-processed text for other branches. A future change of the note wording silently disables the withdrawal. Prefer a structured return value.

**[medium] Withdrawal skips reviewers whose login differs only in case from the request** (werkrevisor) YELLOW; Jev: real 0.54, severity 2.7 of 3 `_withdraw_review_request` passes the logins as Gitea returned them, which is correct. However, the withdrawal is only attempted when the merge note equals exactly `MERGE_SCHEDULED` or `MERGE_ALREADY_SCHEDULED`; the comparison of rendered note strings is fragile, since `_schedule_merge` returns redactor-processed text for other branches. A future change of the note wording silently disables the withdrawal. Prefer a structured return value.
Author
Collaborator

Not changed: the notes are the constants MERGE_SCHEDULED and MERGE_ALREADY_SCHEDULED, which _schedule_merge returns as they are; a change of the wording changes the constant and so the comparison. test_a_scheduled_merge_withdraws_the_review_request_of_an_earlier_round fails if the withdrawal is silently disabled. Left open for the human reviewer.

Not changed: the notes are the constants `MERGE_SCHEDULED` and `MERGE_ALREADY_SCHEDULED`, which `_schedule_merge` returns as they are; a change of the wording changes the constant and so the comparison. `test_a_scheduled_merge_withdraws_the_review_request_of_an_earlier_round` fails if the withdrawal is silently disabled. Left open for the human reviewer.
Author
Collaborator

Answer to the review of 0fcbb0f (ORANGE):

  • ORANGE werkrevisor/cli.py:499-504 (reused comment not in sync after a 500): its text is lost, since 0.20.1 reused my reply 1458. From the title: an edit of a reused comment happens before the review POST and is not affected by a later 500; the new findings of a refused review go into the result comment. Not changed.
  • YELLOW werkrevisor/cli.py:533-537 (note strings as condition): not changed, reasons in the thread.
  • YELLOW spec Scenario#33.07 and hint "contradicts the code for the COMMENT event": fixed in c3ad49e; the Then no longer claims a second review, which only an approval gets.
  • YELLOW risky path werkrevisor/gitea.py: needs the human review.
Answer to the review of `0fcbb0f` (ORANGE): - ORANGE `werkrevisor/cli.py:499-504` (reused comment not in sync after a 500): its text is lost, since 0.20.1 reused my reply 1458. From the title: an edit of a reused comment happens before the review POST and is not affected by a later 500; the new findings of a refused review go into the result comment. Not changed. - YELLOW `werkrevisor/cli.py:533-537` (note strings as condition): not changed, reasons in the thread. - YELLOW spec Scenario#33.07 and hint "contradicts the code for the COMMENT event": fixed in c3ad49e; the Then no longer claims a second review, which only an approval gets. - YELLOW risky path `werkrevisor/gitea.py`: needs the human review.
mi added 1 commit 2026-10-06 16:37:10 +02:00
review: Scenario#33.07 no longer claims a second COMMENT review
werkator build succeeded after 00:08
ai-review/assessment AI findings; a developer approval after checking them unlocks the merge; a human review request is pending; merge already scheduled
ai-review/gate build succeeded after 00:16
werkator/server build succeeded after 00:27
c3ad49e96a
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Collaborator
🟠 werkrevisor ORANGE for c3ad49e, superseded by the result for aa33df4

🟠 werkrevisor: ORANGE / FINDINGS

Change kind: mixed.

Reasons

  • ORANGE 2 finding(s) that Jev rates as probably real and severe: werkrevisor/cli.py:499-500, werkrevisor/cli.py:455-456
  • YELLOW risky paths changed: werkrevisor/gitea.py (werkrevisor/gitea.py)
  • hint 1 build change(s) of low risk according to Jev: server/build.gradle.kts (weakens checks 0.02, risk 0.00)
  • hint 1 finding(s) of low severity or rated minor by Jev: tests/test_line_comments.py:37-39
  • hint model recommends merge with caution

Findings

  1. [medium] HTTP 500 treated as a line-comment refusal hides real server errors at werkrevisor/cli.py:499-500, ORANGE, still open in an earlier line comment

  2. [medium] Edit of reused line comment happens before the result comment and review are posted at werkrevisor/cli.py:455-456, ORANGE, still open in an earlier line comment

  3. [medium] Missing test for failed withdrawal when line comments hit HTTP 500 on APPROVED event at tests/test_line_comments.py:37-39, hint, as a line comment

Classifier

11 of 11 units passed all questions (1 PR document(s)).
4 answers ignored because of a confidence below [classifier] min_confidence.

Usage: model `openrouter/anthropic/claude-sonnet-5.5`, LLM done, tokens counted before the call 79298, calls 1, tokens in 77594 / out 795 (reasoning included in out), duration 13.7 s, cost 0.1631 USD (reported), classifier `jev-1.13.0` calls 11, tokens in 9579, cost 0.0004 USD, finding judge `jev-1.13.0` calls 3, tokens in 4723, cost 0.0002 USD, build file judge `jev-1.13.0` calls 1, tokens in 560, cost 0.0000 USD, config: defaults + host + repository (.werkrevisor.toml at base dcdcf48).
Reviewed head `c3ad49e96a6f68fff353d713b737a23b70ac5c88` against base `dcdcf48bed8f9fc8a7752b3fddce300c185e8358`, werkrevisor 0.20.1.
<!-- werkrevisor:c3ad49e96a6f68fff353d713b737a23b70ac5c88 --> <!-- werkrevisor-verdict:ORANGE --> <!-- werkrevisor-folded --> <details><summary>🟠 werkrevisor ORANGE for <code>c3ad49e</code>, superseded by <a href="http://git.javagil.de/mi/werkrevisor/pulls/33#issuecomment-1491">the result for <code>aa33df4</code></a></summary> ## 🟠 werkrevisor: ORANGE / FINDINGS Change kind: `mixed`. ### Reasons - **ORANGE** 2 finding(s) that Jev rates as probably real and severe: werkrevisor/cli.py:499-500, werkrevisor/cli.py:455-456 - **YELLOW** risky paths changed: werkrevisor/gitea.py (werkrevisor/gitea.py) - **hint** 1 build change(s) of low risk according to Jev: server/build.gradle.kts (weakens checks 0.02, risk 0.00) - **hint** 1 finding(s) of low severity or rated minor by Jev: tests/test_line_comments.py:37-39 - **hint** model recommends merge with caution ### Findings 1. **[medium] HTTP 500 treated as a line-comment refusal hides real server errors** at `werkrevisor/cli.py:499-500`, ORANGE, still open in [an earlier line comment](http://git.javagil.de/mi/werkrevisor/pulls/33#issuecomment-1458) 2. **[medium] Edit of reused line comment happens before the result comment and review are posted** at `werkrevisor/cli.py:455-456`, ORANGE, still open in [an earlier line comment](http://git.javagil.de/mi/werkrevisor/pulls/33#issuecomment-1468) 3. **[medium] Missing test for failed withdrawal when line comments hit HTTP 500 on APPROVED event** at `tests/test_line_comments.py:37-39`, hint, as a line comment ### Classifier 11 of 11 units passed all questions (1 PR document(s)). 4 answers ignored because of a confidence below [classifier] min_confidence. <sub> Usage: model `openrouter/anthropic/claude-sonnet-5.5`, LLM done, tokens counted before the call 79298, calls 1, tokens in 77594 / out 795 (reasoning included in out), duration 13.7 s, cost 0.1631 USD (reported), classifier `jev-1.13.0` calls 11, tokens in 9579, cost 0.0004 USD, finding judge `jev-1.13.0` calls 3, tokens in 4723, cost 0.0002 USD, build file judge `jev-1.13.0` calls 1, tokens in 560, cost 0.0000 USD, config: defaults + host + repository (.werkrevisor.toml at base dcdcf48).<br> Reviewed head `c3ad49e96a6f68fff353d713b737a23b70ac5c88` against base `dcdcf48bed8f9fc8a7752b3fddce300c185e8358`, werkrevisor 0.20.1. </sub> </details>
ai-reviewer left a comment
Collaborator

werkrevisor: ORANGE / FINDINGS for c3ad49e96a6f68fff353d713b737a23b70ac5c88.
The result comment of werkrevisor holds the details.

werkrevisor: ORANGE / FINDINGS for `c3ad49e96a6f68fff353d713b737a23b70ac5c88`. The result comment of werkrevisor holds the details. <!-- werkrevisor-review:c3ad49e96a6f68fff353d713b737a23b70ac5c88 -->
@@ -35,3 +35,3 @@
def test_finding_without_a_usable_line_stays_in_the_result(self):
for name, issues, start, refuse in (("no line", [APP], 0, False), ("other file", ["other.py"], 40, False),
("refused by Gitea", [APP], 40, True)):
("refused by Gitea", [APP], 40, True),
Collaborator

[medium] Missing test for failed withdrawal when line comments hit HTTP 500 on APPROVED event (werkrevisor)

hint; Jev: real 0.78, severity 2.0 of 3

The new HTTP 500 handling in _publish_review retries an APPROVED review without line comments after the refusal. The test test_finding_without_a_usable_line_stays_in_the_result uses finding_run with the default gate="approval" and an ORANGE verdict, so only the COMMENT path is exercised. The retry-without-comments path for GREEN (event != "COMMENT") is not tested with status 500.

**[medium] Missing test for failed withdrawal when line comments hit HTTP 500 on APPROVED event** (werkrevisor) hint; Jev: real 0.78, severity 2.0 of 3 The new HTTP 500 handling in `_publish_review` retries an APPROVED review without line comments after the refusal. The test `test_finding_without_a_usable_line_stays_in_the_result` uses `finding_run` with the default `gate="approval"` and an ORANGE verdict, so only the COMMENT path is exercised. The retry-without-comments path for GREEN (`event != "COMMENT"`) is not tested with status 500.
Author
Collaborator

Fixed in aa33df4: test_a_green_review_refused_with_http_500_approves_without_line_comments.

Fixed in aa33df4: `test_a_green_review_refused_with_http_500_approves_without_line_comments`.
claude-code marked this conversation as resolved
Author
Collaborator

Answer to the review of c3ad49e (ORANGE):

  • ORANGE werkrevisor/cli.py:499-500 (HTTP 500 as refusal) and werkrevisor/cli.py:455-456 (edit before publishing): repeated findings, not changed, reasons in the threads 1458 and 1468, left open for the human reviewer.
  • hint tests/test_line_comments.py:37-39 (no test of HTTP 500 on a GREEN review): test added in aa33df4, resolved.
  • YELLOW risky path werkrevisor/gitea.py: needs the human review.

The spec check passes now. From my side PR#33 is ready for the human review.

Answer to the review of `c3ad49e` (ORANGE): - ORANGE `werkrevisor/cli.py:499-500` (HTTP 500 as refusal) and `werkrevisor/cli.py:455-456` (edit before publishing): repeated findings, not changed, reasons in the threads 1458 and 1468, left open for the human reviewer. - hint `tests/test_line_comments.py:37-39` (no test of HTTP 500 on a GREEN review): test added in aa33df4, resolved. - YELLOW risky path `werkrevisor/gitea.py`: needs the human review. The spec check passes now. From my side PR#33 is ready for the human review.
mi added 1 commit 2026-10-06 16:38:47 +02:00
review: a GREEN review refused with HTTP 500 approves without line comments, tested
werkator build succeeded after 00:10
ai-review/assessment human review required; a human approval unlocks the merge; a human review request is pending; merge already scheduled
ai-review/gate build succeeded after 00:17
werkator/server build succeeded after 00:27
aa33df41af
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Collaborator
🟡 werkrevisor YELLOW for aa33df4, superseded by the result for d9b659a

🟡 werkrevisor: YELLOW / HUMAN_REVIEW

Change kind: mixed.

Reasons

  • YELLOW risky paths changed: werkrevisor/gitea.py (werkrevisor/gitea.py)
  • YELLOW spec: docs/prs/2026-10-06-PR#33-reused-line-comment-gets-new-text.md Scenario#33.05 given_when_then_misused (0.63 > yellow_above = 0.6)
  • YELLOW 1 finding(s) possibly severe, but Jev doubts that they are real: werkrevisor/cli.py:535-536
  • hint 1 build change(s) of low risk according to Jev: server/build.gradle.kts (weakens checks 0.02, risk 0.00)
  • hint 2 finding(s) of low severity or rated minor by Jev: docs/prs/2026-10-06-PR#33-reused-line-comment-gets-new-text.md:85-97, werkrevisor/cli.py:455-456
  • hint model recommends merge with caution

Findings

  1. [medium] Withdrawal skipped when the merge is held but earlier requests remain at werkrevisor/cli.py:535-536, YELLOW, still open in an earlier line comment

  2. [medium] Scenario#33.05 contradicts the test and the code at docs/prs/2026-10-06-PR#33-reused-line-comment-gets-new-text.md:85-97, hint, still open in an earlier line comment

  3. [medium] Edit of a reused comment happens before the review is published at werkrevisor/cli.py:455-456, hint, still open in an earlier line comment

Classifier

Unit Question Problem Confidence Verdict
docs/prs/2026-10-06-PR#33-reused-line-comment-gets-new-text.md Scenario#33.05 given_when_then_misused 0.63 – YELLOW

10 of 11 units passed all questions (1 PR document(s)).
4 answers ignored because of a confidence below [classifier] min_confidence.

Usage: model `openrouter/anthropic/claude-sonnet-5.5`, LLM done, tokens counted before the call 79905, calls 1, tokens in 78208 / out 793 (reasoning included in out), duration 13.8 s, cost 0.1643 USD (reported), classifier `jev-1.13.0` calls 11, tokens in 9579, cost 0.0004 USD, finding judge `jev-1.13.0` calls 3, tokens in 4858, cost 0.0002 USD, build file judge `jev-1.13.0` calls 1, tokens in 560, cost 0.0000 USD, config: defaults + host + repository (.werkrevisor.toml at base dcdcf48).
Reviewed head `aa33df41af752b9071c4ac675f34c0b67042d028` against base `dcdcf48bed8f9fc8a7752b3fddce300c185e8358`, werkrevisor 0.20.1.
<!-- werkrevisor:aa33df41af752b9071c4ac675f34c0b67042d028 --> <!-- werkrevisor-verdict:YELLOW --> <!-- werkrevisor-folded --> <details><summary>🟡 werkrevisor YELLOW for <code>aa33df4</code>, superseded by <a href="http://git.javagil.de/mi/werkrevisor/pulls/33#issuecomment-1495">the result for <code>d9b659a</code></a></summary> ## 🟡 werkrevisor: YELLOW / HUMAN_REVIEW Change kind: `mixed`. ### Reasons - **YELLOW** risky paths changed: werkrevisor/gitea.py (werkrevisor/gitea.py) - **YELLOW** spec: docs/prs/2026-10-06-PR#33-reused-line-comment-gets-new-text.md Scenario#33.05 given_when_then_misused (0.63 > yellow_above = 0.6) - **YELLOW** 1 finding(s) possibly severe, but Jev doubts that they are real: werkrevisor/cli.py:535-536 - **hint** 1 build change(s) of low risk according to Jev: server/build.gradle.kts (weakens checks 0.02, risk 0.00) - **hint** 2 finding(s) of low severity or rated minor by Jev: docs/prs/2026-10-06-PR#33-reused-line-comment-gets-new-text.md:85-97, werkrevisor/cli.py:455-456 - **hint** model recommends merge with caution ### Findings 1. **[medium] Withdrawal skipped when the merge is held but earlier requests remain** at `werkrevisor/cli.py:535-536`, YELLOW, still open in [an earlier line comment](http://git.javagil.de/mi/werkrevisor/pulls/33#issuecomment-1480) 2. **[medium] Scenario#33.05 contradicts the test and the code** at `docs/prs/2026-10-06-PR#33-reused-line-comment-gets-new-text.md:85-97`, hint, still open in [an earlier line comment](http://git.javagil.de/mi/werkrevisor/pulls/33#issuecomment-1441) 3. **[medium] Edit of a reused comment happens before the review is published** at `werkrevisor/cli.py:455-456`, hint, still open in [an earlier line comment](http://git.javagil.de/mi/werkrevisor/pulls/33#issuecomment-1468) ### Classifier | Unit | Question | Problem | Confidence | Verdict | |---|---|---|---|---| | docs/prs/2026-10-06-PR#33-reused-line-comment-gets-new-text.md Scenario#33.05 | given_when_then_misused | 0.63 | – | YELLOW | 10 of 11 units passed all questions (1 PR document(s)). 4 answers ignored because of a confidence below [classifier] min_confidence. <sub> Usage: model `openrouter/anthropic/claude-sonnet-5.5`, LLM done, tokens counted before the call 79905, calls 1, tokens in 78208 / out 793 (reasoning included in out), duration 13.8 s, cost 0.1643 USD (reported), classifier `jev-1.13.0` calls 11, tokens in 9579, cost 0.0004 USD, finding judge `jev-1.13.0` calls 3, tokens in 4858, cost 0.0002 USD, build file judge `jev-1.13.0` calls 1, tokens in 560, cost 0.0000 USD, config: defaults + host + repository (.werkrevisor.toml at base dcdcf48).<br> Reviewed head `aa33df41af752b9071c4ac675f34c0b67042d028` against base `dcdcf48bed8f9fc8a7752b3fddce300c185e8358`, werkrevisor 0.20.1. </sub> </details>
Author
Collaborator

Answer to the review of aa33df4 (YELLOW):

  • YELLOW spec Scenario#33.05 and hint "contradicts the test and the code": fixed in d9b659a; the scenario now has a table for the resolved and the open hint, matching its two tests.
  • YELLOW werkrevisor/cli.py:535-536 (withdrawal skipped while the merge is held): intended, Scenario#33.06; a held merge waits for the author, and the human request stays until the merge is scheduled.
  • hint werkrevisor/cli.py:455-456 (edit before publishing): repeated, see thread 1468.
  • YELLOW risky path werkrevisor/gitea.py: needs the human review.
Answer to the review of `aa33df4` (YELLOW): - YELLOW spec Scenario#33.05 and hint "contradicts the test and the code": fixed in d9b659a; the scenario now has a table for the resolved and the open hint, matching its two tests. - YELLOW `werkrevisor/cli.py:535-536` (withdrawal skipped while the merge is held): intended, Scenario#33.06; a held merge waits for the author, and the human request stays until the merge is scheduled. - hint `werkrevisor/cli.py:455-456` (edit before publishing): repeated, see thread 1468. - YELLOW risky path `werkrevisor/gitea.py`: needs the human review.
mi added 1 commit 2026-10-06 16:50:06 +02:00
review: Scenario#33.05 covers the open and the resolved hint, as its two tests do
werkator/werkrevisor-statistics build succeeded after 00:28
werkator build succeeded after 00:08
ai-review/assessment AI findings; a developer approval after checking them unlocks the merge; approved by mi for this head commit; merge already scheduled
ai-review/gate build succeeded after 00:17
d9b659a7e9
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
mi approved these changes 2026-10-06 16:50:22 +02:00
Collaborator

🟠 werkrevisor: ORANGE / FINDINGS

Change kind: mixed.

Reasons

  • ORANGE 2 finding(s) that Jev rates as probably real and severe: werkrevisor/cli.py:455-456, werkrevisor/cli.py:499-500
  • YELLOW risky paths changed: werkrevisor/gitea.py (werkrevisor/gitea.py)
  • hint 1 build change(s) of low risk according to Jev: server/build.gradle.kts (weakens checks 0.02, risk 0.00)
  • hint 1 finding(s) of low severity or rated minor by Jev: docs/prs/2026-10-06-PR#33-reused-line-comment-gets-new-text.md:14-17
  • hint model recommends changes before merge

Findings

  1. [medium] Line comment edited before the result comment is posted at werkrevisor/cli.py:455-456, ORANGE, still open in an earlier line comment

  2. [medium] HTTP 500 treated as refusal of line comments at werkrevisor/cli.py:499-500, ORANGE, still open in an earlier line comment

  3. [low] PR title and document scope are inconsistent at docs/prs/2026-10-06-PR#33-reused-line-comment-gets-new-text.md:14-17, hint, as a line comment

Classifier

11 of 11 units passed all questions (1 PR document(s)).
3 answers ignored because of a confidence below [classifier] min_confidence.

Usage: model `openrouter/anthropic/claude-sonnet-5.5`, LLM done, tokens counted before the call 79818, calls 1, tokens in 78137 / out 786 (reasoning included in out), duration 14.2 s, cost 0.1641 USD (reported), classifier `jev-1.13.0` calls 11, tokens in 9618, cost 0.0004 USD, finding judge `jev-1.13.0` calls 2, tokens in 3041, cost 0.0001 USD, build file judge `jev-1.13.0` calls 1, tokens in 560, cost 0.0000 USD, config: defaults + host + repository (.werkrevisor.toml at base dcdcf48).
Reviewed head `d9b659a7e9e68faa92b2d62aa3d789cf93eec891` against base `dcdcf48bed8f9fc8a7752b3fddce300c185e8358`, werkrevisor 0.20.1.
<!-- werkrevisor:d9b659a7e9e68faa92b2d62aa3d789cf93eec891 --> <!-- werkrevisor-verdict:ORANGE --> ## 🟠 werkrevisor: ORANGE / FINDINGS Change kind: `mixed`. ### Reasons - **ORANGE** 2 finding(s) that Jev rates as probably real and severe: werkrevisor/cli.py:455-456, werkrevisor/cli.py:499-500 - **YELLOW** risky paths changed: werkrevisor/gitea.py (werkrevisor/gitea.py) - **hint** 1 build change(s) of low risk according to Jev: server/build.gradle.kts (weakens checks 0.02, risk 0.00) - **hint** 1 finding(s) of low severity or rated minor by Jev: docs/prs/2026-10-06-PR#33-reused-line-comment-gets-new-text.md:14-17 - **hint** model recommends changes before merge ### Findings 1. **[medium] Line comment edited before the result comment is posted** at `werkrevisor/cli.py:455-456`, ORANGE, still open in [an earlier line comment](http://git.javagil.de/mi/werkrevisor/pulls/33#issuecomment-1468) 2. **[medium] HTTP 500 treated as refusal of line comments** at `werkrevisor/cli.py:499-500`, ORANGE, still open in [an earlier line comment](http://git.javagil.de/mi/werkrevisor/pulls/33#issuecomment-1458) 3. **[low] PR title and document scope are inconsistent** at `docs/prs/2026-10-06-PR#33-reused-line-comment-gets-new-text.md:14-17`, hint, as a line comment ### Classifier 11 of 11 units passed all questions (1 PR document(s)). 3 answers ignored because of a confidence below [classifier] min_confidence. <sub> Usage: model `openrouter/anthropic/claude-sonnet-5.5`, LLM done, tokens counted before the call 79818, calls 1, tokens in 78137 / out 786 (reasoning included in out), duration 14.2 s, cost 0.1641 USD (reported), classifier `jev-1.13.0` calls 11, tokens in 9618, cost 0.0004 USD, finding judge `jev-1.13.0` calls 2, tokens in 3041, cost 0.0001 USD, build file judge `jev-1.13.0` calls 1, tokens in 560, cost 0.0000 USD, config: defaults + host + repository (.werkrevisor.toml at base dcdcf48).<br> Reviewed head `d9b659a7e9e68faa92b2d62aa3d789cf93eec891` against base `dcdcf48bed8f9fc8a7752b3fddce300c185e8358`, werkrevisor 0.20.1. </sub>
ai-reviewer left a comment
Collaborator

werkrevisor: ORANGE / FINDINGS for d9b659a7e9e68faa92b2d62aa3d789cf93eec891.
The result comment of werkrevisor holds the details.

werkrevisor: ORANGE / FINDINGS for `d9b659a7e9e68faa92b2d62aa3d789cf93eec891`. The result comment of werkrevisor holds the details. <!-- werkrevisor-review:d9b659a7e9e68faa92b2d62aa3d789cf93eec891 -->
@@ -0,0 +11,4 @@
The model words a finding differently in each round and often sharpens it after a fix;
the new wording is lost, and the author answers a text that the latest review no longer holds.
The reviews of this PR showed two more defects of 0.20.1:
Collaborator

[low] PR title and document scope are inconsistent (werkrevisor)

hint

The PR title and the first Problem paragraph name only the reused line comment text. The document also covers reply filtering, withdrawal of review requests and HTTP 500 handling, and presents these as defects "of 0.20.1". The Problem section mixes the three extra defects into one long sentence chain with ; and and, which is hard to read. The unrelated changes make the PR harder to review and revert.

**[low] PR title and document scope are inconsistent** (werkrevisor) hint The PR title and the first Problem paragraph name only the reused line comment text. The document also covers reply filtering, withdrawal of review requests and HTTP 500 handling, and presents these as defects "of 0.20.1". The Problem section mixes the three extra defects into one long sentence chain with `;` and `and`, which is hard to read. The unrelated changes make the PR harder to review and revert.
ai-reviewer merged commit 760f9a5c63 into main 2026-10-06 16:50:30 +02:00
ai-reviewer deleted branch claude-code/reused-comment-gets-new-text 2026-10-06 16:50:31 +02:00
Sign in to join this conversation.
No Reviewers
No labels
3 Participants
Notifications
Due Date
No due date set.
Dependencies

No dependencies set.

Reference: mi/werkrevisor#33