werkrevisor 0.36.0: from the second round on, a hint on unchanged text holds no merge #91
No Assignees
Notifications
Due Date
No due date set.
Dependencies
No dependencies set.
Reference: mi/werkrevisor#91
Reference in New Issue
Block a user
Fewer review rounds: a new GREEN hint on documents or a writing rule holds the merge only on lines changed since the last reviewed head.
🤖 Generated with Claude Code
🟠 werkrevisor ORANGE for
0e127b7, superseded by the result for5554351🟠 werkrevisor: ORANGE / FINDINGS
Change kind:
mixed.Reasons
Findings
werkrevisor/cli.py:787-789, severity unknown, counted as medium, ORANGE, as a line commentClassifier
5 of 6 units passed all questions (1 PR document(s)).
Usage: model `openrouter/minimax/minimax-m3`, LLM done, tokens counted before the call 22731, calls 1, tokens in 21993 / out 42936 (reasoning included in out), duration 374.6 s, cost 0.0579 USD (reported), classifier `jev-1.13.0` calls 6, tokens in 4761, cost 0.0002 USD, finding judge `jev-1.13.0` calls 1, tokens in 1633, cost 0.0001 USD, excerpt selection `openrouter/openai/gpt-6-luna` calls 1, tokens in 22618 / out 488, cost 0.0031 USD (reported), change summary `openrouter/minimax/minimax-m3` via Minimax calls 1, tokens in 3941 / out 1418, cost 0.0029 USD (reported), PR document check `jev-1.13.0` calls 6, tokens in 9405, cost 0.0004 USD, build file judge `jev-1.13.0` calls 5, tokens in 4776, cost 0.0002 USD, phases: path_judge 0.6 s, classifier 0.5 s, change_summary 17.0 s, doc_check 0.4 s, context 6.5 s, model_review 348.9 s, finding_judge 0.2 s, config: defaults + host + repository (.werkrevisor.toml at main1 answer ignored because of a confidence below [classifier] min_confidence.
7ea928f).Reviewed head `0e127b76c4147e83406ef513806bb9530a59540f` against base `7ea928f1cfb4b80b694e56feca447fb5746f3140`, werkrevisor 0.35.0.
werkrevisor: ORANGE / FINDINGS for
0e127b76c4147e83406ef513806bb9530a59540f.The result comment of werkrevisor holds the details.
@@ -132,5 +132,5 @@it stands only in the result comment and holds nothing, unless `[review] doc_hint_line_comments = true`.A frontier model rated 25 such hints of werkrevisor's own PRs: 19 were noise, but each one had cost a review round.YELLOW and ORANGE are not affected: the approving human reads the hints.When a GREEN review schedules the merge, it withdraws the review request of an earlier YELLOW or ORANGE roundfrom the configured human reviewers, since `block_on_official_review_requests` would block the merge.Writing rule: paragraph with 8 sentences, more than 5 (werkrevisor writing rule)
Fix the line, or reply why it stays, and resolve this comment.
Fixed in
5554351: the paragraph is split in two.@@ -755,12 +784,17 @@ def _plan_line_comments(config, gitea, pr, result, bot_login, redactor):if not _update_line_comment(gitea, pr, earlier, report.line_comment(f, v), redactor, updated):result.reused_not_updated.add(i)continueif v is Verdict.GREEN and _on_unchanged_text(config, gitea, pr, result, f.file, first, last, unchanged):doc_hint_line_comments Override, severity unknown, counted as medium (werkrevisor)
ORANGE; Jev: real 0.76, severity 2.9 of 3
The new GREEN-hint branch in
_plan_line_commentsadds a finding todoc_hintswhenever_on_unchanged_textis true, without checkingconfig.review.doc_hint_line_comments. Withdoc_hint_line_comments = true(the override the README advertises to force line comments for GREEN hints on docs), hints on unchanged text in docs still end up indoc_hintsand never get a line comment, so the override is silently partial. Either the new branch should also requirenot config.review.doc_hint_line_comments, or the docs (README and PR doc) must say that the override only applies to changed text. None of the new tests exercisedoc_hint_line_comments = true, so this interaction is uncovered.Fixed in
5554351: withdoc_hint_line_comments = trueevery hint of the model stays a line comment, also on unchanged text;test_doc_hint_line_comments_keeps_the_hintfails without the fix.Codex review of
0e127b76c4147e83406ef513806bb9530a59540f@@ -755,12 +784,18 @@ def _plan_line_comments(config, gitea, pr, result, bot_login, redactor):if not _update_line_comment(gitea, pr, earlier, report.line_comment(f, v), redactor, updated):result.reused_not_updated.add(i)continueif (v is Verdict.GREEN and not config.review.doc_hint_line_comments[high] Changed document sections lose their blocking hints (Codex review)
doccheck.make_findings() sets both finding endpoints to the section or scenario heading. When its body changes but its heading stays unchanged, this condition suppresses even a new contradiction finding about the changed text. Reproduced with an edited Solution section: no line comment is planned and the finding contributes zero blocking hints. Compare the affected section content before exempting these findings.
Fix it, or reply why it does not hold, and resolve this comment.
Fixed in
4dc8396: a finding on a Markdown heading covers the whole section below it;test_a_finding_on_a_heading_covers_its_sectionfails without the fix.@@ -760,4 +793,4 @@result.attached.add(i)for i, v in enumerate(result.writing_violations):if (v.path, v.line) in open_writing:result.writing_reused.add(i)[low] New paragraph-length violations are treated as unchanged (Codex review)
writing.check_text() anchors paragraph-length violations to the paragraph's first line. Adding a sixth sentence to an existing five-sentence paragraph leaves that anchor unchanged, so this branch suppresses the newly introduced violation. Reproduced: the violation is detected, no line comment is posted, and automerge is scheduled. The unchanged check must cover the paragraph rather than only its first line.
Fix it, or reply why it does not hold, and resolve this comment.
Fixed in
4dc8396: a paragraph rule covers its whole paragraph (Violation.last);test_a_paragraph_rule_covers_its_paragraphfails without the fix.⚠️ werkrevisor ERROR for
?, superseded by the result for5554351Codex review: critical
This change weakens the automerge gate by exempting some review hints from blocking merge. Changes to CI gates require human review under the stated criteria.
Findings
werkrevisor/cli.py:787, as a line commentwerkrevisor/cli.py:796, as a line commentThis review only advises; it holds no merge.
Model `gpt-6-astra`, effort high, tokens in 497793 (cached 434688) / out 3478, 122.8 s, 19 commands; reviewed head `0e127b76c4147e83406ef513806bb9530a59540f`.🟢 werkrevisor GREEN for
5554351, superseded by the result for7b3e2a0🟢 werkrevisor: GREEN / PASS
Gate: approved by werkrevisor; merge waits for 2 open hint(s): reply to each line comment and resolve it, then restart the review or push a commit.
Change kind:
mixed.Reasons
Findings
README contradicts PR document behavior at
README.md:137-147, severity unknown, counted as mediumhint; Jev: real 0.78, severity 2.0 of 3
The README keeps "A hint on a PR document (
[pr_doc] globs) is a line comment anyway, since it is the record of the PR" and adds "From the second round on, a new hint on documentation, or a violation of the writing rules, becomes a line comment only on lines that changed since the head of the last round; on older text it stands only in the result comment and holds nothing." For a GREEN hint on a PR document on unchanged text from round 2 on the two statements conflict; the new code (andtest_new_hint_on_old_text, which usesdocs/prs/1.md) silence such hints, so the "anyway" sentence is misleading.Outdated rationale comment at
werkrevisor/cli.py:772-777, severity unknown, counted as medium, hint, as a line commentUnverifiable statistic at
docs/prs/2026-10-12-PR#91-hints-on-changed-lines.md:13, severity unknown, counted as medium, hint, as a line commentClassifier
6 of 6 units passed all questions (1 PR document(s)).
Usage: model `openrouter/minimax/minimax-m3`, LLM done, tokens counted before the call 22759, calls 1, tokens in 22024 / out 33298 (reasoning included in out), duration 341.1 s, cost 0.0465 USD (reported), classifier `jev-1.13.0` calls 6, tokens in 4793, cost 0.0002 USD, finding judge `jev-1.13.0` calls 3, tokens in 4882, cost 0.0002 USD, excerpt selection `openrouter/openai/gpt-6-luna` calls 1, tokens in 22644 / out 480, cost 0.0031 USD (reported), change summary `openrouter/minimax/minimax-m3` via Minimax calls 1, tokens in 4065 / out 991, cost 0.0024 USD (reported), PR document check `jev-1.13.0` calls 6, tokens in 9377, cost 0.0004 USD, build file judge `jev-1.13.0` calls 5, tokens in 4878, cost 0.0002 USD, phases: path_judge 0.4 s, classifier 0.4 s, change_summary 8.8 s, doc_check 0.5 s, context 6.0 s, model_review 324.2 s, finding_judge 0.3 s, config: defaults + host + repository (.werkrevisor.toml at main2 answers ignored because of a confidence below [classifier] min_confidence.
7ea928f).Reviewed head `555435134ae767044badb74331ecf45d0f6cbbb3` against base `7ea928f1cfb4b80b694e56feca447fb5746f3140`, werkrevisor 0.35.0.
werkrevisor: GREEN / PASS for
555435134ae767044badb74331ecf45d0f6cbbb3.The result comment of werkrevisor holds the details.
@@ -0,0 +10,4 @@47 needed one review round, 23 two, 10 three or four, 9 five or more; the mean is 2.0.The 42 PRs with more than one round got 111 line comments of `ai-reviewer`:51 on documents, 43 on code, 13 for writing rules and 4 by Codex.In 24 of the 35 such PRs with line comments, the documents and writing rules outnumber the code.Unverifiable statistic, severity unknown, counted as medium (werkrevisor)
hint; Jev: real 0.87, severity 1.1 of 3
The PR doc states "In 24 of the 35 such PRs with line comments, the documents and writing rules outnumber the code." The "35" is not defined anywhere and does not match the 42 PRs with more than one round given in the previous sentence, so a reader cannot verify the figure.
Fixed in
7b3e2a0: the Problem names the command oftools/review-roundsand that the line comments come from the reviews in Gitea.@@ -768,5 +768,5 @@def _open_hints(gitea, pr, result, bot_login):"""Findings of this review that are no line comment, e.g. refused by Gitea, can only be read in the result comment;a restart reuses the verdict without findings, so it releases them."""in_comment_only = [i for i in range(len(result.findings))if i not in result.attached and i not in result.reused and i not in result.doc_hints]Outdated rationale comment, severity unknown, counted as medium (werkrevisor)
hint; Jev: real 0.70, severity 1.7 of 3
The comment "A PR document is the record of the PR; a wrong statement there stays when nobody answers it." justifies excluding PR documents from
doc_hints. The new_on_unchanged_textpath, however, accepts PR documents and routes them intodoc_hintson unchanged text from round 2 on, so the rationale no longer matches the code.Fixed in
7b3e2a0: the comment names the rule of the second round.New commits pushed, approval review dismissed automatically according to repository settings
Codex review of
555435134ae767044badb74331ecf45d0f6cbbb3@@ -760,4 +794,4 @@result.attached.add(i)for i, v in enumerate(result.writing_violations):if (v.path, v.line) in open_writing:result.writing_reused.add(i)[medium] New paragraph violations are suppressed when their first line is unchanged (Codex review)
Paragraph-length violations are anchored to the paragraph’s first line, but this check examines only that line. Adding a sixth sentence to a previously valid five-sentence paragraph therefore marks the new violation as writing_unchanged when its first line stayed unchanged. I reproduced this: no line comment was created and the merge was scheduled despite the newly introduced violation.
Fix it, or reply why it does not hold, and resolve this comment.
Fixed in
4dc8396, the same point as the comment above: the paragraph rule covers its whole paragraph.⚠️ werkrevisor ERROR for
?, superseded by the result for7b3e2a0Codex review: critical
This change weakens the automated merge gate by allowing previously blocking hints and writing violations to stop blocking merges. Changes to CI gates require human review under your criteria.
Findings
werkrevisor/cli.py:797, as a line commentThis review only advises; it holds no merge.
Model `gpt-6-astra`, effort high, tokens in 314275 (cached 257664) / out 2481, 108.1 s, 13 commands; reviewed head `555435134ae767044badb74331ecf45d0f6cbbb3`.🟡 werkrevisor YELLOW for
7b3e2a0, superseded by the result for4dc8396🟡 werkrevisor: YELLOW / HUMAN_REVIEW
Change kind:
mixed.Reasons
5554351not carried over: 2 approved block(s) are missingFindings
Unclear scope of writing-rule skip at
README.md:142-144, severity unknown, counted as mediumhint; Jev: real 0.78, severity 1.5 of 3
The sentence "a new hint on documentation, or a violation of the writing rules, becomes a line comment only on lines that changed since the head of the last round" reads as if writing rules on any file get the unchanged-text skip. The implementation in
_on_unchanged_text(werkrevisor/cli.py) only returns True for paths matchingconfig.categories.docsorconfig.pr_doc.globs; writing-rule violations on code paths stay line comments. Consider rephrasing, e.g. "...or a writing-rule violation on documentation,..." so the scope matches the code.Then clause drops the writing rule at
docs/prs/2026-10-12-PR#91-hints-on-changed-lines.md:32-34, severity unknown, counted as medium, hint, as a line commentClassifier
5 of 6 units passed all questions (1 PR document(s)).
Usage: model `openrouter/minimax/minimax-m3`, LLM done, tokens counted before the call 24609, calls 1, tokens in 23812 / out 23999 (reasoning included in out), duration 279.4 s, cost 0.0357 USD (reported), classifier `jev-1.13.0` calls 6, tokens in 4821, cost 0.0002 USD, finding judge `jev-1.13.0` calls 2, tokens in 3360, cost 0.0001 USD, excerpt selection `openrouter/openai/gpt-6-luna` calls 1, tokens in 22787 / out 456, cost 0.0031 USD (reported), change summary `openrouter/minimax/minimax-m3` via Minimax calls 1, tokens in 4250 / out 2049, cost 0.0037 USD (reported), PR document check `jev-1.13.0` calls 6, tokens in 8703, cost 0.0004 USD, build file judge `jev-1.13.0` calls 5, tokens in 4993, cost 0.0002 USD, phases: path_judge 0.4 s, classifier 0.4 s, change_summary 37.4 s, doc_check 0.5 s, context 5.8 s, model_review 234.0 s, finding_judge 0.3 s, config: defaults + host + repository (.werkrevisor.toml at main2 answers ignored because of a confidence below [classifier] min_confidence.
7ea928f).Reviewed head `7b3e2a040879a895640280b9525aa8615b74a59c` against base `7ea928f1cfb4b80b694e56feca447fb5746f3140`, werkrevisor 0.35.0.
werkrevisor: YELLOW / HUMAN_REVIEW for
7b3e2a040879a895640280b9525aa8615b74a59c.The result comment of werkrevisor holds the details.
@@ -0,0 +29,4 @@So that a fix of one point does not start a round for the next point in text that the last round had seen.- **Given** a PR after its first review round, and a document line that the new head did not changeThen clause drops the writing rule, severity unknown, counted as medium (werkrevisor)
hint; Jev: real 0.87, severity 1.4 of 3
Scenario#91.01 has "When werkrevisor reports a GREEN hint or a writing rule on that line" but the Then says only "the hint stands only in the result comment". A writing-rule violation on unchanged text is also moved out of the line comments (added to
result.writing_unchanged, see_plan_line_commentsinwerkrevisor/cli.py), so the Then wording does not cover the second trigger. Reword to "the hint or writing rule stands only in the result comment" so both triggers match the testtest_writing_rule_on_old_text.Fixed in
4dc8396: the Then names the hint and the writing rule.Codex review of
7b3e2a040879a895640280b9525aa8615b74a59c@@ -755,12 +797,18 @@ def _plan_line_comments(config, gitea, pr, result, bot_login, redactor):if not _update_line_comment(gitea, pr, earlier, report.line_comment(f, v), redactor, updated):result.reused_not_updated.add(i)[medium] New paragraph violations can be suppressed as unchanged (Codex review)
Paragraph-length violations are anchored to the paragraph’s first line, but this check examines only that line. Adding a sixth sentence to a previously valid five-sentence paragraph leaves its opening line unchanged, so the newly introduced violation is suppressed. Reproducing this case produces no line comment and schedules the merge. Paragraph violations need to account for changes throughout the paragraph.
Fix it, or reply why it does not hold, and resolve this comment.
⚠️ werkrevisor ERROR for
?, superseded by the result for4dc8396Codex review: critical
This change weakens the automatic merge gate by allowing some unresolved hints to stop blocking merges. Changes to CI gates require human review under the stated criteria.
Findings
werkrevisor/cli.py:798, as a line commentThis review only advises; it holds no merge.
Model `gpt-6-astra`, effort high, tokens in 419590 (cached 362496) / out 2327, 92.3 s, 14 commands; reviewed head `7b3e2a040879a895640280b9525aa8615b74a59c`.🟢 werkrevisor: GREEN / PASS
Change kind:
mixed.Reasons
Findings
Inconsistent round counts at
docs/werkrevisor.werkbaum:133-134, severity unknown, counted as mediumhint; Jev: real 0.82, severity 1.4 of 3
The PR document states "47 needed one review round, 23 two, 10 three or four, 9 five or more"
totaling 89 PRs, and its Solution section says the werkbaum has the counts in
#model.rounds.check.The werkbaum instead summarizes as "47 one round, 19 three or more" and omits the two-round group
entirely. The numbers reconcile (47 + 23 + 19 = 89, since "19 three or more" = 10 + 9),
but a reader scanning the entry would see 47 + 19 = 66 PRs and 23 unaccounted for.
Either include the 23 two-round PRs in the entry, or state the multi-round group explicitly
(e.g. "47 one round, 42 with two or more of which 19 with three or more").
Classifier
6 of 6 units passed all questions (1 PR document(s)).
Usage: model `openrouter/minimax/minimax-m3`, LLM done, tokens counted before the call 29358, calls 1, tokens in 28393 / out 42497 (reasoning included in out), duration 492.8 s, cost 0.0595 USD (reported), classifier `jev-1.13.0` calls 6, tokens in 4871, cost 0.0002 USD, finding judge `jev-1.13.0` calls 1, tokens in 2005, cost 0.0001 USD, excerpt selection `openrouter/openai/gpt-6-luna` calls 1, tokens in 23105 / out 354, cost 0.0031 USD (reported), change summary `openrouter/minimax/minimax-m3` via Minimax calls 1, tokens in 4821 / out 1723, cost 0.0035 USD (reported), PR document check `jev-1.13.0` calls 6, tokens in 10487, cost 0.0004 USD, build file judge `jev-1.13.0` calls 6, tokens in 5988, cost 0.0003 USD, phases: path_judge 0.5 s, classifier 0.5 s, change_summary 19.5 s, doc_check 0.5 s, context 6.1 s, model_review 465.0 s, finding_judge 0.3 s, config: defaults + host + repository (.werkrevisor.toml at main2 answers ignored because of a confidence below [classifier] min_confidence.
7ea928f).Reviewed head `4dc839613dca080adcb8619fbd20041c2a231906` against base `7ea928f1cfb4b80b694e56feca447fb5746f3140`, werkrevisor 0.35.0.
werkrevisor: GREEN / PASS for
4dc839613dca080adcb8619fbd20041c2a231906.The result comment of werkrevisor holds the details.