werkrevisor/docs/prs/2026-10-09-PR#66-doc-hints-in-result-comment.md

PR#66: Hints on Documentation Stay in the Result Comment

WARNING: This document describes only the change applied in this PR. It may already be outdated once the next PR is merged. Historic PR documentation is not maintained along with new PRs; treat it as a snapshot.

Related Links

• PR#65: a dry run writes the model prompt, used for the measurement below.

The Problem

Many PRs need several review rounds, each for new small hints, e.g. werkprobe PR#12 with 5 rounds. Each hint is a line comment, and on a GREEN PR every open line comment holds the merge. So the coding agent answers or fixes each hint, and each fix starts a new round with new hints.

A frontier model, anthropic/claude-opus-5.5, rated 42 answered hints of werkrevisor PRs #32 and #48 to #64. It saw the prompt from which the economy model wrote each hint, rebuilt with tools/dry-run --prompt-out from a throwaway PR per reviewed head in mi/werkrevisor-bench. It rated 14 hints as useful and 28 as noise.

Hints (the third row is part of the first)	useful	noise
on a documentation file	6	19
on code	8	9
of these: PR document vs change summary ([doc_check])	1	4

The review round, Jev's values for "real" and the severity of the model did not separate useful hints from noise. The three [doc_check] hints below 0.7 (0.62, 0.65, 0.69) were noise; the useful one had 0.77.

Non-Goals

• No change for hints on code, for YELLOW, ORANGE and RED findings, and for the writing rules.
• No Jev question that rates whether a hint is worth a round: 42 ratings with a confidence of mostly 0.5 to 0.6 are too few for it.

The Scenarios

Feature: a hint on documentation does not cost a review round

Scenario#66.01: a hint on a documentation file is no line comment

So that a small remark on wording does not hold the merge.

• Given a GREEN review with automerge
• And its only finding is a hint on a file of the category docs
• When werkrevisor publishes the review
• Then the hint stands in the result comment with its full text
• And it is no line comment
• And the merge is scheduled

Verified by

• test_doc_hints.DocHintsTest.test_a_hint_on_documentation_is_no_line_comment

Scenario#66.02: a hint on code stays a line comment

So that the hints that most often name a real problem still get an answer.

• Given a GREEN review with automerge
• And its only finding is a hint on a file outside the category docs
• When werkrevisor publishes the review
• Then the hint is a line comment that holds the merge

Verified by

• test_doc_hints.DocHintsTest.test_a_hint_on_code_stays_a_line_comment

Scenario#66.03: the old behaviour stays selectable

So that a repository that wants every hint answered can keep it.

• Given [review] doc_hint_line_comments = true
• And a GREEN review whose only finding is a hint on a file of the category docs
• When werkrevisor publishes the review
• Then the hint is a line comment that holds the merge

Verified by

• test_doc_hints.DocHintsTest.test_doc_hint_line_comments_restores_the_line_comment

Scenario#66.04: a weak contradiction with the change summary is no hint

So that the PR document check reports only contradictions from 0.7 on.

• Given the default config
• When Jev rates a section of the PR document at 0.65 as a contradiction of the change summary
• Then the result has no hint for that section

Verified by

• test_doccheck.DocCheckTest.test_a_contradiction_below_hint_min_is_no_hint

The Solution

• [review] doc_hint_line_comments, default false: a GREEN finding on a file of the category docs stays in the result comment and does not count as an open hint.
• [doc_check] hint_min from 0.6 to 0.7.
• A hint of the PR document check is a hint on documentation too, so it stays in the result comment. It comes back in each round while Jev rates its section as before; the resolved fingerprints of PR#50 apply only with doc_hint_line_comments = true.
• The README describes both; the Werkbaum records the measurement and the next ideas under #model.rounds.
• Version 0.31.0.