104 lines
4.3 KiB
Markdown
104 lines
4.3 KiB
Markdown
# PR#66: Hints on Documentation Stay in the Result Comment
|
|
|
|
> **WARNING:** This document describes only the change applied in this PR.
|
|
> It may already be outdated once the next PR is merged.
|
|
> Historic PR documentation is not maintained along with new PRs; treat it as a snapshot.
|
|
|
|
## Related Links
|
|
|
|
- [PR#65](https://git.javagil.de/mi/werkrevisor/pulls/65): a dry run writes the model prompt, used for the measurement below.
|
|
|
|
## The Problem
|
|
|
|
Many PRs need several review rounds, each for new small hints, e.g. werkprobe PR#12 with 5 rounds.
|
|
Each hint is a line comment, and on a GREEN PR every open line comment holds the merge.
|
|
So the coding agent answers or fixes each hint, and each fix starts a new round with new hints.
|
|
|
|
A frontier model, `anthropic/claude-opus-5.5`, rated 42 answered hints of werkrevisor PRs #32 and #48 to #64.
|
|
It saw the prompt from which the economy model wrote each hint, rebuilt with `tools/dry-run --prompt-out`
|
|
from a throwaway PR per reviewed head in `mi/werkrevisor-bench`.
|
|
It rated 14 hints as useful and 28 as noise.
|
|
|
|
| Hints (the third row is part of the first) | useful | noise |
|
|
|---|---|---|
|
|
| on a documentation file | 6 | 19 |
|
|
| on code | 8 | 9 |
|
|
| of these: PR document vs change summary (`[doc_check]`) | 1 | 4 |
|
|
|
|
The review round, Jev's values for "real" and the severity of the model did not separate useful hints from noise.
|
|
The three `[doc_check]` hints below 0.7 (0.62, 0.65, 0.69) were noise; the useful one had 0.77.
|
|
|
|
## Non-Goals
|
|
|
|
- No change for hints on code, for YELLOW, ORANGE and RED findings, and for the writing rules.
|
|
- No Jev question that rates whether a hint is worth a round: 42 ratings with a confidence of mostly 0.5 to 0.6
|
|
are too few for it.
|
|
|
|
## The Scenarios
|
|
|
|
### Feature: a hint on documentation does not cost a review round
|
|
|
|
#### Scenario#66.01: a hint on a documentation file is no line comment
|
|
|
|
So that a small remark on wording does not hold the merge.
|
|
|
|
- **Given** a GREEN review with automerge
|
|
- **And** its only finding is a hint on a file of the category `docs`
|
|
- **When** werkrevisor publishes the review
|
|
- **Then** the hint stands in the result comment with its full text
|
|
- **And** it is no line comment
|
|
- **And** the merge is scheduled
|
|
|
|
##### Verified by
|
|
|
|
- [test_doc_hints.DocHintsTest.test_a_hint_on_documentation_is_no_line_comment](../../tests/test_doc_hints.py)
|
|
|
|
#### Scenario#66.02: a hint on code stays a line comment
|
|
|
|
So that the hints that most often name a real problem still get an answer.
|
|
|
|
- **Given** a GREEN review with automerge
|
|
- **And** its only finding is a hint on a file outside the category `docs`
|
|
- **When** werkrevisor publishes the review
|
|
- **Then** the hint is a line comment that holds the merge
|
|
|
|
##### Verified by
|
|
|
|
- [test_doc_hints.DocHintsTest.test_a_hint_on_code_stays_a_line_comment](../../tests/test_doc_hints.py)
|
|
|
|
#### Scenario#66.03: the old behaviour stays selectable
|
|
|
|
So that a repository that wants every hint answered can keep it.
|
|
|
|
- **Given** `[review] doc_hint_line_comments = true`
|
|
- **And** a GREEN review whose only finding is a hint on a file of the category `docs`
|
|
- **When** werkrevisor publishes the review
|
|
- **Then** the hint is a line comment that holds the merge
|
|
|
|
##### Verified by
|
|
|
|
- [test_doc_hints.DocHintsTest.test_doc_hint_line_comments_restores_the_line_comment](../../tests/test_doc_hints.py)
|
|
|
|
#### Scenario#66.04: a weak contradiction with the change summary is no hint
|
|
|
|
So that the PR document check reports only contradictions from 0.7 on.
|
|
|
|
- **Given** the default config
|
|
- **When** Jev rates a section of the PR document at 0.65 as a contradiction of the change summary
|
|
- **Then** the result has no hint for that section
|
|
|
|
##### Verified by
|
|
|
|
- [test_doccheck.DocCheckTest.test_a_contradiction_below_hint_min_is_no_hint](../../tests/test_doccheck.py)
|
|
|
|
## The Solution
|
|
|
|
- `[review] doc_hint_line_comments`, default `false`: a GREEN finding on a file of the category `docs`
|
|
stays in the result comment and does not count as an open hint.
|
|
- `[doc_check] hint_min` from 0.6 to 0.7.
|
|
- A hint of the PR document check is a hint on documentation too, so it stays in the result comment.
|
|
It comes back in each round while Jev rates its section as before;
|
|
the resolved fingerprints of PR#50 apply only with `doc_hint_line_comments = true`.
|
|
- The README describes both; the Werkbaum records the measurement and the next ideas under `#model.rounds`.
|
|
- Version 0.31.0.
|