85 lines
3.6 KiB
Markdown
85 lines
3.6 KiB
Markdown
# PR#125: Diagrams for werkrevisor, Drawn by Sonnet for PRs That a Human Reviews
|
||
|
||
Version: minor
|
||
|
||
> **WARNING:** This document describes only the change applied in this PR.
|
||
> It may already be outdated once the next PR is merged.
|
||
> Historic PR documentation is not maintained along with new PRs; treat it as a snapshot.
|
||
|
||
## Related Links
|
||
|
||
- [PR#121](2026-10-18-PR#121-pr-lens-diagrams.md): diagrams of PR Lens in the result comment, default off.
|
||
|
||
## Problem
|
||
|
||
Since PR#121 werkrevisor can attach diagrams of PR Lens, but no repository uses them yet.
|
||
Without a `model`, they take the review model; for werkrevisor that is MiniMax M3,
|
||
whose graph documents do not validate, not even after the repair round of PR Lens.
|
||
|
||
A GREEN PR merges without a human, so its diagrams cost money that nobody looks at.
|
||
|
||
## Non-Goals
|
||
|
||
- No diagrams for `hs-review`.
|
||
- No diagram cost in the usage record yet.
|
||
|
||
## Specification
|
||
|
||
### Feature: only a PR that a human reviews gets diagrams
|
||
|
||
#### Scenario#125.01: a GREEN PR gets no diagrams
|
||
|
||
… so that no money goes into diagrams that nobody looks at.
|
||
|
||
- **Given** a host config with `[diagrams] enabled = true` and the default `skip_green = true`
|
||
- **When** werkrevisor publishes a GREEN result
|
||
- **Then** it calls no PR Lens and attaches nothing
|
||
|
||
##### Verified by
|
||
|
||
- [test_prlens.DiagramsInTheResultCommentTest.test_a_green_pr_gets_no_diagrams](../../tests/test_prlens.py)
|
||
|
||
#### Scenario#125.02: a PR that a human reviews gets diagrams
|
||
|
||
… so that the human reviewer sees what the PR touches.
|
||
|
||
- **Given** a host config with `[diagrams] enabled = true` and `skip_green = true`
|
||
- **When** werkrevisor publishes a YELLOW, ORANGE or RED result
|
||
- **Then** the result comment links the diagrams, unless the diff holds a possible secret (Scenario#121.07)
|
||
|
||
##### Verified by
|
||
|
||
- [test_prlens.DiagramsInTheResultCommentTest.test_a_pr_that_a_human_reviews_gets_diagrams](../../tests/test_prlens.py)
|
||
|
||
## Solution
|
||
|
||
- `[diagrams] skip_green`, default `true`: a GREEN result draws nothing.
|
||
- `config/werkrevisor.toml` switches the diagrams on, with `model = "anthropic/claude-sonnet-5.5"`
|
||
and `timeout_seconds = 120`: with 2 review calls of up to 300 s, the default of 300 s could exceed the 15 minutes
|
||
of the Werkator build.
|
||
|
||
The choice of the model, measured on 2026-10-20 on the merged werkrevisor PR#117, PR#118, PR#122 and PR#123:
|
||
|
||
| Model | Valid | Time | Cost per head | Content |
|
||
|---|---|---|---|---|
|
||
| MiniMax M3, the review model | 0 of 1 | – | – | the graph document misses title, lanes and nodes, also after the repair round |
|
||
| GPT-6 Luna | 4 of 4 | 42–75 s | about 0.01–0.02 USD | system context and rough boxes, e.g. "Codex Runner → bwrap sandbox → Codex CLI" |
|
||
| Sonnet 5.5 | 4 of 4 | 18–43 s | about 0.10–0.20 USD | changed functions with their files, NEW and REMOVED nodes, e.g. the dropped economy-model summary of PR#117 |
|
||
|
||
werkrevisor PRs often have 5 to 8 heads; with `skip_green`, only the heads that a human reviews cost.
|
||
|
||
## Additional Changes
|
||
|
||
- `werkrevisor/pr_lens.mjs` ends a failure with one line: the code and message of PR Lens and its first details.
|
||
Before, the log of werkrevisor showed only the last line of the stack trace, `Node.js v20.19.2`.
|
||
- `docs/werkrevisor.werkbaum`: `#diagrams.on` is `[^]`.
|
||
|
||
Deployment: `config/werkrevisor.toml` reaches mih09 with the release, if PR#124 is merged by then;
|
||
otherwise `tools/sync-host-config --apply mih09-werkator@mih09.hostsharing.net werkrevisor` after the release,
|
||
since 0.52.0 does not know `skip_green`.
|
||
|
||
## Follow-up PRs
|
||
|
||
- `hs-review` puts the SVGs next to its draft for dev.hostsharing.net.
|
||
- The cost of the diagrams in the usage record and in the budget of a review.
|