76 lines
3.2 KiB
Markdown
76 lines
3.2 KiB
Markdown
# PR#90: tools/review-report Drafts a Review by werkrevisor and Codex for a Foreign Repository
|
|
|
|
> **WARNING:** This document describes only the change applied in this PR.
|
|
> It may already be outdated once the next PR is merged.
|
|
> Historic PR documentation is not maintained along with new PRs; treat it as a snapshot.
|
|
|
|
## Problem
|
|
|
|
hs.hsadmin.ng has no account `ai-reviewer` yet, and werkrevisor does not run there.
|
|
The user wants the full review anyway, with the rules, Jev, the model review and Codex,
|
|
started per PR from the command line, and posts the result under the own name after a check.
|
|
For hs.hsadmin.ng #322 this took two commands and a report written by hand.
|
|
|
|
## Non-Goals
|
|
|
|
- No write to Gitea: the user posts the report.
|
|
- No run without a human check of the findings.
|
|
|
|
## Specification
|
|
|
|
### Feature: a draft report of werkrevisor and Codex for one PR
|
|
|
|
#### Scenario#90.01: the report holds both reviews and asks for a check
|
|
|
|
So that the user posts only findings that a human checked.
|
|
|
|
- **Given** a PR of a Gitea without werkrevisor, e.g. hs.hsadmin.ng #322
|
|
- **When** the user runs `tools/review-report` with its URL
|
|
- **Then** `report.md` holds the result of werkrevisor, the criticality and findings of Codex and the usage of both,
|
|
under a note to check each finding before posting
|
|
|
|
##### Verified by
|
|
|
|
- [test_reviewreport.ReviewReportTest.test_the_report_holds_both_reviews_and_asks_for_a_check](../../tests/test_reviewreport.py)
|
|
- [test_reviewreport.ReviewReportTest.test_the_werkrevisor_part_has_no_markers](../../tests/test_reviewreport.py)
|
|
|
|
#### Scenario#90.02: a failed review is named in the report
|
|
|
|
So that a missing part never looks like a review without findings.
|
|
|
|
- **Given** a werkrevisor run or a Codex run that does not complete
|
|
- **When** the user runs `tools/review-report`
|
|
- **Then** the report names the part as not completed, with its cause
|
|
|
|
##### Verified by
|
|
|
|
- [test_reviewreport.ReviewReportTest.test_a_failed_review_is_named_not_left_out](../../tests/test_reviewreport.py)
|
|
|
|
## Solution
|
|
|
|
- `werkrevisor/reviewreport.py` and `tools/review-report` run `tools/dry-run --llm` and `codex.run` in parallel,
|
|
take the result comment of werkrevisor without its markers,
|
|
and write `build/review-report/<server>/<owner>/<repo>-<n>/report.md`.
|
|
- Tokens and the values of the Codex login are removed from the report.
|
|
- The report warns when werkrevisor and Codex reviewed different commits, e.g. after a push between the runs.
|
|
- README: a paragraph below "Codex runner".
|
|
- Werkbaum: `#model.codex.report`; `#model.codex.on` and its nodes werkrevisor and werkator are deployed.
|
|
- Version 0.35.0.
|
|
|
|
## Open Questions
|
|
|
|
None.
|
|
|
|
## Additional Changes
|
|
|
|
- Found by the first Codex review of the runner on this PR: `codex.checkout` kept one mirror per owner and repository
|
|
and fetched from `origin`, so a token of one Gitea could reach another Gitea with a repository of the same name.
|
|
Each server now has its own mirrors and worktrees, and every fetch names the URL of its server;
|
|
a test fails without the fix.
|
|
Mirrors from before are not used any more and can be deleted.
|
|
|
|
## Attachments
|
|
|
|
A trial on hs.hsadmin.ng #322 took 67 s for both reviews in parallel.
|
|
werkrevisor gave ORANGE there, YELLOW in a run before, since the model review differs from run to run.
|