claude-code
6856d04398
werkator build succeeded after 00:05
werkrevisor reviews its own pull requests: PR documents under `docs/prs/` with a fixed workflow, the operator config `config/werkrevisor.toml`, and a CI build for the unit tests. Details: the PR document under `docs/prs/`. 🤖 Generated with [Claude Code](https://claude.com/claude-code) Reviewed-on: #3 Co-authored-by: claude-code <claude-code@javagil.de>
71 lines
2.9 KiB
Markdown
71 lines
2.9 KiB
Markdown
# PR#3: werkrevisor reviews its own pull requests
|
|
|
|
> **WARNING:** This document describes only the change applied in this PR.
|
|
> It may already be outdated once the next PR is merged.
|
|
> Historic PR documentation is not maintained along with new PRs; treat it as a snapshot.
|
|
|
|
## The Problem
|
|
|
|
werkrevisor was tried on the test repository `mi/werkprobe` only, with a handful of constructed PRs.
|
|
Its own development produces many real PRs, and each of them is a chance to learn how good the verdicts are:
|
|
how often a human approval follows an ORANGE, how often a finding was wrong, and what a review costs.
|
|
For that, werkrevisor has to review its own repository, and its PRs need a fixed workflow and a PR document each.
|
|
|
|
## Non-Goals
|
|
|
|
- No change to the review logic or to the installed werkrevisor version (0.6.0 on mih09).
|
|
- No branch protection in this PR; it is set in Gitea, not in the repository.
|
|
- No automatic evaluation of the usage records yet.
|
|
|
|
## The Scenarios
|
|
|
|
### Feature: werkrevisor reviews the pull requests of its own repository
|
|
|
|
#### Background
|
|
|
|
- Werkator on mih09 serves `mi/werkrevisor` and runs the host-only build `ai-review` for every ready PR head.
|
|
- PRs are opened by the account `claude-code`; `mi` reviews and approves, `ai-reviewer` is the bot.
|
|
|
|
#### Scenario#3.01: a ready PR gets a review by the installed werkrevisor
|
|
|
|
So that the review cannot be changed by the PR it judges.
|
|
|
|
- **Given** a PR without the prefix `WIP: `
|
|
- **When** Werkator sees its head commit
|
|
- **Then** the installed werkrevisor reviews it with the operator config from the machine config
|
|
- **and** the PR's own `config/werkrevisor.toml` has no effect on that review
|
|
|
|
##### Verified by
|
|
|
|
- manual check on this PR: the review comment names the config source `env`
|
|
|
|
#### Scenario#3.02: an open question in the PR document blocks the review
|
|
|
|
So that no PR is approved while a question is still open.
|
|
|
|
- **Given** a PR document under `docs/prs/` with an item `- [ ]` under "Open Questions"
|
|
- **When** werkrevisor reviews the PR
|
|
- **Then** the verdict is RED, before any model call
|
|
|
|
##### Verified by
|
|
|
|
- [test_open_questions](../../tests/test_open_questions.py)
|
|
|
|
## The Solution
|
|
|
|
- `docs/prs/README.md` and `docs/prs/TEMPLATE.md` describe the workflow and the structure, after the model of Werkator.
|
|
- `config/werkrevisor.toml` is the operator config for this repository:
|
|
- main code is `werkrevisor/**`, tests are `tests/**`;
|
|
- the operator config, the image, the CI and the modules at the trust boundary are risky paths;
|
|
- PR documents are `docs/prs/*.md`.
|
|
On mih09, the same content is set as `WERKREVISOR_CONFIG_TOML` in the machine config; the file documents it.
|
|
- `.werkator.yml` runs the unit tests on every push, in the werkrevisor rootfs.
|
|
|
|
## Open Questions
|
|
|
|
- [x] Should the review build come from the repository? No: it holds secrets and the gate, so it stays host-only.
|
|
|
|
## Follow-up PRs
|
|
|
|
- A tool that compares verdicts with the later human decisions, from `usage.jsonl` and the Gitea reviews.
|