88 lines
3.4 KiB
Markdown
88 lines
3.4 KiB
Markdown
# PR#107: A Weakened Merge Gate Waits for the Human
|
|
|
|
> **WARNING:** This document describes only the change applied in this PR.
|
|
> It may already be outdated once the next PR is merged.
|
|
> Historic PR documentation is not maintained along with new PRs; treat it as a snapshot.
|
|
|
|
## Related Links
|
|
|
|
- [PR#106](2026-10-14-PR#106-codex-gate-severe-test.md): the severe gate test that werkrevisor did not see.
|
|
- [PR#46](2026-10-08-PR#46-risk-categories.md): the risk categories.
|
|
|
|
## Problem
|
|
|
|
In PR#106 a defect let the approval of the bot count as a human approval for the Codex gate.
|
|
werkrevisor saw the change, but rated it GREEN-worthy:
|
|
Jev rated `weakens_checks` 0.18 and `security` 0.66 for `werkrevisor/codexrunner.py`.
|
|
werkrevisor is the merge gate of all werk repositories, so a change that weakens it must wait for the human.
|
|
|
|
## Non-Goals
|
|
|
|
- No change of the code; the category is configuration of this repository.
|
|
- No ORANGE: four weakened gates are too few to calibrate `orange_min`.
|
|
- Werkator gets its own category in its repository; see Follow-up PRs.
|
|
|
|
## Specification
|
|
|
|
### Feature: a weakened merge gate waits for the human
|
|
|
|
#### Scenario#107.01: Jev rates a weakened gate as likely
|
|
|
|
So that a change that lets PRs merge with less review never merges without the human.
|
|
|
|
- **Given** the configuration of werkrevisor
|
|
- **When** Jev rates a main-code file at 0.75 or more for `gate_weakened`
|
|
- **Then** the review is YELLOW with the reason `risk_gate_weakened`
|
|
|
|
##### Verified by
|
|
|
|
- [test_risk: test_werkrevisor_holds_a_weakened_gate_for_the_human](../../tests/test_risk.py)
|
|
|
|
#### Scenario#107.02: a gate change below the threshold passes
|
|
|
|
So that the usual changes of the review, which all touch the gate, need no human.
|
|
|
|
- **Given** the configuration of werkrevisor
|
|
- **When** Jev rates a main-code file below 0.75 for `gate_weakened`
|
|
- **Then** the category gives no YELLOW
|
|
|
|
##### Verified by
|
|
|
|
- [test_risk: test_werkrevisor_holds_a_weakened_gate_for_the_human](../../tests/test_risk.py)
|
|
|
|
## Solution
|
|
|
|
- `.werkrevisor.toml`: a category `gate_weakened` of its own, for main code, `action = "yellow"`, `yellow_min = 0.75`.
|
|
- Version 0.42.0.
|
|
|
|
## Measurement
|
|
|
|
On 2026-10-14 Jev judged each changed main-code file of the 22 merged PRs #85 to #106,
|
|
the defects of PR#105 and PR#106, and three weakenings made for the measurement.
|
|
Two questions were compared:
|
|
`merge_gate` asked whether the change alters who or what may approve, hold or release a merge;
|
|
`gate_weakened` asked whether it makes it easier to merge without the required review.
|
|
|
|
| Case | `merge_gate` | `gate_weakened` |
|
|
|---|---|---|
|
|
| PR#106 defect: the bot approval counts | 0.91 | 0.84 |
|
|
| `hints_hold_merge` defaults to false | 0.95 | 0.82 |
|
|
| an approval before the result counts | 0.90 | 0.88 |
|
|
| a failed Codex review gives `success` | 0.91 | 0.89 |
|
|
| highest of the 22 merged PRs | 0.91 (PR#94) | 0.68 (PR#98, a diagram generator) |
|
|
| PR#105 defect: one review more per run | 0.87 | 0.56 |
|
|
|
|
`merge_gate` rates most feature PRs of werkrevisor high, since werkrevisor is the gate.
|
|
`gate_weakened` separates: the four weakenings from 0.82 on, the merged PRs at most 0.68.
|
|
0.75 lies between both; two runs differed by about 0.02.
|
|
The measurement cost less than 0.01 USD.
|
|
|
|
## Open Questions
|
|
|
|
None.
|
|
|
|
## Follow-up PRs
|
|
|
|
- [werkator PR#159](https://git.javagil.de/mi/werkator/pulls/159): Werkator gets the category `status_misreported`
|
|
and a hard path, since `gate_weakened` rated its weakened statuses at most 0.46.
|