68 lines
3.0 KiB
Markdown
68 lines
3.0 KiB
Markdown
# PR#159: A Misreported Build Status Waits for the Human
|
|
|
|
> **WARNING:** This document describes only the change applied in this PR.
|
|
> It may already be outdated once the next PR is merged.
|
|
> Historic PR documentation is not maintained along with new PRs; treat it as a snapshot.
|
|
|
|
## Related Links
|
|
|
|
- [werkrevisor PR#106](https://git.javagil.de/mi/werkrevisor/pulls/106): werkrevisor did not see a weakened gate, Codex did.
|
|
- [werkrevisor PR#107](https://git.javagil.de/mi/werkrevisor/pulls/107): the category `gate_weakened` of werkrevisor.
|
|
|
|
## Problem
|
|
|
|
The commit statuses of Werkator are required checks of all werk repositories.
|
|
A change that reports a failed or unfinished build as a success lets PRs merge without their checks.
|
|
A change that lets a pull-request target build take its settings from the PR lets the PR choose its own checks.
|
|
werkrevisor rated neither as risky in a measurement:
|
|
`weakens_checks` and `security` stayed at 0.07 or less for a timed-out build reported as `success`.
|
|
|
|
## Non-Goals
|
|
|
|
- No change of the code of Werkator.
|
|
- No ORANGE: two misreported statuses are too few to calibrate `orange_min`.
|
|
|
|
## Solution
|
|
|
|
- `.werkrevisor.toml`: a risk category `status_misreported` of its own, for main code,
|
|
`action = "yellow"`, `yellow_min = 0.6`.
|
|
- `.werkrevisor.toml`: `**/BuildSettingsSource*` is a hard path:
|
|
it decides whether a pull-request target build takes its settings from the primary checkout or from the PR;
|
|
like the other hard paths, it also matches the tests of the class.
|
|
- Werkbaum: the node of the Codex review below `#process` gets the two rules.
|
|
- Version v1.42.3, internal.
|
|
|
|
## Measurement
|
|
|
|
On 2026-10-14 Jev judged each changed main-code file of two kinds of changes:
|
|
the 22 merged PRs #136 to #157, which need no human,
|
|
and three weakenings of the gate that were made for the measurement only.
|
|
Each value is the probability that Jev gave to one question.
|
|
|
|
The questions:
|
|
|
|
- `weakens_checks` and `security`: the questions that werkrevisor asks today.
|
|
- `gate_weakened`: the category of werkrevisor PR#107.
|
|
- `status_misreported`: the new category of this PR.
|
|
- `pr_decides_checks`: a candidate for the trust boundary, not used.
|
|
|
|
| Change | `weakens_checks` | `security` | `gate_weakened` | `status_misreported` | `pr_decides_checks` |
|
|
|---|---|---|---|---|---|
|
|
| an interrupted build reports `success` | 0.05 | 0.02 | 0.35 | 0.85 | 0.04 |
|
|
| a timed-out build reports `success` | 0.07 | 0.02 | 0.46 | 0.75 | 0.04 |
|
|
| a pull-request target build takes the settings of the PR | 0.50 | 0.04 | 0.15 | 0.14 | 0.39 |
|
|
| the highest value of the 22 merged PRs | 0.50 | 0.52 | 0.17 | 0.25 | 0.25 |
|
|
|
|
The results:
|
|
|
|
- Only `status_misreported` finds the two misreported statuses: 0.75 and 0.85, against at most 0.25 for the merged PRs.
|
|
The threshold 0.6 lies between 0.25 and 0.75.
|
|
- No question finds the settings taken from the PR.
|
|
The diff is one line, and Jev does not see what `target` means there.
|
|
The file is small and is the trust boundary of pull-request target builds, so it becomes a hard path.
|
|
- The measurement cost less than 0.01 USD.
|
|
|
|
## Open Questions
|
|
|
|
None.
|