71 lines
3.8 KiB
Markdown
71 lines
3.8 KiB
Markdown
# PR#87: Measure a Review by the Codex CLI on 10 hs.hsadmin.ng PRs
|
|
|
|
> **WARNING:** This document describes only the change applied in this PR.
|
|
> It may already be outdated once the next PR is merged.
|
|
> Historic PR documentation is not maintained along with new PRs; treat it as a snapshot.
|
|
|
|
## Related Links
|
|
|
|
- Werkbaum: `#model.codex` below `#model.external`.
|
|
- [docs/anleitung-dev.hostsharing.net.md](../anleitung-dev.hostsharing.net.md), section 6: the dry runs of 2026-10-06.
|
|
|
|
## Problem
|
|
|
|
The model review by PR-Agent sees the diff and excerpts only, and the economy model finds few concrete issues.
|
|
An agent that reads the whole repository may find more, and the Codex CLI can run with a frontier model
|
|
in the subscription of the user, without cost per token.
|
|
Before werkrevisor uses it, a measurement must show what it finds, how it rates the criticality of a PR,
|
|
and how much of the subscription quota a review uses.
|
|
|
|
## Non-Goals
|
|
|
|
- No use of the Codex CLI in the review gate; this PR measures only.
|
|
- No run on mih09 and no write to Gitea.
|
|
|
|
## Solution
|
|
|
|
- `tools/codex-eval` clones the repository of a PR, checks the PR head out as a worktree
|
|
and runs `codex exec --sandbox read-only` in it, with the PR description and two questions:
|
|
is the PR critical in the sense of the triage goal, and which concrete issues does it have.
|
|
The answer follows a JSON schema; the tool records it with the token use, the duration and the commands Codex ran.
|
|
- Codex gets neither a token nor network access; the Gitea token reaches git only through its environment.
|
|
- The Codex CLI runs on the workstation only, with the subscription login of the user; werkrevisor does not depend on it.
|
|
- Werkbaum: `#model.codex.eval`, and `#model.codex.gate` for the use in the review.
|
|
- Version 0.33.0.
|
|
|
|
## Measurement
|
|
|
|
`gpt-6-astra` with `model_reasoning_effort = "high"`, Codex CLI 0.162.1, on 2026-10-11,
|
|
on the last head of 10 merged hs.hsadmin.ng PRs.
|
|
"Critical" in the table is the classification of 2026-10-06 by the user.
|
|
|
|
| PR | Critical | Codex: critical | Findings | Input tokens (cached) | Output tokens | Seconds |
|
|
|---|---|---|---|---|---|---|
|
|
| #283 | yes | yes | 2 medium | 402494 (313984) | 3194 | 116 |
|
|
| #288 | yes | yes | 1 high | 421270 (350848) | 2940 | 104 |
|
|
| #297 | yes | yes | 1 low | 191225 (149632) | 1123 | 53 |
|
|
| #306 | yes | yes | none | 592413 (510208) | 2624 | 104 |
|
|
| #310 | yes | yes | 1 high | 413331 (340096) | 2375 | 92 |
|
|
| #313 | maybe | yes | none | 575334 (493184) | 2749 | 109 |
|
|
| #304 | no | yes | 1 medium | 369544 (299776) | 3188 | 117 |
|
|
| #307 | no | yes | none | 385787 (317824) | 2619 | 101 |
|
|
| #290 | no | no | none | 112201 (75648) | 535 | 28 |
|
|
| #299 | no | no | 1 low | 160544 (111488) | 1257 | 47 |
|
|
|
|
- Codex found all 5 critical PRs and rated the borderline #313 as critical: an endpoint without authentication.
|
|
- Of the 4 uncritical PRs it rated 2 as critical, with reasons from the definition:
|
|
#304 raises the budgets of the PR-document check, a weakened CI gate;
|
|
#307 overwrites the bank directory, an irreversible change of existing data.
|
|
- The two high findings name concrete RBAC defects:
|
|
#288 loses permission grants to imported roles in a single case, and on #310
|
|
a changed relation of one debitor revokes the access of another debitor with the same relation.
|
|
- Together 3.6 million input tokens, 82 % of them from the cache, 23000 output tokens and 15 minutes.
|
|
- On #306, `medium` used 355133 input tokens (286464 cached) and 1348 output tokens in 60 s,
|
|
with the same classification and no finding either.
|
|
- A first run with a question about whether the PR "causes" a security problem rated #306 as not critical;
|
|
the question now asks what the PR changes, not whether a defect was found.
|
|
|
|
## Open Questions
|
|
|
|
- [ ] TODO.verify: Check the high findings on #288 and #310 with the hs.hsadmin.ng team.
|