werkrevisor/docs/prs/2026-10-11-PR#87-codex-eval.md

PR#87: Measure a Review by the Codex CLI on 10 hs.hsadmin.ng PRs

WARNING: This document describes only the change applied in this PR. It may already be outdated once the next PR is merged. Historic PR documentation is not maintained along with new PRs; treat it as a snapshot.

Related Links

• Werkbaum: #model.codex below #model.external.
• docs/anleitung-dev.hostsharing.net.md, section 6: the dry runs of 2026-10-06.

Problem

The model review by PR-Agent sees the diff and excerpts only, and the economy model finds few concrete issues. An agent that reads the whole repository may find more, and the Codex CLI can run with a frontier model in the subscription of the user, without cost per token. Before werkrevisor uses it, a measurement must show what it finds, how it rates the criticality of a PR, and how much of the subscription quota a review uses.

Non-Goals

• No use of the Codex CLI in the review gate; this PR measures only.
• No run on mih09 and no write to Gitea.

Solution

• tools/codex-eval clones the repository of a PR, checks the PR head out as a worktree and runs codex exec --sandbox read-only in it, with the PR description and two questions: is the PR critical in the sense of the triage goal, and which concrete issues does it have. The answer follows a JSON schema; the tool records it with the token use, the duration and the commands Codex ran.
• Codex gets neither a token nor network access; the Gitea token reaches git only through its environment.
• The Codex CLI runs on the workstation only, with the subscription login of the user; werkrevisor does not depend on it.
• Werkbaum: #model.codex.eval, and #model.codex.gate for the use in the review.
• Version 0.33.0.

Measurement

gpt-6-astra with model_reasoning_effort = "high", Codex CLI 0.162.1, on 2026-10-11, on the last head of 10 merged hs.hsadmin.ng PRs. "Critical" in the table is the classification of 2026-10-06 by the user.

PR	Critical	Codex: critical	Findings	Input tokens (cached)	Output tokens	Seconds
#283	yes	yes	2 medium	402494 (313984)	3194	116
#288	yes	yes	1 high	421270 (350848)	2940	104
#297	yes	yes	1 low	191225 (149632)	1123	53
#306	yes	yes	none	592413 (510208)	2624	104
#310	yes	yes	1 high	413331 (340096)	2375	92
#313	maybe	yes	none	575334 (493184)	2749	109
#304	no	yes	1 medium	369544 (299776)	3188	117
#307	no	yes	none	385787 (317824)	2619	101
#290	no	no	none	112201 (75648)	535	28
#299	no	no	1 low	160544 (111488)	1257	47

• Codex found all 5 critical PRs and rated the borderline #313 as critical: an endpoint without authentication.
• Of the 4 uncritical PRs it rated 2 as critical, with reasons from the definition: #304 raises the budgets of the PR-document check, a weakened CI gate; #307 overwrites the bank directory, an irreversible change of existing data.
• The two high findings name concrete RBAC defects: #288 loses permission grants to imported roles in a single case, and on #310 a changed relation of one debitor revokes the access of another debitor with the same relation.
• Together 3.6 million input tokens, 82 % of them from the cache, 23000 output tokens and 15 minutes.
• On #306, medium used 355133 input tokens (286464 cached) and 1348 output tokens in 60 s, with the same classification and no finding either.
• A first run with a question about whether the PR "causes" a security problem rated #306 as not critical; the question now asks what the PR changes, not whether a defect was found.

Open Questions

• TODO.verify: Check the high findings on #288 and #310 with the hs.hsadmin.ng team.