werkrevisor 0.71.0: a security fix that only hardens needs no human review by Codex #143
No Assignees
Notifications
Due Date
No due date set.
Dependencies
No dependencies set.
Reference: mi/werkrevisor#143
Reference in New Issue
Block a user
The Codex review rates a PR as not critical when it only hardens the security, fails closed and has a test of the closed case.
See
docs/prs/for the details.🤖 Generated with Claude Code
WIP: werkrevisor 0.70.0: a security fix that only hardens needs no human review by Codexto WIP: werkrevisor 0.71.0: a security fix that only hardens needs no human review by Codex930a63470eto2aaac100ad🟢 werkrevisor GREEN for
2aaac10, superseded by the result for225368d🟢 werkrevisor: GREEN / PASS
Gate: approved by werkrevisor; merge waits for 3 open hint(s): reply to each line comment and resolve it, then restart the review or push a commit.
Change kind:
mixed.Reasons
Findings
[high] Weakened coverage at
tests/test_codex.py:50-57, hint, as a line comment[low] Misleading rating at
docs/prs/2026-10-24-PR#143-codex-hardening.md:73, hint, as a line commentClassifier
7 of 7 units passed all questions (1 PR document(s)).
Usage: model `openrouter/minimax/minimax-m3`, LLM done, tokens counted before the call 62346, calls 1, tokens in 59895 / out 50708 (reasoning included in out), duration 451.0 s, cost 0.0788 USD (reported), classifier `jev-1.14.0` calls 7, tokens in 5850, cost 0.0002 USD, finding judge `jev-1.14.0` calls 1, tokens in 2073, cost 0.0001 USD, PR document check `jev-1.14.0` calls 13, tokens in 33514, cost 0.0014 USD, build file judge `jev-1.14.0` calls 11, tokens in 9711, cost 0.0004 USD, phases: path_judge 0.6 s, classifier 0.3 s, doc_check 0.6 s, context 0.4 s, model_review 448.5 s, finding_judge 0.2 s, config: defaults + host + repository (.werkrevisor.toml at main3 answers ignored because of a confidence below [classifier] min_confidence.
113f634).Reviewed head `2aaac100ad70a2546266a52d05c4eb5c80997bd5` against base `113f6342e6bb78d936f864a6f49e3dc33eaf9971`, werkrevisor 0.69.0.
werkrevisor: GREEN / PASS for
2aaac100ad70a2546266a52d05c4eb5c80997bd5.The result comment of werkrevisor holds the details.
@@ -1041,5 +1041,5 @@`statistics/` holds a Kotlin Spring Boot service that collects the usage records of all runsand shows them in a dashboard: costs for 24 hours, 7 and 30 days, per day, week and month,reviews, pull requests and costs per pull request.Per day, week and month, a trend chart shows the cost, the number of pull requests and the cost per pull request,Writing rule: paragraph with 6 sentences, more than 5 (werkrevisor writing rule)
Fix the line, or reply why it stays, and resolve this comment.
Fixed in
225368d: the paragraph is split before the security effect.@@ -0,0 +70,4 @@- `werkrevisor/codexrunner.py`: the result comment rates the PR by `codex.rating`and names the three answers in a line of its own.The marker `werkrevisor-codex-critical` holds `needs_human`, so the result comments of before keep their meaning.- `werkrevisor/reviewreport.py` and `tools/codex-eval`: the same rating and the security effect.[low] Misleading rating (werkrevisor)
hint
The Solution says
tools/codex-evaluses the same rating as the gate, but the tool prints the rawcriticalfield. A critical tested hardening is therefore shown ascritical True, whilecodex.rating()and the gate call itnot critical: a tested hardening that fails closed. An operator evaluating the tool could reach the opposite conclusion. Print the derived rating and exemption flags, or state that the tool intentionally reports raw criticality.Fixed in
225368d:tools/codex-evalprints the rawcritical, the security effect and the rating of the gate.@@ -48,3 +48,3 @@def test_only_a_complete_answer_counts_as_a_review(self):finding = {"severity": "high", "file": "a.py", "line": 3, "title": "t", "explanation": "e"}self.assertTrue(codex_eval.valid_answer({"critical": False, "critical_reason": "", "findings": [finding]}))self.assertTrue(codex_eval.valid_answer({"critical": False, "critical_reason": "", "security_effect": "none", "fails_closed": False, "hardening_tested": False, "findings": [finding]}))[high] Weakened coverage (werkrevisor)
hint; Jev: real 0.79, severity 2.1 of 3
The new required fields are supplied only for the malformed
line,title, andexplanationfixtures. The other legacy malformed fixtures still omit them, sovalid_answerrejects them for the missing fields before the old checks are exercised. Removing validation forcritical,critical_reason,findings, orseveritywould therefore leave this test green, although the runner relies on this manual validator. Keep each legacy fixture complete except for the field under test.Fixed in
225368d: each broken answer is the complete answer with only the field under test changed or removed; it now also covers a missingcritical,critical_reasonandfindings. Without thecritical_reasoncheck the test fails.Codex review: critical
This change weakens the CI merge gate by allowing security-sensitive PRs to bypass human approval based on the model’s hardening assessment. Tests verify the decision logic but cannot establish that the model will classify real security changes correctly.
No findings.
The merge waits for the status
Model `gpt-6-astra`, effort high, tokens in 326320 (cached 281600) / out 2081, 47.4 s, 9 commands; reviewed head `2aaac100ad70a2546266a52d05c4eb5c80997bd5`.ai-review/codex: every line comment resolved, and the approval of a human for a critical PR or for a finding that is only here.New commits pushed, approval review dismissed automatically according to repository settings
🟢 werkrevisor: GREEN / PASS
Change kind:
mixed.Reasons
2aaac10not carried over: 6 approved block(s) are missingFindings
[medium] Missing Diagrams at
docs/prs/2026-10-24-PR#143-codex-hardening.md:82hint; Jev: real 0.85, severity 2.3 of 3
Adding the tested-hardening exemption changes a review decision, but this line states that no diagram update is needed.
AGENTS.mdrequires the diagrams underdocs/to be updated whenever a PR changes the steps or decisions of a review. Update the affected review-flow diagram(s) to show when human approval is bypassed; otherwise the shipped diagrams do not document this security-gate branch and can mislead maintainers about when a human must review.Classifier
7 of 7 units passed all questions (1 PR document(s)).
Usage: model `openrouter/minimax/minimax-m3`, LLM done, tokens counted before the call 62999, calls 1, tokens in 60520 / out 36389 (reasoning included in out), duration 370.5 s, cost 0.0618 USD (reported), classifier `jev-1.14.0` calls 7, tokens in 5899, cost 0.0002 USD, finding judge `jev-1.14.0` calls 1, tokens in 1458, cost 0.0001 USD, PR document check `jev-1.14.0` calls 13, tokens in 34446, cost 0.0014 USD, build file judge `jev-1.14.0` calls 11, tokens in 10005, cost 0.0004 USD, phases: path_judge 0.5 s, classifier 0.3 s, doc_check 0.6 s, context 0.5 s, model_review 367.7 s, finding_judge 0.2 s, config: defaults + host + repository (.werkrevisor.toml at main4 answers ignored because of a confidence below [classifier] min_confidence.
113f634).Reviewed head `225368df8afa330b943f56bf47b5640feab093f6` against base `113f6342e6bb78d936f864a6f49e3dc33eaf9971`, werkrevisor 0.69.0.
werkrevisor: GREEN / PASS for
225368df8afa330b943f56bf47b5640feab093f6.The result comment of werkrevisor holds the details.
Codex review: critical
This change relaxes the merge gate by allowing some PRs classified as critical to merge without human approval. An incorrect model classification could bypass security review, and the tests verify gate logic rather than classification accuracy.
No findings.
The merge waits for the status
Model `gpt-6-astra`, effort high, tokens in 242502 (cached 205184) / out 1622, 41.8 s, 13 commands; reviewed head `225368df8afa330b943f56bf47b5640feab093f6`.ai-review/codex: every line comment resolved, and the approval of a human for a critical PR or for a finding that is only here.Answer to the review of
225368d:The diagrams show no decision of the Codex review.
docs/werkrevisor-pr-workflow.svgshows it as one box: "Codex review after werkrevisor, optional; per repository; in gate mode it holds the merge".review_decisions.pyandreview_inside.pydraw only the review of werkrevisor; neither names Codex.When Codex needs a human stays inside that box, as before this PR.