werkrevisor/docs/prs/2026-10-11-PR#88-codex-runner.md

PR#88: A Runner on the Workstation Adds a Review by the Codex CLI

WARNING: This document describes only the change applied in this PR. It may already be outdated once the next PR is merged. Historic PR documentation is not maintained along with new PRs; treat it as a snapshot.

Related Links

• PR#87: the measurement on 10 hs.hsadmin.ng PRs.
• Werkbaum: #model.codex.gate.

Problem

On 10 hs.hsadmin.ng PRs, the Codex CLI with gpt-6-astra rated all 5 critical PRs as critical and found two real RBAC defects that the model review by PR-Agent had not found. werkrevisor does not use it yet.

Non-Goals

• No Codex CLI on mih09: the subscription login stays on the workstation of the user.

Specification

Feature: a runner on the workstation reviews ready PRs with the Codex CLI

Scenario#88.01: a repository without a Codex mode gets no review

So that the subscription quota goes only to repositories that chose the review.

• Given a repository whose .werkrevisor.toml on the default branch has no [codex] section or mode = "off"
• When the runner runs
• Then it runs no Codex review, writes no comment and sets no status

Verified by

• test_codexrunner.CodexRunnerTest.test_a_repository_without_a_codex_mode_gets_no_review
• test_config_layers.LayeringTest.test_the_repository_chooses_the_codex_mode

Scenario#88.02: a PR waits for the result of werkrevisor

So that no quota goes to a PR that is not ready.

• Given a PR with a "WIP: " title, or a PR without a result comment of werkrevisor for its head
• When the runner runs
• Then it runs no Codex review
• and in gate mode, a PR without "WIP: " gets the status ai-review/codex pending

Verified by

• test_codexrunner.CodexRunnerTest.test_a_pr_waits_for_werkrevisor_and_its_wip_end

Scenario#88.03: a RED head gets no Codex review

So that no quota goes to a head that needs a new commit anyway.

• Given a head that werkrevisor rated RED
• When the runner runs
• Then it runs no Codex review

Verified by

• test_codexrunner.CodexRunnerTest.test_a_red_head_gets_no_codex_review

Scenario#88.04: advise comments once and holds nothing

So that a repository can see the Codex review before it lets it hold the merge.

• Given a repository with mode = "advise" and a ready PR with a result of werkrevisor
• When the runner runs twice
• Then one result comment and one line comment per finding in the diff exist, and no status is set

Verified by

• test_codexrunner.CodexRunnerTest.test_advise_comments_once_and_sets_no_status

Scenario#88.05: gate releases the merge only when nothing is open

So that a finding or a critical PR is not merged unseen.

• Given a repository with mode = "gate" and a Codex review of the head
• When the runner runs again
• Then ai-review/codex is success only when every Codex line comment is resolved and, for a critical PR or a finding outside the diff, a human reviewer approved the head after the review

Verified by

• test_codexrunner.CodexRunnerTest.test_gate_passes_a_pr_without_findings
• test_codexrunner.CodexRunnerTest.test_gate_holds_until_each_line_comment_is_resolved
• test_codexrunner.CodexRunnerTest.test_a_critical_pr_waits_for_a_later_human_approval
• test_codexrunner.CodexRunnerTest.test_a_finding_outside_the_diff_needs_a_human
• test_codexrunner.CodexRunnerTest.test_an_unchanged_status_is_not_repeated

Scenario#88.06: a failed Codex run is retried, then reported

So that a broken run never passes as a review without findings.

• Given two failed Codex runs for a head, each with an error or without a valid answer
• When the third run fails too
• Then the runner writes an error comment
• and in gate mode the status ai-review/codex is error until a human reviewer approves the head

Verified by

• test_codexrunner.CodexRunnerTest.test_a_failed_run_is_retried_and_then_reported
• test_codex.CodexTest.test_only_a_complete_answer_counts_as_a_review

Scenario#88.07: no token and no login reaches the PR

So that a prompt injection in a PR cannot publish a secret through the answer of Codex.

• Given an answer of Codex that contains the Gitea token or a value of the Codex login
• When the runner posts it
• Then neither value appears in a comment

Verified by

• test_codexrunner.CodexRunnerTest.test_no_token_or_login_reaches_the_pr
• test_codex.CodexTest.test_the_login_values_are_secrets

Scenario#88.08: werkrevisor does not wait for Codex comments

So that a Codex review in advise mode never stops a merge.

• Given a PR without findings of werkrevisor, whose only unresolved line comment comes from the Codex review
• When werkrevisor reviews the PR
• Then the merge of the PR is scheduled

Verified by

• test_hints_hold_merge.HintsHoldTheMergeTest.test_a_codex_line_comment_holds_no_merge

Scenario#88.09: Codex cannot read the secrets of the workstation

So that a prompt injection in a PR cannot make Codex quote a token from the home directory.

• Given a home directory with ~/.config/ai-coding-agents/secrets.env
• When the runner starts Codex
• Then Codex can read only its login, the worktree with its repository and the programs on the PATH

Verified by

• test_codex.CodexTest.test_the_jail_hides_the_home_except_the_login_and_the_programs

Solution

Decisions

• The Codex review runs on the workstation, as a second stage after werkrevisor: a runner asks Gitea for PRs whose head has a result of werkrevisor but no Codex review, runs the Codex CLI on them and writes its result into the PR as ai-reviewer. Reason: the login of the subscription must not reach a shared server or a build container; a Codex review exists only while the workstation runs.

• .werkrevisor.toml chooses per repository whether and how the Codex review acts, with [codex] mode: off, the default, means no Codex review; advise writes comments only and holds nothing; gate lets the merge wait for the Codex review of the head, makes its findings line comments that hold the merge, and asks for the approval of a human when Codex rates the PR as critical or reports a finding outside the diff. Repositories that want a Codex review start with advise. Reason: in the measurement Codex rated 8 of 10 PRs as critical, 2 of them against the classification of the user, and with gate a PR waits as long as the workstation is off.

• In gate mode the runner sets its own commit status ai-review/codex: pending while the review runs or while a finding is open, success when no finding is open, and for a critical PR or a finding outside the diff success only after the approval of a human for the head. The branch protection of a gate repository requires this status, so the automerge waits for it. Reason: Werkator sets ai-review/gate from the exit code of werkrevisor on mih09, and nothing restarts that review when the runner is done; werkrevisor only accepts [codex] and ignores the line comments of Codex.

Implementation

• werkrevisor/codex.py holds the review by the Codex CLI: checkout, prompt, JSON schema, run and the check of the answer; tools/codex-eval of PR#87 uses it too; tests/test_codex_eval.py now tests the tool itself.
• werkrevisor/codexrunner.py and tools/codex-runner look once at the open PRs of config/codex-runner.toml and exit; the units in tools/systemd/ start it every 5 minutes.
• Markers in the result comment hold the head, the criticality and the number of findings outside the diff, so a later run decides the status without a new Codex run.
• codex.run starts Codex in bwrap with an empty home directory, except the login, the worktree with its bare repository, the node_modules of the CLI and the program directories on the PATH. Without bwrap no review runs. A probe on 2026-10-11 showed: inside, secrets.env is missing, and curl ends with exit 6, since the sandbox of Codex already blocks the network of its commands.
• Values of the Codex login from 8 characters on are secrets; shorter ones are words like chatgpt.
• [codex] mode in .werkrevisor.toml; werkrevisor accepts it and skips line comments with the Codex marker.
• GiteaClient lists open PRs and the statuses of a commit.
• README: [codex] and "Codex runner"; the PR workflow diagram shows the optional Codex review.
• Werkbaum: #model.codex.gate.
• Version 0.34.0.

Open Questions

• How do the criticality and the findings of Codex affect the merge? Per repository, off, advise or gate; see "Decisions" under "Solution".

Additional Changes

• The four tests of tests/test_codex_eval.py from PR#87 moved to tests/test_codex.py with the code they test; tests/test_codex_eval.py now tests tools/codex-eval itself.

Follow-up PRs

• Switch on advise in the .werkrevisor.toml of the first repositories; each change needs the approval of the human.
• Install the systemd units on the workstation.