werkrevisor/docs/prs/2026-10-11-PR#88-codex-runner.md

# PR#88: A Runner on the Workstation Adds a Review by the Codex CLI

> **WARNING:** This document describes only the change applied in this PR.
> It may already be outdated once the next PR is merged.
> Historic PR documentation is not maintained along with new PRs; treat it as a snapshot.

## Related Links

- [PR#87](2026-10-11-PR#87-codex-eval.md): the measurement on 10 hs.hsadmin.ng PRs.
- Werkbaum: `#model.codex.gate`.

## Problem

On 10 hs.hsadmin.ng PRs, the Codex CLI with `gpt-6-astra` rated all 5 critical PRs as critical
and found two real RBAC defects that the model review by PR-Agent had not found.
werkrevisor does not use it yet.

## Non-Goals

- No Codex CLI on mih09: the subscription login stays on the workstation of the user.

## Specification

### Feature: a runner on the workstation reviews ready PRs with the Codex CLI

#### Scenario#88.01: a repository without a Codex mode gets no review

So that the subscription quota goes only to repositories that chose the review.

- **Given** a repository whose `.werkrevisor.toml` on the default branch has no `[codex]` section or `mode = "off"`
- **When** the runner runs
- **Then** it runs no Codex review, writes no comment and sets no status

##### Verified by

- [test_codexrunner.CodexRunnerTest.test_a_repository_without_a_codex_mode_gets_no_review](../../tests/test_codexrunner.py)
- [test_config_layers.LayeringTest.test_the_repository_chooses_the_codex_mode](../../tests/test_config_layers.py)

#### Scenario#88.02: a PR waits for the result of werkrevisor

So that no quota goes to a PR that is not ready.

- **Given** a PR with a "WIP: " title, or a PR without a result comment of werkrevisor for its head
- **When** the runner runs
- **Then** it runs no Codex review
- **and** in `gate` mode, a PR without "WIP: " gets the status `ai-review/codex` `pending`

##### Verified by

- [test_codexrunner.CodexRunnerTest.test_a_pr_waits_for_werkrevisor_and_its_wip_end](../../tests/test_codexrunner.py)

#### Scenario#88.03: a RED head gets no Codex review

So that no quota goes to a head that needs a new commit anyway.

- **Given** a head that werkrevisor rated RED
- **When** the runner runs
- **Then** it runs no Codex review

##### Verified by

- [test_codexrunner.CodexRunnerTest.test_a_red_head_gets_no_codex_review](../../tests/test_codexrunner.py)

#### Scenario#88.04: `advise` comments once and holds nothing

So that a repository can see the Codex review before it lets it hold the merge.

- **Given** a repository with `mode = "advise"` and a ready PR with a result of werkrevisor
- **When** the runner runs twice
- **Then** one result comment and one line comment per finding in the diff exist, and no status is set

##### Verified by

- [test_codexrunner.CodexRunnerTest.test_advise_comments_once_and_sets_no_status](../../tests/test_codexrunner.py)

#### Scenario#88.05: `gate` releases the merge only when nothing is open

So that a finding or a critical PR is not merged unseen.

- **Given** a repository with `mode = "gate"` and a Codex review of the head
- **When** the runner runs again
- **Then** `ai-review/codex` is `success` only when every Codex line comment is resolved
  and, for a critical PR or a finding outside the diff, a human reviewer approved the head after the review

##### Verified by

- [test_codexrunner.CodexRunnerTest.test_gate_passes_a_pr_without_findings](../../tests/test_codexrunner.py)
- [test_codexrunner.CodexRunnerTest.test_gate_holds_until_each_line_comment_is_resolved](../../tests/test_codexrunner.py)
- [test_codexrunner.CodexRunnerTest.test_a_critical_pr_waits_for_a_later_human_approval](../../tests/test_codexrunner.py)
- [test_codexrunner.CodexRunnerTest.test_a_finding_outside_the_diff_needs_a_human](../../tests/test_codexrunner.py)
- [test_codexrunner.CodexRunnerTest.test_an_unchanged_status_is_not_repeated](../../tests/test_codexrunner.py)

#### Scenario#88.06: a failed Codex run is retried, then reported

So that a broken run never passes as a review without findings.

- **Given** two failed Codex runs for a head, each with an error or without a valid answer
- **When** the third run fails too
- **Then** the runner writes an error comment
- **and** in `gate` mode the status `ai-review/codex` is `error` until a human reviewer approves the head

##### Verified by

- [test_codexrunner.CodexRunnerTest.test_a_failed_run_is_retried_and_then_reported](../../tests/test_codexrunner.py)
- [test_codex.CodexTest.test_only_a_complete_answer_counts_as_a_review](../../tests/test_codex.py)

#### Scenario#88.07: no token and no login reaches the PR

So that a prompt injection in a PR cannot publish a secret through the answer of Codex.

- **Given** an answer of Codex that contains the Gitea token or a value of the Codex login
- **When** the runner posts it
- **Then** neither value appears in a comment

##### Verified by

- [test_codexrunner.CodexRunnerTest.test_no_token_or_login_reaches_the_pr](../../tests/test_codexrunner.py)
- [test_codex.CodexTest.test_the_login_values_are_secrets](../../tests/test_codex.py)

#### Scenario#88.08: werkrevisor does not wait for Codex comments

So that a Codex review in `advise` mode never stops a merge.

- **Given** a PR without findings of werkrevisor, whose only unresolved line comment comes from the Codex review
- **When** werkrevisor reviews the PR
- **Then** the merge of the PR is scheduled

##### Verified by

- [test_hints_hold_merge.HintsHoldTheMergeTest.test_a_codex_line_comment_holds_no_merge](../../tests/test_hints_hold_merge.py)

#### Scenario#88.09: Codex cannot read the secrets of the workstation

So that a prompt injection in a PR cannot make Codex quote a token from the home directory.

- **Given** a home directory with `~/.config/ai-coding-agents/secrets.env`
- **When** the runner starts Codex
- **Then** Codex can read only its login, the worktree with its repository and the programs on the `PATH`

##### Verified by

- [test_codex.CodexTest.test_the_jail_hides_the_home_except_the_login_and_the_programs](../../tests/test_codex.py)

## Solution

### Decisions

- The Codex review runs on the workstation, as a second stage after werkrevisor:
  a runner asks Gitea for PRs whose head has a result of werkrevisor but no Codex review,
  runs the Codex CLI on them and writes its result into the PR as `ai-reviewer`.
  Reason: the login of the subscription must not reach a shared server or a build container;
  a Codex review exists only while the workstation runs.
- `.werkrevisor.toml` chooses per repository whether and how the Codex review acts, with `[codex] mode`:
  `off`, the default, means no Codex review;
  `advise` writes comments only and holds nothing;
  `gate` lets the merge wait for the Codex review of the head, makes its findings line comments that hold the merge,
  and asks for the approval of a human when Codex rates the PR as critical or reports a finding outside the diff.
  Repositories that want a Codex review start with `advise`.
  Reason: in the measurement Codex rated 8 of 10 PRs as critical, 2 of them against the classification of the user,
  and with `gate` a PR waits as long as the workstation is off.

- In `gate` mode the runner sets its own commit status `ai-review/codex`:
  `pending` while the review runs or while a finding is open, `success` when no finding is open,
  and for a critical PR or a finding outside the diff `success` only after the approval of a human for the head.
  The branch protection of a `gate` repository requires this status, so the automerge waits for it.
  Reason: Werkator sets `ai-review/gate` from the exit code of werkrevisor on mih09,
  and nothing restarts that review when the runner is done;
  werkrevisor only accepts `[codex]` and ignores the line comments of Codex.

### Implementation

- `werkrevisor/codex.py` holds the review by the Codex CLI: checkout, prompt, JSON schema, run and the check of the answer;
  `tools/codex-eval` of PR#87 uses it too; `tests/test_codex_eval.py` now tests the tool itself.
- `werkrevisor/codexrunner.py` and `tools/codex-runner` look once at the open PRs of `config/codex-runner.toml` and exit;
  the units in `tools/systemd/` start it every 5 minutes.
- Markers in the result comment hold the head, the criticality and the number of findings outside the diff,
  so a later run decides the status without a new Codex run.
- `codex.run` starts Codex in `bwrap` with an empty home directory, except the login,
  the worktree with its bare repository, the `node_modules` of the CLI and the program directories on the `PATH`.
  Without `bwrap` no review runs.
  A probe on 2026-10-11 showed: inside, `secrets.env` is missing, and `curl` ends with exit 6,
  since the sandbox of Codex already blocks the network of its commands.
- Values of the Codex login from 8 characters on are secrets; shorter ones are words like `chatgpt`.
- `[codex] mode` in `.werkrevisor.toml`; werkrevisor accepts it and skips line comments with the Codex marker.
- `GiteaClient` lists open PRs and the statuses of a commit.
- README: `[codex]` and "Codex runner"; the PR workflow diagram shows the optional Codex review.
- Werkbaum: `#model.codex.gate`.
- Version 0.34.0.

## Open Questions

- [x] How do the criticality and the findings of Codex affect the merge?
  Per repository, `off`, `advise` or `gate`; see "Decisions" under "Solution".

## Additional Changes

- The four tests of `tests/test_codex_eval.py` from PR#87 moved to `tests/test_codex.py` with the code they test;
  `tests/test_codex_eval.py` now tests `tools/codex-eval` itself.

## Follow-up PRs

- Switch on `advise` in the `.werkrevisor.toml` of the first repositories; each change needs the approval of the human.
- Install the systemd units on the workstation.