werkrevisor 0.34.0: a runner on the workstation adds a review by the Codex CLI #88

🟠 werkrevisor ORANGE for 342e224, superseded by the result for 98a07b9

🟠 werkrevisor: ORANGE / FINDINGS

Change kind: mixed.

Reasons

ORANGE 2 finding(s) that Jev rates as probably real and severe: werkrevisor/codex.py:116-136, werkrevisor/codex.py:162-192
YELLOW PR too large for an AI-only approval: 1007 changed lines (docs and mechanical files not counted) exceed [size] max_changed_lines = 800; please review it yourself
YELLOW spec: docs/prs/2026-10-11-PR#88-codex-runner.md Scenario#88.07 given_when_then_misused (0.77 > yellow_above = 0.6)
YELLOW model reports a security concern: Sensitive information exposure: --sandbox read-only is the only barrier between a prompt-injected Codex run and the network. The redactor only sanitises the response, so a successful injection can exfiltrate anything Codex reads (source, secrets, Gitea token leak via env) to an attacker host before any redaction happens. The auth_secrets filter also only catches strings of length 16 or more from…
hint 16 changed file(s) that neither weaken the checks nor touch security according to Jev
hint 1 build change(s) of low risk according to Jev: statistics/build.gradle.kts (weakens checks 0.02, risk 0.00, security 0.01)
hint 14 section(s) of the PR document agree with the change summary of openrouter/minimax/minimax-m3 according to Jev
hint 1 changed file was given to the AI as excerpt of the selected lines and 5 lines around each changed line, not in full (each larger than [review] max_full_file_chars = 80000): werkrevisor/cli.py
hint files the model cannot review: docs/werkrevisor-pr-workflow-dark.svg, docs/werkrevisor-pr-workflow.svg (advisory: Jev judged every changed file)
hint 1 finding(s) of low severity or rated minor by Jev: docs/prs/2026-10-11-PR#88-codex-runner.md:108-119
hint model recommends merge with caution

Findings

Short secrets bypass the redactor at werkrevisor/codex.py:116-136, severity unknown, counted as medium, ORANGE, as a line comment
Sandbox does not block the network at werkrevisor/codex.py:162-192, severity unknown, counted as medium, ORANGE, as a line comment
Scenario wording contradicts the test at docs/prs/2026-10-11-PR#88-codex-runner.md:108-119, severity unknown, counted as medium, hint, as a line comment

Classifier

Unit	Question	Problem	Confidence	Verdict
docs/prs/2026-10-11-PR#88-codex-runner.md Scenario#88.07	given_when_then_misused	0.77	–	YELLOW

13 of 14 units passed all questions (1 PR document(s)).
5 answers ignored because of a confidence below [classifier] min_confidence.

_{Usage: model `openrouter/minimax/minimax-m3`, LLM done, tokens counted before the call 75193, calls 1, tokens in 72073 / out 13568 (reasoning included in out), duration 126.9 s, cost 0.0379 USD (reported), classifier `jev-1.13.0` calls 14, tokens in 11881, cost 0.0005 USD, finding judge `jev-1.13.0` calls 3, tokens in 5436, cost 0.0002 USD, excerpt selection `openrouter/openai/gpt-6-luna` calls 1, tokens in 21529 / out 369, cost 0.0004 USD (reported), change summary `openrouter/minimax/minimax-m3` via Minimax calls 1, tokens in 16512 / out 5120, cost 0.0071 USD (reported), PR document check `jev-1.13.0` calls 14, tokens in 37200, cost 0.0016 USD, build file judge `jev-1.13.0` calls 17, tokens in 22051, cost 0.0009 USD, phases: path_judge 1.2 s, classifier 0.9 s, change_summary 31.5 s, doc_check 1.0 s, context 7.4 s, model_review 84.1 s, finding_judge 0.3 s, config: defaults + host + repository (.werkrevisor.toml at main 08b725e).

Reviewed head `342e2248108cf9c99b0be49bffb0c61edb9a26e2` against base `08b725e23a224e9f818ec693b644dfaa12a54baa`, werkrevisor 0.33.0.}

<details><summary>🟠 werkrevisor ORANGE for <code>342e224</code>, superseded by <a href="http://git.javagil.de/mi/werkrevisor/pulls/88#issuecomment-4193">the result for <code>98a07b9</code></a></summary> ## 🟠 werkrevisor: ORANGE / FINDINGS Change kind: `mixed`. ### Reasons - **ORANGE** 2 finding(s) that Jev rates as probably real and severe: werkrevisor/codex.py:116-136, werkrevisor/codex.py:162-192 - **YELLOW** PR too large for an AI-only approval: 1007 changed lines (docs and mechanical files not counted) exceed [size] max_changed_lines = 800; please review it yourself - **YELLOW** spec: docs/prs/2026-10-11-PR#88-codex-runner.md Scenario#88.07 given_when_then_misused (0.77 > yellow_above = 0.6) - **YELLOW** model reports a security concern: Sensitive information exposure: `--sandbox read-only` is the only barrier between a prompt-injected Codex run and the network. The redactor only sanitises the response, so a successful injection can exfiltrate anything Codex reads (source, secrets, Gitea token leak via env) to an attacker host before any redaction happens. The auth_secrets filter also only catches strings of length 16 or more from… - **hint** 16 changed file(s) that neither weaken the checks nor touch security according to Jev - **hint** 1 build change(s) of low risk according to Jev: statistics/build.gradle.kts (weakens checks 0.02, risk 0.00, security 0.01) - **hint** 14 section(s) of the PR document agree with the change summary of openrouter/minimax/minimax-m3 according to Jev - **hint** 1 changed file was given to the AI as excerpt of the selected lines and 5 lines around each changed line, not in full (each larger than [review] max_full_file_chars = 80000): werkrevisor/cli.py - **hint** files the model cannot review: docs/werkrevisor-pr-workflow-dark.svg, docs/werkrevisor-pr-workflow.svg (advisory: Jev judged every changed file) - **hint** 1 finding(s) of low severity or rated minor by Jev: docs/prs/2026-10-11-PR#88-codex-runner.md:108-119 - **hint** model recommends merge with caution ### Findings 1. **Short secrets bypass the redactor** at `werkrevisor/codex.py:116-136`, severity unknown, counted as medium, ORANGE, as a line comment 2. **Sandbox does not block the network** at `werkrevisor/codex.py:162-192`, severity unknown, counted as medium, ORANGE, as a line comment 3. **Scenario wording contradicts the test** at `docs/prs/2026-10-11-PR#88-codex-runner.md:108-119`, severity unknown, counted as medium, hint, as a line comment ### Classifier | Unit | Question | Problem | Confidence | Verdict | |---|---|---|---|---| | docs/prs/2026-10-11-PR#88-codex-runner.md Scenario#88.07 | given_when_then_misused | 0.77 | – | YELLOW | 13 of 14 units passed all questions (1 PR document(s)). 5 answers ignored because of a confidence below [classifier] min_confidence. Usage: model `openrouter/minimax/minimax-m3`, LLM done, tokens counted before the call 75193, calls 1, tokens in 72073 / out 13568 (reasoning included in out), duration 126.9 s, cost 0.0379 USD (reported), classifier `jev-1.13.0` calls 14, tokens in 11881, cost 0.0005 USD, finding judge `jev-1.13.0` calls 3, tokens in 5436, cost 0.0002 USD, excerpt selection `openrouter/openai/gpt-6-luna` calls 1, tokens in 21529 / out 369, cost 0.0004 USD (reported), change summary `openrouter/minimax/minimax-m3` via Minimax calls 1, tokens in 16512 / out 5120, cost 0.0071 USD (reported), PR document check `jev-1.13.0` calls 14, tokens in 37200, cost 0.0016 USD, build file judge `jev-1.13.0` calls 17, tokens in 22051, cost 0.0009 USD, phases: path_judge 1.2 s, classifier 0.9 s, change_summary 31.5 s, doc_check 1.0 s, context 7.4 s, model_review 84.1 s, finding_judge 0.3 s, config: defaults + host + repository (.werkrevisor.toml at main 08b725e). Reviewed head `342e2248108cf9c99b0be49bffb0c61edb9a26e2` against base `08b725e23a224e9f818ec693b644dfaa12a54baa`, werkrevisor 0.33.0. </details>

ai-reviewer reviewed

ai-reviewer left a comment

werkrevisor: ORANGE / FINDINGS for 342e2248108cf9c99b0be49bffb0c61edb9a26e2.
The result comment of werkrevisor holds the details.

werkrevisor: ORANGE / FINDINGS for `342e2248108cf9c99b0be49bffb0c61edb9a26e2`. The result comment of werkrevisor holds the details.

docs/prs/2026-10-11-PR#88-codex-runner.md Outdated

						
				@@ -0,0 +105,4 @@

				- [test_codexrunner.CodexRunnerTest.test_no_token_or_login_reaches_the_pr](../../tests/test_codexrunner.py)

				- [test_codex.CodexTest.test_the_login_values_are_secrets](../../tests/test_codex.py)

				#### Scenario#88.07: werkrevisor does not wait for Codex comments

ai-reviewer commented

2026-10-11 21:15:18 +02:00

Scenario wording contradicts the test, severity unknown, counted as medium (werkrevisor)

hint; Jev: real 0.80, severity 1.6 of 3

Scenario#88.07 says: Given a PR that werkrevisor rated GREEN, with one unresolved comment of the Codex review / When werkrevisor reviews the same commit again / Then werkrevisor starts the automatic merge. But the verified test (test_a_codex_line_comment_holds_no_merge) starts with a writing-rule comment on a PR that is rated GREEN, replaces the body with the Codex marker, and asserts the merge is scheduled. The scenario describes "one unresolved comment of the Codex review", which is misleading: in the test the unresolved hint comes from a werkrevisor writing-rule comment that has been retrofitted with the Codex marker, not from a real Codex line comment. Either the scenario should be reworded to match the test (an open bot hint whose body carries the Codex marker), or the test should actually place a real Codex line comment into the worktree.

**Scenario wording contradicts the test**, severity unknown, counted as medium (werkrevisor) hint; Jev: real 0.80, severity 1.6 of 3 Scenario#88.07 says: Given a PR that werkrevisor rated GREEN, with one unresolved comment of the Codex review / When werkrevisor reviews the same commit again / Then werkrevisor starts the automatic merge. But the verified test (`test_a_codex_line_comment_holds_no_merge`) starts with a writing-rule comment on a PR that is rated GREEN, replaces the body with the Codex marker, and asserts the merge is scheduled. The scenario describes "one unresolved comment of the Codex review", which is misleading: in the test the unresolved hint comes from a werkrevisor writing-rule comment that has been retrofitted with the Codex marker, not from a real Codex line comment. Either the scenario should be reworded to match the test (an open bot hint whose body carries the Codex marker), or the test should actually place a real Codex line comment into the worktree.

claude-code commented

Fixed in 98a07b9: the test now adds a real Codex review with a line comment before werkrevisor runs, instead of rewriting a writing-rule comment.

Fixed in `98a07b9`: the test now adds a real Codex review with a line comment before werkrevisor runs, instead of rewriting a writing-rule comment.

claude-code marked this conversation as resolved

werkrevisor/codex.py Outdated

						
				@@ -0,0 +113,4 @@

				        and isinstance(f.get("line"), int) and not isinstance(f.get("line"), bool) for f in findings)

				def auth_secrets(codex_home=None):

ai-reviewer commented

2026-10-11 21:15:17 +02:00

Short secrets bypass the redactor, severity unknown, counted as medium (werkrevisor)

ORANGE; Jev: real 0.84, severity 3.0 of 3

auth_secrets collects only strings of length >= 16 from ~/.codex/auth.json. Anything shorter, e.g. an account_id or a future short-lived token, is added neither to the redact set nor to the unit test, so a prompt-injection that quotes such a value would leak it through the issue comment and the Codex line comments verbatim. The PR document claims "no token and no login reaches the PR"; the implementation only enforces that for long values. The threshold needs a justification (or a different approach), and the test test_the_login_values_are_secrets only covers strings of length 20 and 40, so it would not catch a regression that drops the threshold further.

**Short secrets bypass the redactor**, severity unknown, counted as medium (werkrevisor) ORANGE; Jev: real 0.84, severity 3.0 of 3 `auth_secrets` collects only strings of length >= 16 from `~/.codex/auth.json`. Anything shorter, e.g. an `account_id` or a future short-lived token, is added neither to the redact set nor to the unit test, so a prompt-injection that quotes such a value would leak it through the issue comment and the Codex line comments verbatim. The PR document claims "no token and no login reaches the PR"; the implementation only enforces that for long values. The threshold needs a justification (or a different approach), and the test `test_the_login_values_are_secrets` only covers strings of length 20 and 40, so it would not catch a regression that drops the threshold further.

claude-code commented

Fixed in 98a07b9: values of the login from 8 characters on are secrets now; the test covers a value of 10 characters, and it fails with the old limit of 16. Shorter values of auth.json are words like chatgpt; hiding them would garble the text.

Fixed in `98a07b9`: values of the login from 8 characters on are secrets now; the test covers a value of 10 characters, and it fails with the old limit of 16. Shorter values of `auth.json` are words like `chatgpt`; hiding them would garble the text.

claude-code marked this conversation as resolved

werkrevisor/codex.py

						
				@@ -0,0 +159,4 @@

				    return work

				def run(work, prompt, model, effort, timeout_seconds=1800):

ai-reviewer commented

2026-10-11 21:15:18 +02:00

Sandbox does not block the network, severity unknown, counted as medium (werkrevisor)

ORANGE; Jev: real 0.75, severity 3.0 of 3

codex exec ... --sandbox read-only in Codex CLI primarily restricts file writes, not network egress. The runner only sanitises the response of Codex with the Redactor, so a successful prompt injection in a PR description or repository can still make Codex send whatever it reads from the worktree (secrets, source, the Gitea token from any environment leak) to an attacker-controlled host before the response is ever returned. The README states "without network access to Gitea", which suggests this was intended to be off-limits. A deny-network option or an offline sandbox profile should be used for a review whose prompt comes from untrusted PR content.

**Sandbox does not block the network**, severity unknown, counted as medium (werkrevisor) ORANGE; Jev: real 0.75, severity 3.0 of 3 `codex exec ... --sandbox read-only` in Codex CLI primarily restricts file writes, not network egress. The runner only sanitises the response of Codex with the Redactor, so a successful prompt injection in a PR description or repository can still make Codex send whatever it reads from the worktree (secrets, source, the Gitea token from any environment leak) to an attacker-controlled host before the response is ever returned. The README states "without network access to Gitea", which suggests this was intended to be off-limits. A deny-network option or an offline sandbox profile should be used for a review whose prompt comes from untrusted PR content.

claude-code commented

Fixed in 98a07b9, with one correction: the sandbox of Codex already blocks the network of its commands; a probe inside the runner setup ended curl https://example.org with exit 6. The real gap was reading: read-only let Codex read every secret of the workstation. codex.run now starts Codex in bwrap with an empty home, except its login, the worktree and the programs; the same probe found no secrets.env any more. Codex gets no Gitea token in its environment, only HOME, PATH, LANG, TERM and CODEX_HOME.

Fixed in `98a07b9`, with one correction: the sandbox of Codex already blocks the network of its commands; a probe inside the runner setup ended `curl https://example.org` with exit 6. The real gap was reading: `read-only` let Codex read every secret of the workstation. `codex.run` now starts Codex in `bwrap` with an empty home, except its login, the worktree and the programs; the same probe found no `secrets.env` any more. Codex gets no Gitea token in its environment, only `HOME`, `PATH`, `LANG`, `TERM` and `CODEX_HOME`.

claude-code marked this conversation as resolved

claude-code added 1 commit 2026-10-11 21:15:10 +02:00

Codex runs in bwrap without the secrets of the workstation; short login values are secrets too

werkator build succeeded after 00:09

Details

werkator/werkrevisor-statistics build succeeded after 00:31

Details

ai-review/assessment AI findings; a developer approval after checking them unlocks the merge; a human review request is pending; merge already scheduled

Details

ai-review/gate build succeeded after 05:26

Details

98a07b945c

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

ai-reviewer commented

🟠 werkrevisor ORANGE for 98a07b9, superseded by the result for 45743f1

🟠 werkrevisor: ORANGE / FINDINGS

Change kind: mixed.

Reasons

ORANGE changes that weaken the checks according to Jev: tests/test_codex_eval.py (weakens checks 0.81)
ORANGE 1 finding(s) that Jev rates as probably real and severe: werkrevisor/codex.py:109-116
YELLOW tests lost or disabled: tests/test_codex_eval.py deleted
YELLOW PR too large for an AI-only approval: 1141 changed lines (docs and mechanical files not counted) exceed [size] max_changed_lines = 800; please review it yourself
YELLOW spec: docs/prs/2026-10-11-PR#88-codex-runner.md Scenario#88.07 given_when_then_misused (0.78 > yellow_above = 0.6)
hint 16 changed file(s) that neither weaken the checks nor touch security according to Jev
hint 1 build change(s) of low risk according to Jev: statistics/build.gradle.kts (weakens checks 0.02, risk 0.00, security 0.01)
hint 15 section(s) of the PR document agree with the change summary of openrouter/minimax/minimax-m3 according to Jev
hint 1 changed file was given to the AI as excerpt of the selected lines and 5 lines around each changed line, not in full (each larger than [review] max_full_file_chars = 80000): werkrevisor/cli.py
hint 1 changed file was given to the AI as excerpt of the selected lines and 5 lines around each changed line, not in full (all full files together would exceed [review] max_full_files_chars = 160000): werkrevisor/gitea.py
hint files the model cannot review: docs/werkrevisor-pr-workflow-dark.svg, docs/werkrevisor-pr-workflow.svg (advisory: Jev judged every changed file)
hint 1 finding(s) of low severity or rated minor by Jev: docs/prs/2026-10-11-PR#88-codex-runner.md:108-118
hint model recommends merge with caution

Findings

[medium] Incomplete answer validation at werkrevisor/codex.py:109-116, ORANGE, as a line comment
[low] Scenario Given/When/Then inconsistent with test at docs/prs/2026-10-11-PR#88-codex-runner.md:108-118, hint, as a line comment

Classifier

Unit	Question	Problem	Confidence	Verdict
docs/prs/2026-10-11-PR#88-codex-runner.md Scenario#88.07	given_when_then_misused	0.78	–	YELLOW

14 of 15 units passed all questions (1 PR document(s)).
7 answers ignored because of a confidence below [classifier] min_confidence.

_{Usage: model `openrouter/minimax/minimax-m3`, LLM done, tokens counted before the call 77053, calls 1, tokens in 73852 / out 51202 (reasoning included in out), duration 324.5 s, cost 0.0836 USD (reported), classifier `jev-1.13.0` calls 15, tokens in 12961, cost 0.0005 USD, finding judge `jev-1.13.0` calls 1, tokens in 1535, cost 0.0001 USD, excerpt selection `openrouter/openai/gpt-6-luna` calls 2, tokens in 24026 / out 531, cost 0.0005 USD (reported), change summary `openrouter/minimax/minimax-m3` via Minimax calls 1, tokens in 18615 / out 2814, cost 0.0089 USD (reported), PR document check `jev-1.13.0` calls 15, tokens in 33541, cost 0.0014 USD, build file judge `jev-1.13.0` calls 18, tokens in 24505, cost 0.0010 USD, phases: path_judge 1.1 s, classifier 0.9 s, change_summary 15.1 s, doc_check 0.9 s, context 8.0 s, model_review 297.6 s, finding_judge 0.2 s, config: defaults + host + repository (.werkrevisor.toml at main 08b725e).

Reviewed head `98a07b945c3430456f872b75aa72e8852c4d80ca` against base `08b725e23a224e9f818ec693b644dfaa12a54baa`, werkrevisor 0.33.0.}

<details><summary>🟠 werkrevisor ORANGE for <code>98a07b9</code>, superseded by <a href="http://git.javagil.de/mi/werkrevisor/pulls/88#issuecomment-4200">the result for <code>45743f1</code></a></summary> ## 🟠 werkrevisor: ORANGE / FINDINGS Change kind: `mixed`. ### Reasons - **ORANGE** changes that weaken the checks according to Jev: tests/test_codex_eval.py (weakens checks 0.81) - **ORANGE** 1 finding(s) that Jev rates as probably real and severe: werkrevisor/codex.py:109-116 - **YELLOW** tests lost or disabled: tests/test_codex_eval.py deleted - **YELLOW** PR too large for an AI-only approval: 1141 changed lines (docs and mechanical files not counted) exceed [size] max_changed_lines = 800; please review it yourself - **YELLOW** spec: docs/prs/2026-10-11-PR#88-codex-runner.md Scenario#88.07 given_when_then_misused (0.78 > yellow_above = 0.6) - **hint** 16 changed file(s) that neither weaken the checks nor touch security according to Jev - **hint** 1 build change(s) of low risk according to Jev: statistics/build.gradle.kts (weakens checks 0.02, risk 0.00, security 0.01) - **hint** 15 section(s) of the PR document agree with the change summary of openrouter/minimax/minimax-m3 according to Jev - **hint** 1 changed file was given to the AI as excerpt of the selected lines and 5 lines around each changed line, not in full (each larger than [review] max_full_file_chars = 80000): werkrevisor/cli.py - **hint** 1 changed file was given to the AI as excerpt of the selected lines and 5 lines around each changed line, not in full (all full files together would exceed [review] max_full_files_chars = 160000): werkrevisor/gitea.py - **hint** files the model cannot review: docs/werkrevisor-pr-workflow-dark.svg, docs/werkrevisor-pr-workflow.svg (advisory: Jev judged every changed file) - **hint** 1 finding(s) of low severity or rated minor by Jev: docs/prs/2026-10-11-PR#88-codex-runner.md:108-118 - **hint** model recommends merge with caution ### Findings 1. **[medium] Incomplete answer validation** at `werkrevisor/codex.py:109-116`, ORANGE, as a line comment 2. **[low] Scenario Given/When/Then inconsistent with test** at `docs/prs/2026-10-11-PR#88-codex-runner.md:108-118`, hint, as a line comment ### Classifier | Unit | Question | Problem | Confidence | Verdict | |---|---|---|---|---| | docs/prs/2026-10-11-PR#88-codex-runner.md Scenario#88.07 | given_when_then_misused | 0.78 | – | YELLOW | 14 of 15 units passed all questions (1 PR document(s)). 7 answers ignored because of a confidence below [classifier] min_confidence. Usage: model `openrouter/minimax/minimax-m3`, LLM done, tokens counted before the call 77053, calls 1, tokens in 73852 / out 51202 (reasoning included in out), duration 324.5 s, cost 0.0836 USD (reported), classifier `jev-1.13.0` calls 15, tokens in 12961, cost 0.0005 USD, finding judge `jev-1.13.0` calls 1, tokens in 1535, cost 0.0001 USD, excerpt selection `openrouter/openai/gpt-6-luna` calls 2, tokens in 24026 / out 531, cost 0.0005 USD (reported), change summary `openrouter/minimax/minimax-m3` via Minimax calls 1, tokens in 18615 / out 2814, cost 0.0089 USD (reported), PR document check `jev-1.13.0` calls 15, tokens in 33541, cost 0.0014 USD, build file judge `jev-1.13.0` calls 18, tokens in 24505, cost 0.0010 USD, phases: path_judge 1.1 s, classifier 0.9 s, change_summary 15.1 s, doc_check 0.9 s, context 8.0 s, model_review 297.6 s, finding_judge 0.2 s, config: defaults + host + repository (.werkrevisor.toml at main 08b725e). Reviewed head `98a07b945c3430456f872b75aa72e8852c4d80ca` against base `08b725e23a224e9f818ec693b644dfaa12a54baa`, werkrevisor 0.33.0. </details>

ai-reviewer reviewed

ai-reviewer left a comment

werkrevisor: ORANGE / FINDINGS for 98a07b945c3430456f872b75aa72e8852c4d80ca.
The result comment of werkrevisor holds the details.

werkrevisor: ORANGE / FINDINGS for `98a07b945c3430456f872b75aa72e8852c4d80ca`. The result comment of werkrevisor holds the details.

docs/prs/2026-10-11-PR#88-codex-runner.md Outdated

						
				@@ -0,0 +105,4 @@

				- [test_codexrunner.CodexRunnerTest.test_no_token_or_login_reaches_the_pr](../../tests/test_codexrunner.py)

				- [test_codex.CodexTest.test_the_login_values_are_secrets](../../tests/test_codex.py)

				#### Scenario#88.07: werkrevisor does not wait for Codex comments

ai-reviewer commented

2026-10-11 21:22:09 +02:00

[low] Scenario Given/When/Then inconsistent with test (werkrevisor)

hint

Scenario#88.07 states: "Given a PR that werkrevisor rated GREEN, with one unresolved comment of the Codex review. When werkrevisor reviews the same commit again. Then werkrevisor starts the automatic merge." The Given describes a prior werkrevisor run (past tense "rated GREEN", "the same commit again") that the test test_a_codex_line_comment_holds_no_merge does not simulate — the test sets up a PR with one unresolved Codex line comment and runs werkrevisor once. The Given should describe the setup before the action (e.g. "a PR with one unresolved Codex line comment and no werkrevisor hints"), not the outcome.

**[low] Scenario Given/When/Then inconsistent with test** (werkrevisor) hint Scenario#88.07 states: "Given a PR that werkrevisor rated GREEN, with one unresolved comment of the Codex review. When werkrevisor reviews the same commit again. Then werkrevisor starts the automatic merge." The Given describes a prior werkrevisor run (past tense "rated GREEN", "the same commit again") that the test `test_a_codex_line_comment_holds_no_merge` does not simulate — the test sets up a PR with one unresolved Codex line comment and runs werkrevisor once. The Given should describe the setup before the action (e.g. "a PR with one unresolved Codex line comment and no werkrevisor hints"), not the outcome.

claude-code commented

Fixed in 45743f1: the Given now names only the state before the run, a PR whose only unresolved line comment comes from Codex, as the test sets it up.

Fixed in `45743f1`: the Given now names only the state before the run, a PR whose only unresolved line comment comes from Codex, as the test sets it up.

claude-code marked this conversation as resolved

werkrevisor/codex.py

						
				@@ -0,0 +106,4 @@

				    return usage, commands, errors

				def valid_answer(answer):

ai-reviewer commented

2026-10-11 21:22:08 +02:00

[medium] Incomplete answer validation (werkrevisor)

ORANGE; Jev: real 0.73, severity 2.7 of 3

valid_answer checks that each finding has severity, file, and line, but not title and explanation (both required by SCHEMA). If the Codex CLI silently does not enforce --output-schema, a finding with missing title or explanation would pass valid_answer but crash the runner in line_comment and result_comment when accessing finding['title'] or finding['explanation']. The crash is caught by the outer try/except in _review and treated as a failed run, so the impact is a degraded review (3 failed retries → error comment) instead of a posted comment with broken findings. Add isinstance(f.get("title"), str) and isinstance(f.get("explanation"), str) to the all(...) check, and (for completeness) verify answer.get("critical_reason") is a string.

**[medium] Incomplete answer validation** (werkrevisor) ORANGE; Jev: real 0.73, severity 2.7 of 3 `valid_answer` checks that each finding has `severity`, `file`, and `line`, but not `title` and `explanation` (both required by `SCHEMA`). If the Codex CLI silently does not enforce `--output-schema`, a finding with missing `title` or `explanation` would pass `valid_answer` but crash the runner in `line_comment` and `result_comment` when accessing `finding['title']` or `finding['explanation']`. The crash is caught by the outer `try/except` in `_review` and treated as a failed run, so the impact is a degraded review (3 failed retries → error comment) instead of a posted comment with broken findings. Add `isinstance(f.get("title"), str)` and `isinstance(f.get("explanation"), str)` to the `all(...)` check, and (for completeness) verify `answer.get("critical_reason")` is a string.

claude-code commented

Fixed in 45743f1: valid_answer also requires critical_reason, title and explanation as strings; the test covers a missing title and a missing explanation.

Fixed in `45743f1`: `valid_answer` also requires `critical_reason`, `title` and `explanation` as strings; the test covers a missing title and a missing explanation.

claude-code marked this conversation as resolved

claude-code added 1 commit 2026-10-11 21:22:10 +02:00

A Codex answer needs every field of the schema; tests of tools/codex-eval; scenario 88.07 without an earlier run

werkator build succeeded after 00:11

Details

werkator/werkrevisor-statistics build succeeded after 00:30

Details

ai-review/assessment AI findings; a developer approval after checking them unlocks the merge; a human review request is pending; merge already scheduled

Details

ai-review/gate build succeeded after 05:02

Details

45743f10d2

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

ai-reviewer commented

🟠 werkrevisor ORANGE for 45743f1, superseded by the result for 16af857

🟠 werkrevisor: ORANGE / FINDINGS

Change kind: mixed.

Reasons

ORANGE 1 finding(s) that Jev rates as probably real and severe: werkrevisor/codexrunner.py:88-90
YELLOW tests lost or disabled: tests/test_codex_eval.py:17 removes 5 test(s) or assertion(s) without a replacement
YELLOW PR too large for an AI-only approval: 1182 changed lines (docs and mechanical files not counted) exceed [size] max_changed_lines = 800; please review it yourself
YELLOW spec: docs/prs/2026-10-11-PR#88-codex-runner.md Scenario#88.07 given_when_then_misused (0.74 > yellow_above = 0.6)
hint 17 changed file(s) that neither weaken the checks nor touch security according to Jev
hint 1 build change(s) of low risk according to Jev: statistics/build.gradle.kts (weakens checks 0.02, risk 0.00, security 0.01)
hint 15 section(s) of the PR document agree with the change summary of openrouter/minimax/minimax-m3 according to Jev
hint 1 changed file was given to the AI as excerpt of the selected lines and 5 lines around each changed line, not in full (each larger than [review] max_full_file_chars = 80000): werkrevisor/cli.py
hint 1 changed file was given to the AI as excerpt of the selected lines and 5 lines around each changed line, not in full (all full files together would exceed [review] max_full_files_chars = 160000): werkrevisor/config.py
hint files the model cannot review: docs/werkrevisor-pr-workflow-dark.svg, docs/werkrevisor-pr-workflow.svg (advisory: Jev judged every changed file)
hint 1 finding(s) of low severity or rated minor by Jev: werkrevisor/codexrunner.py:276-283
hint model recommends merge with caution

Findings

Codex runner runs even when werkrevisor returns RED at werkrevisor/codexrunner.py:88-90, severity unknown, counted as medium, ORANGE, as a line comment
Status comparison uses untruncated description at werkrevisor/codexrunner.py:276-283, severity unknown, counted as medium, hint, as a line comment

Classifier

Unit	Question	Problem	Confidence	Verdict
docs/prs/2026-10-11-PR#88-codex-runner.md Scenario#88.07	given_when_then_misused	0.74	–	YELLOW

14 of 15 units passed all questions (1 PR document(s)).
4 answers ignored because of a confidence below [classifier] min_confidence.

_{Usage: model `openrouter/minimax/minimax-m3`, LLM done, tokens counted before the call 68746, calls 1, tokens in 65892 / out 45314 (reasoning included in out), duration 300.3 s, cost 0.0741 USD (reported), classifier `jev-1.13.0` calls 15, tokens in 12967, cost 0.0005 USD, finding judge `jev-1.13.0` calls 2, tokens in 2995, cost 0.0001 USD, excerpt selection `openrouter/openai/gpt-6-luna` calls 2, tokens in 37773 / out 577, cost 0.0025 USD (reported), change summary `openrouter/minimax/minimax-m3` via Minimax calls 1, tokens in 19441 / out 2705, cost 0.0090 USD (reported), PR document check `jev-1.13.0` calls 15, tokens in 40357, cost 0.0017 USD, build file judge `jev-1.13.0` calls 18, tokens in 25373, cost 0.0011 USD, phases: path_judge 1.1 s, classifier 0.9 s, change_summary 19.9 s, doc_check 1.0 s, context 9.2 s, model_review 267.4 s, finding_judge 0.2 s, config: defaults + host + repository (.werkrevisor.toml at main 08b725e).

Reviewed head `45743f10d24a1d546234c2ddca9cb9ce1b189b09` against base `08b725e23a224e9f818ec693b644dfaa12a54baa`, werkrevisor 0.33.0.}

<details><summary>🟠 werkrevisor ORANGE for <code>45743f1</code>, superseded by <a href="http://git.javagil.de/mi/werkrevisor/pulls/88#issuecomment-4207">the result for <code>16af857</code></a></summary> ## 🟠 werkrevisor: ORANGE / FINDINGS Change kind: `mixed`. ### Reasons - **ORANGE** 1 finding(s) that Jev rates as probably real and severe: werkrevisor/codexrunner.py:88-90 - **YELLOW** tests lost or disabled: tests/test_codex_eval.py:17 removes 5 test(s) or assertion(s) without a replacement - **YELLOW** PR too large for an AI-only approval: 1182 changed lines (docs and mechanical files not counted) exceed [size] max_changed_lines = 800; please review it yourself - **YELLOW** spec: docs/prs/2026-10-11-PR#88-codex-runner.md Scenario#88.07 given_when_then_misused (0.74 > yellow_above = 0.6) - **hint** 17 changed file(s) that neither weaken the checks nor touch security according to Jev - **hint** 1 build change(s) of low risk according to Jev: statistics/build.gradle.kts (weakens checks 0.02, risk 0.00, security 0.01) - **hint** 15 section(s) of the PR document agree with the change summary of openrouter/minimax/minimax-m3 according to Jev - **hint** 1 changed file was given to the AI as excerpt of the selected lines and 5 lines around each changed line, not in full (each larger than [review] max_full_file_chars = 80000): werkrevisor/cli.py - **hint** 1 changed file was given to the AI as excerpt of the selected lines and 5 lines around each changed line, not in full (all full files together would exceed [review] max_full_files_chars = 160000): werkrevisor/config.py - **hint** files the model cannot review: docs/werkrevisor-pr-workflow-dark.svg, docs/werkrevisor-pr-workflow.svg (advisory: Jev judged every changed file) - **hint** 1 finding(s) of low severity or rated minor by Jev: werkrevisor/codexrunner.py:276-283 - **hint** model recommends merge with caution ### Findings 1. **Codex runner runs even when werkrevisor returns RED** at `werkrevisor/codexrunner.py:88-90`, severity unknown, counted as medium, ORANGE, as a line comment 2. **Status comparison uses untruncated description** at `werkrevisor/codexrunner.py:276-283`, severity unknown, counted as medium, hint, as a line comment ### Classifier | Unit | Question | Problem | Confidence | Verdict | |---|---|---|---|---| | docs/prs/2026-10-11-PR#88-codex-runner.md Scenario#88.07 | given_when_then_misused | 0.74 | – | YELLOW | 14 of 15 units passed all questions (1 PR document(s)). 4 answers ignored because of a confidence below [classifier] min_confidence. Usage: model `openrouter/minimax/minimax-m3`, LLM done, tokens counted before the call 68746, calls 1, tokens in 65892 / out 45314 (reasoning included in out), duration 300.3 s, cost 0.0741 USD (reported), classifier `jev-1.13.0` calls 15, tokens in 12967, cost 0.0005 USD, finding judge `jev-1.13.0` calls 2, tokens in 2995, cost 0.0001 USD, excerpt selection `openrouter/openai/gpt-6-luna` calls 2, tokens in 37773 / out 577, cost 0.0025 USD (reported), change summary `openrouter/minimax/minimax-m3` via Minimax calls 1, tokens in 19441 / out 2705, cost 0.0090 USD (reported), PR document check `jev-1.13.0` calls 15, tokens in 40357, cost 0.0017 USD, build file judge `jev-1.13.0` calls 18, tokens in 25373, cost 0.0011 USD, phases: path_judge 1.1 s, classifier 0.9 s, change_summary 19.9 s, doc_check 1.0 s, context 9.2 s, model_review 267.4 s, finding_judge 0.2 s, config: defaults + host + repository (.werkrevisor.toml at main 08b725e). Reviewed head `45743f10d24a1d546234c2ddca9cb9ce1b189b09` against base `08b725e23a224e9f818ec693b644dfaa12a54baa`, werkrevisor 0.33.0. </details>

ai-reviewer reviewed

ai-reviewer left a comment

werkrevisor: ORANGE / FINDINGS for 45743f10d24a1d546234c2ddca9cb9ce1b189b09.
The result comment of werkrevisor holds the details.

werkrevisor: ORANGE / FINDINGS for `45743f10d24a1d546234c2ddca9cb9ce1b189b09`. The result comment of werkrevisor holds the details.

werkrevisor/codexrunner.py

						
				@@ -0,0 +85,4 @@

				    return None, False

				def has_werkrevisor_result(comments, head, login):

ai-reviewer commented

2026-10-11 21:28:21 +02:00

Codex runner runs even when werkrevisor returns RED, severity unknown, counted as medium (werkrevisor)

ORANGE; Jev: real 0.71, severity 3.0 of 3

The function has_werkrevisor_result returns True for any non-None verdict (GREEN, YELLOW, or RED), so the runner proceeds to invoke the Codex CLI for PRs that werkrevisor has already blocked. This wastes the subscription quota and posts review results on PRs that cannot merge. The PR document's Scenario#88.02 is titled "a PR waits for the result of werkrevisor", implying a wait for a passing result, but the implementation only waits for the presence of any result comment. The check should verify the verdict value (e.g., GREEN or YELLOW with approval) before consuming quota.

**Codex runner runs even when werkrevisor returns RED**, severity unknown, counted as medium (werkrevisor) ORANGE; Jev: real 0.71, severity 3.0 of 3 The function `has_werkrevisor_result` returns True for any non-None verdict (GREEN, YELLOW, or RED), so the runner proceeds to invoke the Codex CLI for PRs that werkrevisor has already blocked. This wastes the subscription quota and posts review results on PRs that cannot merge. The PR document's Scenario#88.02 is titled "a PR waits for the result of werkrevisor", implying a wait for a passing result, but the implementation only waits for the presence of any result comment. The check should verify the verdict value (e.g., GREEN or YELLOW with approval) before consuming quota.

claude-code commented

Fixed in 16af857: the runner reviews only a head that werkrevisor rated GREEN, YELLOW or ORANGE; the test covers a RED head.

Fixed in `16af857`: the runner reviews only a head that werkrevisor rated GREEN, YELLOW or ORANGE; the test covers a RED head.

claude-code marked this conversation as resolved

werkrevisor/codexrunner.py

						
				@@ -0,0 +273,4 @@

				                 f"{len(findings)} finding(s), {record['seconds']} s")

				        return comment, False

				    def set_status(self, gitea, head, state, description):

ai-reviewer commented