werkrevisor 0.35.0: tools/review-report drafts a review by werkrevisor and Codex #90
No Assignees
Notifications
Due Date
No due date set.
Dependencies
No dependencies set.
Reference: mi/werkrevisor#90
Reference in New Issue
Block a user
One command runs the full werkrevisor dry run and the Codex review on a PR of any Gitea and writes a draft report to check and post by hand.
🤖 Generated with Claude Code
🟢 werkrevisor GREEN for
4801163, superseded by the result fore456b1d🟢 werkrevisor: GREEN / PASS
Gate: approved by werkrevisor; merge waits for 3 open hint(s): reply to each line comment and resolve it, then restart the review or push a commit.
Change kind:
mixed.Reasons
Findings
Werkbaum Scope Creep at
docs/werkrevisor.werkbaum:110-122, severity unknown, counted as mediumhint; Jev: real 0.73, severity 1.6 of 3
The PR updates the
werkator (XS)entry from[ ]to[^]with the comment"werkator PR#153", which asserts a merged PR in a different repository (werkator) from inside the werkrevisor repo. The PR description and the Open Questions section (which says "None") do not mention this. The associated claim cannot be verified from this PR's diff, and the change is outside the stated scope (tools/review-report). It should be either split into its own PR or at least surfaced in the PR description and Open Questions. The companion change ofwerkrevisor (XS)from[x]to[^]also changes a verified-status line without explanation.Empty critical_reason yields trailing colon at
werkrevisor/reviewreport.py:38-39, severity unknown, counted as medium, hint, as a line commentCodex error truncation can split a message at
werkrevisor/reviewreport.py:102, severity unknown, counted as medium, hint, as a line comment1 more finding(s) not shown.
Classifier
6 of 6 units passed all questions (1 PR document(s)).
Usage: model `openrouter/minimax/minimax-m3`, LLM done, tokens counted before the call 13594, calls 1, tokens in 13093 / out 7959 (reasoning included in out), duration 68.3 s, cost 0.0134 USD (reported), classifier `jev-1.13.0` calls 6, tokens in 4673, cost 0.0002 USD, finding judge `jev-1.13.0` calls 4, tokens in 6051, cost 0.0003 USD, change summary `openrouter/minimax/minimax-m3` via Minimax calls 1, tokens in 3647 / out 1567, cost 0.0029 USD (reported), PR document check `jev-1.13.0` calls 6, tokens in 11555, cost 0.0005 USD, build file judge `jev-1.13.0` calls 5, tokens in 5127, cost 0.0002 USD, phases: path_judge 0.5 s, classifier 0.4 s, change_summary 12.4 s, doc_check 0.5 s, context 0.2 s, model_review 53.8 s, finding_judge 0.3 s, config: defaults + host + repository (.werkrevisor.toml at main1 answer ignored because of a confidence below [classifier] min_confidence.
ee7311e).Reviewed head `480116399956a7c4530728ffad3baee8476994de` against base `ee7311eeaeca4f43c960c6ca6f1664cce5488246`, werkrevisor 0.34.1.
werkrevisor: GREEN / PASS for
480116399956a7c4530728ffad3baee8476994de.The result comment of werkrevisor holds the details.
@@ -0,0 +51,4 @@- `werkrevisor/reviewreport.py` and `tools/review-report` run `tools/dry-run --llm` and `codex.run` in parallel,take the result comment of werkrevisor without its markers, and write `build/review-report/<repo>-<n>/report.md`.- Tokens and the values of the Codex login are removed from the report.- On hs.hsadmin.ng #322 it took 67 s; werkrevisor gave ORANGE this time, YELLOW in the run before,Narration of a specific run in Solution, severity unknown, counted as medium (werkrevisor)
hint; Jev: real 0.65, severity 0.9 of 3
The Solution section contains a single-run observation ("On hs.hsadmin.ng #322 it took 67 s; werkrevisor gave ORANGE this time, YELLOW in the run before, since the model review differs from run to run.") that narrates one execution rather than describing the feature. It does not change with the code and is filler in the Solution block.
Fixed in
e456b1d: the trial on #322 moved to a section "Attachments".@@ -0,0 +35,4 @@answer = record.get("answer") or {}if record.get("exit_code") != 0 or not codex.valid_answer(answer):return Nonelines = [f"Codex rates the PR as {'critical' if answer['critical'] else 'not critical'}: "Empty critical_reason yields trailing colon, severity unknown, counted as medium (werkrevisor)
hint; Jev: real 0.78, severity 1.1 of 3
In
codex_part, whenanswer['critical_reason']is the empty string (a state accepted bycodex.valid_answer, as exercised by the test at line 57 oftests/test_reviewreport.py), the rendered line becomes"Codex rates the PR as not critical: "with a dangling colon and space. The format string should trim or omit the colon when the reason is empty.Fixed in
e456b1d: without a reason the line ends after the rating; the test covers a blank reason.@@ -0,0 +99,4 @@head, base = pr["head"]["sha"], pr.get("merge_base") or pr["base"]["sha"]work = codex.checkout(base_url, owner, repo, number, head, base, token, os.path.join(args.out, "codex"))record = codex.run(work, codex.prompt_for(pr, base), args.model, args.effort)codex_error = "; ".join(record["errors"])[-500:]Codex error truncation can split a message, severity unknown, counted as medium (werkrevisor)
hint; Jev: real 0.72, severity 2.3 of 3
Line 102 joins all errors then slices the joined string from the end (
"; ".join(record["errors"])[-500:]). Truncating the joined blob, rather than trimming per-message, can cut the first error in the middle and leave the report with a misleading leading fragment of a different error than the surrounding text suggests.Fixed in
e456b1d:codex.error_textcuts each of the last two messages on its own; the runner uses it too.New commits pushed, approval review dismissed automatically according to repository settings
🟢 werkrevisor GREEN for
e456b1d, superseded by the result for07e512f🟢 werkrevisor: GREEN / PASS
Gate: approved by werkrevisor; merge waits for 1 open hint(s): reply to each line comment and resolve it, then restart the review or push a commit.
Change kind:
mixed.Reasons
4801163not carried over: 3 approved block(s) are missingFindings
werkrevisor/reviewreport.py:9, severity unknown, counted as medium, hint, as a line commentClassifier
7 of 7 units passed all questions (1 PR document(s)).
Usage: model `openrouter/minimax/minimax-m3`, LLM done, tokens counted before the call 23134, calls 1, tokens in 22214 / out 22746 (reasoning included in out), duration 273.2 s, cost 0.0339 USD (reported), classifier `jev-1.13.0` calls 7, tokens in 5299, cost 0.0002 USD, finding judge `jev-1.13.0` calls 1, tokens in 972, cost 0.0000 USD, change summary `openrouter/minimax/minimax-m3` via Minimax calls 1, tokens in 4039 / out 1508, cost 0.0030 USD (reported), PR document check `jev-1.13.0` calls 7, tokens in 11000, cost 0.0005 USD, build file judge `jev-1.13.0` calls 7, tokens in 6336, cost 0.0003 USD, phases: path_judge 0.6 s, classifier 0.5 s, change_summary 16.6 s, doc_check 0.5 s, context 0.3 s, model_review 254.0 s, finding_judge 0.2 s, config: defaults + host + repository (.werkrevisor.toml at main2 answers ignored because of a confidence below [classifier] min_confidence.
ee7311e).Reviewed head `e456b1d117f947e1d959e36e629fa84dd7fe82fa` against base `ee7311eeaeca4f43c960c6ca6f1664cce5488246`, werkrevisor 0.34.1.
werkrevisor: GREEN / PASS for
e456b1d117f947e1d959e36e629fa84dd7fe82fa.The result comment of werkrevisor holds the details.
@@ -0,0 +6,4 @@import osimport reimport subprocessimport sysUnused import, severity unknown, counted as medium (werkrevisor)
hint; Jev: real 0.79, severity 0.5 of 3
The
sysmodule is imported at the top of the file but never referenced anywhere in the module. It is dead code that should be removed.Fixed in
07e512f:sysis gone.Codex review of
e456b1d117f947e1d959e36e629fa84dd7fe82fa@@ -0,0 +98,4 @@try:pr = GiteaClient(base_url, owner, repo, token, timeout=60).get_pr(number)head, base = pr["head"]["sha"], pr.get("merge_base") or pr["base"]["sha"]work = codex.checkout(base_url, owner, repo, number, head, base, token, os.path.join(args.out, "codex"))[high] Isolate checkout caches by Gitea server (Codex review)
Every server shares the same checkout root, while codex.checkout keys its mirror only by owner and repository and retains the existing origin. Reviewing the same owner/repository on server B after server A therefore fetches from A using B’s GITEA_TOKEN, disclosing B’s credential to A.
Fix it, or reply why it does not hold, and resolve this comment.
Fixed in
07e512f: one mirror and worktree per server, and every fetch names the URL of its server instead oforigin;test_a_token_reaches_only_its_own_serverfails without the fix.@@ -0,0 +104,4 @@except Exception as e:head, codex_error = "unknown", f"{type(e).__name__}: {e}"thread.join()report = compose(head, werkrevisor_part(result.get("werkrevisor", "")), record and codex_part(record),[medium] Verify that both reviews cover the same commit (Codex review)
The parallel reviewers fetch the PR head independently. A push between those fetches can make Codex review the old head and werkrevisor review the new one. The report uses only Codex’s head and strips werkrevisor’s reviewed-head metadata, silently presenting both results as reviews of one commit.
Fix it, or reply why it does not hold, and resolve this comment.
Fixed in
07e512f: the report compares the head in the result of werkrevisor with the head of Codex and warns when they differ.⚠️ werkrevisor ERROR for
?, superseded by the result for07e512fCodex review: critical
The new command handles live Gitea credentials across arbitrary servers. Incorrect checkout isolation can disclose a server’s token to another server, which requires human security review.
Findings
werkrevisor/reviewreport.py:101, as a line commentwerkrevisor/reviewreport.py:107, as a line commentThis review only advises; it holds no merge.
Model `gpt-6-astra`, effort high, tokens in 323100 (cached 274816) / out 2479, 82.6 s, 7 commands; reviewed head `e456b1d117f947e1d959e36e629fa84dd7fe82fa`.New commits pushed, approval review dismissed automatically according to repository settings
🟢 werkrevisor GREEN for
07e512f, superseded by the result for581e1f0🟢 werkrevisor: GREEN / PASS
Gate: approved by werkrevisor; merge waits for 1 open hint(s): reply to each line comment and resolve it, then restart the review or push a commit.
Change kind:
mixed.Reasons
e456b1dnot carried over: 3 approved block(s) are missingClassifier
8 of 8 units passed all questions (1 PR document(s)).
Usage: model `openrouter/minimax/minimax-m3`, LLM done, tokens counted before the call 27497, calls 1, tokens in 26407 / out 14353 (reasoning included in out), duration 108.6 s, cost 0.0251 USD (reported), classifier `jev-1.13.0` calls 8, tokens in 6051, cost 0.0003 USD, change summary `openrouter/minimax/minimax-m3` via Minimax calls 1, tokens in 5172 / out 1590, cost 0.0034 USD (reported), PR document check `jev-1.13.0` calls 8, tokens in 14445, cost 0.0006 USD, build file judge `jev-1.13.0` calls 8, tokens in 7856, cost 0.0003 USD, phases: path_judge 0.5 s, classifier 0.5 s, change_summary 11.7 s, doc_check 0.5 s, context 0.3 s, model_review 94.5 s, config: defaults + host + repository (.werkrevisor.toml at main2 answers ignored because of a confidence below [classifier] min_confidence.
ee7311e).Reviewed head `07e512f826e6da2a8e245fb06373ccb9ee89e0a4` against base `ee7311eeaeca4f43c960c6ca6f1664cce5488246`, werkrevisor 0.34.1.
werkrevisor: GREEN / PASS for
07e512f826e6da2a8e245fb06373ccb9ee89e0a4.The result comment of werkrevisor holds the details.
@@ -0,0 +65,4 @@- Found by the first Codex review of the runner on this PR: `codex.checkout` kept one mirror per owner and repositoryand fetched from `origin`, so a token of one Gitea could reach another Gitea with a repository of the same name.Each server now has its own mirrors and worktrees, and every fetch names the URL of its server;a test fails without the fix. Mirrors from before are not used any more and can be deleted.Writing rule: more than one sentence on a line (werkrevisor writing rule)
Fix the line, or reply why it stays, and resolve this comment.
Fixed in
581e1f0: one sentence per line.Codex review of
07e512f826e6da2a8e245fb06373ccb9ee89e0a4@@ -0,0 +89,4 @@redact = Redactor([v for k, v in env.items() if k in ("GITEA_TOKEN", "OPENROUTER_API_KEY", "TYPESAFE_API_KEY")]+ codex.auth_secrets(env.get("CODEX_HOME")))base_url, owner, repo, number = codex.parse_pr_url(args.pr_url)out = os.path.join(args.out, f"{repo}-{number}")[medium] Reports for different repositories overwrite each other (Codex review)
The output directory includes only the repository name and PR number, omitting server and owner. Reviewing a.example.org/team/app/pulls/7 and b.example.org/other/app/pulls/7 writes both drafts to app-7/report.md; the second run silently overwrites the first, including any manual corrections.
Fix it, or reply why it does not hold, and resolve this comment.
Fixed in
581e1f0: the draft goes to<server>/<owner>/<repo>-<n>/report.md.@@ -0,0 +110,4 @@record = codex.run(work, codex.prompt_for(pr, base), args.model, args.effort)codex_error = codex.error_text(record["errors"])except Exception as e:head, codex_error = "unknown", f"{type(e).__name__}: {e}"[low] Codex failures discard the known commit (Codex review)
If checkout or Codex execution raises after the PR was fetched, this handler replaces the known head with 'unknown'. The report then identifies its commit as 'unknown' and falsely warns that the reviews covered different commits, even when werkrevisor reviewed the fetched head. Preserve the known head and distinguish a failed Codex review from a commit mismatch.
Fix it, or reply why it does not hold, and resolve this comment.
Fixed in
581e1f0: a failed Codex run keeps the known head and gives no warning about different commits; a test covers it.⚠️ werkrevisor ERROR for
?, superseded by the result for581e1f0Codex review: critical
The PR changes authenticated Git fetching and server isolation in the shared Codex runner. Mistakes there could disclose Gitea credentials to another server, so human review is required.
Findings
werkrevisor/reviewreport.py:92, as a line commentwerkrevisor/reviewreport.py:113, as a line commentThis review only advises; it holds no merge.
Model `gpt-6-astra`, effort high, tokens in 276743 (cached 236032) / out 2752, 89.0 s, 11 commands; reviewed head `07e512f826e6da2a8e245fb06373ccb9ee89e0a4`.New commits pushed, approval review dismissed automatically according to repository settings
🟢 werkrevisor GREEN for
581e1f0, superseded by the result forb76479d🟢 werkrevisor: GREEN / PASS
Gate: approved by werkrevisor; merge waits for 1 open hint(s): reply to each line comment and resolve it, then restart the review or push a commit.
Change kind:
mixed.Reasons
07e512fnot carried over: 4 approved block(s) are missingFindings
docs/prs/2026-10-11-PR#90-review-report.md:57, severity unknown, counted as medium, hint, as a line commentClassifier
8 of 8 units passed all questions (1 PR document(s)).
Usage: model `openrouter/minimax/minimax-m3`, LLM done, tokens counted before the call 28028, calls 1, tokens in 26919 / out 28723 (reasoning included in out), duration 171.1 s, cost 0.0425 USD (reported), classifier `jev-1.13.0` calls 8, tokens in 6057, cost 0.0003 USD, finding judge `jev-1.13.0` calls 1, tokens in 1319, cost 0.0001 USD, change summary `openrouter/minimax/minimax-m3` via Minimax calls 1, tokens in 5378 / out 1819, cost 0.0038 USD (reported), PR document check `jev-1.13.0` calls 8, tokens in 15651, cost 0.0007 USD, build file judge `jev-1.13.0` calls 8, tokens in 8072, cost 0.0003 USD, phases: path_judge 0.5 s, classifier 0.5 s, change_summary 11.7 s, doc_check 0.6 s, context 0.3 s, model_review 156.5 s, finding_judge 0.2 s, config: defaults + host + repository (.werkrevisor.toml at main3 answers ignored because of a confidence below [classifier] min_confidence.
ee7311e).Reviewed head `581e1f0eb9c2c7fd30e59dac309aee300f1a64b9` against base `ee7311eeaeca4f43c960c6ca6f1664cce5488246`, werkrevisor 0.34.1.
werkrevisor: GREEN / PASS for
581e1f0eb9c2c7fd30e59dac309aee300f1a64b9.The result comment of werkrevisor holds the details.
@@ -0,0 +54,4 @@- Tokens and the values of the Codex login are removed from the report.- The report warns when werkrevisor and Codex reviewed different commits, e.g. after a push between the runs.- README: a paragraph below "Codex runner".- Werkbaum: `#model.codex.report`.Undocumented werkbaum changes, severity unknown, counted as medium (werkrevisor)
hint; Jev: real 0.71, severity 1.8 of 3
The "Solution" section mentions only "Werkbaum:
#model.codex.report.", but the diff todocs/werkrevisor.werkbaumalso flips#model.codex.onfrom in progress to done, changes the mark ofwerkrevisor (XS)and — most notably — markswerkator (XS)from not started to done with the cross-repo reference "werkator PR#153". This side change is not described in the PR document, so a reviewer reading only the document will miss that this PR completes werkator's task in the plan.Fixed in
b76479d: the item names all changed plan nodes.New commits pushed, approval review dismissed automatically according to repository settings
🟢 werkrevisor: GREEN / PASS
Change kind:
mixed.Reasons
581e1f0not carried over: 1 approved block(s) are missingFindings
Werkbaum indentation error at
docs/werkrevisor.werkbaum:116-122, severity unknown, counted as mediumhint; Jev: real 0.84, severity 2.1 of 3
The new entries are indented as siblings of
#model.externaland#model.codexinstead of as children of#model.codex. In the new file#model.codex.onand#model.codex.reportsit at 6 spaces, the same indent as#model.external(line 110), while the existing children#model.codex.evaland#model.codex.gateare at 10 spaces (lines 112, 114). The new childrenwerkrevisorandwerkatorare at 8 spaces, the same indent as#model.codex(line 111), instead of 12 spaces. As a result the four new items visually belong to#model.externalrather than#model.codex. The pre-existing#model.codex.onwas already off by one space (8 spaces versus 9 for#model.codex.eval); this PR makes the inconsistency worse by moving it to 6 spaces, and the comment line" PR#89(line 118) is left at 12 spaces, no longer matching the 2-space gap used elsewhere (e.g." PR#87under#model.codex.eval). The Werkbaum uses indentation to express hierarchy, so the planning hierarchy of this PR is misrepresented.Classifier
8 of 8 units passed all questions (1 PR document(s)).
Usage: model `openrouter/minimax/minimax-m3`, LLM done, tokens counted before the call 27996, calls 1, tokens in 26889 / out 37079 (reasoning included in out), duration 216.7 s, cost 0.0525 USD (reported), classifier `jev-1.13.0` calls 8, tokens in 6076, cost 0.0003 USD, finding judge `jev-1.13.0` calls 1, tokens in 2149, cost 0.0001 USD, change summary `openrouter/minimax/minimax-m3` via Minimax calls 1, tokens in 5378 / out 1972, cost 0.0027 USD (reported), PR document check `jev-1.13.0` calls 8, tokens in 14102, cost 0.0006 USD, build file judge `jev-1.13.0` calls 8, tokens in 8072, cost 0.0003 USD, phases: path_judge 0.6 s, classifier 0.5 s, change_summary 12.8 s, doc_check 0.5 s, context 1.6 s, model_review 199.6 s, finding_judge 0.3 s, config: defaults + host + repository (.werkrevisor.toml at main2 answers ignored because of a confidence below [classifier] min_confidence.
ee7311e).Reviewed head `b76479d47ae5f5db2ce2e2ea3d8390ee58e05908` against base `ee7311eeaeca4f43c960c6ca6f1664cce5488246`, werkrevisor 0.34.1.
werkrevisor: GREEN / PASS for
b76479d47ae5f5db2ce2e2ea3d8390ee58e05908.The result comment of werkrevisor holds the details.