werkrevisor 0.30.2: a dry run writes the model prompt to a file #65
No Assignees
Notifications
Due Date
No due date set.
Dependencies
No dependencies set.
Reference: mi/werkrevisor#65
Reference in New Issue
Block a user
See the PR document.
🤖 Generated with Claude Code
🟡 werkrevisor YELLOW for
dbb5256, superseded by the result forbcdf1b6🟡 werkrevisor: YELLOW / HUMAN_REVIEW
Change kind:
mixed.Reasons
Findings
Incomplete help text at
werkrevisor/cli.py:1122-1123, severity unknown, counted as medium, hint, as a line commentScenario misses the --no-llm case at
docs/prs/2026-10-09-PR#65-prompt-out.md:52-58, severity unknown, counted as medium, hint, as a line commentClassifier
5 of 6 units passed all questions (1 PR document(s)).
Usage: model `openrouter/minimax/minimax-m3`, LLM done, tokens counted before the call 33667, calls 1, tokens in 32573 / out 13908 (reasoning included in out), duration 79.4 s, cost 0.0264 USD (reported), classifier `jev-1.13.0` calls 6, tokens in 4904, cost 0.0002 USD, finding judge `jev-1.13.0` calls 2, tokens in 2860, cost 0.0001 USD, change summary `openrouter/minimax/minimax-m3` via Minimax calls 1, tokens in 3498 / out 1212, cost 0.0025 USD (reported), PR document check `jev-1.13.0` calls 6, tokens in 9263, cost 0.0004 USD, build file judge `jev-1.13.0` calls 5, tokens in 4706, cost 0.0002 USD, phases: path_judge 0.6 s, classifier 0.4 s, change_summary 7.5 s, doc_check 0.6 s, context 0.2 s, model_review 69.4 s, finding_judge 0.3 s, config: defaults + host + repository (.werkrevisor.toml at main3 answers ignored because of a confidence below [classifier] min_confidence.
e65eea0).Reviewed head `dbb5256187b0e8a336dd7c4e59c5af9e509839c1` against base `e65eea0026c4b499de6c8868310fc4baf18e2c9d`, werkrevisor 0.30.1.
werkrevisor: YELLOW / HUMAN_REVIEW for
dbb5256187b0e8a336dd7c4e59c5af9e509839c1.The result comment of werkrevisor holds the details.
@@ -0,0 +49,4 @@- [test_prompt_out.PromptOutTest.test_the_file_holds_the_prompt_after_the_size_checks](../../tests/test_prompt_out.py)#### Scenario#65.03: only a dry run writes the promptScenario misses the --no-llm case, severity unknown, counted as medium (werkrevisor)
hint; Jev: real 0.82, severity 1.8 of 3
The title of Scenario#65.03 says "only a dry run writes the prompt" and the Given/When/Then only describes
--prompt-outwithout--no-publish, but the test (and the implementation inmain()) also reject--prompt-outtogether with--no-llm. A reader of the PR document who only looks at the scenario will not know that--prompt-out --no-llmis also forbidden; the scenario should add an "And" line covering the--no-llmcase so the scenario matches the verification.Fixed in
bcdf1b6: Scenario#65.03 covers--no-llmtoo, as the test does.@@ -1103,6 +1119,8 @@ def parse_args(argv):parser.add_argument("--no-llm", action="store_true", help="apply the deterministic rules only")parser.add_argument("--allow-closed", action="store_true",help="with --no-publish: also review a merged or closed PR (for offline measurement)")parser.add_argument("--prompt-out", metavar="FILE",Incomplete help text, severity unknown, counted as medium (werkrevisor)
hint; Jev: real 0.69, severity 2.1 of 3
The CLI help for
--prompt-outstates it is "with --no-publish", but the validation inmain()also rejects--prompt-outcombined with--no-llm. A user invoking--prompt-out FILE --no-llmwill get a configuration error that the help text did not warn about, since the help implies that omitting the model call (--no-llm) is allowed. Either drop--no-llmfrom the check (and still pass--prompt-outwithout it) or document the second constraint in the help string.Fixed in
bcdf1b6: the help names both conditions, with--no-publishand without--no-llm.Review round of
dbb52561: Scenario#65.01 has a plain Given/When/Then now, and both hints are fixed inbcdf1b6.🟡 werkrevisor YELLOW for
bcdf1b6, superseded by the result for9289382🟡 werkrevisor: YELLOW / HUMAN_REVIEW
Change kind:
mixed.Reasons
Findings
Misleading Error at
tools/dry-run:39, severity unknown, counted as medium, hint, as a line commentUnclear Rationale at
docs/prs/2026-10-09-PR#65-prompt-out.md:54-55, severity unknown, counted as medium, hint, as a line commentClassifier
5 of 6 units passed all questions (1 PR document(s)).
Usage: model `openrouter/minimax/minimax-m3`, LLM done, tokens counted before the call 33565, calls 1, tokens in 32482 / out 22159 (reasoning included in out), duration 105.7 s, cost 0.0363 USD (reported), classifier `jev-1.13.0` calls 6, tokens in 4937, cost 0.0002 USD, finding judge `jev-1.13.0` calls 2, tokens in 2577, cost 0.0001 USD, change summary `openrouter/minimax/minimax-m3` via Minimax calls 1, tokens in 3509 / out 2168, cost 0.0036 USD (reported), PR document check `jev-1.13.0` calls 6, tokens in 8252, cost 0.0003 USD, build file judge `jev-1.13.0` calls 5, tokens in 4717, cost 0.0002 USD, phases: path_judge 0.4 s, classifier 0.7 s, change_summary 8.7 s, doc_check 0.5 s, context 1.6 s, model_review 93.3 s, finding_judge 0.3 s, config: defaults + host + repository (.werkrevisor.toml at maine65eea0).Reviewed head `bcdf1b6e0648bbbdeef78ce60f90fbc9e06e1daa` against base `e65eea0026c4b499de6c8868310fc4baf18e2c9d`, werkrevisor 0.30.1.
werkrevisor: YELLOW / HUMAN_REVIEW for
bcdf1b6e0648bbbdeef78ce60f90fbc9e06e1daa.The result comment of werkrevisor holds the details.
@@ -0,0 +51,4 @@#### Scenario#65.03: only a dry run writes the promptSo that a review on the review machine never ends without a model review because of this option,Unclear Rationale, severity unknown, counted as medium (werkrevisor)
hint; Jev: real 0.75, severity 1.3 of 3
The "So that" paragraph of Scenario#65.03 does not make clear why --prompt-out combined with --no-llm must fail; the second sentence about a dry run with --no-llm not ending before the prompt exists is hard to parse.
Fixed in
9289382: the--no-llmcase is Scenario#65.04 with its own reason: with--no-llm, werkrevisor never builds a prompt.@@ -33,1 +36,4 @@mkdir -p "$(dirname "$4")"fiif [[ "${3:-}" == "--llm" || "${3:-}" == "--prompt-out" ]]; then: "${OPENROUTER_API_KEY:?OPENROUTER_API_KEY must be set for --llm}"Misleading Error, severity unknown, counted as medium (werkrevisor)
hint; Jev: real 0.91, severity 1.6 of 3
The OPENROUTER_API_KEY guard triggers for both --llm and --prompt-out, but the message only mentions --llm, so a user invoking --prompt-out without the key gets a confusing error.
Fixed in
9289382: the message names--llmand--prompt-out.Review round of
bcdf1b6e: Scenario#65.03 is split into two plain scenarios, and both hints are fixed in9289382.🟠 werkrevisor ORANGE for
9289382, superseded by the result fore0e74ff🟠 werkrevisor: ORANGE / FINDINGS
Change kind:
mixed.Reasons
Findings
tools/dry-run:34-37, severity unknown, counted as medium, ORANGE, as a line commentClassifier
7 of 7 units passed all questions (1 PR document(s)).
Usage: model `openrouter/minimax/minimax-m3`, LLM done, tokens counted before the call 33844, calls 1, tokens in 32740 / out 20737 (reasoning included in out), duration 129.1 s, cost 0.0347 USD (reported), classifier `jev-1.13.0` calls 7, tokens in 5786, cost 0.0002 USD, finding judge `jev-1.13.0` calls 1, tokens in 1617, cost 0.0001 USD, change summary `openrouter/minimax/minimax-m3` via Minimax calls 1, tokens in 3543 / out 1236, cost 0.0025 USD (reported), PR document check `jev-1.13.0` calls 7, tokens in 9453, cost 0.0004 USD, build file judge `jev-1.13.0` calls 5, tokens in 4757, cost 0.0002 USD, phases: path_judge 0.7 s, classifier 0.5 s, change_summary 12.7 s, doc_check 0.5 s, context 0.3 s, model_review 113.5 s, finding_judge 0.3 s, config: defaults + host + repository (.werkrevisor.toml at main4 answers ignored because of a confidence below [classifier] min_confidence.
e65eea0).Reviewed head `9289382cef0f9af963956679c3ad0b717c56a6b5` against base `e65eea0026c4b499de6c8868310fc4baf18e2c9d`, werkrevisor 0.30.1.
werkrevisor: ORANGE / FINDINGS for
9289382cef0f9af963956679c3ad0b717c56a6b5.The result comment of werkrevisor holds the details.
@@ -31,3 +33,2 @@werkrevisor_args=(--no-llm)if [[ "${3:-}" == "--llm" ]]; then: "${OPENROUTER_API_KEY:?OPENROUTER_API_KEY must be set for --llm}"if [[ "${3:-}" == "--prompt-out" ]]; thenLoose path check, severity unknown, counted as medium (werkrevisor)
ORANGE; Jev: real 0.81, severity 2.9 of 3
The bash pattern
[[ "${4:-}" == build/* ]]matches any string starting withbuild/, includingbuild/../foo. The Docker mount-v "$PWD/build":/work/buildcontains the impact (writes outside the bind mount go to the container's ephemeral filesystem and are discarded by--rm), so this is not a security vulnerability. However, the check does not fulfill its stated purpose of ensuring the file is underbuild/: a typo likebuild/../prompt.jsonis accepted and the file is silently lost. A stricter check (rejecting..segments or canonicalizing withrealpath --relative-to=build) would catch such typos.Fixed in
e0e74ff: the check refuses a path with a..segment and a barebuild/.Review round of
9289382c: the ORANGE finding is fixed ine0e74ff.🟢 werkrevisor: GREEN / PASS
Change kind:
mixed.Reasons
Classifier
7 of 7 units passed all questions (1 PR document(s)).
Usage: model `openrouter/minimax/minimax-m3`, LLM done, tokens counted before the call 33881, calls 1, tokens in 32776 / out 16894 (reasoning included in out), duration 105.2 s, cost 0.0301 USD (reported), classifier `jev-1.13.0` calls 7, tokens in 5786, cost 0.0002 USD, change summary `openrouter/minimax/minimax-m3` via Minimax calls 1, tokens in 3559 / out 2859, cost 0.0039 USD (reported), PR document check `jev-1.13.0` calls 7, tokens in 10272, cost 0.0004 USD, build file judge `jev-1.13.0` calls 5, tokens in 4773, cost 0.0002 USD, phases: path_judge 0.5 s, classifier 0.6 s, change_summary 21.1 s, doc_check 0.5 s, context 0.2 s, model_review 81.8 s, config: defaults + host + repository (.werkrevisor.toml at main4 answers ignored because of a confidence below [classifier] min_confidence.
e65eea0).Reviewed head `e0e74ff2ec04bfd2c23d5450548e93eb0653aa22` against base `e65eea0026c4b499de6c8868310fc4baf18e2c9d`, werkrevisor 0.30.1.
werkrevisor: GREEN / PASS for
e0e74ff2ec04bfd2c23d5450548e93eb0653aa22.The result comment of werkrevisor holds the details.