werkrevisor 0.67.0: the cost of the PR Lens diagrams in the usage record, the result comment and the statistics #140
mi
No Assignees
Notifications
Due Date
No due date set.
Dependencies
No dependencies set.
Reference: mi/werkrevisor#140
Reference in New Issue
Block a user
See the PR document under docs/prs/.
🤖 Generated with Claude Code
WIP: werkrevisor 0.66.0: the cost of the PR Lens diagrams in the usage record, the result comment and the statisticsto WIP: werkrevisor 0.67.0: the cost of the PR Lens diagrams in the usage record, the result comment and the statistics🟡 werkrevisor YELLOW for
f0b4e4a, superseded by the result for15dd358🟡 werkrevisor: YELLOW / HUMAN_REVIEW
Change kind:
mixed.Reasons
Classifier
9 of 9 units passed all questions (1 PR document(s)).
Usage: model `openrouter/minimax/minimax-m3`, LLM failed (TimeoutError), tokens counted before the call 56964, duration 733.0 s, cost unknown, classifier `jev-1.14.0` calls 9, tokens in 7181, cost 0.0003 USD, excerpt selection `openrouter/openai/gpt-6-luna` calls 1, tokens in 25495 / out 311, cost 0.0033 USD (reported), PR document check `jev-1.14.0` calls 12, tokens in 35192, cost 0.0015 USD, build file judge `jev-1.14.0` calls 10, tokens in 10982, cost 0.0005 USD, phases: path_judge 0.6 s, classifier 0.5 s, doc_check 0.6 s, context 8.1 s, model_review 722.7 s, config: defaults + host + repository (.werkrevisor.toml at main5 answers ignored because of a confidence below [classifier] min_confidence.
bad4782).Reviewed head `f0b4e4aa186b8cf5db81a890c520f2f6e4f1fdc3` against base `bad4782b99657b8a662db8b9333722c04619962e`, werkrevisor 0.65.2.
Diagrams
Drawn by a model from the diff with PR Lens: a view of what the PR touches, not a statement of what it does.
werkrevisor: YELLOW / HUMAN_REVIEW for
f0b4e4aa186b8cf5db81a890c520f2f6e4f1fdc3.The result comment of werkrevisor holds the details.
@@ -0,0 +91,4 @@### Test of the Release of a Held MergeThis paragraph breaks the writing rule of at most 5 sentences on purpose.Writing rule: paragraph with 6 sentences, more than 5 (werkrevisor writing rule)
Fix the line, or reply why it stays, and resolve this comment.
Fixed in
15dd358: the deliberate paragraph is removed; the test of the release moves to a simpler PR.Answer to the review of
f0b4e4a:/usr/bin/node, the same Node.js that runs PR Lens in production. No existing test is disabled. This is left to the human.Codex review of
f0b4e4aa186b8cf5db81a890c520f2f6e4f1fdc3@@ -203,1 +204,3 @@boundedSum("cost_usd, excerpt_cost_usd and doc_summary_cost_usd", listOf(it, excerptCost, docSummaryCost))boundedSum("cost_usd, excerpt_cost_usd, doc_summary_cost_usd and diagram_cost_usd",listOf(it, excerptCost, docSummaryCost, diagramCost),[medium] Unknown diagram costs are presented as a complete total (Codex review)
When diagram_calls is positive but diagram_cost_usd is null, boundedSum silently drops the unknown cost and preserves the review's cost_status. For a reported review cost, statistics therefore show the partial total as known, with unknownCostReviews unchanged and no estimated-cost indicator. Distinguish an unpriced drawing from a run that drew nothing.
Fix it, or reply why it does not hold, and resolve this comment.
Fixed in
00a7b82: a record withdiagram_calls > 0and nodiagram_cost_usdmakes the cost of the review unknown, so it counts inunknownCostReviews. Test: Scenario#140.03 inUsageApiIntegrationTest.@@ -68,5 +94,6 @@ try {? error.details.split("\n").map((l) => l.trim()).filter(Boolean).slice(0, 2).join(" ") : "";process.stderr.write(`pr-lens: ${error?.code ?? error?.name ?? "Error"}: ${error?.message ?? error}`+ `${detail ? ` (${detail})` : ""}\n`);process.stdout.write(JSON.stringify({ usage: usageOut() }));[medium] A timeout loses usage from completed calls (Codex review)
Usage is emitted only when drawing finishes or throws. If the first model call completes but the repair call exceeds timeout_seconds, subprocess.run kills Node without executing this catch block. prlens.draw then raises DiagramError without usage, so the record reports zero diagram calls and loses the first call's known cost.
Fix it, or reply why it does not hold, and resolve this comment.
Fixed in
00a7b82:pr_lens.mjswrites the usage to stderr after each call; on a timeoutprlens.drawreads the last line from the killed process. Test:test_a_killed_pr_lens_names_the_usage_of_its_finished_calls.Codex review: not critical
The changes add diagram usage accounting and reporting without changing authorization, credentials, merge decisions, or existing data. The identified risks concern incomplete cost reporting.
Findings
werkrevisor/pr_lens.mjs:97, as a line commentstatistics/src/main/kotlin/de/javagil/werkrevisor/statistics/UsageApiController.kt:206, as a line commentThe merge waits for the status
Model `gpt-6-astra`, effort high, tokens in 459108 (cached 411648) / out 2508, 100.2 s, 14 commands; reviewed head `f0b4e4aa186b8cf5db81a890c520f2f6e4f1fdc3`.ai-review/codex: every line comment resolved, and the approval of a human for a critical PR or for a finding that is only here.🟠 werkrevisor ORANGE for
15dd358, superseded by the result for00a7b82🟠 werkrevisor: ORANGE / FINDINGS
Change kind:
mixed.Reasons
Findings
[medium] Failed Call at
werkrevisor/pr_lens.mjs:22-30, ORANGE, as a line comment[medium] Callback Contract at
werkrevisor/cli.py:1323-1328, ORANGE, as a line comment[medium] Missing Cost at
werkrevisor/cli.py:1349-1362, hint, as a line commentClassifier
8 of 8 units passed all questions (1 PR document(s)).
Usage: model `openrouter/minimax/minimax-m3`, LLM done, tokens counted before the call 57780, calls 1, tokens in 55744 / out 34265 (reasoning included in out), duration 404.1 s, cost 0.0578 USD (reported), classifier `jev-1.14.0` calls 8, tokens in 6459, cost 0.0003 USD, finding judge `jev-1.14.0` calls 3, tokens in 5047, cost 0.0002 USD, excerpt selection `openrouter/openai/gpt-6-luna` calls 1, tokens in 25495 / out 380, cost 0.0034 USD (reported), PR document check `jev-1.14.0` calls 12, tokens in 33992, cost 0.0014 USD, build file judge `jev-1.14.0` calls 10, tokens in 10982, cost 0.0005 USD, phases: path_judge 0.6 s, classifier 0.4 s, doc_check 0.7 s, context 7.7 s, model_review 393.5 s, finding_judge 0.2 s, config: defaults + host + repository (.werkrevisor.toml at main3 answers ignored because of a confidence below [classifier] min_confidence.
bad4782).Reviewed head `15dd3582cc28789f6a20e96f0611848837dd5636` against base `bad4782b99657b8a662db8b9333722c04619962e`, werkrevisor 0.65.2.
Diagrams
Drawn by a model from the diff with PR Lens: a view of what the PR touches, not a statement of what it does.
werkrevisor: ORANGE / FINDINGS for
15dd3582cc28789f6a20e96f0611848837dd5636.The result comment of werkrevisor holds the details.
@@ -1321,0 +1320,4 @@def _draw(config, pr, deps, diff, result):"""The usage goes into result.diagrams, also of a drawing that failed after a model call."""try:drawing = deps.draw_diagrams(config, pr, diff)[medium] Callback Contract (werkrevisor)
ORANGE; Jev: real 0.79, severity 2.9 of 3
Deps.draw_diagramsis still documented as returning a list of(name, SVG)tuples, but_drawnow unconditionally reads.usageand.filesfrom aDrawing. A callback following the documented contract raisesAttributeError, so its diagrams and usage are skipped. Update the contract and callers, or support the former return shape.Fixed in
00a7b82: the comment ofDeps.draw_diagramsnames the new contract,prlens.Drawingandprlens.DiagramErrorwith its usage; all callers return aDrawing.@@ -1336,3 +1347,3 @@if not diff:returndrawn = deps.draw_diagrams(config, pr, diff)drawn = _draw(config, pr, deps, diff, result)[medium] Missing Cost (werkrevisor)
hint; Jev: real 0.80, severity 2.4 of 3
The new cost line is added only when at least one asset was attached. If PR Lens returns no SVGs, or if every
create_comment_assetcall fails,_drawhas already recorded a paid model call but the result comment receives no Diagrams section and never names that cost. Add the usage section when usage exists even if no asset could be attached.Fixed in
00a7b82: a drawing that cost something but attached no diagram gets the section "Diagrams" with "No diagram was attached." and the cost line. Test:test_a_drawing_without_a_diagram_names_its_cost_in_the_comment.@@ -13,0 +19,4 @@// OpenRouter adds the cost of the call to its usage only on request.request = { ...init, body: JSON.stringify({ ...JSON.parse(init.body), usage: { include: true } }) };}const response = await originalFetch(url, request);[medium] Failed Call (werkrevisor)
ORANGE; Jev: real 0.90, severity 2.9 of 3
If
originalFetchrejects after OpenRouter accepted the request, such as when the connection drops while reading the response, the wrapper never reaches its accounting block. The outer handler then emitscalls: 0and a known$0cost even though the request may have been billed. Count the attempted call and mark its cost unknown whenfetchrejects.Fixed in
00a7b82: the wrapper counts the call beforefetch, and a rejectedfetchmakes the cost unknown. Test:test_a_call_without_an_answer_counts_with_an_unknown_cost.Codex review of
15dd3582cc28789f6a20e96f0611848837dd5636@@ -203,1 +204,3 @@boundedSum("cost_usd, excerpt_cost_usd and doc_summary_cost_usd", listOf(it, excerptCost, docSummaryCost))boundedSum("cost_usd, excerpt_cost_usd, doc_summary_cost_usd and diagram_cost_usd",listOf(it, excerptCost, docSummaryCost, diagramCost),[medium] Unknown diagram costs become apparently complete totals (Codex review)
The script deliberately returns diagram_cost_usd=null when a response lacks cost information. boundedSum drops that null, while costStatus remains the review's original status. Thus a reported $0.10 review with paid diagram calls of unknown cost appears as a fully known $0.10 total, without increasing unknownCostReviews or marking the total estimated. Distinguish missing diagram costs after calls from diagrams that were never requested.
Fix it, or reply why it does not hold, and resolve this comment.
Fixed in
00a7b82, the same finding as onf0b4e4a(this review was of15dd358):diagram_calls > 0withoutdiagram_cost_usdmakes the review cost unknown.@@ -60,6 +85,7 @@ try {flows: document.flows.length,attempts,truncated: truncatedAt !== undefined,usage: usageOut(),[medium] Completed-call usage is lost when drawing times out (Codex review)
Usage is emitted only when drawing finishes or throws. If the first paid response completes but a repair call exceeds timeout_seconds, subprocess.run kills Node before either output path runs. The Python timeout handler then supplies no usage, so the record reports zero diagram calls and no cost despite the completed paid call. Preserve usage incrementally and recover it on timeout.
Fix it, or reply why it does not hold, and resolve this comment.
Fixed in
00a7b82, the same finding as onf0b4e4a(this review was of15dd358): the usage goes to stderr after each call and survives a timeout.Codex review: not critical
The change adds diagram usage accounting and display without changing authorization, merge decisions, CI gates, or existing data. Its failure modes affect reported costs rather than security or merge safety.
Findings
werkrevisor/pr_lens.mjs:88, as a line commentstatistics/src/main/kotlin/de/javagil/werkrevisor/statistics/UsageApiController.kt:206, as a line commentThe merge waits for the status
Model `gpt-6-astra`, effort high, tokens in 398128 (cached 351360) / out 2045, 84.7 s, 8 commands; reviewed head `15dd3582cc28789f6a20e96f0611848837dd5636`.ai-review/codex: every line comment resolved, and the approval of a human for a critical PR or for a finding that is only here.⚠️ werkrevisor: ERROR / review not completed, human action needed
Change kind:
mixed.Reasons
Findings
Issue at
werkrevisor/pr_lens.mjs, severity unknown, counted as mediumhint; Jev: real 0.15, severity 0.5 of 3
Classifier
8 of 8 units passed all questions (1 PR document(s)).
Usage: model `openrouter/minimax/minimax-m3`, LLM done, tokens counted before the call 60003, calls 1, tokens in 57881 / out 64000 (reasoning included in out), duration 676.1 s, cost 0.0941 USD (reported), classifier `jev-1.14.0` calls 8, tokens in 6576, cost 0.0003 USD, finding judge `jev-1.14.0` calls 1, tokens in 576, cost 0.0000 USD, excerpt selection `openrouter/openai/gpt-6-luna` calls 1, tokens in 25869 / out 380, cost 0.0034 USD (reported), PR document check `jev-1.14.0` calls 12, tokens in 36682, cost 0.0015 USD, build file judge `jev-1.14.0` calls 10, tokens in 12224, cost 0.0005 USD, phases: path_judge 0.5 s, classifier 0.4 s, doc_check 0.5 s, context 6.5 s, model_review 667.3 s, finding_judge 0.2 s, config: defaults + host + repository (.werkrevisor.toml at main3 answers ignored because of a confidence below [classifier] min_confidence.
bad4782).Reviewed head `00a7b8264a859317fcf75eb1a0e83443f0e5b48a` against base `bad4782b99657b8a662db8b9333722c04619962e`, werkrevisor 0.65.2.
View command line instructions
Checkout
From your project repository, check out a new branch and test the changes.