werkrevisor 0.11.0: exact token count, focused style review #9

Merged
mi merged 9 commits from claude-code/exact-tokens-and-focused-style-review into main 2026-10-05 18:30:13 +02:00
Collaborator

werkrevisor 0.11.0: exact token count before the model call, and a focused style review.

  • werkrevisor counts the exact prompt PR-Agent sends, with o200k_base and [model] token_count_factor (default 1.15), instead of estimating chars/3.5.
  • The model no longer reports line length or sentence counts; werkrevisor checks them itself.

Details: the PR document under docs/prs/.

🤖 Generated with Claude Code

werkrevisor 0.11.0: exact token count before the model call, and a focused style review. - werkrevisor counts the exact prompt PR-Agent sends, with o200k_base and `[model] token_count_factor` (default 1.15), instead of estimating chars/3.5. - The model no longer reports line length or sentence counts; werkrevisor checks them itself. Details: the PR document under `docs/prs/`. 🤖 Generated with [Claude Code](https://claude.com/claude-code)
claude-code added 5 commits 2026-10-05 17:44:23 +02:00
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
werkrevisor wraps PR-Agent's AI handler and counts the system and user prompt
right before each model call, times the new host key [model] token_count_factor.
The count decides about the input limit, the worst-case cost and the key budget query.
PR-Agent gets a window of twice the input limit, so a too long prompt reaches the count unshortened;
a prompt PR-Agent still prunes or refuses is YELLOW.
Without the tokenizer, the estimate of 0.10.0 applies and is named "token count estimated".
The usage record holds the count next to the reported prompt tokens.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
The model miscounted them: on PR#8 it called a 120-character line too long
and a wrap after a comma a violation.
It now reports for docs only bloated or mannered prose, unclear statements,
and contradictions between the PR document and the code.
.werkrevisor.toml is read from the base commit, so this applies to werkrevisor's own PRs after the merge.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
The README describes the token count, the evidence against the old estimate,
the PR-Agent window and the fallback; the example host configs set token_count_factor.
The test of the version skew reason uses the installed version.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
docs: complete the PR#9 document with test links, answered questions and follow-ups
ai-review/assessment human review required; a human approval unlocks the merge; review requested from mi
ai-review/gate build succeeded after 00:03
werkator build succeeded after 00:19
e9193fe66b
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
claude-code marked the pull request as ready for review 2026-10-05 17:44:23 +02:00
Collaborator

🟡 werkrevisor: YELLOW / HUMAN_REVIEW

Change kind: mixed.

Reasons

  • YELLOW test changes are not purely additive: tests/fakes.py:150 changes an existing test line, tests/fakes.py:161 changes an existing test line, tests/test_config_budget_redact.py:39 changes an existing test line and 25 more
  • YELLOW change kind 'mixed' (main code, tests, other files, docs); AI-approvable are only: docs only, tests only added, docs and tests, refactoring with unchanged tests, feature with purely additive tests, harmless files only
  • YELLOW risky paths changed: .werkrevisor.toml (built in), config/hsadmin-ng.toml (config/), config/werkprobe.toml (config/) and 2 more
  • YELLOW possible secrets in added lines: tests/test_token_count.py:99 (private_key)
Usage: model `openrouter/anthropic/claude-sonnet-5.5`, LLM skipped (possible secrets in the diff), duration 0.3 s, no model call, cost 0 USD, config: defaults + host + repository (.werkrevisor.toml at base 4cf8b99).
Reviewed head `e9193fe66b37facab2e08444f06e84816255166b` against base `4cf8b9953535fa47f7a08f058ee4b44045e4c83d`, werkrevisor 0.10.0.
<!-- werkrevisor:e9193fe66b37facab2e08444f06e84816255166b --> <!-- werkrevisor-verdict:YELLOW --> ## 🟡 werkrevisor: YELLOW / HUMAN_REVIEW Change kind: `mixed`. ### Reasons - **YELLOW** test changes are not purely additive: tests/fakes.py:150 changes an existing test line, tests/fakes.py:161 changes an existing test line, tests/test_config_budget_redact.py:39 changes an existing test line and 25 more - **YELLOW** change kind 'mixed' (main code, tests, other files, docs); AI-approvable are only: docs only, tests only added, docs and tests, refactoring with unchanged tests, feature with purely additive tests, harmless files only - **YELLOW** risky paths changed: .werkrevisor.toml (built in), config/hsadmin-ng.toml (config/**), config/werkprobe.toml (config/**) and 2 more - **YELLOW** possible secrets in added lines: tests/test_token_count.py:99 (private_key) <sub> Usage: model `openrouter/anthropic/claude-sonnet-5.5`, LLM skipped (possible secrets in the diff), duration 0.3 s, no model call, cost 0 USD, config: defaults + host + repository (.werkrevisor.toml at base 4cf8b99).<br> Reviewed head `e9193fe66b37facab2e08444f06e84816255166b` against base `4cf8b9953535fa47f7a08f058ee4b44045e4c83d`, werkrevisor 0.10.0. </sub>
ai-reviewer requested review from mi 2026-10-05 17:44:27 +02:00
claude-code added 1 commit 2026-10-05 17:45:28 +02:00
Test data assembles the private-key marker, so the secret scan does not skip the model review
ai-review/assessment review not completed, human action needed
werkator build succeeded after 00:05
ai-review/gate build failed after 00:14
eb2823f3d7
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Collaborator

⚠️ werkrevisor: ERROR / review not completed, human action needed

Change kind: mixed.

Reasons

  • YELLOW test changes are not purely additive: tests/fakes.py:150 changes an existing test line, tests/fakes.py:161 changes an existing test line, tests/test_config_budget_redact.py:39 changes an existing test line and 25 more
  • YELLOW change kind 'mixed' (main code, tests, other files, docs); AI-approvable are only: docs only, tests only added, docs and tests, refactoring with unchanged tests, feature with purely additive tests, harmless files only
  • YELLOW risky paths changed: .werkrevisor.toml (built in), config/hsadmin-ng.toml (config/), config/werkprobe.toml (config/) and 2 more
  • YELLOW 4 changed files were given to the AI only as diff, not in full, so it could not check them against the rest of their file (all full files together would exceed [review] max_full_files_chars = 120000): werkrevisor/cli.py, werkrevisor/config.py, werkrevisor/report.py and 1 more
  • ERROR PR-Agent could not read: werkrevisor/review.py, docs/prs/2026-10-05-PR#9-exact-token-count-and-focused-style-review.md
  • ORANGE model reports 2 key issues ([review] red_min_findings = 2)
  • YELLOW model recommends merge with caution

Findings

  1. Possible Bug at werkrevisor/review.py:223-229

    The admit callback runs inside Admitting.chat_completion on every PR-Agent model call, including the retry. On the second attempt, admit queries the OpenRouter key again. A transient key-query failure then raises NoModelCall after the first call has already cost money, with result.cost_usd set to 0.0 and cost status "none". cli.py also overwrites budget_remaining_before on every call. The retry path is a real scenario because max_retries = 1 is the default, but how PR-Agent handles an exception raised in the retry is not visible in the diff.

  2. Cost Misreported at werkrevisor/cli.py:160-164

    The NoModelCall handler sets cost_usd = 0.0 and cost_status = "none" regardless of whether an earlier model call already happened (for example in a retry or a second PR-Agent call). The reported cost can then understate real spend. The first call is unlikely to be followed by a stop, so the impact is narrow.

Usage: model `openrouter/anthropic/claude-sonnet-5.5`, LLM done, calls 1, tokens in 91385 / out 526 (reasoning included in out), duration 12.7 s, cost 0.1880 USD (reported), config: defaults + host + repository (.werkrevisor.toml at base 4cf8b99).
Reviewed head `eb2823f3d7305937f286d4e485a71ddca4064dfc` against base `4cf8b9953535fa47f7a08f058ee4b44045e4c83d`, werkrevisor 0.10.0.
<!-- werkrevisor-error:eb2823f3d7305937f286d4e485a71ddca4064dfc --> ## ⚠️ werkrevisor: ERROR / review not completed, human action needed Change kind: `mixed`. ### Reasons - **YELLOW** test changes are not purely additive: tests/fakes.py:150 changes an existing test line, tests/fakes.py:161 changes an existing test line, tests/test_config_budget_redact.py:39 changes an existing test line and 25 more - **YELLOW** change kind 'mixed' (main code, tests, other files, docs); AI-approvable are only: docs only, tests only added, docs and tests, refactoring with unchanged tests, feature with purely additive tests, harmless files only - **YELLOW** risky paths changed: .werkrevisor.toml (built in), config/hsadmin-ng.toml (config/**), config/werkprobe.toml (config/**) and 2 more - **YELLOW** 4 changed files were given to the AI only as diff, not in full, so it could not check them against the rest of their file (all full files together would exceed [review] max_full_files_chars = 120000): werkrevisor/cli.py, werkrevisor/config.py, werkrevisor/report.py and 1 more - **ERROR** PR-Agent could not read: werkrevisor/review.py, docs/prs/2026-10-05-PR#9-exact-token-count-and-focused-style-review.md - **ORANGE** model reports 2 key issues ([review] red_min_findings = 2) - **YELLOW** model recommends merge with caution ### Findings 1. **Possible Bug** at `werkrevisor/review.py:223-229` The `admit` callback runs inside `Admitting.chat_completion` on every PR-Agent model call, including the retry. On the second attempt, `admit` queries the OpenRouter key again. A transient key-query failure then raises `NoModelCall` after the first call has already cost money, with `result.cost_usd` set to 0.0 and cost status "none". `cli.py` also overwrites `budget_remaining_before` on every call. The retry path is a real scenario because `max_retries = 1` is the default, but how PR-Agent handles an exception raised in the retry is not visible in the diff. 2. **Cost Misreported** at `werkrevisor/cli.py:160-164` The `NoModelCall` handler sets `cost_usd = 0.0` and `cost_status = "none"` regardless of whether an earlier model call already happened (for example in a retry or a second PR-Agent call). The reported cost can then understate real spend. The first call is unlikely to be followed by a stop, so the impact is narrow. <sub> Usage: model `openrouter/anthropic/claude-sonnet-5.5`, LLM done, calls 1, tokens in 91385 / out 526 (reasoning included in out), duration 12.7 s, cost 0.1880 USD (reported), config: defaults + host + repository (.werkrevisor.toml at base 4cf8b99).<br> Reviewed head `eb2823f3d7305937f286d4e485a71ddca4064dfc` against base `4cf8b9953535fa47f7a08f058ee4b44045e4c83d`, werkrevisor 0.10.0. </sub>
claude-code added 2 commits 2026-10-05 17:51:19 +02:00
A retry of the model call queried the key again; a failed second query stopped a review
whose first call already cost money, and overwrote budget_remaining_before.
Now only the first call queries the key; a later call passes unless its prompt grew above the limit.
A stop after an earlier call keeps PR-Agent's run details and cost instead of "no model call".
The run details are read inside PR-Agent's event loop, where its context variable lives.
The PR document gets Scenario#9.08 and #9.09 and names the false "could not read" ERROR as a follow-up.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
review: only PR-Agent's notice below a file header marks a file unreadable (PR#9)
werkator build succeeded after 00:06
ai-review/assessment review not completed, human action needed
ai-review/gate build failed after 00:14
91561fb9d2
The coverage check flagged a file when the notice text appeared anywhere in its section,
also in a diff line or the appended full files; the self-review of PR#9 got a false ERROR
for werkrevisor/review.py and its PR document.
The check now matches the notice line that PR-Agent 0.47.0 writes directly below the header.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Collaborator

⚠️ werkrevisor: ERROR / review not completed, human action needed

Change kind: mixed.

Reasons

  • YELLOW test changes are not purely additive: tests/fakes.py:126 changes an existing test line, tests/fakes.py:133 changes an existing test line, tests/fakes.py:150 changes an existing test line and 28 more
  • YELLOW change kind 'mixed' (main code, tests, other files, docs); AI-approvable are only: docs only, tests only added, docs and tests, refactoring with unchanged tests, feature with purely additive tests, harmless files only
  • YELLOW risky paths changed: .werkrevisor.toml (built in), config/hsadmin-ng.toml (config/), config/werkprobe.toml (config/) and 2 more
  • YELLOW 4 changed files were given to the AI only as diff, not in full, so it could not check them against the rest of their file (all full files together would exceed [review] max_full_files_chars = 120000): werkrevisor/cli.py, werkrevisor/config.py, werkrevisor/report.py and 1 more
  • ERROR PR-Agent could not read: tests/test_pr_agent_integration.py, werkrevisor/review.py, docs/prs/2026-10-05-PR#9-exact-token-count-and-focused-style-review.md
  • ORANGE model reports 2 key issues ([review] red_min_findings = 2)
  • YELLOW model recommends merge with caution

Findings

  1. Doc contradicts code at docs/prs/2026-10-05-PR#9-exact-token-count-and-focused-style-review.md:75-88

    Scenario#9.03 and the Solution section say a prompt that PR-Agent prunes or refuses yields YELLOW with a reason naming the window. In run_model_review, the window_reason is raised only when PromptShortened is caught or when no call was made and FallbackEligibleError is the cause. A refusal after an earlier model call (for example a retry) is not mapped and re-raises as a generic error. The document also gives no explanation for the 9.03 case beyond the first call.

  2. Pruning flag unused at werkrevisor/review.py:216-228

    _admitting receives pruned as a list, so if pruned: is true as soon as recording appends. But ModelRun.pruned and remaining_files are still declared, and the handler raises PromptShortened for every later call, so a retry after pruning is converted to a NoModelCall with details that may hold usage. This is uncertain: the visible diff does not show whether ModelRun is still populated with pruned or remaining_files, and coverage_reasons may still read them. If it does, they are now dead and the "clipped" ERROR path is unreachable, which should be confirmed.

Usage: model `openrouter/anthropic/claude-sonnet-5.5`, LLM done, calls 1, tokens in 99424 / out 621 (reasoning included in out), duration 13.0 s, cost 0.2051 USD (reported), config: defaults + host + repository (.werkrevisor.toml at base 4cf8b99).
Reviewed head `91561fb9d2d4429e727fa0fe59af443e5116d6be` against base `4cf8b9953535fa47f7a08f058ee4b44045e4c83d`, werkrevisor 0.10.0.
<!-- werkrevisor-error:91561fb9d2d4429e727fa0fe59af443e5116d6be --> ## ⚠️ werkrevisor: ERROR / review not completed, human action needed Change kind: `mixed`. ### Reasons - **YELLOW** test changes are not purely additive: tests/fakes.py:126 changes an existing test line, tests/fakes.py:133 changes an existing test line, tests/fakes.py:150 changes an existing test line and 28 more - **YELLOW** change kind 'mixed' (main code, tests, other files, docs); AI-approvable are only: docs only, tests only added, docs and tests, refactoring with unchanged tests, feature with purely additive tests, harmless files only - **YELLOW** risky paths changed: .werkrevisor.toml (built in), config/hsadmin-ng.toml (config/**), config/werkprobe.toml (config/**) and 2 more - **YELLOW** 4 changed files were given to the AI only as diff, not in full, so it could not check them against the rest of their file (all full files together would exceed [review] max_full_files_chars = 120000): werkrevisor/cli.py, werkrevisor/config.py, werkrevisor/report.py and 1 more - **ERROR** PR-Agent could not read: tests/test_pr_agent_integration.py, werkrevisor/review.py, docs/prs/2026-10-05-PR#9-exact-token-count-and-focused-style-review.md - **ORANGE** model reports 2 key issues ([review] red_min_findings = 2) - **YELLOW** model recommends merge with caution ### Findings 1. **Doc contradicts code** at `docs/prs/2026-10-05-PR#9-exact-token-count-and-focused-style-review.md:75-88` Scenario#9.03 and the Solution section say a prompt that PR-Agent prunes or refuses yields YELLOW with a reason naming the window. In `run_model_review`, the `window_reason` is raised only when `PromptShortened` is caught or when no call was made and `FallbackEligibleError` is the cause. A refusal after an earlier model call (for example a retry) is not mapped and re-raises as a generic error. The document also gives no explanation for the 9.03 case beyond the first call. 2. **Pruning flag unused** at `werkrevisor/review.py:216-228` `_admitting` receives `pruned` as a list, so `if pruned:` is true as soon as `recording` appends. But `ModelRun.pruned` and `remaining_files` are still declared, and the handler raises `PromptShortened` for every later call, so a retry after pruning is converted to a NoModelCall with `details` that may hold usage. This is uncertain: the visible diff does not show whether `ModelRun` is still populated with `pruned` or `remaining_files`, and `coverage_reasons` may still read them. If it does, they are now dead and the "clipped" ERROR path is unreachable, which should be confirmed. <sub> Usage: model `openrouter/anthropic/claude-sonnet-5.5`, LLM done, calls 1, tokens in 99424 / out 621 (reasoning included in out), duration 13.0 s, cost 0.2051 USD (reported), config: defaults + host + repository (.werkrevisor.toml at base 4cf8b99).<br> Reviewed head `91561fb9d2d4429e727fa0fe59af443e5116d6be` against base `4cf8b9953535fa47f7a08f058ee4b44045e4c83d`, werkrevisor 0.10.0. </sub>
claude-code added 1 commit 2026-10-05 17:54:15 +02:00
review: a refused prompt is YELLOW in every attempt, the dead pruning path is removed (PR#9)
werkator build succeeded after 00:06
ai-review/assessment review not completed, human action needed
ai-review/gate build failed after 00:15
bd3ea67252
A refusal is now told apart per attempt: an attempt that fails as fallback-eligible
without reaching the model refused its prompt, also after an earlier call;
an eligible error after a call, e.g. an invalid answer, stays an ERROR.
A pruned prompt never reaches the model, so ModelRun.pruned, remaining_files
and the ERROR reasons clipped and "PR-Agent left out files" were dead and are removed.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Collaborator

⚠️ werkrevisor: ERROR / review not completed, human action needed

Change kind: mixed.

Reasons

  • YELLOW test changes are not purely additive: tests/fakes.py:126 changes an existing test line, tests/fakes.py:133 changes an existing test line, tests/fakes.py:150 changes an existing test line and 33 more
  • YELLOW change kind 'mixed' (main code, tests, other files, docs); AI-approvable are only: docs only, tests only added, docs and tests, refactoring with unchanged tests, feature with purely additive tests, harmless files only
  • YELLOW risky paths changed: .werkrevisor.toml (built in), config/hsadmin-ng.toml (config/), config/werkprobe.toml (config/) and 2 more
  • YELLOW 7 changed files were given to the AI only as diff, not in full, so it could not check them against the rest of their file (all full files together would exceed [review] max_full_files_chars = 120000): tests/test_pr_agent_run.py, tests/test_token_count.py, werkrevisor/budget.py and 4 more
  • ERROR PR-Agent could not read: tests/test_pr_agent_integration.py, werkrevisor/review.py, docs/prs/2026-10-05-PR#9-exact-token-count-and-focused-style-review.md
  • ORANGE model reports 2 key issues ([review] red_min_findings = 2)
  • YELLOW model recommends merge with caution

Findings

  1. Possible Bug at werkrevisor/review.py:228-240

    _admitting raises PromptShortened when pruned is non-empty. pruned is only filled by recording when pr_generate_compressed_diff runs. In the diff recording is now installed together with pruned, called = [], [], but _admitting receives the same list object, so this works only while the list is never rebound. The result of _prepare_prediction wrapping also converts any FallbackEligibleError raised before a model call into PromptShortened. This includes errors unrelated to prompt size, for example a template rendering failure or a git provider error. Such errors would be reported as a YELLOW too_large_tokens reason naming the PR-Agent window, which hides a real technical failure that should be an ERROR. It is uncertain which exceptions PR-Agent 0.47.0 raises as FallbackEligibleError.

  2. Possible Bug at werkrevisor/cli.py:138-159

    In admit, a retry with a prompt that has fewer tokens than the first returns without updating result.token_count. A retry with a larger prompt that stays within the limit replaces result.token_count but does not re-check the key budget. Only the worst-case cost is checked again by precheck. The key limit_remaining was checked against max_usd_per_review once, so this is acceptable. The usage record, however, then reports the count of the last (larger) prompt while the first call was the one decided on. This is a minor inconsistency.

Usage: model `openrouter/anthropic/claude-sonnet-5.5`, LLM done, calls 1, tokens in 101689 / out 685 (reasoning included in out), duration 13.0 s, cost 0.2102 USD (reported), config: defaults + host + repository (.werkrevisor.toml at base 4cf8b99).
Reviewed head `bd3ea67252da559238ac029c83788f481048bf67` against base `4cf8b9953535fa47f7a08f058ee4b44045e4c83d`, werkrevisor 0.10.0.
<!-- werkrevisor-error:bd3ea67252da559238ac029c83788f481048bf67 --> ## ⚠️ werkrevisor: ERROR / review not completed, human action needed Change kind: `mixed`. ### Reasons - **YELLOW** test changes are not purely additive: tests/fakes.py:126 changes an existing test line, tests/fakes.py:133 changes an existing test line, tests/fakes.py:150 changes an existing test line and 33 more - **YELLOW** change kind 'mixed' (main code, tests, other files, docs); AI-approvable are only: docs only, tests only added, docs and tests, refactoring with unchanged tests, feature with purely additive tests, harmless files only - **YELLOW** risky paths changed: .werkrevisor.toml (built in), config/hsadmin-ng.toml (config/**), config/werkprobe.toml (config/**) and 2 more - **YELLOW** 7 changed files were given to the AI only as diff, not in full, so it could not check them against the rest of their file (all full files together would exceed [review] max_full_files_chars = 120000): tests/test_pr_agent_run.py, tests/test_token_count.py, werkrevisor/budget.py and 4 more - **ERROR** PR-Agent could not read: tests/test_pr_agent_integration.py, werkrevisor/review.py, docs/prs/2026-10-05-PR#9-exact-token-count-and-focused-style-review.md - **ORANGE** model reports 2 key issues ([review] red_min_findings = 2) - **YELLOW** model recommends merge with caution ### Findings 1. **Possible Bug** at `werkrevisor/review.py:228-240` `_admitting` raises `PromptShortened` when `pruned` is non-empty. `pruned` is only filled by `recording` when `pr_generate_compressed_diff` runs. In the diff `recording` is now installed together with `pruned, called = [], []`, but `_admitting` receives the same list object, so this works only while the list is never rebound. The result of `_prepare_prediction` wrapping also converts any `FallbackEligibleError` raised before a model call into `PromptShortened`. This includes errors unrelated to prompt size, for example a template rendering failure or a git provider error. Such errors would be reported as a YELLOW `too_large_tokens` reason naming the PR-Agent window, which hides a real technical failure that should be an ERROR. It is uncertain which exceptions PR-Agent 0.47.0 raises as `FallbackEligibleError`. 2. **Possible Bug** at `werkrevisor/cli.py:138-159` In `admit`, a retry with a prompt that has fewer tokens than the first returns without updating `result.token_count`. A retry with a larger prompt that stays within the limit replaces `result.token_count` but does not re-check the key budget. Only the worst-case cost is checked again by `precheck`. The key `limit_remaining` was checked against `max_usd_per_review` once, so this is acceptable. The usage record, however, then reports the count of the last (larger) prompt while the first call was the one decided on. This is a minor inconsistency. <sub> Usage: model `openrouter/anthropic/claude-sonnet-5.5`, LLM done, calls 1, tokens in 101689 / out 685 (reasoning included in out), duration 13.0 s, cost 0.2102 USD (reported), config: defaults + host + repository (.werkrevisor.toml at base 4cf8b99).<br> Reviewed head `bd3ea67252da559238ac029c83788f481048bf67` against base `4cf8b9953535fa47f7a08f058ee4b44045e4c83d`, werkrevisor 0.10.0. </sub>
mi merged commit 488b1498a5 into main 2026-10-05 18:30:13 +02:00
mi deleted branch claude-code/exact-tokens-and-focused-style-review 2026-10-05 18:30:13 +02:00
Sign in to join this conversation.
No Reviewers
No labels
2 Participants
Notifications
Due Date
No due date set.
Dependencies

No dependencies set.

Reference: mi/werkrevisor#9