werkrevisor/docs/prs/2026-10-05-PR#9-exact-token-count-and-focused-style-review.md

PR#9: Exact token count and a focused style review

WARNING: This document describes only the change applied in this PR. It may already be outdated once the next PR is merged. Historic PR documentation is not maintained along with new PRs; treat it as a snapshot.

The Problem

werkrevisor 0.10.0 estimates the input tokens before the model call as characters / 3.5, with a fixed prompt overhead and an expansion factor for the diff. The estimate decides whether a PR gets an AI review at all: the input limit, the worst-case cost and the YELLOW reason. It is much too low:

PR	Estimate	Prompt tokens reported by the model
hsadmin-ng PR#305	about 168000	268735
werkrevisor PR#7	below 80000	87212 to 88369

A PR above the limit therefore reaches the model, and a review can cost more than [budget] max_usd_per_review.

The example configs tell the model to judge documentation against the writing style in AGENTS.md. The model then reports the countable rules, and it counts wrongly. On PR#8 it claimed that a 120-character line and a sentence wrapped after a comma violate the rules, although the limit is 128 characters and a wrap after a comma is explicitly allowed. werkrevisor 0.9.0 already checks these rules deterministically in [writing].

Non-Goals

• No tokenizer per model: the image has only tiktoken with o200k_base; Anthropic and other providers count differently, so a configurable factor covers the difference.
• No call to a token-count API of a provider: it would be a network call with the key before the budget check.
• No change to the deterministic writing rules.

The Scenarios

Feature: the token count decides about the AI review

Scenario#9.01: werkrevisor counts the prompt that PR-Agent sends

So that the input limit applies to the real prompt, not to an estimate.

• Given PR-Agent has rendered the system and the user prompt, with the diff, the full files and the repository context files
• When PR-Agent is about to call the model
• Then werkrevisor counts both prompts with the tokenizer o200k_base
  • and multiplies the count with [model] token_count_factor, rounded up

Verified by

• test_token_count.TokenCountTest.test_count_uses_the_tokenizer_and_the_factor
• test_token_count.TokenCountTest.test_real_o200k_base_tokenizer_loads_offline
• test_pr_agent_run.PrAgentRunTest.test_admit_sees_the_exact_prompt_before_the_model_call

Scenario#9.02: a prompt above the input limit gets no AI review

So that the model never gets a prompt that it reviews badly or that costs more than the budget.

• Given the effective input limit is <limit>
• When the counted prompt has more tokens than the limit
• Then werkrevisor does not call the model
  • and it does not query the key budget
  • and the verdict is YELLOW with a reason too_large_tokens that names the count, the factor and <limit>

limit
[model] max_context_tokens = 1000
the limit of 250 derived from [budget] max_usd_per_review = 0.081

Verified by

• test_token_count.TokenCountFlowTest.test_limit_is_decided_on_the_count_times_the_factor
• test_flow.BudgetFlowTest.test_too_long_for_the_model_is_yellow_without_llm_call
• test_flow.BudgetFlowTest.test_too_expensive_is_yellow_without_llm_call
• test_pr_agent_run.PrAgentRunTest.test_too_long_prompt_never_reaches_the_model

Scenario#9.03: a prompt that PR-Agent had to shorten gets no AI review

So that a PR far above the limit is YELLOW, not an ERROR of a clipped review.

• Given the prompt does not fit PR-Agent's window of twice the input limit plus [model] max_output_tokens
• When PR-Agent prunes the diff or refuses the prompt, in the first or in a later attempt
• Then werkrevisor does not call the model with this prompt
  • and the verdict is YELLOW with a reason too_large_tokens that names the window
  • and an error that PR-Agent raises after a model call of the same attempt stays an ERROR, e.g. an invalid answer

Verified by

• test_pr_agent_run.PrAgentRunTest.test_pruned_prompt_never_reaches_the_model
• test_pr_agent_run.PrAgentRunTest.test_prompt_refused_by_pr_agent_never_reaches_the_model
• test_pr_agent_run.RefusalTrackingTest.test_refusal_after_an_earlier_call_is_still_a_refusal
• test_pr_agent_run.RefusalTrackingTest.test_eligible_error_after_a_call_stays_a_model_error

Scenario#9.04: without the tokenizer werkrevisor falls back to the estimate and says so

So that a broken image does not hide that the decision rests on an estimate.

• Given the tokenizer o200k_base cannot load
• When werkrevisor decides about the model call
• Then it uses the estimate of 0.10.0
  • and the reason and the comment say token count estimated
  • and the usage record has token_count_method = "estimated"

Verified by

• test_token_count.TokenCountTest.test_fallback_to_the_estimate_is_flagged
• test_token_count.TokenCountFlowTest.test_estimated_count_is_named_in_reason_comment_and_usage

Scenario#9.05: the usage record holds the count and the reported prompt tokens

So that the operator can calibrate [model] token_count_factor from real reviews.

• Given a review with a model call
• When werkrevisor writes the usage record
• Then it has counted_input_tokens, counted_raw_tokens, token_count_factor, token_count_method and worst_case_usd
  • and prompt_tokens as the model reported it

Verified by

• test_token_count.TokenCountFlowTest.test_usage_record_has_count_and_reported_prompt_tokens

Scenario#9.06: only the host sets [model] token_count_factor, and it is positive

So that a PR cannot lower the count of its own review.

• Given <config>
• When werkrevisor builds the config
• Then it refuses the config with <message>

config	message
host token_count_factor = 0	[model] token_count_factor must be a positive number
host token_count_factor = true	[model] token_count_factor must be a positive number
.werkrevisor.toml with token_count_factor = 0.5	must not set [model] token_count_factor: only the host config sets it

Verified by

• test_token_count.TokenCountConfigTest.test_factor_must_be_positive
• test_token_count.TokenCountConfigTest.test_factor_is_host_only

Scenario#9.08: a retry of the model call uses the decision of the first call

So that a retry never stops a review whose first call already cost money, because of a second key query.

• Given werkrevisor admitted the first model call of a review
• When PR-Agent calls the model again with a prompt of <size>
• Then werkrevisor does not query the key budget again
  • and <result>

size	result
the same or fewer tokens	the call goes ahead; budget_remaining_before_usd stays the value of the first query
more tokens, above the input limit	the review stops with YELLOW too_large_tokens

Verified by

• test_token_count.RetryTest.test_retry_with_the_same_prompt_queries_the_key_only_once
• test_token_count.RetryTest.test_retry_with_a_smaller_prompt_keeps_the_first_decision
• test_token_count.RetryTest.test_retry_with_a_grown_prompt_above_the_limit_stops_and_keeps_the_usage

Scenario#9.09: a stop after an earlier model call keeps its usage

So that the comment and the usage record never hide the cost of a call that happened.

• Given a review stops before a model call
• When an earlier model call of this review <happened>
• Then the usage shows <usage>

happened	usage
happened	the reported calls, tokens and cost, or cost unknown
did not happen	no model call, cost 0 USD

Verified by

• test_token_count.RetryTest.test_retry_with_a_grown_prompt_above_the_limit_stops_and_keeps_the_usage
• test_token_count.RetryTest.test_stop_without_an_earlier_call_costs_nothing
• test_pr_agent_run.PrAgentRunTest.test_stop_carries_the_run_details_of_pr_agent

Feature: a focused style review

Scenario#9.07: the model does not report the countable writing rules

So that the model reports only what werkrevisor cannot count.

• Given the example configs and .werkrevisor.toml
• When werkrevisor builds the review instructions
• Then they tell the model not to report line length, sentences per line and sentences per paragraph
  • and to report bloated or mannered prose, unclear statements and contradictions between the PR document and the code

Verified by

• test_config_budget_redact.ConfigTest.test_examples_leave_the_countable_writing_rules_to_werkrevisor

The Solution

• werkrevisor wraps the AI handler of PR-Agent. Before each model call, it counts the system and the user prompt that PR-Agent passes to the handler, with tiktoken o200k_base from litellm's bundled tokenizer data. The count times [model] token_count_factor decides about the input limit and the worst-case cost. Only then werkrevisor queries the key budget.
• New host key [model] token_count_factor, default 1.15, because Anthropic and other tokenizers count more tokens than o200k_base.
• PR-Agent gets a window of twice the input limit plus max_output_tokens, unless [model] context_tokens is set. Thus a prompt reaches the count unshortened, and the reason names the exact count. A prompt that PR-Agent still has to prune or refuse is YELLOW, also in a later attempt; the usage of an earlier call stays, see Scenario#9.09.
• If the tokenizer cannot load, werkrevisor uses the estimate of 0.10.0 and names it token count estimated. [budget] chars_per_token, prompt_overhead_chars and diff_expansion_factor apply only to this fallback.
• The usage record replaces estimated_input_tokens and estimated_worst_case_usd with the fields of Scenario#9.05.
• The review instructions in .werkrevisor.toml, config/werkprobe.toml and config/hsadmin-ng.toml leave the countable writing rules to werkrevisor. .werkrevisor.toml is a risky path, and the review reads it from the base commit, so the new instructions apply to the PRs of werkrevisor only after the merge of this PR.
• werkrevisor 0.11.0; the README and docs/betrieb-git.javagil.de.md describe the change.

Open Questions

• Does werkrevisor count before PR-Agent builds the prompt? No: only PR-Agent knows its rendered templates, the diff with context lines and the repository context files. werkrevisor counts inside the run, in the AI handler, before the model call; nothing reaches the model without the count.
• Does a dry run without --llm still report a token count? No: without --llm werkrevisor starts no PR-Agent run, so there is no prompt to count; the usage fields are empty. tools/dry-run … --llm reports the count.
• Does the key budget query still come before the token check? No: the count comes first, so a too long PR is YELLOW without a query of the key.

Additional Changes

• The test of the version skew reason uses the installed version instead of a fixed 0.10.0.
• A prompt that PR-Agent prunes is YELLOW now; before, it was an ERROR clipped after the model call. A pruned prompt never reaches the model, so the ERROR reasons clipped and PR-Agent left out files are removed. A changed file missing in the prompt still gives ERROR skipped_files.
• The coverage check reported a file as unreadable when the text This file could not be read. appeared anywhere in its diff section, also in a diff line or in the appended full files. The self-review of this PR got a false ERROR for werkrevisor/review.py and this document, because review.py contains that text. Now only the notice line that PR-Agent writes directly below the ## File: header counts.

Follow-up PRs

• Calibrate [model] token_count_factor from the usage records: prompt_tokens / counted_raw_tokens of real reviews.