werkrevisor/docs/prs/2026-10-06-PR#31-excerpt-selection.md

PR#31: a cheap model selects the excerpt of a file too large for the review model

WARNING: This document describes only the change applied in this PR. It may already be outdated once the next PR is merged. Historic PR documentation is not maintained along with new PRs; treat it as a snapshot.

The Problem

Since werkrevisor 0.19.2, a file beyond the full-file limits goes to the review model as excerpt: each changed line with 40 lines before and after. The lines that matter for a change are often elsewhere, though: the function that the changed code calls, the callers of a changed function, a constant at the top of the file. A fixed window around each change does not find them.

The cheap models of the model comparison of 2026-10-06 found too few defects to review on their own, but reading a file and naming the relevant lines is a simpler task.

Non-Goals

• No text of the cheap model reaches the review model: it returns line numbers only, which werkrevisor checks.
• No selection for files that go in full: they need none.
• No default switch-on before a measurement shows that the review model finds the known defects with the selected excerpt.

The Scenarios

Feature: a cheap model selects the excerpt

Scenario#31.01: the excerpt holds the changes and the lines that the cheap model selects

So that the review model sees what the change depends on, not only what surrounds it.

• Given [excerpt] select = "model"
  • and a PR changes line 150 of a file beyond the full-file limits
  • and the cheap model selects the lines 10 to 20 of that file
• When werkrevisor reviews the PR
• Then the review model gets an excerpt of the lines 10 to 20 and 145 to 155

Verified by

• test_excerpt: test_the_excerpt_holds_the_changes_and_the_selected_lines
• test_excerpt: test_the_cheap_model_gets_the_diff_and_the_numbered_file

Scenario#31.02: an unusable selection falls back to the fixed window

So that a failure of the cheap model never costs the review its context.

• Given [excerpt] select = "model"
• When the cheap model <fails>
• Then the excerpt holds each changed line with [review] excerpt_context_lines lines before and after, as without selection

fails
answers with an error or not in time
answers with something that is no list of line ranges
selects more lines than [excerpt] max_lines

Verified by

• test_excerpt: test_an_unusable_selection_falls_back_to_the_fixed_window
• test_excerpt: test_failures_give_none_and_name_the_cause
• test_excerpt: test_anything_but_valid_line_numbers_is_refused_as_a_whole

Scenario#31.03: the selection costs are part of the run

So that the dashboard shows what the selection costs.

• Given the cheap model selected the excerpt of one file
• When werkrevisor writes the usage record
• Then the record holds the model, the calls, the tokens and the costs of the selection
  • and the statistics service counts the costs as LLM costs

Verified by

• test_excerpt: test_the_usage_record_holds_the_selection
• UsageApiIntegrationTest: the excerpt selection counts as LLM cost

Feature: a too large prompt keeps its review

Scenario#31.04: a prompt beyond the input limit first loses the full files, then the excerpts

So that a large PR still gets an AI review, as before werkrevisor 0.19.2 raised the full-file limits.

• Given the prompt with the full files exceeds the input limit of the review model
• When werkrevisor reviews the PR
• Then the review model gets <context>
  • and a hint says that the context was reduced

fits the input limit	context
with excerpts	excerpts with the fixed window instead of the full files
only without excerpts	the diff only

Verified by

• test_fullfiles: test_a_too_large_prompt_first_loses_the_full_files_then_the_excerpts

The Solution

• [excerpt], a host-only section, sets select = "context" (default) or "model", the model, its prices and limits.
• werkrevisor/excerpt.py asks the model through the OpenRouter chat API for JSON line ranges and refuses an answer as a whole unless it holds only valid line numbers within max_lines.
• The excerpt holds each changed line with context_lines lines and the selected ranges; a failed selection gets the fixed window of [review] excerpt_context_lines and the hint excerpt_selection_failed.
• No selection without a model call, for a file above 400000 characters, or for a diff with a secret; with a secret, werkrevisor collects no full files at all, as before.
• A malformed usage field of OpenRouter makes the selection cost an estimate instead of ending the review.
• The usage record and the comment name the model, calls, tokens and cost of the selection; the statistics service counts the cost as LLM cost.
• The README diagram shows the selection as optional step between the context and the review model.
• The host configs of werkrevisor, werkprobe and werkator switch the selection on at the deployment; hsadmin-ng keeps the fixed window, since the selection sends code to another provider.
• A prompt beyond the input limit gets excerpts with the fixed window instead of the full files, and then the diff only, with a hint context_reduced; the selection is not repeated, so the retry costs no more, and the usage record marks a selection that the reduction dropped as excerpt_discarded. PR#29 raised the total limit of the full files to 160000 characters but forgot token_count_factor = 1.55 in its estimate, so the first review of this PR found no room for the model at all.
• werkrevisor 0.20.0.

Measurement

On 2026-10-06, Claude Sonnet 5.5 reviewed 4 heads with 10 known real findings, copied into the private repositories mi/werkrevisor-bench and mi/hsadmin-ng-bench, since a dry run always reviews the current head of a pull request. max_full_file_chars = 8000 made each changed source file an excerpt; [review] max_findings stayed 3.

head (known findings)	full files	fixed window	GPT-6 Luna
werkrevisor PR#9 at eb2823f (2)	too large	1	1
werkrevisor PR#15 at b5bb590 (2)	0	0	0
hs.hsadmin.ng PR#302 at 6de8bd6 (2)	1	0	1
hs.hsadmin.ng PR#305 at aed0d4d (4)	too large	0	0

The selection never found less than the fixed window; on PR#302 it gave Sonnet the bypassable BIC check, which the fixed window missed and the full files showed. It cost 0.002 to 0.005 USD and 45 to 70 seconds per review. DeepSeek V4 Flash answered 7 of 10 selections of PR#305 with no JSON and took 296 seconds, so it is no candidate. The 4 findings of PR#305 contradict the documentation; with 3 findings per review, Sonnet reported code findings instead. The sample is small, so the default stays the fixed window.

Additional Changes

• Follow-up of PR#30, which the bot merged at GREEN before its hints were addressed: the docstring of the approval gate no longer says that RED requests changes, the review event has one branch less, and a test no longer repeats an assertion.

Prerequisite PRs

• PR#29: larger full files and excerpts.
• PR#30: a RED review no longer blocks later commits.