werkrevisor/docs/prs/2026-10-08-PR#47-pr-doc-vs-change-summary.md

PR#47: Jev compares the PR document with a change summary of an economy model

WARNING: This document describes only the change applied in this PR. It may already be outdated once the next PR is merged. Historic PR documentation is not maintained along with new PRs; treat it as a snapshot.

The Problem

werkrevisor shall separate the PRs that need a human from the others, and the exchange between the coding agent and werkrevisor shall remove issues before a human spends time on the PR. The target verdict rule needs concrete issues: ORANGE for a likely high risk or a concrete issue, RED for both in the same file or category. Jev writes no findings, and a full review by a model is expensive and finds different defects each time.

The PR document states what the author intended and implemented. A PR document that contradicts the code, or that omits a relevant change, is a concrete issue: the author fixes the document, fixes the code, or reverts a change the task does not need. An economy model can summarize the diff cheaply, and Jev can compare the two texts. The summary must not see the PR document, or it repeats its claims instead of checking them.

Non-Goals

• No PR-wide classification over all signals, and no RED from Jev; that is a later PR.
• No change of the existing questions on sections and scenarios, e.g. readability or slop.
• No replacement of the model review by PR-Agent.
• No questions in .werkrevisor.toml yet; they are built into werkrevisor, since they do not depend on the ecosystem.

The Scenarios

Feature: Jev compares the PR document with an independent change summary

Background

• Given a [doc_check] section with an economy model
  • and a PR that changes code and its PR document

Scenario#47.01: the economy model summarizes the diff without the PR document

So that the summary checks the claims of the PR document instead of repeating them.

• When werkrevisor asks the economy model for the change summary
• Then the prompt holds the diff of every changed file outside the PR documents
  • and the prompt holds no line of a PR document

Verified by

• test_doccheck: test_the_summary_sees_the_diff_without_the_pr_document

Scenario#47.02: a test claim that contradicts the change summary gets ORANGE as a line comment

So that the author fixes the document or the tests before a human reads the PR.

• Given a section of the PR document claims which tests the PR adds, changes or removes
• When Jev rates a contradiction between this claim and the summary at or above orange_min
• Then the review is ORANGE
  • and a line comment at the heading of the section says that its test claim contradicts the summary

Verified by

• test_doccheck: test_a_conflicting_test_claim_gets_orange_as_a_line_comment

Scenario#47.03: another contradiction is a hint as a line comment

So that the author checks the section, and the merge waits for the answer, without a human.

• Given a section of the PR document describes the change, an implementation or a requirement
• When Jev rates a contradiction between this section and the summary at or above hint_min
• Then werkrevisor writes a hint as a line comment at the heading of the section
  • and the merge waits until the author resolves the hint

Verified by

• test_doccheck: test_another_contradiction_is_a_hint_as_a_line_comment

Scenario#47.04: a low value gives no comment

So that a consistent PR document passes without noise.

• When Jev rates every contradiction of every section below hint_min
• Then the comparison adds only a hint with the count of compared sections

Verified by

• test_doccheck: test_a_low_value_gives_no_comment

Scenario#47.05: a failed summary leaves the comparison out

So that a problem of the economy model never makes a PR GREEN, and never blocks it either.

• When the economy model fails, times out, or answers with no usable summary
• Then werkrevisor leaves out the comparison
  • and a hint names the cause

Verified by

• test_doccheck: test_a_failed_summary_leaves_the_comparison_out

Scenario#47.06: a secret in the diff keeps the diff from the economy model

So that a leaked secret does not reach a third party.

• Given the diff adds a line that looks like a private key
• When werkrevisor reviews the PR with [doc_check]
• Then werkrevisor sends no request to the economy model
  • and a hint says that the comparison was left out because of a possible secret

Verified by

• test_doccheck: test_a_secret_keeps_the_diff_from_the_economy_model

Scenario#47.07: the usage record logs the summary and the comparison

So that the thresholds can be calibrated on real PRs.

• When werkrevisor writes the usage record
• Then the record holds model, tokens and cost of the summary
  • and the value of each comparison question per section

Verified by

• test_doccheck: test_the_usage_record_logs_the_summary_and_the_comparison
• UsageApiIntegrationTest: the change summary counts as LLM cost, its comparison as Jev cost

The Solution

• A new optional host section [doc_check]: model, prices, max_output_tokens, timeout_seconds, max_diff_chars, orange_min and hint_min.
• The economy model gets the diff of the changed files outside the PR documents and answers with JSON: a list of changes, each with files, kind and a short statement.
• Jev answers four contradiction questions once per section and scenario of each changed PR document, with the summary as context, so that a line comment can point to the section: wrong change description, claimed but not implemented, conflict with a requirement, conflicting test claim.
• A conflicting test claim gives ORANGE from orange_min on; every contradiction from hint_min on gives a hint.
• The statistics service counts the summary as LLM cost and the comparison as Jev cost.
• config/werkrevisor.toml enables the comparison; the deployed host configs stay unchanged until the merge.
• werkrevisor 0.23.0.

Measurement

On 2026-10-08 a prototype tested the comparison on 22 merged PRs: werkrevisor #36 to #46 and hs.hsadmin.ng #289, #297, #301 to #303, #306, #308, #310, #311, #313 and #316. MiniMax M3 wrote the summaries without the PR documents, for 0.001 to 0.011 USD per PR. Jev cost less than 0.01 USD per PR.

Jev compared each PR document twice: unchanged, and with one planted defect. The defect notests adds the claim that no test changed. The defect nofile adds the claim that a changed main file stays unchanged.

question	highest value, unchanged	value with planted defect
conflicting test claim	0.18	0.71 to 0.93 with notests, in 14 of 14 documents
wrong change description	0.50	0.10 to 0.79 with nofile, above 0.5 in 5 of 15 documents
claimed but not implemented	0.27	not tested
conflict with a requirement	0.50	not tested

Thus a test claim gives ORANGE from 0.7 on, and each question gives a hint from 0.6 on. Real contradictions are less obvious than the planted ones, so real results can be worse.

Eight further questions asked whether the PR document omits a change, e.g. of the API or the database. They did not separate: unchanged documents got up to 0.89. This PR does not ask them.

Follow-up PRs

• Better questions for omitted changes, e.g. one question per summarized change.
• One classification of the whole PR from the risk categories, the comparison and the findings. It gives ORANGE for a likely high risk or a concrete issue, and RED for both in the same file or category.

Prerequisite PRs

• PR#46: werkrevisor 0.22.0, merged.