werkrevisor/docs/prs/2026-10-08-PR#47-pr-doc-vs-change-summary.md

# PR#47: Jev compares the PR document with a change summary of an economy model

> **WARNING:** This document describes only the change applied in this PR.
> It may already be outdated once the next PR is merged.
> Historic PR documentation is not maintained along with new PRs; treat it as a snapshot.

## The Problem

werkrevisor shall separate the PRs that need a human from the others,
and the exchange between the coding agent and werkrevisor shall remove issues before a human spends time on the PR.
The target verdict rule needs concrete issues:
ORANGE for a likely high risk or a concrete issue, RED for both in the same file or category.
Jev writes no findings, and a full review by a model is expensive and finds different defects each time.

The PR document states what the author intended and implemented.
A PR document that contradicts the code, or that omits a relevant change, is a concrete issue:
the author fixes the document, fixes the code, or reverts a change the task does not need.
An economy model can summarize the diff cheaply, and Jev can compare the two texts.
The summary must not see the PR document, or it repeats its claims instead of checking them.

## Non-Goals

- No PR-wide classification over all signals, and no RED from Jev; that is a later PR.
- No change of the existing questions on sections and scenarios, e.g. readability or slop.
- No replacement of the model review by PR-Agent.
- No questions in `.werkrevisor.toml` yet; they are built into werkrevisor, since they do not depend on the ecosystem.

## The Scenarios

### Feature: Jev compares the PR document with an independent change summary

#### Background

- **Given** a `[doc_check]` section with an economy model
  - **and** a PR that changes code and its PR document

#### Scenario#47.01: the economy model summarizes the diff without the PR document

So that the summary checks the claims of the PR document instead of repeating them.

- **When** werkrevisor asks the economy model for the change summary
- **Then** the prompt holds the diff of every changed file outside the PR documents
  - **and** the prompt holds no line of a PR document

##### Verified by

- [test_doccheck: test_the_summary_sees_the_diff_without_the_pr_document](../../tests/test_doccheck.py)

#### Scenario#47.02: a test claim that contradicts the change summary gets ORANGE as a line comment

So that the author fixes the document or the tests before a human reads the PR.

- **Given** a section of the PR document claims which tests the PR adds, changes or removes
- **When** Jev rates a contradiction between this claim and the summary at or above `orange_min`
- **Then** the review is ORANGE
  - **and** a line comment at the heading of the section says that its test claim contradicts the summary

##### Verified by

- [test_doccheck: test_a_conflicting_test_claim_gets_orange_as_a_line_comment](../../tests/test_doccheck.py)

#### Scenario#47.03: another contradiction is a hint as a line comment

So that the author checks the section, and the merge waits for the answer, without a human.

- **Given** a section of the PR document describes the change, an implementation or a requirement
- **When** Jev rates a contradiction between this section and the summary at or above `hint_min`
- **Then** werkrevisor writes a hint as a line comment at the heading of the section
  - **and** the merge waits until the author resolves the hint

##### Verified by

- [test_doccheck: test_another_contradiction_is_a_hint_as_a_line_comment](../../tests/test_doccheck.py)

#### Scenario#47.04: a low value gives no comment

So that a consistent PR document passes without noise.

- **When** Jev rates every contradiction of every section below `hint_min`
- **Then** the comparison adds only a hint with the count of compared sections

##### Verified by

- [test_doccheck: test_a_low_value_gives_no_comment](../../tests/test_doccheck.py)

#### Scenario#47.05: a failed summary leaves the comparison out

So that a problem of the economy model never makes a PR GREEN, and never blocks it either.

- **When** the economy model fails, times out, or answers with no usable summary
- **Then** werkrevisor leaves out the comparison
  - **and** a hint names the cause

##### Verified by

- [test_doccheck: test_a_failed_summary_leaves_the_comparison_out](../../tests/test_doccheck.py)

#### Scenario#47.06: a secret in the diff keeps the diff from the economy model

So that a leaked secret does not reach a third party.

- **Given** the diff adds a line that looks like a private key
- **When** werkrevisor reviews the PR with `[doc_check]`
- **Then** werkrevisor sends no request to the economy model
  - **and** a hint says that the comparison was left out because of a possible secret

##### Verified by

- [test_doccheck: test_a_secret_keeps_the_diff_from_the_economy_model](../../tests/test_doccheck.py)

#### Scenario#47.07: the usage record logs the summary and the comparison

So that the thresholds can be calibrated on real PRs.

- **When** werkrevisor writes the usage record
- **Then** the record holds model, tokens and cost of the summary
  - **and** the value of each comparison question per section

##### Verified by

- [test_doccheck: test_the_usage_record_logs_the_summary_and_the_comparison](../../tests/test_doccheck.py)
- [UsageApiIntegrationTest: the change summary counts as LLM cost, its comparison as Jev cost](../../statistics/src/test/kotlin/de/hoennig/werkrevisor/statistics/UsageApiIntegrationTest.kt)

## The Solution

- A new optional host section `[doc_check]`: model, prices, `max_output_tokens`, `timeout_seconds`, `max_diff_chars`,
  `orange_min` and `hint_min`.
- The economy model gets the diff of the changed files outside the PR documents
  and answers with JSON: a list of changes, each with files, kind and a short statement.
- Jev answers four contradiction questions once per section and scenario of each changed PR document,
  with the summary as context, so that a line comment can point to the section:
  wrong change description, claimed but not implemented, conflict with a requirement, conflicting test claim.
- A conflicting test claim gives ORANGE from `orange_min` on; every contradiction from `hint_min` on gives a hint.
- The statistics service counts the summary as LLM cost and the comparison as Jev cost.
- config/werkrevisor.toml enables the comparison; the deployed host configs stay unchanged until the merge.
- werkrevisor 0.23.0.

## Measurement

On 2026-10-08 a prototype tested the comparison on 22 merged PRs:
werkrevisor #36 to #46 and hs.hsadmin.ng #289, #297, #301 to #303, #306, #308, #310, #311, #313 and #316.
MiniMax M3 wrote the summaries without the PR documents, for 0.001 to 0.011 USD per PR.
Jev cost less than 0.01 USD per PR.

Jev compared each PR document twice: unchanged, and with one planted defect.
The defect `notests` adds the claim that no test changed.
The defect `nofile` adds the claim that a changed main file stays unchanged.

| question | highest value, unchanged | value with planted defect |
|---|---|---|
| conflicting test claim | 0.18 | 0.71 to 0.93 with `notests`, in 14 of 14 documents |
| wrong change description | 0.50 | 0.10 to 0.79 with `nofile`, above 0.5 in 5 of 15 documents |
| claimed but not implemented | 0.27 | not tested |
| conflict with a requirement | 0.50 | not tested |

Thus a test claim gives ORANGE from 0.7 on, and each question gives a hint from 0.6 on.
Real contradictions are less obvious than the planted ones, so real results can be worse.

Eight further questions asked whether the PR document omits a change, e.g. of the API or the database.
They did not separate: unchanged documents got up to 0.89.
This PR does not ask them.

## Follow-up PRs

- Better questions for omitted changes, e.g. one question per summarized change.
- One classification of the whole PR from the risk categories, the comparison and the findings.
  It gives ORANGE for a likely high risk or a concrete issue, and RED for both in the same file or category.

## Prerequisite PRs

- PR#46: werkrevisor 0.22.0, merged.