werkrevisor/docs/prs/2026-10-17-PR#117-doccheck-variant-b.md

# PR#117: Jev Checks Each Changed File Against the PR Document

Version: minor

> **WARNING:** This document describes only the change applied in this PR.
> It may already be outdated once the next PR is merged.
> Historic PR documentation is not maintained along with new PRs; treat it as a snapshot.

## Problem

The document check of PR#47 and PR#116 runs in two models:
an economy model summarizes the diff without the PR document,
then Jev compares each section of the PR document and each summarized change with the summary.
The summary is a second provider, a second cost and a second source of error:
a change that the summary words loosely or drops never reaches Jev.
It also hides where a change sits: the summary names files, but Jev never sees the diff itself.

For the main use of the check, the hs.hsadmin.ng backend, the summary buys nothing.
Jev reads about 32000 tokens of state, enough for the PR document and one file's diff.
A measurement on 2026-10-17 showed that Jev reads a file's diff directly as well as it reads a summary of it,
and that a backdoor woven into an on-topic file, which the summary hides, is caught by a question about data outflow.

## Non-Goals

- No change of the four contradiction questions of PR#47 beyond rewording them from the summary to the file diff.
- The PR document, written by the author, must not lower the judgments about security or weakened checks;
  the data-outflow question is a judgment of the diff, not of the document.
- No per-language detection of comment-only changes; the existing mechanical rule decides what is trivial.
- A PR document that claims every change in vague words can still hide one; Jev judges the text as written.

## Specification

### Feature: Jev checks each changed file against the PR document

#### Scenario#117.01: a file diff that contradicts a test claim gives ORANGE

… so that a PR document that claims tests the code does not add waits for a human.

- **Given** Jev rates the diff of a changed file as contradicting a test claim of the PR document from `orange_min` on
- **When** werkrevisor reviews the PR with `[doc_check]`
- **Then** a line comment at the file states the contradiction as ORANGE

##### Verified by

- [test_doccheck: test_a_conflicting_test_claim_gets_orange_at_the_file](../../tests/test_doccheck.py)

#### Scenario#117.02: a file unrelated to the PR document gives a hint

… so that a by-catch gets declared, or a change that does not belong to the PR gets reverted.

- **Given** Jev rates the diff of a changed file as unrelated to the PR document from `unrelated_min` on
- **When** werkrevisor reviews the PR with `[doc_check]`
- **Then** a hint at the file names the change as not required by the PR document

##### Verified by

- [test_doccheck: test_an_unrelated_file_gets_a_hint](../../tests/test_doccheck.py)

#### Scenario#117.03: a file that sends customer data outside gives a data-outflow hint

… so that a backdoor woven into an on-topic file is caught before a human reads the PR.

- **Given** Jev rates the diff of a changed file as sending personal or customer data to a URL from `exfil_min` on
- **When** werkrevisor reviews the PR with `[doc_check]`
- **Then** a hint at the file names the possible data outflow

##### Verified by

- [test_doccheck: test_a_data_outflow_gets_a_hint](../../tests/test_doccheck.py)

#### Scenario#117.04: each changed code file is checked against the PR document in one step

… so that the check needs no economy model and no second provider.

- **Given** a PR with a changed PR document and two changed code files
- **When** werkrevisor reviews the PR with `[doc_check]`
- **Then** the review checks each code file against the PR document for a contradiction, an unrelated change and a data outflow
  - **and** no economy model summarizes the diff

##### Verified by

- [test_doccheck: test_jev_gets_each_file_with_the_pr_document_and_all_questions](../../tests/test_doccheck.py)

#### Scenario#117.05: a mechanical file is not sent to Jev

… so that an import-only or whitespace-only change costs no Jev call.

- **Given** a changed file whose every changed line is an import or whitespace
- **When** werkrevisor reviews the PR with `[doc_check]`
- **Then** Jev gets no state for that file

##### Verified by

- [test_doccheck: test_a_mechanical_file_is_not_checked](../../tests/test_doccheck.py)

#### Scenario#117.06: a resolved hint is not repeated while the file and the PR document stay unchanged

… so that the author can answer a hint once.

- **Given** the author resolved the hint about a file
- **When** werkrevisor reviews a new commit with the same PR document and the same diff of that file
- **Then** werkrevisor does not repeat the hint

##### Verified by

- [test_doccheck: test_a_resolved_hint_is_not_repeated](../../tests/test_doccheck.py)

#### Scenario#117.07: the usage record logs the values per file

… so that the thresholds can be calibrated on real PRs.

- **When** werkrevisor writes the usage record
- **Then** the record holds the file, the line and the value of each question for every judged file

##### Verified by

- [test_doccheck: test_the_usage_record_logs_the_values_per_file](../../tests/test_doccheck.py)

#### Scenario#117.08: a deleted file is checked and its hint sits on the PR document

… so that a deletion that the PR document does not require does not pass unseen.

- **Given** a PR that deletes a file, which Jev rates as unrelated to the PR document from `unrelated_min` on
- **When** werkrevisor reviews the PR with `[doc_check]`
- **Then** one hint at the PR document names the deleted file

##### Verified by

- [test_doccheck: test_a_deleted_file_is_checked_and_anchored_at_the_pr_document](../../tests/test_doccheck.py)

#### Scenario#117.09: a hint on a documentation file holds the merge

… so that an unrelated change or a data outflow in a Markdown file does not pass unseen.

- **Given** Jev flags a changed documentation file, and `[review] doc_hint_line_comments` is off
- **When** werkrevisor reviews the PR with `[doc_check]`
- **Then** the hint is a line comment on that file, so it holds the merge until it is resolved

##### Verified by

- [test_doccheck: test_a_hint_on_a_markdown_file_holds_the_merge](../../tests/test_doccheck.py)

## Solution

- `doccheck.judge` sends one Jev call per changed file outside the PR documents and the mechanical files.
  The state holds the PR document, the diff of that one file and the conventions of the repository.
- The economy-model summary is gone:
  no `SUMMARY_PROMPT`, no `summarize`, no `parse_summary`, no OpenRouter `doc_check_client`.
  The four contradiction questions of PR#47 are reworded from the summary to the diff of the file;
  the unrelated question of PR#116 and a new data-outflow question join them in the same call.
- A new key `[doc_check] exfil_min`, default 0.7, sets the value from which a data outflow gives a hint.
  `[doc_check]` no longer needs `model`, `input_price_usd_per_mtok`, `output_price_usd_per_mtok`,
  `max_output_tokens`, `max_diff_chars` or `provider_order`; the check is Jev alone.
- A flagged file gets one line comment at its first changed line, as ORANGE for a test-claim contradiction,
  else as a hint; like every hint, it holds the merge until it is resolved.
  A deleted or deletion-only file has no line in its new version,
  so the check judges it too and collects its hint in one comment at the PR document.
- A PR document check finding carries `from_doc_check`, so it holds the merge even on a documentation file;
  the documentation exception of `_plan_line_comments`, which turns a model hint on a doc file into a free hint,
  does not apply to it.
  The comment holds a fingerprint of the file path, its diff and the PR document,
  so a resolved hint does not come back while both stay unchanged, but an ORANGE one does.
- The usage record holds the values per file as `doc_check_judgments`;
  `doc_summary_*` and `doc_coverage_judgments` are gone.
  The check is Jev cost only; the statistics service already tolerates the missing `doc_summary_*`.
- `[doc_check]` is switched on in [config/hsadmin-ng.toml](../../config/hsadmin-ng.toml).
- The diagrams drop the economy-model summary step; the check is one Jev step "PR document vs each file".
- werkrevisor 0.49.0.

## Measurement

Re-run live on 2026-10-17 with the state and questions of this PR, against the hs.hsadmin.ng backend
(`hostsharing/hs.hsadmin.ng`), Jev `jev-latest`.
The false-positive set is 103 non-trivial main-code file diffs from 15 merged PRs (#299 to #329),
each judged with its own PR document.
The positive is a backdoor woven into a topic-matching file,
which sends `customerEmail` and the invoice total by an HTTP query to an external host, disguised as a template fetch.

| question | backdoor (6 runs) | 103 real files |
|---|---|---|
| data-outflow | 6 of 6 from 0.97, median 0.98 | 0 of 103 from 0.5, max 0.13, median 0.02 |
| unrelated (PR#116) | 0 of 6, max 0.25 | 0 of 103 from 0.3, max 0.29, median 0.03 |

The data-outflow question separates the backdoor from every real file;
the unrelated question misses the backdoor, because the file is on topic, and gives no hint on a real file.
The real files stay below 0.3 for both questions, while a planted change reached 0.7 to 0.98 in PR#116,
so `unrelated_min` = 0.7 and `exfil_min` = 0.7 sit in a wide gap and stay unchanged.
The four contradiction questions keep the separation of PR#47 when they read the file diff instead of the summary:
a planted "no test changed" still reaches 0.71 and more, unchanged documents stay below 0.2.

## Release Notes

- `[doc_check]` now checks each changed file directly against the PR document, without an economy-model summary,
  and flags a change that sends personal or customer data to an external service.

## Open Questions

- [x] Keep the data-outflow question as a hint or make it ORANGE?
  A hint from `exfil_min` on; it holds the merge until the author resolves it.
  ORANGE needs a calibration on real flagged PRs first.
- [x] What counts as a trivial file that needs no Jev call?
  The existing mechanical rule: a file whose every changed line is an import, a whitespace change or a rename.
  Comment-only changes are rare and cheap and are not detected separately.
- [x] Which value for `unrelated_min` and `exfil_min`?
  Both stay at 0.7: on 103 real file diffs both questions stay below 0.3, see "Measurement",
  while a planted change reached 0.7 to 0.98 in PR#116.

## Follow-up PRs

- Calibrate `hint_min` on the usage records of real PRs, and decide whether the data-outflow question becomes ORANGE.