werkrevisor/docs/prs/2026-10-17-PR#117-doccheck-variant-b.md

PR#117: Jev Checks Each Changed File Against the PR Document

Version: minor

WARNING: This document describes only the change applied in this PR. It may already be outdated once the next PR is merged. Historic PR documentation is not maintained along with new PRs; treat it as a snapshot.

Problem

The document check of PR#47 and PR#116 runs in two models: an economy model summarizes the diff without the PR document, then Jev compares each section of the PR document and each summarized change with the summary. The summary is a second provider, a second cost and a second source of error: a change that the summary words loosely or drops never reaches Jev. It also hides where a change sits: the summary names files, but Jev never sees the diff itself.

For the main use of the check, the hs.hsadmin.ng backend, the summary buys nothing. Jev reads about 32000 tokens of state, enough for the PR document and one file's diff. A measurement on 2026-10-17 showed that Jev reads a file's diff directly as well as it reads a summary of it, and that a backdoor woven into an on-topic file, which the summary hides, is caught by a question about data outflow.

Non-Goals

• No change of the four contradiction questions of PR#47 beyond rewording them from the summary to the file diff.
• The PR document, written by the author, must not lower the judgments about security or weakened checks; the data-outflow question is a judgment of the diff, not of the document.
• No per-language detection of comment-only changes; the existing mechanical rule decides what is trivial.
• A PR document that claims every change in vague words can still hide one; Jev judges the text as written.

Specification

Feature: Jev checks each changed file against the PR document

Scenario#117.01: a file diff that contradicts a test claim gives ORANGE

… so that a PR document that claims tests the code does not add waits for a human.

• Given Jev rates the diff of a changed file as contradicting a test claim of the PR document from orange_min on
• When werkrevisor reviews the PR with [doc_check]
• Then a line comment at the file states the contradiction as ORANGE

Verified by

• test_doccheck: test_a_conflicting_test_claim_gets_orange_at_the_file

Scenario#117.02: a file unrelated to the PR document gives a hint

… so that a by-catch gets declared, or a change that does not belong to the PR gets reverted.

• Given Jev rates the diff of a changed file as unrelated to the PR document from unrelated_min on
• When werkrevisor reviews the PR with [doc_check]
• Then a hint at the file names the change as not required by the PR document

Verified by

• test_doccheck: test_an_unrelated_file_gets_a_hint

Scenario#117.03: a file that sends customer data outside gives a data-outflow hint

… so that a backdoor woven into an on-topic file is caught before a human reads the PR.

• Given Jev rates the diff of a changed file as sending personal or customer data to a URL from exfil_min on
• When werkrevisor reviews the PR with [doc_check]
• Then a hint at the file names the possible data outflow

Verified by

• test_doccheck: test_a_data_outflow_gets_a_hint

Scenario#117.04: each changed code file is checked against the PR document in one step

… so that the check needs no economy model and no second provider.

• Given a PR with a changed PR document and two changed code files
• When werkrevisor reviews the PR with [doc_check]
• Then the review checks each code file against the PR document for a contradiction, an unrelated change and a data outflow
  • and no economy model summarizes the diff

Verified by

• test_doccheck: test_jev_gets_each_file_with_the_pr_document_and_all_questions

Scenario#117.05: a mechanical file is not sent to Jev

… so that an import-only or whitespace-only change costs no Jev call.

• Given a changed file whose every changed line is an import or whitespace
• When werkrevisor reviews the PR with [doc_check]
• Then Jev gets no state for that file

Verified by

• test_doccheck: test_a_mechanical_file_is_not_checked

Scenario#117.06: a resolved hint is not repeated while the file and the PR document stay unchanged

… so that the author can answer a hint once.

• Given the author resolved the hint about a file
• When werkrevisor reviews a new commit with the same PR document and the same diff of that file
• Then werkrevisor does not repeat the hint

Verified by

• test_doccheck: test_a_resolved_hint_is_not_repeated

Scenario#117.07: the usage record logs the values per file

… so that the thresholds can be calibrated on real PRs.

• When werkrevisor writes the usage record
• Then the record holds the file, the line and the value of each question for every judged file

Verified by

• test_doccheck: test_the_usage_record_logs_the_values_per_file

Scenario#117.08: a deleted file is checked and its hint sits on the PR document

… so that a deletion that the PR document does not require does not pass unseen.

• Given a PR that deletes a file, which Jev rates as unrelated to the PR document from unrelated_min on
• When werkrevisor reviews the PR with [doc_check]
• Then one hint at the PR document names the deleted file

Verified by

• test_doccheck: test_a_deleted_file_is_checked_and_anchored_at_the_pr_document

Scenario#117.09: a hint on a documentation file holds the merge

… so that an unrelated change or a data outflow in a Markdown file does not pass unseen.

• Given Jev flags a changed documentation file, and [review] doc_hint_line_comments is off
• When werkrevisor reviews the PR with [doc_check]
• Then the hint is a line comment on that file, so it holds the merge until it is resolved

Verified by

• test_doccheck: test_a_hint_on_a_markdown_file_holds_the_merge

Solution

• doccheck.judge sends one Jev call per changed file outside the PR documents and the mechanical files. The state holds the PR document, the diff of that one file and the conventions of the repository.
• The economy-model summary is gone: no SUMMARY_PROMPT, no summarize, no parse_summary, no OpenRouter doc_check_client. The four contradiction questions of PR#47 are reworded from the summary to the diff of the file; the unrelated question of PR#116 and a new data-outflow question join them in the same call.
• A new key [doc_check] exfil_min, default 0.7, sets the value from which a data outflow gives a hint. [doc_check] no longer needs model, input_price_usd_per_mtok, output_price_usd_per_mtok, max_output_tokens, max_diff_chars or provider_order; the check is Jev alone.
• A flagged file gets one line comment at its first changed line, as ORANGE for a test-claim contradiction, else as a hint; like every hint, it holds the merge until it is resolved. A deleted or deletion-only file has no line in its new version, so the check judges it too and collects its hint in one comment at the PR document.
• A PR document check finding carries from_doc_check, so it holds the merge even on a documentation file; the documentation exception of _plan_line_comments, which turns a model hint on a doc file into a free hint, does not apply to it. The comment holds a fingerprint of the file path, its diff and the PR document, so a resolved hint does not come back while both stay unchanged, but an ORANGE one does.
• The usage record holds the values per file as doc_check_judgments; doc_summary_* and doc_coverage_judgments are gone. The check is Jev cost only; the statistics service already tolerates the missing doc_summary_*.
• [doc_check] is switched on in config/hsadmin-ng.toml.
• The diagrams drop the economy-model summary step; the check is one Jev step "PR document vs each file".
• werkrevisor 0.49.0.

Measurement

Re-run live on 2026-10-17 with the state and questions of this PR, against the hs.hsadmin.ng backend (hostsharing/hs.hsadmin.ng), Jev jev-latest. The false-positive set is 103 non-trivial main-code file diffs from 15 merged PRs (#299 to #329), each judged with its own PR document. The positive is a backdoor woven into a topic-matching file, which sends customerEmail and the invoice total by an HTTP query to an external host, disguised as a template fetch.

question	backdoor (6 runs)	103 real files
data-outflow	6 of 6 from 0.97, median 0.98	0 of 103 from 0.5, max 0.13, median 0.02
unrelated (PR#116)	0 of 6, max 0.25	0 of 103 from 0.3, max 0.29, median 0.03

The data-outflow question separates the backdoor from every real file; the unrelated question misses the backdoor, because the file is on topic, and gives no hint on a real file. The real files stay below 0.3 for both questions, while a planted change reached 0.7 to 0.98 in PR#116, so unrelated_min = 0.7 and exfil_min = 0.7 sit in a wide gap and stay unchanged. The four contradiction questions keep the separation of PR#47 when they read the file diff instead of the summary: a planted "no test changed" still reaches 0.71 and more, unchanged documents stay below 0.2.

Release Notes

• [doc_check] now checks each changed file directly against the PR document, without an economy-model summary, and flags a change that sends personal or customer data to an external service.

Open Questions

• Keep the data-outflow question as a hint or make it ORANGE? A hint from exfil_min on; it holds the merge until the author resolves it. ORANGE needs a calibration on real flagged PRs first.
• What counts as a trivial file that needs no Jev call? The existing mechanical rule: a file whose every changed line is an import, a whitespace change or a rename. Comment-only changes are rare and cheap and are not detected separately.
• Which value for unrelated_min and exfil_min? Both stay at 0.7: on 103 real file diffs both questions stay below 0.3, see "Measurement", while a planted change reached 0.7 to 0.98 in PR#116.

Follow-up PRs

• Calibrate hint_min on the usage records of real PRs, and decide whether the data-outflow question becomes ORANGE.