werkrevisor/README.md

# werkrevisor

werkrevisor is an AI review gate for Gitea pull requests.
Simple PRs can merge after an automated review.
Risky PRs get human attention.

Werkator starts werkrevisor for each ready (non-draft) PR head commit.
werkrevisor runs on a checkout of the base branch and never executes PR files.
It reads the PR through the Gitea API, applies deterministic rules,
and asks a model through PR-Agent 0.47.0 for a review.
The exit code is the merge gate: Werkator publishes it as a commit status on the PR head.

Status: v0.1, trial.

## Verdicts and exit codes

| Verdict | Label | Exit code | Meaning |
|---|---|---|---|
| GREEN | PASS | 0 | All files are AI-approvable, no hard rule fired, the model found no relevant issue or every changed file is mechanical. |
| YELLOW | HUMAN_REVIEW | 10 | A human must look: hard rule, change kind not AI-approvable, unclear model answer, `merge_with_caution`, minor model findings, no AI review possible because the PR is too long for the model or its budget. |
| ORANGE | FINDINGS | 15 | Soft: a critical path changed, the model reports a finding of high severity or enough findings of medium severity, the classifier rates a PR-document unit as clearly bad, a changed doc violates a writing rule, or the PR is too large for an AI-only approval. A developer checks the PR knowingly; an approval unlocks the merge. |
| RED | CHANGES_REQUIRED | 20 | Hard: a deterministic rule is violated, e.g. an open question in the PR document. Only a new commit that fixes it lifts it. |
| ERROR | review not completed, human action needed | 30 | Technical error, invalid model answer, timeout, budget, skipped files, superseded head. |

The order is GREEN < YELLOW < ORANGE < RED < ERROR; the worst reason decides.
ERROR is never GREEN.
A run without a complete model review is never GREEN,
except a PR of mechanical changes only; see [`[mechanical]`](#mechanical).

Only deterministic rules give RED.
A model or a classifier can be wrong, so their findings give at most ORANGE and never block hard.
On dev.hostsharing.net the repository administrator is an operator, not a developer;
an administrator merge override is therefore no way out of a false RED.

With `yellow_gate = "block"` or `"pass_with_review_request"`, YELLOW and ORANGE can still pass the gate (exit 0):

- A configured human reviewer approved exactly this head commit (Gitea review `commit_id`).
- `yellow_gate = "pass_with_review_request"` is set and a review request to a configured human reviewer is pending.

With `yellow_gate = "approval"`, the missing approval is the lock; see [Approval mode](#approval-mode).

An approval does not unlock RED.
Only a new head commit that fixes the violated rule can change RED.

## Approval mode

In this mode the branch protection requires one approval, and the bot is one of the allowed approvers.
werkrevisor acts as a reviewer:

| Verdict | Gitea action | Exit code |
|---|---|---|
| GREEN | Submit a review `APPROVED` with `commit_id` = head commit. | 0 |
| YELLOW | No bot review. Request an official review from `human_reviewers`, unless a request is pending or a human approved this head. | 0; 10 if the request failed |
| ORANGE | As YELLOW: no bot review, neither `REQUEST_CHANGES` nor `APPROVED`; request an official review from `human_reviewers`. The comment lists the findings. | 0; 15 if the request failed |
| RED | Submit a review `REQUEST_CHANGES` with `commit_id` = head commit and up to 3 violated rules. | 20 |
| ERROR | No review. | 30 |
| Superseded head | No review, no comment. | 30 |

werkrevisor reads the PR again directly before it writes anything.
It posts the result comment for every verdict, including GREEN, and links it from the review.
A rerun on the same head does not submit a second review.
An earlier bot review of the PR stays as it is; Gitea marks a review on an older head as stale.
The bot never approves a PR that it opened itself: that is ERROR.

Branch protection of the base branch (Gitea 1.27 API field names):

| Field | Value | Why |
|---|---|---|
| `required_approvals` | `1` | The approval is the lock. |
| `enable_approvals_whitelist` | `true` | Only listed users count. |
| `approvals_whitelist_username` | the bot user and `human_reviewers` | A GREEN bot review or a human approval unlocks the merge. |
| `block_on_rejected_reviews` | `true` | A RED review `REQUEST_CHANGES` blocks the merge. |
| `block_on_official_review_requests` | `true` | The review request of a YELLOW or ORANGE blocks until the human reviews. |
| `dismiss_stale_approvals` | `true` | A new commit removes the bot approval of the old head. |
| `block_on_outdated_branch` | `true` | Werkator starts no new review when only the base moves. |
| `enable_status_check` | `true` | The Werkator status is required. |
| `status_check_contexts` | `["ai-review/gate"]` | The pinned `statusContext` of the Werkator build. |

The bot user needs write access to the repository to submit official reviews.

## How a run works

1. Read the PR.
   If its head differs from `WERKATOR_PR_HEAD_SHA` or the PR is not open, stop with exit 30 and write nothing.
   Read `.werkrevisor.toml` at the base commit and put it on top of the host config; see [Config layers](#config-layers).
2. Look for an own result comment with the marker `<!-- werkrevisor:<head_sha> -->`.
   If it exists, reuse its verdict, do not call the model, and do not comment again.
3. Apply the deterministic rules to the changed files and the diff.
   A changed critical path (see [`[hard_orange]`](#hard_orange)) or a PR above the `[size]` limits is ORANGE;
   the model still reviews it, so its findings help the human reviewer.
   Check the changed PR documents for open questions; an open question gives RED without a model call.
   Check the changed lines of the docs against the writing rules; a violation gives ORANGE, and the model still reviews the PR.
   With `full_files = true`, read the new version of every changed text file at the head commit.
4. With `[engine] mode = "classifier"` or `"combined"`, Jev judges the sections and scenarios of the changed PR documents first;
   see [Classifier](#classifier).
5. Run the PR-Agent reviewer with all settings forced by werkrevisor.
   It leaves out the mechanical files; a PR of mechanical files only skips Jev and the model.
   Right before the model call, count the tokens of the prompt that PR-Agent built; see [Token count](#token-count).
   A count above the effective input limit (see [`[budget]`](#budget)) gives YELLOW without a model call.
6. Query the OpenRouter key budget.
   Too little budget gives ERROR without a model call.
7. Check that the model saw every changed file completely.
8. Read the PR again.
   If the head moved meanwhile, stop with exit 30 and write nothing.
9. Write one PR comment, the commit status `ai-review/assessment`, and one usage line.

The deterministic rules come only from the host config and the repository config of the base commit.
The model can make a verdict worse, never better than the rules allow.
If the secret scan fires, the diff is not sent to the model.

## Configuration

The operator config is TOML.
`WERKREVISOR_CONFIG_TOML` holds it inline; if that is empty, `--config` or `WERKREVISOR_CONFIG` names a file.
If both are set, the inline TOML wins.
An invalid inline TOML is ERROR (exit 30); the message names the line and column, never the content.
Unknown keys are errors, so a typo cannot weaken a rule.
Examples: [config/werkprobe.toml](config/werkprobe.toml) (cheap) and [config/hsadmin-ng.toml](config/hsadmin-ng.toml) (1 USD per review).

### Config layers

Three layers make the config; a later layer wins per key:

1. the built-in defaults of werkrevisor,
2. the host config (`WERKREVISOR_CONFIG_TOML` or `WERKREVISOR_CONFIG`),
3. the repository config `.werkrevisor.toml` at the repository root.

werkrevisor reads `.werkrevisor.toml` through the Gitea API at the base commit of the PR, never at its head,
so a PR cannot change its own review.
Without the file there is no repository layer.
After the merge of both layers, every section except `[engine]` and `[classifier]` must be set; else the result is ERROR.

The repository config may set these keys:

| Section | Keys | Rule |
|---|---|---|
| `[size]` | all | replaces the host value |
| `[categories]` | all | replaces the host value |
| `[hard_orange]` | `paths` | added to the host paths, never replaces them |
| `[hard_yellow]` | `paths` | added to the host paths, never replaces them |
| `[hard_yellow]` | `secret_scan_exclude` | replaces the host value |
| `[hard_yellow]` | `secret_scan` | can only stay on |
| `[pr_doc]` | all | replaces the host value |
| `[writing]` | all | replaces the host value |
| `[mechanical]` | `enabled`, `whitespace_significant` | replaces the host value |
| `[mechanical]` | `imports` | replaces the host regexes per file extension |
| `[review]` | `extra_instructions` | appended to the host instructions |
| `[review]` | `max_findings`, `orange_min_medium`, `repo_context_files`, `full_files` | replaces the host value |
| `[review]` | `max_full_file_chars`, `max_full_files_chars` | only lower or equal |
| `[classifier]` | `pr_doc_globs`, `min_confidence`, `scenario_questions`, `section_questions` | replaces the host value; needs a host `[classifier]` |
| `[model]` | `max_context_tokens`, `max_output_tokens` | only lower or equal |
| `[budget]` | `max_usd_per_review` | only lower or equal |

Every other key is a host key: the model, its prices and reasoning, the rest of the budget, `[gate]`, `[engine]`,
and the classifier endpoint, model and limits.
A repository config that sets one of them is ERROR, and the message names the key.
Model, money and merge rights stay with the host operator.

`.werkrevisor.toml` is always a risky path (see [`[hard_yellow]`](#hard_yellow)), so a change needs a human approval.
werkrevisor also checks the changed file of the PR against the host config;
an invalid file is RED, because after the merge every review would end with ERROR.
A section, key or change kind unknown to the installed werkrevisor version gives YELLOW instead,
if the rest of the file is valid:
a newer version may know it, so deploy that version right after the merge.
The change takes effect only after the merge.

[config/werkrevisor.toml](config/werkrevisor.toml) holds only the host part for werkrevisor itself;
the repository part is [.werkrevisor.toml](.werkrevisor.toml).

### `[model]`

| Key | Default | Meaning |
|---|---|---|
| `id` | required | LiteLLM model id, e.g. `openrouter/google/gemini-3.8-flash`. No fallback models are used. |
| `input_price_usd_per_mtok` | required | USD per 1M input tokens, maintained by the operator. |
| `output_price_usd_per_mtok` | required | USD per 1M output tokens, maintained by the operator. |
| `max_context_tokens` | required | Quality cap for the input: long prompts degrade a review even within the context window. About 150000 for frontier models, 80000 for cheap models. Replaces `max_input_tokens` of 0.6.0. |
| `max_output_tokens` | required | Output limit including reasoning tokens. |
| `reasoning_effort` | `"low"` | `none`, `minimal`, `low`, `medium`, `high`, `xhigh` or `max`. |
| `reasoning_max_tokens` | `0` | OpenRouter reasoning budget; 0 leaves it unset. |
| `context_tokens` | `0` | PR-Agent's window; 0 means twice the input limit plus `max_output_tokens`, see [Token count](#token-count). |
| `timeout_seconds` | `120` | Timeout per model call. |
| `max_retries` | `1` | 0 or 1; PR-Agent retries a failed call at most once. |
| `token_count_factor` | `1.15` | Multiplies the `o200k_base` count of the prompt; host key. See [Token count](#token-count). |

### `[budget]`

| Key | Default | Meaning |
|---|---|---|
| `max_usd_per_review` | required | Worst-case cost limit per review; it derives an input limit, see below. |
| `min_key_remaining_usd` | `0.0` | The key must keep at least this budget, else ERROR. |
| `chars_per_token` | `3.5` | Fallback estimate without a tokenizer, see [Token count](#token-count). |
| `prompt_overhead_chars` | `16000` | Fallback estimate: PR-Agent prompt size plus repository context files. |
| `diff_expansion_factor` | `1.5` | Fallback estimate: PR-Agent adds context lines and line numbers to the diff. |

The worst case is (counted input × input price + `max_output_tokens` × output price) × (`max_retries` + 1).
The budget-derived input limit is the largest input whose worst case stays within `max_usd_per_review`.
The effective input limit is the lower of this limit and `[model] max_context_tokens`;
a PR whose token count exceeds it gets no AI review and is YELLOW, and the reason names the count and the limit that applied.
Example: Sonnet 5.5 at 2 / 10 USD per 1M tokens, `max_output_tokens = 4000`, 2 attempts and 0.40 USD give 80000 input tokens.
A config whose budget does not even pay for `max_output_tokens` is refused.

After the token count and before the model call, werkrevisor reads `GET https://openrouter.ai/api/v1/key`.
If `limit_remaining` is below `max_usd_per_review` or `min_key_remaining_usd`, the result is ERROR "budget".
A key without a limit passes this check.

#### Token count

Right before the model call, werkrevisor counts the system and the user prompt that PR-Agent passes to its AI handler:
the instructions, the PR description, the diff, the full files, the classifier hints and the `repo_context_files`.
It counts with tiktoken `o200k_base`; litellm ships its data in the image, so nothing is downloaded.
The count times `[model] token_count_factor` decides about the input limit and the worst-case cost,
because Anthropic and other providers count more tokens than `o200k_base`.
Only the host config sets the factor, so a PR cannot lower the count of its own review.

werkrevisor 0.10.0 estimated characters / 3.5 instead, and the estimate was much too low:

| PR | Estimate | Prompt tokens reported by the model |
|---|---|---|
| hsadmin-ng PR#305 | about 168000 | 268735 |
| werkrevisor PR#7 | below 80000 | 87212 to 88369 |

Calibrate the factor with the usage records of real reviews: `prompt_tokens` / `counted_raw_tokens`.
On werkrevisor PR#10 the factor 1.15 gave 49841 tokens, and Anthropic reported 65792 prompt tokens for Sonnet:
1.15 × 65792 / 49841 ≈ 1.52, so the example configs use 1.55.

PR-Agent gets a window of twice the input limit plus `max_output_tokens`, or `[model] context_tokens` if set.
Thus PR-Agent passes a too long prompt unshortened to the count, and the reason names the exact count.
A prompt beyond that window, which PR-Agent prunes or refuses, gets no AI review either, and the reason names the window.
If the tokenizer cannot load, werkrevisor uses the estimate from the `[budget]` keys above
and says `token count estimated` in the reason, the comment and the usage line.

### `[size]`

| Key | Default | Meaning |
|---|---|---|
| `max_changed_lines` | `800` | Changed lines (additions plus deletions) of all files outside the `docs` category, except mechanical files. |
| `max_files` | `30` | Changed files outside the `docs` category, except mechanical files. |

Above one of these limits, the PR is too large for an AI-only approval: the verdict is at least ORANGE,
and a human approves it knowingly.
The limits do not stop the model review; if the model can take the PR, its findings help the human.
Docs do not count, because a PR document alone has 100 to 200 lines.

This limit is separate from the model limit in [`[budget]`](#budget):
a PR can be too large for an AI-only approval, too long for the model, or both; then both reasons appear and ORANGE wins.

Changed lines outside the docs per merged PR, measured over one month:

| Repository | PRs | Median | p75 | p90 | Max |
|---|---|---|---|---|---|
| Werkator | 92 | 194 | 489 | 865 | 1748 |
| hsadmin-ng | 16 | 441 | 974 | 1098 | 1146 |

With 800 lines, about every tenth Werkator PR needs a knowing human approval.
hsadmin-ng uses 1000 lines: that is about every fourth PR,
and it lies just below the hard limit of hsadmin-ng's own `tools/pr-quick-check` (about 1200 to 1400 lines).

### `[categories]`

Glob lists per category: `docs`, `tests`, `main`, `api_spec`, `harmless`.
The first match wins in the order `api_spec`, `tests`, `docs`, `main`, `harmless`; else the file is `other`.
`harmless` (default `[]`) lists files, e.g. `.editorconfig`, that do not spoil an AI-approvable change kind.
A pattern without `/` matches the file name in any directory; `**` matches any number of directories.

`auto_approvable` lists the change kinds that can become GREEN:

| Change kind | Condition |
|---|---|
| `docs_only` | Only docs. |
| `tests_only_added` | Only tests, and the test changes are purely additive. |
| `docs_and_tests` | Docs and tests; the test changes are purely additive. |
| `refactoring` | Main code, optionally docs; no tests, no API spec. |
| `feature_with_tests` | Main code and tests, optionally docs; the test changes are purely additive. |
| `harmless_only` | Only files of the `harmless` category. |
| `mechanical_only` | Only mechanical files, see [`[mechanical]`](#mechanical); no model call. |

Harmless files may come along with every change kind.
Mechanical files do not count for the change kind,
e.g. main code with a test file that changed only its imports is a `refactoring`.

Purely additive test changes add test files, test methods or lines, e.g. a new assert line in an existing test method.
They delete no test file and remove or change no line of a test file; blank lines do not count.
A removed line that the diff adds again unchanged, e.g. a closing brace after an appended test, also does not count,
except a line with `@Test`, `assert`, `should` or `expect`.

An added line must not disable a test: `@Disabled`, `@Ignore`, `.skip(`, `.only(`, `pytest.mark.skip`, `pytest.mark.xfail`,
`unittest.skip`, `skipTest(`, `xit(`, `xdescribe(` or `xtest(`.

All other change kinds (`tests_modified`, `feature_with_modified_tests`, `mixed`) are at most YELLOW.
The reason names the kinds of the changed files and the AI-approvable change kinds,
e.g. with `auto_approvable = ["docs_only", "refactoring"]`:
`change kind 'mixed' (main code, other files, docs); AI-approvable are only: docs only, refactoring with unchanged tests`.
werkrevisor tells the model the change kind, so the model can object if, for example, a refactoring changes behavior.

### `[hard_orange]`

`paths` lists globs of critical files, e.g. RBAC, authentication, authorization and secrets.
A change of such a file gives ORANGE: a developer reviews it deliberately, and the model still reviews the PR.
The section is required; `paths = []` is allowed.

A renamed file counts with its old and new path.
Files that match the `docs` category globs never match `paths`.
A mechanical change counts too, because a critical path stays critical.

All critical files share one reason line, e.g.
`critical paths changed, review them deliberately: werkrevisor/gitea.py ([hard_orange] werkrevisor/gitea.py) and 2 more`.
A file that matches `[hard_orange]` and a YELLOW path, also a built-in one, appears only in this reason.

### `[hard_yellow]`

`paths` lists globs of risky files, e.g. migrations, CI and deploy config, dependency manifests.
A renamed file counts with its old and new path.
Files that match the `docs` category globs never match `paths`,
so a PR document named after a risky topic, e.g. `doc/PR/…-rbac-generator-hardening.md`, is no risky file.
`secret_scan = true` scans added lines for private keys, `-----BEGIN`, AWS keys, generic `token|secret|password|api_key = "..."` assignments,
and tokens with the prefixes `hsak_` and `sk-or-`.
`secret_scan_exclude` lists globs of files with fake secrets, e.g. `["src/test/**"]` for test data.
There, only real key formats count: private keys (`-----BEGIN … PRIVATE KEY`), AWS key ids,
`hsak_<name>.` with 64 hex digits and `sk-or-v1-` with 64 hex digits.

`.werkrevisor.toml` is a built-in risky path; no layer can remove it.
DB schema changes are built-in risky paths too, outside Markdown files:
`**/*.sql`, `**/db/changelog/**`, `**/migrations/**`, `**/migration/**`, `**/liquibase/**`, `**/flyway/**`,
`**/*.changelog.*` and `**/db.changelog-*`.
Their reason names them, e.g. `DB schema changed: src/main/resources/db/changelog/x.sql (built in)`.
These hard rules give YELLOW; the model cannot override them.

### `[pr_doc]`

| Key | Default | Meaning |
|---|---|---|
| `globs` | `[]` | Globs of the PR documents, e.g. `doc/PR/*.md`; `TEMPLATE*.md` and `README.md` never count. |
| `open_question_headings` | `["Open Questions", "Offene Fragen"]` | Headings of the open-questions section, compared without case and a trailing colon. |

A PR never passes while a PR document that it adds or changes has open questions.
The section starts at such a heading of any level and ends at the next heading of the same or a higher level;
deeper subheadings stay in it.
Allowed are an empty section and list items checked as done with `[x]`, e.g. `- [x] TODO.spec: …`.
Every item with `[ ]` and every list item or paragraph without a checkbox gives RED.
Blank lines, comments (`<!-- … -->`) and continuation lines of a done item do not count.
The check is deterministic and runs before any model call; the model cannot lift it, and no model call follows.
The reason lists up to 3 open items as `file:line: text`, then the count of the rest.

### `[writing]`

| Key | Default | Meaning |
|---|---|---|
| `enabled` | `true` | Check the writing rules. |
| `max_line_length` | `128` | Characters per line. |
| `max_sentences_per_paragraph` | `5` | Sentences per paragraph; each list item is a paragraph of its own. |
| `globs` | `[]` | Globs of the checked files; empty means the `docs` category. The PR documents of `[pr_doc]` always count. |

werkrevisor checks the countable rules of the writing style in `AGENTS.md` on changed Markdown files:
at most one sentence per line, at most `max_line_length` characters per line,
and at most `max_sentences_per_paragraph` sentences per paragraph.
Only lines that the PR adds or changes count, and a paragraph only if the PR changes one of its lines.
A table row, a heading, a line with only a link or URL, a fenced code block, an HTML comment and the front matter
have no line limit and no sentences.

The sentence split is conservative, so a false finding is rare:
a sentence ends at `.` or `!` before a capital letter, but not after an abbreviation such as `e.g.`, `z. B.` or `Nr.`,
an initial or a number.
A question mark ends no sentence, because a PR document answers an open question on the same line.
Code spans, URLs and link targets do not count.

A violation gives ORANGE: a style rule must not block hard, so a human approves knowingly.
The check runs before the model call and does not skip it.
The reason lists at most 3 violations as `file:line: rule`, then the count of the rest.

### `[mechanical]`

| Key | Default | Meaning |
|---|---|---|
| `enabled` | `true` | Find the mechanical files. |
| `whitespace_significant` | `*.py`, `*.yml`, `*.yaml`, Makefiles, `*.mk`, `*.md`, `*.haml`, `*.pug`, `*.coffee`, `*.nim`, `*.fs` | Globs of files in which whitespace has a meaning. |
| `imports` | see below | Table from a file extension without the dot to the regexes of lines that an import change may add or remove. |

A changed file is mechanical only if one of these rules explains every changed line:

- `renamed`: Gitea reports the file as renamed with no added or removed line.
- `whitespace`: the lines differ only in the amount of whitespace, like `git diff -b`, or are blank lines.
  A file of `whitespace_significant` is never mechanical by this rule.
- `imports`: the line matches a regex of its file extension, or it is part of a multi-line import statement
  whose lines all appear in the same diff hunk and match together.
  Blank lines count too.

A file with an added or removed line that no rule explains is not mechanical.
Neither is an added, deleted or binary file, a renamed file with changes, or a file whose diff Gitea shortened.

The built-in import regexes:

| Extensions | Lines |
|---|---|
| `java`, `kt`, `kts` | `import a.b.C;`, `import a.b.*;`, `package a.b;` |
| `ts`, `tsx`, `js`, `jsx`, `mjs` | named, default and namespace imports with `from`; never a side-effect import like `import './x'` |
| `go` | `import "x"`, `import y "x"` and `import ( … )` blocks; never `_ "x"` or `. "x"` |
| `rs` | `use a::b;`, `pub use a::{b, c};`; never `mod x;` |

Python and all other extensions have no import rule.
A `[mechanical.imports]` table in a config layer replaces the regexes of the extensions that it names, e.g.
`scala = ['^\s*import\s+[\w.]+\s*$']`; an empty list switches an extension off.
An invalid regex is a config error.

A mechanical file does not go to the model, neither its diff nor its full file;
werkrevisor passes it to PR-Agent as `ignore.regex`, and the coverage check does not miss it.
It does not count for the change kind and for the check of purely additive tests.
The hard rules apply to it as to every file, e.g. a mechanical change of a risky path gives YELLOW,
of a critical path ORANGE.
The `[size]` limits do not count it, so a package move across many files does not make a PR too large.

The comment lists the mechanical files in one line, with at most 3 examples and the count of the rest, e.g.
`3 files changed only mechanically and were not reviewed by the model: A.java (imports), b.c (whitespace), c (renamed)`.

### `[review]`

| Key | Default | Meaning |
|---|---|---|
| `extra_instructions` | `""` | Project review rules for the model. |
| `max_findings` | `3` | Findings requested from the model and shown in the comment. |
| `orange_min_medium` | `2` | This many findings of medium severity give ORANGE; 0 disables it. See [Severity of the findings](#severity-of-the-findings). |
| `repo_context_files` | `[]` | Files PR-Agent reads from the default branch, e.g. `AGENTS.md`. |
| `full_files` | `false` | Add the complete new version of every changed file to the prompt. |
| `max_full_file_chars` | `40000` | Limit per rendered file; a larger file is left out. |
| `max_full_files_chars` | `120000` | Limit for all rendered files together; files beyond it are left out. |

#### Severity of the findings

The answer schema of PR-Agent 0.47.0 has no severity field.
So the `extra_instructions` of the examples tell the model to start every `issue_header` with `[high]`, `[medium]` or `[low]`:

- `[high]`: wrong behavior, data loss, a security problem or a broken contract in a realistic scenario.
- `[medium]`: a defect with limited impact, or a missing test for changed behavior.
- `[low]`: cosmetics, wording, log noise or style.

werkrevisor reads the prefix case-insensitively and removes it from the title.
A missing or unknown prefix counts as medium; the comment says `severity unknown, counted as medium`.
One finding of high severity or `orange_min_medium` findings of medium severity give ORANGE.
Any other finding gives YELLOW with the reason `model reports minor findings`.
`merge_recommendation: changes_required` alone gives YELLOW; only the findings give ORANGE.

werkrevisor 0.14.0 replaced `red_min_findings` with `orange_min_medium`; the old key is a config error.

The full files share the effective input limit with the diff and the prompt overhead.
Choose `max_full_files_chars` well below that limit, at about 3.5 characters per token,
so that the diff of a PR at the `[size]` limit still fits;
e.g. 120000 chars are about 34000 of 80000 tokens.

A diff with a few context lines hides constraints stated elsewhere in a file, e.g. in its header comment.
PR-Agent 0.47.0 caps the extra context lines per hunk at 10, so werkrevisor adds the full files itself:

- It reads each changed, not deleted file at the head commit through the Gitea API (`/raw/{path}?ref=<sha>`).
- It skips files that PR-Agent cannot review (binary extensions); the coverage rules already make them YELLOW.
- It skips files of the `docs` category; they reach the model only as diff and do not count against the limits.
  The model reviews docs for their wording against the writing style, which the diff shows;
  full docs would crowd out the code files.
- It appends one section after the diff, in the user prompt, never in the system prompt, because PR content is untrusted:
  `## Full content of the changed files (new version at <sha>, …)`, one `### Full file: '<path>' (<n> lines)` block per file,
  and `## End of the full content of the changed files`.
- Every content line starts with its line number and ` | `,
  so file content can never look like a section marker or a `## File:` diff header.
- A file that is not UTF-8 text or exceeds a limit is left out.
  One YELLOW reason per cause names the cause, the count and at most 3 files, e.g.
  `20 changed files were given to the AI only as diff, not in full, so it could not check them against the rest of their file
  (all full files together would exceed [review] max_full_files_chars = 120000): README.md, werkrevisor/cli.py, werkrevisor/config.py and 17 more`.
  A review with incomplete context is never GREEN.

The hook wraps `get_pr_diff` as the PR-Agent reviewer module calls it, so the section becomes part of the prompt variable `diff`.
The operator `extra_instructions` should tell the model to check the changed lines against constraints stated elsewhere in the full files.
They should also tell it not to report the countable writing rules, which [`[writing]`](#writing) checks deterministically:
a model miscounts them, e.g. on PR#8 it called a line of 120 characters too long and a wrap after a comma a violation.
For documentation the model reports only bloated or mannered prose, unclear statements,
and contradictions between the PR document and the code.

### `[engine]`

Optional; without it the mode is `llm`.

| `mode` | Meaning |
|---|---|
| `llm` | The LLM review alone, as before 0.5.0. |
| `classifier` | Jev judges the PR documents; no LLM call. Only a `docs_only` PR whose changed files are all judged PR documents can become GREEN. |
| `combined` | Jev judges the PR documents, then the LLM reviews the PR with Jev's flagged units as hints. |

### `[classifier]`

Optional unless `[engine] mode` uses it; when present, every key is required.

| Key | Meaning |
|---|---|
| `endpoint` | `https://api.typesafe.ai/v1/systemone`; https only. |
| `model` | `jev-latest`. |
| `input_price_usd_per_mtok` | USD per 1M input tokens, maintained by the operator; Jev's output is free. |
| `max_state_chars` | A unit with a larger JSON state is not sent; the result is YELLOW "classifier context incomplete". |
| `max_units` | Units beyond this number are not sent; the result is YELLOW. |
| `timeout_seconds` | Timeout per Jev request. |
| `pr_doc_globs` | Globs of the PR documents, e.g. `doc/PR/*.md`; `TEMPLATE*.md` and `README.md` never count. |
| `min_confidence` | A choice or score answer below this confidence does not count for the verdict; 0 disables it. |
| `scenario_questions`, `section_questions` | Arrays of tables, see below. |

Each question has `name`, `type` (`noul`, `choice` or `score`), `instructions`, `red_above` and `yellow_above`.
Every question asks for a problem, so a high value always means "bad":

- `noul`: the value is the probability of "yes"; optional `criteria = { true = "…", false = "…" }`.
- `choice`: `criteria` is a table of options; `problem_options` lists the bad ones, and the value is their summed probability.
- `score`: `criteria` is a list of 2 to 10 levels from "no problem" to "worst problem";
  the value is Jev's expected level divided by the top level.

A value above `red_above` gives ORANGE, a value above `yellow_above` YELLOW.
The key `red_above` keeps its name from 0.5.0, when this gave RED.

An answer below `min_confidence` is ignored for the verdict, whatever its value:
in the calibration on hsadmin-ng such answers came mostly from short sections and named no real problem.
A noul has no confidence; its `yellow_above` band covers the uncertain answers.

The example configs contain default questions for readability, one aspect per scenario, implementation details,
Given/When/Then use and vague outcomes in scenarios, and for readability and slop in sections.
The writing rules are no Jev question; `[writing]` checks them deterministically.

### `[gate]`

| Key | Default | Meaning |
|---|---|---|
| `yellow_gate` | `"block"` | `block` (exit 10 for YELLOW, 15 for ORANGE), `pass_with_review_request`, or `approval` (see [Approval mode](#approval-mode)). |
| `human_reviewers` | `[]` | Gitea logins whose approval unlocks YELLOW and ORANGE and who get review requests; required unless `block`. |
| `assessment_context` | `"ai-review/assessment"` | Context of the informative commit status. |

The informative status maps GREEN to `success`, YELLOW to `pending` ("human review required"),
ORANGE to `warning` ("AI findings; a developer approval after checking them unlocks the merge"), RED to `failure` and ERROR to `error`.
Gitea 1.27 supports the state `warning`.
The gate itself is the Werkator status of the build (planned context `ai-review/gate`).

## Environment

| Variable | Source | Meaning |
|---|---|---|
| `WERKATOR_PR_NUMBER`, `WERKATOR_PR_HEAD_SHA` | Werkator | PR and head commit to review. |
| `WERKATOR_PR_HEAD_REF`, `WERKATOR_PR_BASE_REF`, `WERKATOR_PR_BASE_SHA` | Werkator | Informative. |
| `WERKATOR_GITEA_BASE_URL`, `WERKATOR_GITEA_OWNER`, `WERKATOR_GITEA_REPO` | Werkator | Repository; the URL must use https. |
| `GITEA_BOT_TOKEN` | host secret | Bot token: read PR, write issue comments, write commit status, request reviewers. |
| `OPENROUTER_API_KEY` | host secret | OpenRouter key with a credit limit; not needed with `[engine] mode = "classifier"`. |
| `TYPESAFE_API_KEY` | host secret | TypeSafe key for Jev; needed when `[engine] mode` is not `llm`. |
| `WERKREVISOR_CONFIG_TOML` | host config | The operator TOML itself; wins over `WERKREVISOR_CONFIG`. |
| `WERKREVISOR_CONFIG` | host config | Path to the operator TOML, used when `WERKREVISOR_CONFIG_TOML` is empty. |
| `PYTHONNOUSERSITE` | host config | Must be `1`; see [Hardening against a writable HOME](#hardening-against-a-writable-home). |

werkrevisor never prints tokens or keys.
It redacts known secret values and token patterns from comments, logs and usage lines.
HTTP redirects are not followed, so the token cannot reach another host.

## Classifier

Jev from TypeSafe is a classifier: it answers typed questions with probabilities, fast and cheap, but writes no findings.
werkrevisor calls it directly with the standard library; TLS stays verified, redirects are not followed, the key never appears in output.
A request that fails with HTTP 5xx or a timeout is retried once.

For each changed PR document werkrevisor reads the new version at the head commit and splits it into units:

- each `##` or `###` section with its text, without the scenarios and without `Verified by` blocks;
- each Gherkin scenario `#### Scenario#<id>: <title>` (or `#### Scenario: <title>`) with its steps,
  its feature and its `#### Background` as context, without the `##### Verified by` block.

Headings in code blocks do not count.
Jev gets one request per unit with all questions of its kind; up to 4 requests run in parallel.
The verdict per question follows the thresholds above; a failed Jev request is ERROR.
A clearly bad unit gives ORANGE without an LLM finding, also for the specification questions on Gherkin and PR documents:
a classifier can be wrong too, so it never blocks hard, and a developer checks the finding.
A leaked secret in the diff keeps the PR away from Jev, as from the LLM.

In `combined` mode, werkrevisor appends a hint list to the user prompt after the full files:
file, line, scenario id, question and value of each flagged unit, but no text from the PR.
An answer ignored for its low confidence is in the list too, marked as uncertain.
The LLM checks these units and writes concrete findings.

The comment gets a table "Classifier" with the flagged units, the count of passed units
and the count of answers ignored for their low confidence.

## Output

- One PR comment per head commit: verdict, reasons, up to `max_findings` findings with file and lines,
  the mechanical files in one line,
  usage (model, calls, token count before the call, reported tokens, duration, cost), and the reviewed head and base commit.
  The reasons are sorted from the most to the least severe: ERROR, RED, ORANGE, YELLOW.
  A reason from a limit names its config key and value, e.g. `[size] max_changed_lines = 800`.
  A reason that concerns many files is one line with at most 3 examples and the count of the rest.
  The usage names the config sources, e.g. `config: defaults + host + repository (.werkrevisor.toml at base 3d174cc)`.
- The findings in the comment show their severity, e.g. `1. **[high] Possible Bug** at …`.
  They are sorted from high to low severity, before the cut at `max_findings`.
- The commit status `ai-review/assessment` on the head commit.
- One JSON line per run on stdout and in `build/werkrevisor/usage.jsonl`.
- A short summary on stderr.

The usage line has timestamp, repository, PR, head and base commit, config sources (`config_source`)
and host config (`host_config`: `env` or the file path), verdict, exit code, change kind, count of mechanical files
(`mechanical_files`), reasons, model, calls,
prompt, completion and cache tokens, cost and cost status, the token count before the call
(`counted_input_tokens`, `counted_raw_tokens`, `token_count_factor`, `token_count_method`) and `worst_case_usd`, duration,
and the key budget before and after the run.
With the classifier it also has the engine mode, the Jev model, units, flagged and ignored answers,
calls, input tokens and cost.
Reasoning tokens are part of the completion tokens and are not counted twice.
The cost is `reported` when PR-Agent could price every call, `estimated` from the operator prices otherwise,
`none` with `cost_usd` 0.0 when no model call was made, or `unknown` when a model call reported no usage.
The comment then says "no model call, cost 0 USD" or "cost unknown".

An ERROR comment has its own marker, so a successful retry can still post its result.
werkrevisor posts at most one ERROR comment per head commit.

## Build and test

The runtime is the pinned image `pragent/pr-agent:0.47.0@sha256:7d98954c…` (Debian 13, Python 3.14, PR-Agent under `/app`).
werkrevisor uses only the Python standard library and the packages in that image.

Run the tests inside the image:

```sh
docker run --rm -v "$PWD":/werkrevisor -w /werkrevisor --entrypoint python \
    pragent/pr-agent:0.47.0@sha256:7d98954c29289846a08faf00aaea0f1f1375b1676cd8a2ed303e0af07010af7a -m unittest -v
```

The tests need no network, no LLM and no Gitea.
One test runs the real PR-Agent reviewer with a fake Gitea provider and a fake model.

Build the image and the rootfs archive for hosts without Docker:

```sh
tools/build-rootfs
```

This creates the image `werkrevisor:<version>` and `werkrevisor-rootfs-<version>.tar.zst`.

Evaluate the classifier questions on local PR documents, without Gitea:

```sh
tools/jev-eval --units-only ../hsadmin-ng/doc/PR/*.md                          # extraction only, no network
TYPESAFE_API_KEY=… tools/jev-eval --config config/hsadmin-ng.toml --no-repo-config ../hsadmin-ng/doc/PR/*.md
TYPESAFE_API_KEY=… tools/jev-eval --config config/werkrevisor.toml docs/prs/*.md
```

It uses the config layers of the review.
`--repo-config FILE` puts a repository config on top of the host config, with the same rules as the review;
without it, `.werkrevisor.toml` of the current directory is used if present, unless `--no-repo-config` is given.

It prints one line per unit and question (file, unit, question, problem value, confidence, verdict)
and the totals (verdicts, calls, input tokens, cost).
An answer below `min_confidence` shows `IGNORED` with the confidence and counts apart from the verdicts.
The archive has no image metadata.
Start it with `PATH=/app/.venv/bin:/usr/local/bin:/usr/bin:/bin`, `PYTHONPATH=/werkrevisor:/app`, `PYTHONNOUSERSITE=1`
and the command `python -s -m werkrevisor`.

Measure real PRs without writing to Gitea and without a model call:

```sh
GITEA_BOT_TOKEN=… tools/dry-run https://git.example.org/owner/repo/pulls/302 config/hsadmin-ng.toml
```

## Operations

### Werkator build

werkrevisor runs as a Werkator (v1.36.0) build with `trigger.onPullRequestTarget: true`.
Such a build is host-only: put it into the machine config `.git/werkator/.werkator.yml` of the checkout, never into a committed file.
Werkator checks out the base branch, sets the `WERKATOR_PR_*` and `WERKATOR_GITEA_*` variables,
and publishes the exit code under the pinned `statusContext` on the PR head.
The werkdock sandbox clears the environment and the rootfs archive keeps no image metadata,
so `werkdock.env` sets `PATH` and `PYTHONPATH`.
The host config comes from the host, never from the checkout:
`WERKREVISOR_CONFIG_TOML` holds the TOML itself, else `WERKREVISOR_CONFIG` names a file in the image (`/werkrevisor/config/`).
The repository config comes from the base commit through the Gitea API, never from the PR head.

### Hardening against a writable HOME

werkdock may bind a home directory as `/root` that ordinary PR builds of the same repository can write.
werkrevisor therefore loads no code and no data from HOME:

- Start it with `python -s -m werkrevisor` and `PYTHONNOUSERSITE=1`; the image entrypoint does so.
- At startup it refuses to run (exit 30) if user site-packages are enabled or a `sys.path` entry lies under HOME.
- It removes the working directory, a repository checkout, from `sys.path`.
- Before PR-Agent and litellm load, it points `HOME` and the XDG directories to a fresh temporary directory,
  sets `LITELLM_MODE=PRODUCTION` (no `.env` file is loaded) and `LITELLM_LOCAL_MODEL_COST_MAP=True` (no price map download),
  and forces `TIKTOKEN_CACHE_DIR` to the tokenizer that litellm ships.

### Machine config

mih09, werkdock with the rootfs archive:

```yaml
# .git/werkator/.werkator.yml (not committed)
builds:
  ai-review:
    trigger:
      onPullRequestTarget: true
    statusContext: ai-review/gate
    cleanCommand: ""
    buildCommand: python -s -m werkrevisor
    timeout: 10m
    artifactDirs: [build/werkrevisor]
    werkdock:
      enabled: true
      rootfs: /path/to/werkrevisor-rootfs-0.15.0.tar.zst
      env:
        PATH: /app/.venv/bin:/usr/local/bin:/usr/bin:/bin
        PYTHONPATH: /werkrevisor:/app
        PYTHONNOUSERSITE: "1"
        GITEA_BOT_TOKEN: "…"
        OPENROUTER_API_KEY: "…"
        # The complete operator config; start from config/werkprobe.toml and change what you need.
        WERKREVISOR_CONFIG_TOML: |
          [model]
          id = "openrouter/anthropic/claude-sonnet-5.5"
          input_price_usd_per_mtok = 2.00
          output_price_usd_per_mtok = 10.00
          max_context_tokens = 150000
          max_output_tokens = 4000
          reasoning_effort = "low"

          [budget]
          max_usd_per_review = 0.40
          min_key_remaining_usd = 0.50

          # … all other sections of config/werkprobe.toml …
```

vm4006, Docker:

```yaml
# .git/werkator/.werkator.yml (not committed)
builds:
  ai-review:
    trigger:
      onPullRequestTarget: true
    statusContext: ai-review/gate
    cleanCommand: ""
    buildCommand: python -s -m werkrevisor
    timeout: 10m
    artifactDirs: [build/werkrevisor]
    docker:
      enabled: true
      image: werkrevisor:0.15.0
      env:
        PYTHONNOUSERSITE: "1"
        GITEA_BOT_TOKEN: "…"
        OPENROUTER_API_KEY: "…"
        WERKREVISOR_CONFIG_TOML: |
          # the complete content of config/hsadmin-ng.toml, adapted
```

To switch the model or the reasoning level, edit `id`, the two prices, `reasoning_effort` and, if needed, `max_context_tokens`
in `WERKREVISOR_CONFIG_TOML` of the machine config; the next review uses them, no new image is needed.
The inline TOML replaces the config file completely, so paste the complete host config.
A config that lacks a section after the merge with `.werkrevisor.toml` is refused,
because a default section would have, for example, no risky paths.
Each run logs the config sources on stderr; the usage line has them as `config_source` and `host_config`.

A changed `rootfs` path makes werkdock import a new image, so give each version its own archive name.
Werkator needs `gitea.baseUrl`, `gitea.owner`, `gitea.repo` and `git.token` for this trigger.

Set the build timeout above `timeout_seconds` × (`max_retries` + 1) + 120 seconds;
werkrevisor stops itself at that limit.
Make `ai-review/gate` a required status in the Gitea branch protection only after the trial.
A review approval starts no Werkator build; restart the build by hand to let werkrevisor see a new human approval.

### Secrets

- `GITEA_BOT_TOKEN` belongs to a dedicated bot user.
  It needs repository read access, issue comment write access, commit status write access,
  and, for `pass_with_review_request` and `approval`, the right to request reviewers.
  For `approval`, it needs write access to submit official reviews and must be in `approvals_whitelist_username`.
- `OPENROUTER_API_KEY` is a dedicated key per host with a credit limit in OpenRouter.
- `TYPESAFE_API_KEY` is a TypeSafe key for Jev; it is needed only when `[engine] mode` is not `llm`.
- Store them in the machine config `.git/werkator/.werkator.yml` (`werkdock.env` or `docker.env`) with file mode 0600,
  never in the repository or the operator config.

### Key exchange

1. Create a new key in OpenRouter (or a new Gitea token for the bot) with the same limits.
2. Replace the value in `werkdock.env` or `docker.env` of the machine config.
3. Restart Werkator, so it reads the machine config again.
4. Start a review, e.g. with a new commit on a test PR, and check the usage line.
5. Delete the old key in OpenRouter or Gitea.

### Stop, restart, remove

- Stop: disable the `ai-review` build in the Werkator config and reload Werkator.
  If `ai-review/gate` is a required status, remove that requirement first, else no PR can merge.
  In approval mode, also remove the bot from `approvals_whitelist_username`, so humans approve alone.
- Restart: enable the build again.
  A new commit or a re-run of the build reviews the current head again;
  an existing result comment for that head is reused without a model call.
- Remove: stop it, delete the build config with its secrets, the rootfs archive or image,
  and revoke the OpenRouter key and the bot token.