# werkrevisor werkrevisor is an AI review gate for Gitea pull requests. Simple PRs can merge after an automated review. Risky PRs get human attention. Werkator starts werkrevisor for each ready (non-draft) PR head commit; any other CI system can do the same, see [Integration](#integration). werkrevisor runs on a checkout of the base branch and never executes PR files. It reads the PR through the Gitea API, applies deterministic rules, and asks a model through PR-Agent 0.47.0 for a review. The exit code is the merge gate: Werkator publishes it as a commit status on the PR head. Status: in use for the werk repositories on git.javagil.de; see [Operations](#operations). ## Verdicts and exit codes | Verdict | Label | Exit code | Meaning | |---|---|---|---| | GREEN | PASS | 0 | Nothing below; hints may remain, e.g. low findings, writing rule violations or a cautious recommendation of the model. | | YELLOW | HUMAN_REVIEW | 10 | Dangerous change detected, but no finding: a dangerous path, the DB schema, a lost or disabled test, a PR above the `[size]` limits, a possibly severe finding that Jev doubts, a high risk or a security concern of the model, no AI review possible. | | ORANGE | FINDINGS | 15 | Probably real finding: Jev rates a model finding as probably real and severe, the classifier rates a PR-document unit as clearly bad. A developer checks the findings; an approval unlocks the merge. | | RED | CHANGES_REQUIRED | 20 | Certain finding: a deterministic rule is violated, e.g. an open question in the PR document; or a very likely risk and a finding that Jev confirms in the same file. Only a new commit that fixes it lifts it. | | ERROR | review not completed, human action needed | 30 | Technical error, invalid model answer, timeout, budget, skipped files, superseded head. | The order is GREEN < YELLOW < ORANGE < RED < ERROR; the worst reason decides. ERROR is never GREEN. A run without a complete model review is never GREEN, except in these cases: - a PR of mechanical changes and images only; see [`[mechanical]`](#mechanical) and [`[images]`](#images); - a human approval carried over from an earlier head; see [Approval after a rebase](#approval-after-a-rebase); - Jev judged every changed file, with `[engine] model_role = "advises"` or `mode = "classifier"`; see [`[path_judge]`](#path_judge); - in classifier mode, a PR of PR documents only, all judged by Jev. Only deterministic rules give RED. A model or a classifier can be wrong, so their findings give at most ORANGE and never block hard. A reason with GREEN is a hint: the comment lists it as `hint`, and the bot still approves. On dev.hostsharing.net the repository administrator is an operator, not a developer; an administrator merge override is therefore no way out of a false RED. With `yellow_gate = "block"` or `"pass_with_review_request"`, YELLOW and ORANGE can still pass the gate (exit 0): - A configured human reviewer approved exactly this head commit (Gitea review `commit_id`). - `yellow_gate = "pass_with_review_request"` is set and a review request to a configured human reviewer is pending. With `yellow_gate = "approval"`, the missing approval is the lock; see [Approval mode](#approval-mode). An approval does not unlock RED. Only a new head commit that fixes the violated rule can change RED. ## Approval mode In this mode the branch protection requires one approval, and the bot is one of the allowed approvers. werkrevisor acts as a reviewer: | Verdict | Gitea action | Exit code | |---|---|---| | GREEN | Submit a review `APPROVED` with `commit_id` = head commit. | 0 | | YELLOW | A bot review `COMMENT` only if it has line comments. Request an official review from `human_reviewers`, unless a request is pending or a human approved this head. | 0; 10 if the request failed | | ORANGE | As YELLOW: neither `REQUEST_CHANGES` nor `APPROVED`; request an official review from `human_reviewers`. The findings are line comments. | 0; 15 if the request failed | | RED | Submit a review `COMMENT` with `commit_id` = head commit and up to 3 violated rules; the exit code fails the required status check. | 20 | | ERROR | No review. | 30 | | Superseded head | No review, no comment. | 30 | werkrevisor never submits `REQUEST_CHANGES`: the bot cannot dismiss it without admin rights, and Gitea keeps it across later commits, so a fixed RED would still block the merge. The failed status check of a RED head blocks it, and the next commit gets a new status. werkrevisor reads the PR again directly before it writes anything. It posts the result comment for every verdict, including GREEN. The bot review carries every finding with a line in a changed file as a line comment, which a human resolves in Gitea and Gitea marks as outdated when the line changes; the result comment lists such a finding in one line and gives the full text only of the others. The result comment holds the full text of every finding until Gitea has accepted the line comments, so a refused or failed review loses no finding, even if the following edit of the comment fails too. An unresolved bot line comment within two lines of a new finding is reused instead of repeated. The bot edits it to the text of the new finding and names the updating head; if the edit fails, the result comment holds the new text. A new result comment folds the bot's results for earlier commits into a `
` block of one line. A rerun on the same head does not submit a second review. An earlier bot review of the PR stays as it is; Gitea marks a review on an older head as stale. The bot never approves a PR that it opened itself: that is ERROR. Branch protection of the base branch (Gitea 1.27 API field names): | Field | Value | Why | |---|---|---| | `required_approvals` | `1` | The approval is the lock. | | `enable_approvals_whitelist` | `true` | Only listed users count. | | `approvals_whitelist_username` | the bot user and `human_reviewers` | A GREEN bot review or a human approval unlocks the merge. | | `block_on_rejected_reviews` | `true` | A human `REQUEST_CHANGES` blocks the merge. | | `block_on_official_review_requests` | `true` | The review request of a YELLOW or ORANGE blocks until the human reviews. | | `dismiss_stale_approvals` | `true` | A new commit removes the bot approval of the old head. | | `block_on_outdated_branch` | `true` | Werkator starts no new review when only the base moves. | | `enable_status_check` | `true` | The Werkator status is required. | | `status_check_contexts` | `["ai-review/gate"]` | The pinned `statusContext` of the Werkator build. | The bot user needs write access to the repository to submit official reviews. A repository admin sets these fields with `tools/branch-protection`; it saves the previous rule as a JSON file first and keeps its status contexts, e.g. `GITEA_TOKEN=… tools/branch-protection https://git.javagil.de/mi/werkator main` `--contexts werkator,ai-review/gate --approvers ai-reviewer,mi`. `--dry-run` shows the changes only, `--block-admin-override` also stops admins. A human approval counts only when Gitea recorded it after the result comment of its head: an approval given while werkrevisor still reviewed the head was given without its findings. For such an approval of a YELLOW or ORANGE head, the bot requests a review from that human again; Gitea then no longer counts the earlier approval, and the next one merges a scheduled merge. ### PR workflow and automerge An agent opens its PR as the Gitea account `claude-code`, with the title prefix "WIP: "; Werkator starts no review for a WIP PR. Removing the prefix marks the PR as finished, in the Gitea UI or with `PATCH /api/v1/repos///pulls/` and the new `title`. With `[gate] require_wip_transition = true`, werkrevisor reviews only a PR whose title Gitea recorded as changed from "WIP: " to one without. A PR opened without "WIP: " gets ERROR, no review and no approval; setting "WIP: ", pushing the next commit and removing "WIP: " brings it back into the workflow. With `[gate] automerge = true`, the bot schedules the merge with Gitea's "merge when checks succeed" after GREEN, YELLOW and ORANGE, in the merge style and branch deletion of the repository. Gitea merges a GREEN PR at once, and a YELLOW or ORANGE one when a human approves it; RED and ERROR never. `automerge` needs `yellow_gate = "approval"` and `require_wip_transition`, so that a PR opened without "WIP: " can never merge after its first green review. In a repository with `[codex] mode = "gate"`, the branch protection also requires the status `ai-review/codex`, so a scheduled merge waits for the [Codex runner](#codex-runner) as well. The status is `success` when every Codex line comment is resolved, and for a critical PR or a finding outside the diff only after the approval of a human for the head; see [`[codex]`](#codex). Gitea refuses to schedule the merge of a PR in conflict with its base branch, and for a moment after a merge into it. The bot then tries once more after 5 seconds. If Gitea refuses again and calls the PR not mergeable, the result comment asks the author to rebase onto the base branch and push, and a YELLOW or ORANGE review requests no human review: an approval could not merge the PR. The push starts a new round, which asks the human. With `[gate] hints_hold_merge = true`, the default, a GREEN review that leaves unresolved line comments of the bot, from this or an earlier round, approves but schedules no merge, and cancels a merge scheduled for an earlier head; the result comment says how many hints are open. A violation of the writing rules is such a line comment too, at most 10 per review; further violations stay in the result comment and hold the merge as well. The author replies to each line comment and resolves it. In a repository of the [Codex runner](#codex-runner), the runner then schedules the merge in its next run; elsewhere the author restarts the review build in Werkator or pushes a commit, and a review of a GREEN head without open bot line comments schedules the merge. A hint without a line comment, e.g. one that Gitea refused, stands only in the result comment; the note asks to answer it in a PR comment, and a restart of the review releases it. A hint on a file that the PR does not change stands only in the result comment and holds nothing: no line comment could be resolved there. A hint of the model on a file of the category `docs` is no line comment: it stands only in the result comment and holds nothing, unless `[review] doc_hint_line_comments = true`. A frontier model rated 25 such hints of werkrevisor's own PRs: 19 were noise, but each one had cost a review round. A hint on a PR document (`[pr_doc] globs`) is a line comment anyway, since it is the record of the PR, in the first round on every line, later only on changed lines, see below. From the second round on, a new hint on documentation, or a violation of the writing rules, becomes a line comment only on lines that changed since the head of the last round; on older text it stands only in the result comment and holds nothing. Of 111 line comments in the 42 PRs that needed more than one round from 2026-10-09 to 2026-10-12, 51 were on documents and 13 were writing rules; a fix of one point found the next one in text the last round had seen. If the earlier version of the file cannot be read, the hint holds as before; `[review] doc_hint_line_comments = true` keeps every hint of the model a line comment. YELLOW and ORANGE are not affected: the approving human reads the hints. When a GREEN review schedules the merge, it withdraws the review request of an earlier YELLOW or ORANGE round from the configured human reviewers, since `block_on_official_review_requests` would block the merge. ### Approval after a rebase Gitea dismisses an approval when new commits arrive (`dismiss_stale_approvals`), also after a rebase onto `main`. Every result comment therefore holds a hidden fingerprint per changed block: the path, the removed and the added lines, without line numbers and context. With `[gate] carry_over_approvals = true`, the default, a review looks for the newest verdict of `human_reviewers`. If it approved another head after the result comment of that head and only a push dismissed it, werkrevisor compares the blocks of both heads: 1. The new head has the same blocks: werkrevisor approves without Jev and model. 2. It has new blocks too, e.g. a resolved merge conflict: Jev judges the files with new blocks; without risk, werkrevisor approves without a model call. 3. Jev sees a risk: the model reviews the files with new blocks only; a finding on approved lines is a hint. A block of the approved head that the new head lacks ends the carry-over, since it can be a check the PR added. An approval that a human dismissed by hand, or a later "request changes", is not carried over. After a GREEN review of werkrevisor itself, the next head gets a full review, but a finding on lines of that GREEN head is a hint. The CI must still pass: the blocks do not show whether the newer `main` still fits the change. ## How a run works Three parts decide a review, each with what it does best: the workflow code applies fixed rules and owns every write to Gitea, Jev answers fixed questions with calibrated probabilities, and the LLM reads the code and reports findings. Only the workflow code turns their results into a verdict, so neither Jev nor the LLM can make a PR better than the rules allow. ```mermaid flowchart TD classDef code fill:#e4eef3,stroke:#0e6b7d,color:#141616 classDef jev fill:#e8ddf1,stroke:#6a3d9a,color:#141616 classDef llm fill:#dfeee0,stroke:#2e7d32,color:#141616 classDef io fill:#ffffff,stroke:#9e9e9e,color:#141616 subgraph LEGEND["Legend"] direction LR LCODE["workflow code,
deterministic"]:::code LJEV["Jev: fixed questions,
probabilities"]:::jev LLLM["LLM via OpenRouter"]:::llm end PR(["Pull request in Gitea"]):::io READ["Read the PR and the config layers;
reuse the result of an already reviewed head"]:::code RULES["Rules: change kind, hard paths, secrets, size,
open questions, writing rules, mechanical files, images"]:::code JDOC["PR documents:
sections and scenarios"]:::jev JDB["DB changesets:
deploy risk, data loss"]:::jev JFILE["Each changed file: weakened checks,
security, risk categories"]:::jev JDOCCHECK["Each changed file against the
PR document: contradiction,
unrelated change, data outflow"]:::jev CONTEXT["Context: full files or excerpts;
an economy model selects the lines"]:::code REVIEW["Model review by PR-Agent,
with the hints of Jev"]:::llm JFIND["Each finding: real defect,
severity, risk category"]:::jev COMBINE["Very likely risk and confirmed finding
in one file or category: RED"]:::code VERDICT["Verdict: the worst reason wins"]:::code WRITE["Result comment, line comments, status,
approval; open hints hold the merge"]:::code STATS(["Statistics service:
usage record, seconds per phase"]):::io PR --> READ --> RULES RULES --> JDOC & JDB & JFILE & JDOCCHECK & CONTEXT CONTEXT --> REVIEW JDOC -- "hints" --> REVIEW REVIEW --> JFIND JFILE --> COMBINE JFIND --> COMBINE RULES -- "RED from rules" --> VERDICT JDOC & JDB & JFILE & JDOCCHECK & JFIND --> VERDICT COMBINE --> VERDICT VERDICT --> WRITE --> PR WRITE --> STATS ``` A secret in the diff keeps it from Jev and from the LLM; a failed Jev call never makes a path or a finding GREEN. The same review from the view of a pull request: who acts after each verdict, and how a new commit starts a new round. How a pull request gets merged with werkrevisor: the author, werkrevisor, a human reviewer and Gitea The inside of one review: which files configure it, which services each step calls, and when. Inside a werkrevisor review: configuration files, the ten steps and the services they call Where a review ends early or takes a shortcut, e.g. RED for an open question without a model call. Decisions in a werkrevisor review: each check, its exit and the result of the exit The steps have the numbers of the inside diagram. 1. Read the pull request. If its head differs from `WERKATOR_PR_HEAD_SHA` or the PR is not open, stop with exit 30 and write nothing. Read `.werkrevisor.toml` at the head of the default branch and put it on top of the host config; see [Config layers](#config-layers). With `[gate] require_wip_transition = true`, a PR whose title never changed from "WIP: " gets ERROR and no review. If an own result comment with the marker `` exists, reuse its verdict, do not call the model, and do not comment again. Look for a human approval of an earlier head that a push dismissed; see [Approval after a rebase](#approval-after-a-rebase). 2. Apply the deterministic rules to the changed files and the diff. A PR above the `[size]` limits is YELLOW; the model still reviews it, so its findings help the human reviewer. Check the changed PR documents for open questions; an open question gives RED without a model call. Check the changed lines of the docs against the writing rules; a violation is a hint and a line comment, and the model still reviews the PR. Check the size and the signature of each changed raster image; see [`[images]`](#images). A possible secret in the added lines gives YELLOW, and nothing goes to Jev or a model. Without a secret, a PR of mechanical changes and images only ends here, with the verdict of the rules; so does a PR whose blocks a human approved on an earlier head. 3. With `[engine] mode = "classifier"` or `"combined"`, Jev judges the sections and scenarios of the changed PR documents; see [Classifier](#classifier). 4. With `[db_schema] judge = "jev"`, Jev judges the changed changesets; see [`[db_schema]`](#db_schema). 5. With `[path_judge] judge = "jev"`, Jev judges the changed judged paths; with `scope = "all"` every changed file, and with `risk_template` also by the risk categories. 6. With `[doc_check]`, Jev compares the diff of each changed file with the changed PR documents; see [`[doc_check]`](#doc_check). 7. With `full_files = true`, read the new version of every changed text file at the head commit; a large file goes as an excerpt, see [`[excerpt]`](#excerpt). 8. Run the PR-Agent reviewer with all settings forced by werkrevisor, except in classifier mode. It leaves out the mechanical files and the images. Right before the model call, count the tokens of the prompt that PR-Agent built; see [Token count](#token-count). Above the effective input limit (see [`[budget]`](#budget)) the prompt loses the full files, then the excerpts; if it is still too large, the result is YELLOW without a model call. Too little budget on the OpenRouter key gives ERROR without a model call. Check that the model saw every changed file completely. With `[engine] model_role = "advises"` and Jev's judgment of every file, the doubts of the model and a failed model call are hints. 9. Jev judges each finding of the model: real defect, severity and, with a risk template, its risk category. A very likely risk and a confirmed finding in the same file or category give RED. 10. The worst reason decides the verdict; a GREEN without a complete model review becomes YELLOW, except in the cases under [Verdicts and exit codes](#verdicts-and-exit-codes). After `[gate] max_red_rounds` heads with RED in a row, a new RED also requests the human reviewers. Read the PR again; if the head moved meanwhile, stop with exit 30 and write nothing. Write the result comment, the commit status `ai-review/assessment`, the line comments, and in approval mode the approval, the review request or the scheduled merge; see [Approval mode](#approval-mode). Write one usage record and send it to the [statistics service](#statistics-service). The deterministic rules come only from the host config and the repository config of the default branch. The model can make a verdict worse, never better than the rules allow. If the secret scan fires, the diff is not sent to the model. ## Configuration The operator config is TOML. `WERKREVISOR_CONFIG_TOML` holds it inline; if that is empty, `--config` or `WERKREVISOR_CONFIG` names a file. If both are set, the inline TOML wins. An invalid inline TOML is ERROR (exit 30); the message names the line and column, never the content. Unknown keys are errors, so a typo cannot weaken a rule. Examples: [config/werkprobe.toml](config/werkprobe.toml) (small budget), [config/hsadmin-ng.toml](config/hsadmin-ng.toml) (1 USD per review) and [config/werkcrappit.toml](config/werkcrappit.toml) (Jev decides, an economy model advises). ### Config layers Three layers make the config; a later layer wins per key: 1. the built-in defaults of werkrevisor, 2. the host config (`WERKREVISOR_CONFIG_TOML` or `WERKREVISOR_CONFIG`), 3. the repository config `.werkrevisor.toml` at the repository root. werkrevisor reads `.werkrevisor.toml` through the Gitea API at the head of the default branch, e.g. `main`, never at the PR, so a PR cannot change its own review, also not one stacked on another branch. Without the file there is no repository layer. After the merge of both layers, every section except `[engine]` and `[classifier]` must be set; else the result is ERROR. The repository config may set these keys: | Section | Keys | Rule | |---|---|---| | `[size]` | all | replaces the host value | | `[categories]` | all | replaces the host value | | `[hard_yellow]` | `paths` | replaces the host paths; `.werkrevisor.toml` itself stays hard | | `[hard_yellow]` | `judged_paths` | replaces the host paths | | `[db_schema]` | `paths` | replaces the DB schema paths | | `[path_judge]` | `judge` | `"never"` always; `"jev"` only if the host keeps `repository_may_enable = true` | | `[hard_yellow]` | `secret_scan_exclude` | replaces the host value | | `[hard_yellow]` | `secret_scan` | can only stay on | | `[pr_doc]` | all | replaces the host value | | `[writing]` | all | replaces the host value | | `[mechanical]` | `enabled`, `whitespace_significant` | replaces the host value | | `[mechanical]` | `imports` | replaces the host regexes per file extension | | `[images]` | `globs` | replaces the host value | | `[images]` | `max_bytes`, `max_total_bytes` | only lower or equal | | `[review]` | `extra_instructions` | appended to the host instructions | | `[review]` | `max_findings`, `orange_min_medium`, `doc_hint_line_comments`, `repo_context_files`, `full_files` | replaces the host value | | `[review]` | `max_full_file_chars`, `max_full_files_chars` | only lower or equal | | `[classifier]` | `pr_doc_globs`, `min_confidence`, `scenario_questions`, `section_questions` | replaces the host value; needs a host `[classifier]` | | `[model]` | `max_context_tokens`, `max_output_tokens` | only lower or equal | | `[budget]` | `max_usd_per_review` | only lower or equal | Every other key is a host key: the model, its prices and reasoning, the rest of the budget, `[gate]`, `[engine]`, and the classifier endpoint, model and limits. A repository config that sets one of them is ERROR, and the message names the key. Model, money and merge rights stay with the host operator. `.werkrevisor.toml` is always a risky path (see [`[hard_yellow]`](#hard_yellow)), so a change needs a human approval. werkrevisor also checks the changed file of the PR against the host config; an invalid file is RED, because after the merge every review would end with ERROR. A section, key or change kind unknown to the installed werkrevisor version gives YELLOW instead, if the rest of the file is valid: a newer version may know it, so deploy that version right after the merge. The change takes effect only after the merge. [config/werkrevisor.toml](config/werkrevisor.toml) holds only the host part for werkrevisor itself; the repository part is [.werkrevisor.toml](.werkrevisor.toml). ### `[model]` | Key | Default | Meaning | |---|---|---| | `id` | required | LiteLLM model id, e.g. `openrouter/google/gemini-3.8-flash`. No fallback models are used. | | `input_price_usd_per_mtok` | required | USD per 1M input tokens, maintained by the operator. | | `output_price_usd_per_mtok` | required | USD per 1M output tokens, maintained by the operator. | | `max_context_tokens` | required | Quality cap for the input: long prompts degrade a review even within the context window. About 150000 for frontier models, 80000 for economy models. Replaces `max_input_tokens` of 0.6.0. | | `max_output_tokens` | required | Output limit including reasoning tokens; depends on the model, see [Model-dependent values](#model-dependent-values). | | `reasoning_effort` | `"low"` | `none`, `minimal`, `low`, `medium`, `high`, `xhigh` or `max`; depends on the model, see [Model-dependent values](#model-dependent-values). | | `reasoning_max_tokens` | `0` | OpenRouter reasoning budget; 0 leaves it unset. | | `context_tokens` | `0` | PR-Agent's window; 0 means twice the input limit plus `max_output_tokens`, see [Token count](#token-count). | | `timeout_seconds` | `120` | Timeout per model call; depends on the model, see [Model-dependent values](#model-dependent-values). | | `max_retries` | `1` | 0 or 1; PR-Agent retries a failed call at most once. | | `token_count_factor` | `1.15` | Multiplies the `o200k_base` count of the prompt; host key. See [Token count](#token-count). | | `provider_order` | `[]` | OpenRouter providers to ask first, e.g. `["minimax"]`; others answer when these fail. | #### Model-dependent values `max_output_tokens`, `timeout_seconds` and `reasoning_effort` have no value that fits every model. Check them whenever the model changes: - `max_output_tokens` includes the reasoning tokens. A model that thinks long needs a high limit; if the limit is reached, the model returns empty content with `finish_reason: length`, and the review fails. - The duration of a call grows with the output and differs between providers, so `timeout_seconds` must cover the slowest calls of the chosen model. - A higher `reasoning_effort` usually means more output tokens and more time. - The budget check assumes the worst case: 2 attempts, each with the full input and `max_output_tokens`. A higher output limit therefore lowers the input limit within the same `[budget] max_usd_per_review`; raise the budget with the output limit, or the prompt gets smaller. Values from the model comparison of 2026-10-06 and 2026-10-07, where each model reviewed the same 4 PR heads: | Model | `reasoning_effort` | Duration per review | `max_output_tokens` | Note | |---|---|---|---|---| | Claude Sonnet 5.5 | `medium` | 24–63 s | 10000 | | | Grok 4.7 | `low` | 120–209 s | 32000 | `medium` exceeded 32000 output tokens and 600 s | | MiniMax M3 | `medium` | 181–242 s | 64000 | some calls hang at some providers; see `provider_order`; up to 47050 output tokens (PR#84) | | MiniMax M3 | `none` | 32–103 s | 32000 | not effective at every provider | | GPT-6 Astra | `high` | 140–221 s | 32000 | | | Mistral Large 4 | `low` / `medium` | up to 447 s | – | 3 of 4 reviews exceeded 360 s or 600 s on 2026-10-07 | ### `[budget]` | Key | Default | Meaning | |---|---|---| | `max_usd_per_review` | required | Worst-case cost limit per review; it derives an input limit, see below. | | `min_key_remaining_usd` | `0.0` | The key must keep at least this budget, else ERROR. | | `chars_per_token` | `3.5` | Fallback estimate without a tokenizer, see [Token count](#token-count). | | `prompt_overhead_chars` | `16000` | Fallback estimate: PR-Agent prompt size plus repository context files. | | `diff_expansion_factor` | `1.5` | Fallback estimate: PR-Agent adds context lines and line numbers to the diff. | The worst case is (counted input × input price + `max_output_tokens` × output price) × (`max_retries` + 1). The budget-derived input limit is the largest input whose worst case stays within `max_usd_per_review`. The effective input limit is the lower of this limit and `[model] max_context_tokens`; a PR whose token count exceeds it gets no AI review and is YELLOW, and the reason names the count and the limit that applied. Example: Sonnet 5.5 at 2 / 10 USD per 1M tokens, `max_output_tokens = 4000`, 2 attempts and 0.40 USD give 80000 input tokens. A config whose budget does not even pay for `max_output_tokens` is refused. After the token count and before the model call, werkrevisor reads `GET https://openrouter.ai/api/v1/key`. If `limit_remaining` is below `max_usd_per_review` or `min_key_remaining_usd`, the result is ERROR "budget". A key without a limit passes this check. #### Token count Right before the model call, werkrevisor counts the system and the user prompt that PR-Agent passes to its AI handler: the instructions, the PR description, the diff, the full files, the classifier hints and the `repo_context_files`. It counts with tiktoken `o200k_base`; litellm ships its data in the image, so nothing is downloaded. The count times `[model] token_count_factor` decides about the input limit and the worst-case cost, because Anthropic and other providers count more tokens than `o200k_base`. Only the host config sets the factor, so a PR cannot lower the count of its own review. werkrevisor 0.10.0 estimated characters / 3.5 instead, and the estimate was much too low: | PR | Estimate | Prompt tokens reported by the model | |---|---|---| | hsadmin-ng PR#305 | about 168000 | 268735 | | werkrevisor PR#7 | below 80000 | 87212 to 88369 | Calibrate the factor with the usage records of real reviews: `prompt_tokens` / `counted_raw_tokens`. On werkrevisor PR#10 the factor 1.15 gave 49841 tokens, and Anthropic reported 65792 prompt tokens for Sonnet: 1.15 × 65792 / 49841 ≈ 1.52, so the example configs use 1.55. PR-Agent gets a window of twice the input limit plus `max_output_tokens`, or `[model] context_tokens` if set. Thus PR-Agent passes a too long prompt unshortened to the count, and the reason names the exact count. A prompt beyond that window, which PR-Agent prunes or refuses, gets no AI review either, and the reason names the window. If the tokenizer cannot load, werkrevisor uses the estimate from the `[budget]` keys above and says `token count estimated` in the reason, the comment and the usage line. ### `[size]` | Key | Default | Meaning | |---|---|---| | `max_changed_lines` | `800` | Changed lines (additions plus deletions) of all files outside the `docs` category, except mechanical files. | | `max_files` | `30` | Changed files outside the `docs` category, except mechanical files. | Above one of these limits, the PR is too large for an AI-only approval: the verdict is at least YELLOW, and a human approves it knowingly. The limits do not stop the model review; if the model can take the PR, its findings help the human. Docs do not count, because a PR document alone has 100 to 200 lines. This limit is separate from the model limit in [`[budget]`](#budget): a PR can be too large for an AI-only approval, too long for the model, or both; then both reasons appear. Changed lines outside the docs per merged PR, measured over one month: | Repository | PRs | Median | p75 | p90 | Max | |---|---|---|---|---|---| | Werkator | 92 | 194 | 489 | 865 | 1748 | | hsadmin-ng | 16 | 441 | 974 | 1098 | 1146 | With 800 lines, about every tenth Werkator PR needs a knowing human approval. hsadmin-ng uses 1000 lines: that is about every fourth PR, and it lies just below the hard limit of hsadmin-ng's own `tools/pr-quick-check` (about 1200 to 1400 lines). ### `[categories]` Glob lists per category: `docs`, `tests`, `main`, `api_spec`, `harmless`. The first match wins in the order `api_spec`, `tests`, `docs`, `main`, `harmless`; else the file is `other`. `harmless` (default `[]`) lists files, e.g. `.editorconfig`, that do not spoil an AI-approvable change kind. A pattern without `/` matches the file name in any directory; `**` matches any number of directories. `auto_approvable` lists the change kinds that can become GREEN: | Change kind | Condition | |---|---| | `docs_only` | Only docs. | | `tests_only_added` | Only tests, and the test changes are purely additive. | | `docs_and_tests` | Docs and tests; the test changes are purely additive. | | `refactoring` | Main code, optionally docs; no tests, no API spec. | | `feature_with_tests` | Main code and tests, optionally docs; the test changes are purely additive. | | `harmless_only` | Only files of the `harmless` category. | | `mechanical_only` | Only mechanical files, see [`[mechanical]`](#mechanical); no model call. | | `mixed` | Any other combination without a lost or disabled test, e.g. main code with an API spec or a build file. | Harmless files may come along with every change kind. Mechanical files do not count for the change kind, e.g. main code with a test file that changed only its imports is a `refactoring`. Purely additive test changes lose no test: they delete no test file, and per test file they remove no more lines with `@Test`, `assert`, `should` or `expect` than they add. So an assertion that checks a new value is a change, not a loss. werkrevisor lists every other changed line of an existing test in the model instruction; the model checks that the test stays as strict as before and reports a weakened test as `[high]`. Blank lines and a removed line without content that the diff adds again, e.g. a closing brace, do not count. An added line must not disable a test: `@Disabled`, `@Ignore`, `.skip(`, `.only(`, `pytest.mark.skip`, `pytest.mark.xfail`, `unittest.skip`, `skipTest(`, `xit(`, `xdescribe(` or `xtest(`. A lost or disabled test makes the kind `tests_modified` or `feature_with_modified_tests` and gives YELLOW. A change kind outside `auto_approvable` gives YELLOW too; the reason names the kinds of the changed files and the AI-approvable change kinds, e.g. with `auto_approvable = ["docs_only", "refactoring"]`: `change kind 'mixed' (main code, other files, docs); AI-approvable are only: docs only, refactoring with unchanged tests`. werkrevisor tells the model the change kind, so the model can object if, for example, a refactoring changes behavior. ### `[hard_yellow]` `paths` lists globs of risky files, e.g. authorization, RBAC, migrations, secrets, CI and deploy config, dependency manifests. A renamed file counts with its old and new path. Files that match the `docs` category globs never match `paths`, so a PR document named after a risky topic, e.g. `doc/PR/…-rbac-generator-hardening.md`, is no risky file. `secret_scan = true` scans added lines for private keys, `-----BEGIN`, AWS keys, generic `token|secret|password|api_key = "..."` assignments, and tokens with the prefixes `hsak_` and `sk-or-`. `secret_scan_exclude` lists globs of files with fake secrets, e.g. `["src/test/**"]` for test data. There, only real key formats count: private keys (`-----BEGIN … PRIVATE KEY`), AWS key ids, `hsak_.` with 64 hex digits and `sk-or-v1-` with 64 hex digits. `.werkrevisor.toml` is a built-in risky path; no layer can remove it. DB schema changes are risky paths too, outside Markdown files and test directories: by default `**/*.sql`, `**/db/changelog/**`, `**/migrations/**`, `**/migration/**`, `**/liquibase/**`, `**/flyway/**`, `**/*.changelog.*` and `**/db.changelog-*`; `[db_schema] paths` replaces them. The test directories `src/test/`, `**/src/test/`, `test/` and `tests/` are built in too. Their reason names them, e.g. `DB schema changed: src/main/resources/db/changelog/x.sql (built in)`; with `[db_schema] judge = "jev"`, Jev judges them instead, see [`[db_schema]`](#db_schema). Hard rules give YELLOW; the model cannot override them. `judged_paths` lists globs of files that Jev judges instead, e.g. build files, if `[path_judge] judge = "jev"`; see [`[path_judge]`](#path_judge). A file that also matches `paths` or a built-in risky path, with its old or new name, stays a fixed YELLOW. Without the judge, a judged path gives YELLOW like a path in `paths`. ### `[db_schema]` Optional, host only: only the operator knows whether a repository deploys through stages and can restore its database. | Key | Default | Meaning | |---|---|---| | `judge` | `"never"` | `never`: every DB schema change gives YELLOW. `jev`: Jev judges every new or changed changeset; needs `[classifier]`. | | `risk_min` | `2.5` | A changeset with at least this deploy risk, from 0 to 3, gives YELLOW. | | `data_loss_min` | `0.6` | A changeset with at least this probability of data loss gives YELLOW. | | `paths` | `**/*.sql`, `**/db/changelog/**`, `**/migrations/**` and other migration paths | The DB schema paths; the repository config may replace them. | werkrevisor splits Liquibase formatted SQL at `--changeset` lines and compares the changesets of the head and the merge base by their id. A changed changeset without `runOnChange:true`, `runAlways:true` or `--validCheckSum: ANY`, a removed changeset and a deleted schema file give YELLOW without Jev. Jev gets each new or changed changeset, a changed one with its version at the merge base, and rates the deploy risk and the data loss; a file without changesets, e.g. a master changelog, goes to Jev as its diff. A changeset below both thresholds is a hint `db_schema_judged`; above, it gives YELLOW `db_schema_risky`. A changeset that Jev cannot judge, e.g. with `--no-llm` or a secret in the diff, gives YELLOW `db_schema_not_judged`. On 34 hsadmin-ng changesets, only a dropped column reached the defaults: risk 2.99, data loss 0.84. Jev separates purely additive changesets (mean 0.55) from the others (1.4 to 1.9), but not DB code from access control. The usage record logs the values per changeset as `db_schema_judgments` for a later calibration. ### `[excerpt]` Optional, host only. An economy model selects the lines of an excerpt, so that the review model sees what a change depends on. | Key | Default | Meaning | |---|---|---| | `select` | `"context"` | `context`: `[review] excerpt_context_lines` around each changed line. `model`: the economy model selects the lines. | | `model` | `"openrouter/openai/gpt-6-luna"` | The OpenRouter model of the selection. | | `input_price_usd_per_mtok`, `output_price_usd_per_mtok` | `0.10`, `0.50` | Prices for the estimate when OpenRouter reports no cost. | | `max_lines` | `400` | A selection of more lines is refused. | | `max_output_tokens`, `timeout_seconds` | `2000`, `60` | Limits of one selection call. | | `context_lines` | `5` | Lines around each changed line besides the selection. | The economy model gets the diff and the numbered new version of the file and answers with JSON line ranges only. werkrevisor refuses an answer as a whole if it is no JSON, holds anything but pairs of line numbers inside the file, or selects more than `max_lines` lines; then the excerpt gets the fixed window, and a hint `excerpt_selection_failed` names the cause. No text of the economy model reaches the review model. A file above 400000 characters, a run without model call and a diff with a secret get no selection. The usage record holds `excerpt_model`, `excerpt_calls`, the tokens and `excerpt_cost_usd`; the statistics service counts the cost as LLM cost. On 4 heads with 10 known real findings, Sonnet found 2 with the selection of GPT-6 Luna and 1 with the fixed window; DeepSeek V4 Flash answered 7 of 10 selections with no JSON. ### `[doc_check]` Optional, host only; needs `[engine] mode = "classifier"` or `"combined"`. The check is Jev alone: for each changed file outside the PR documents and the mechanical files, Jev gets the PR document and the diff of that one file and answers, in one call, whether the file contradicts the PR document, is unrelated to it, or sends data outside. So a PR document that contradicts the code, a change that the PR document does not require, or a change that exfiltrates customer data gets a line comment before a human reads the PR. The check uses the model, endpoint, price and limits of `[classifier]`; it needs no OpenRouter key. | Key | Default | Meaning | |---|---|---| | `orange_min` | `0.7` | A test claim that the file contradicts from this value on gives ORANGE. | | `hint_min` | `0.7` | Each contradiction from this value on gives a hint; PR#66 measured the ones below 0.7 as noise. | | `unrelated_min` | `0.7` | A file that Jev rates as unrelated to the PR document from this value on gives a hint. | | `exfil_min` | `0.7` | A file that Jev rates as sending personal or customer data outside from this value on gives a hint. | Jev answers six questions per file, each a `noul` value: the four contradiction questions of PR#47, reworded from the summary to the diff of the file (does the file contradict a description, a claimed implementation, a requirement, or a claim about tests), the unrelated question of PR#116, and a data-outflow question (does the change send personal or customer data to an external service, a network endpoint, or a URL). The state also holds the conventions of the repository: a version raise, a plan update, release notes, tests, documentation and diagrams belong to a PR without a mention. A flagged file gets one line comment at its first changed line. A test claim that the file contradicts from `orange_min` on is ORANGE; every other flag is a hint. Like every hint, it holds the merge until it is resolved. The comment names a fingerprint of the file path, the diff and the PR document; a hint that the author resolved does not come back while both stay unchanged, but an ORANGE one does. A deleted or deletion-only file has no line in its new version, so its hint joins one comment at the PR document. A secret in the diff leaves the check out, with a hint; a file whose diff and the PR document exceed the `[classifier] max_state_chars` is not judged, with a hint. The usage record holds `doc_check_*` and the values per file as `doc_check_judgments`; the statistics service counts the check as Jev cost. On 22 merged PRs, unchanged PR documents reached at most 0.18 for a contradicting test claim, and a planted claim "no test changed" reached 0.71 to 0.93 in 14 of 14 documents; the other three contradiction questions separated less, so they give hints only. On the hs.hsadmin.ng backend, re-run live on 2026-10-17: the data-outflow question caught a backdoor woven into an on-topic file, which sends customer data by an HTTP query, in 6 of 6 runs from 0.97 on, with 0 false alarms on 103 real main-code file diffs of 15 merged PRs; the unrelated question of PR#116 misses such a backdoor, because the file is on topic. ### `[path_judge]` Optional. Jev judges each changed file below `[hard_yellow] judged_paths`, so that a routine build change needs no human; with `scope = "all"`, Jev judges every other changed file too, so that only a change that needs a second look waits for a human. | Key | Default | Meaning | |---|---|---| | `judge` | `"never"` | `never`: a judged path gives YELLOW. `jev`: Jev judges it; needs `[classifier]`. | | `repository_may_enable` | `true` | Host only: `false` forbids `judge = "jev"` in `.werkrevisor.toml`. | | `orange_weakens_min` | `0.8` | Host only: a probability of weakened checks from this value on gives ORANGE. | | `orange_risk_min` | `2.8` | Host only: a build risk from this value on, from 0 to 3, gives ORANGE. | | `green_weakens_max` | `0.3` | Host only: below this probability of weakened checks … | | `green_risk_max` | `1.5` | Host only: … and at or below this build risk, the change is a hint. | | `scope` | `"judged_paths"` | Host only: `all` also judges every changed file outside the docs, the mechanical files and the hard paths. | | `yellow_security_min` | `0.93` | Host only: a probability that a change touches security from this value on gives YELLOW. | | `risk_template` | `""` | A template of risk categories in `werkrevisor/risk/`; needs `scope = "all"`. The repository may set it. | | `repository_may_tune_risks` | `true` | Host only: `false` forbids `risk_template` and `[[risk.categories]]` in `.werkrevisor.toml`. | Jev gets the path and the diff hunks with their context lines and answers two questions: `weakens_checks`, e.g. tests disabled, failures ignored or a threshold lowered, and `build_risk`. ORANGE is `judged_paths_weakened`, YELLOW `judged_paths_risky`, a hint `judged_paths_judged`. A file that Jev cannot judge gives YELLOW `judged_paths_not_judged`: a deleted file, a file without a text diff, a file with a secret in the diff, or every file with `--no-llm`. A renamed judged file gives YELLOW as a risky path, since the diff of its content does not show the move. On 41 build file diffs, all 10 harmful ones got ORANGE, also 2 with a comment that called them harmless; of 27 real changes in hsadmin-ng and werkrevisor, 12 got GREEN and 15 YELLOW, none ORANGE. The usage record logs the values per file as `path_judgments`. With `scope = "all"`, Jev also answers `security`: does the change affect authentication, authorization, roles, API-key scopes, which endpoints need a login, or the handling of passwords, tokens or keys. With `scope = "all"`, a main-code file gets the questions `weakens_checks` and `security`, a test file only `weakens_checks`, since a test that names a role tests the access rule instead of changing it. ORANGE is `files_weaken_checks`, YELLOW `files_touch_security`, a hint `files_judged`. On the 25 merged hs.hsadmin.ng PRs #281 to #316, these two questions found 7 of the 9 PRs that needed a human and none of the 15 others; the build risk did not separate main code, so it counts only for judged paths. With `risk_template`, Jev also answers the risk categories of the template for each main-code file, e.g. object-level authorization, transaction integrity or a migration of existing data; the templates are `jvm-spring-backend` with 16 categories and `typescript-frontend` with 15. The backend category `data_correction` asks whether a mistake can write wrong data that would need corrections; it gives YELLOW from 0.85 on and never ORANGE. Each category has a question, the file kinds, `yellow_min`, an optional `orange_min` and an `action`: with `yellow`, a value from `yellow_min` on gives YELLOW and from `orange_min` on ORANGE; with `hint`, e.g. for performance or API compatibility, it gives a hint only. The reason `risk_` names the category and the files, and tells the author to revert a change that the task does not need. The question `security` stays: on the 25 PRs, the narrower categories alone missed 2 of the 9 PRs that needed a human. `.werkrevisor.toml` may choose the template and tune it with `[[risk.categories]]`, unless the host forbids it: an entry with the name of a template category changes its `yellow_min`, `orange_min` or `action`, `action = "off"` removes it, and an entry with a new name and all keys of a template category adds a category of its own. The host config must set `[path_judge] judge = "jev"` and `scope = "all"`; then a `.werkrevisor.toml` may say: ```toml [path_judge] risk_template = "jvm-spring-backend" [[risk.categories]] name = "sql_injection" action = "off" [[risk.categories]] name = "build_slots" title = "Build slots" question = "Does the change alter how the slots of parallel builds are taken and released?" yellow_min = 0.9 action = "yellow" ``` A file that Jev rates as very likely risky, by `weakens_checks` or a category from its `orange_min` on, and an ORANGE finding of the model that Jev confirms in the same file give RED with the reason `risk_with_issue`: only a new commit lifts it, while each of the two alone stays ORANGE, which a human can approve. With a risk template, Jev also assigns each finding to a category of the template or to "none". A confirmed finding in main code, assigned with a confidence of at least 0.7 to a category that Jev rates as very likely risky in any changed file, gives RED with the same reason. A finding in test code never counts by its category: there, 3 of 4 measured assignments were wrong. When Jev judged every changed file and `[engine] model_role = "advises"`, the model only advises: a finding that Jev doubts, a high risk, a file that reached the model only as diff or excerpt, and a PR too large for the model become hints; a security concern of the model and a finding that Jev confirms stay. The reason of a security concern quotes the model's text on one line, at most 400 characters. An answer "No", "False" or "None", in any case and also as a YAML bool, is no security concern. With `[engine] mode = "classifier"`, Jev's judgment of every file suffices for GREEN without a model call. ### `[pr_doc]` | Key | Default | Meaning | |---|---|---| | `globs` | `[]` | Globs of the PR documents, e.g. `doc/PR/*.md`; `TEMPLATE*.md` and `README.md` never count. | | `open_question_headings` | `["Open Questions", "Offene Fragen"]` | Headings of the open-questions section, compared without case and a trailing colon. | A PR never passes while a PR document that it adds or changes has open questions. The section starts at such a heading of any level and ends at the next heading of the same or a higher level; deeper subheadings stay in it. Allowed are an empty section and list items checked as done with `[x]`, e.g. `- [x] TODO.spec: …`. Every item with `[ ]` and every list item or paragraph without a checkbox gives RED. An open item that starts with `TODO.:`, e.g. `- [ ] TODO.tool: …`, is deferred work and only a hint. Blank lines, comments (``) and continuation lines of a done item do not count. The check is deterministic and runs before any model call; the model cannot lift it, and no model call follows. The reason lists up to 3 open items as `file:line: text`, then the count of the rest. ### `[writing]` | Key | Default | Meaning | |---|---|---| | `enabled` | `true` | Check the writing rules. | | `max_line_length` | `128` | Characters per line. | | `max_sentences_per_paragraph` | `5` | Sentences per paragraph; each list item is a paragraph of its own. | | `globs` | `[]` | Globs of the checked files; empty means the `docs` category. The PR documents of `[pr_doc]` always count. | werkrevisor checks the countable rules of the writing style in `AGENTS.md` on changed Markdown files: at most one sentence per line, at most `max_line_length` characters per line, and at most `max_sentences_per_paragraph` sentences per paragraph. Only lines that the PR adds or changes count, and a paragraph only if the PR changes one of its lines. A table row, a heading, a line with only a link or URL, a fenced code block, an HTML comment and the front matter have no line limit and no sentences. A tool writes the section from a comment `` up to the next heading; no writing rule checks it, and the prompts of the model and of Codex say to report nothing about it. The sentence split is conservative, so a false finding is rare: a sentence ends at `.` or `!` before a capital letter, but not after an abbreviation such as `e.g.`, `z. B.` or `Nr.`, an initial or a number. A question mark ends no sentence, because a PR document answers an open question on the same line. Code spans, URLs and link targets do not count. A violation is a hint: it is certain, but rarely worth a review cycle, so the comment shows it and nothing blocks. The check runs before the model call and does not skip it. The reason lists at most 3 violations as `file:line: rule`, then the count of the rest. ### `[mechanical]` | Key | Default | Meaning | |---|---|---| | `enabled` | `true` | Find the mechanical files. | | `whitespace_significant` | `*.py`, `*.yml`, `*.yaml`, Makefiles, `*.mk`, `*.md`, `*.haml`, `*.pug`, `*.coffee`, `*.nim`, `*.fs` | Globs of files in which whitespace has a meaning. | | `imports` | see below | Table from a file extension without the dot to the regexes of lines that an import change may add or remove. | A changed file is mechanical only if one of these rules explains every changed line: - `renamed`: Gitea reports the file as renamed with no added or removed line. A renamed file with changed lines is mechanical if the rules below explain each of them, e.g. "imports, renamed" for a package move, if it keeps its file extension, and if its old and new path are in the same category. A whitespace-significant name counts under the old and the new name. - `whitespace`: the lines differ only in the amount of whitespace, like `git diff -b`, or are blank lines. A file of `whitespace_significant` is never mechanical by this rule. - `imports`: the line matches a regex of its file extension, or it is part of a multi-line import statement whose lines all appear in the same diff hunk and match together. Blank lines count too. A file with an added or removed line that no rule explains is not mechanical. Neither is an added, deleted or binary file, or a file whose diff Gitea shortened. The built-in import regexes: | Extensions | Lines | |---|---| | `java`, `kt`, `kts` | `import a.b.C;`, `import a.b.*;`, `package a.b;` | | `ts`, `tsx`, `js`, `jsx`, `mjs` | named, default and namespace imports with `from`; never a side-effect import like `import './x'` | | `go` | `import "x"`, `import y "x"` and `import ( … )` blocks; never `_ "x"` or `. "x"` | | `rs` | `use a::b;`, `pub use a::{b, c};`; never `mod x;` | Python and all other extensions have no import rule. A `[mechanical.imports]` table in a config layer replaces the regexes of the extensions that it names, e.g. `scala = ['^\s*import\s+[\w.]+\s*$']`; an empty list switches an extension off. An invalid regex is a config error. A mechanical file does not go to the model, neither its diff nor its full file; werkrevisor passes it to PR-Agent as `ignore.regex`, and the coverage check does not miss it. It does not count for the change kind and for the check of purely additive tests. The hard rules apply to it as to every file, e.g. a mechanical change of a risky path gives YELLOW. The `[size]` limits do not count it, so a package move across many files does not make a PR too large. The comment lists the mechanical files in one line, with at most 3 examples and the count of the rest, e.g. `3 files changed only mechanically and were not reviewed by the model: A.java (imports), b.c (whitespace), c (renamed)`. ### `[images]` | Key | Default | Meaning | |---|---|---| | `globs` | `**/*.png`, `**/*.jpg`, `**/*.jpeg`, `**/*.gif`, `**/*.webp`, `**/*.ico` | Globs of the raster images. | | `max_bytes` | `512000` | Size limit of one image. | | `max_total_bytes` | `2000000` | Size limit of all added and changed images of a PR. | Neither Jev nor the model can read a raster image, so werkrevisor checks its size and its first bytes instead. Only a file with one of the extensions above counts, also with other `globs`; SVG is text and gets the normal review. werkrevisor reads at most `max_bytes` plus one byte of each image at the head commit. - An image above `max_bytes`, or all images above `max_total_bytes` together, give YELLOW. - An image whose first bytes do not match its extension gives YELLOW, e.g. an archive named `logo.png`. - Images that pass give a hint with their sizes, e.g. `images checked by size and signature only: doc/images/dashboard.png (177 KB)`. An image does not go to Jev or the model, and the coverage check does not miss it. An image outside the other categories counts as `harmless` for the change kind. The hard paths apply to it as to every file. ### `[review]` | Key | Default | Meaning | |---|---|---| | `extra_instructions` | `""` | Project review rules for the model. | | `max_findings` | `3` | Findings requested from the model and shown in the comment. | | `doc_hint_line_comments` | `false` | `true` makes a hint on a file of the category `docs` a line comment, like a hint on code. | | `orange_min_medium` | `2` | Without Jev: this many findings of medium severity give ORANGE; 0 disables it. See [Severity of the findings](#severity-of-the-findings). | | `finding_real_min` | `0.6` | Host only. With Jev: a finding is probably real from this probability on. | | `finding_severity_min` | `2.5` | Host only. With Jev: a finding is severe from this level on, on Jev's scale from 0 to 3. | | `repo_context_files` | `[]` | Files PR-Agent reads from the default branch, e.g. `AGENTS.md`. | | `full_files` | `false` | Add the complete new version of every changed file to the prompt. | | `max_full_file_chars` | `80000` | Limit per rendered file; a larger file goes as excerpt. | | `max_full_files_chars` | `160000` | Limit for all rendered files and excerpts together; a file beyond it goes as excerpt. | | `excerpt_context_lines` | `40` | Host only: lines before and after each changed line in an excerpt. | #### Severity of the findings The answer schema of PR-Agent 0.47.0 has no severity field. So the `extra_instructions` of the examples tell the model to start every `issue_header` with `[high]`, `[medium]` or `[low]`: - `[high]`: wrong behavior, data loss, a security problem or a broken contract in a realistic scenario. - `[medium]`: a defect with limited impact, or a missing test for changed behavior. - `[low]`: cosmetics, wording, log noise or style. werkrevisor reads the prefix case-insensitively and removes it from the title. A missing or unknown prefix counts as medium; the comment says `severity unknown, counted as medium`. A finding of low severity is a hint and never blocks. With `[engine] mode = "combined"`, Jev judges every other finding; the model cannot tell well whether its own finding is real. Jev gets the finding and 25 lines of code above and below it at the head commit, and answers two questions: the probability of a real defect, and the severity from 0 (no problem) to 3 (wrong behavior, data loss, security). | Jev | Verdict of the finding | |---|---| | severity ≥ `finding_severity_min` and real ≥ `finding_real_min` | ORANGE: probably real | | severity ≥ `finding_severity_min`, real below | YELLOW: possibly severe, Jev doubts it | | severity below `finding_severity_min` | hint | | Jev failed | YELLOW with the reason `finding_not_judged` | On 42 findings of known truth, 29 real and 13 false, the defaults gave 10 ORANGE findings: 9 of the 15 real medium or high findings, no real low finding, and 1 false finding. The comment shows Jev's two values per finding, and the usage record logs them for a later calibration. Without Jev, one finding of high severity or `orange_min_medium` findings of medium severity give ORANGE, and another finding of medium severity gives YELLOW. `merge_with_caution` and `changes_required` are hints; only the findings decide. werkrevisor 0.14.0 replaced `red_min_findings` with `orange_min_medium`; the old key is a config error. The full files share the effective input limit with the diff and the prompt overhead. Choose `max_full_files_chars` well below that limit, at about 3.5 characters per token, so that the diff of a PR at the `[size]` limit still fits; e.g. 160000 chars are about 46000 tokens of o200k_base, with `token_count_factor = 1.55` about 71000 of 100000. If the prompt still exceeds the input limit, werkrevisor sends excerpts instead of the full files, and if that is too large too, the diff only, each with a hint `context_reduced`; only a too large diff skips the review. A diff with a few context lines hides constraints stated elsewhere in a file, e.g. in its header comment. PR-Agent 0.47.0 caps the extra context lines per hunk at 10, so werkrevisor adds the full files itself: - It reads each changed, not deleted file at the head commit through the Gitea API (`/raw/{path}?ref=`). - It skips files that PR-Agent cannot review (binary extensions); the coverage rules already make them YELLOW. werkrevisor removes `gradle` from these extensions, since Gradle files hold build code. - It skips files of the `docs` category; they reach the model only as diff and do not count against the limits. The model reviews docs for their wording against the writing style, which the diff shows; full docs would crowd out the code files. - It appends one section after the diff, in the user prompt, never in the system prompt, because PR content is untrusted: `## Full content of the changed files (new version at , …)`, one `### Full file: '' ( lines)` block per file, and `## End of the full content of the changed files`. - Every content line starts with its line number and ` | `, so file content can never look like a section marker or a `## File:` diff header. - A file that exceeds a limit goes as excerpt: `### Excerpt of file: '' (lines 101-181, 240-320 of 1200)`, with `excerpt_context_lines` lines before and after each changed line, numbered like a full file. PR-Agent caps its own diff context at 10 lines, so the excerpt shows the enclosing function where the diff does not. One hint `full_files_partial` per cause names the files that went as excerpt. - With `[excerpt] select = "model"`, an economy model chooses the lines of such an excerpt; see [`[excerpt]`](#excerpt). - A file that is not UTF-8 text, or whose excerpt would exceed a limit too, goes only as diff. One YELLOW reason per cause names the cause, the count and at most 3 files, e.g. `2 changed files were given to the AI only as diff, not in full, so it could not check them against the rest of their file (not UTF-8 text): data/export.csv, data/legacy.txt`. A review with a file only as diff is never GREEN. The hook wraps `get_pr_diff` as the PR-Agent reviewer module calls it, so the section becomes part of the prompt variable `diff`. The operator `extra_instructions` should tell the model to check the changed lines against constraints stated elsewhere in the full files. They should also tell it not to report the countable writing rules, which [`[writing]`](#writing) checks deterministically: a model miscounts them, e.g. on PR#8 it called a line of 120 characters too long and a wrap after a comma a violation. For documentation the model reports only bloated or mannered prose, unclear statements, and contradictions between the PR document and the code. ### `[engine]` Optional; without it the mode is `llm`. | `mode` | Meaning | |---|---| | `llm` | The LLM review alone, as before 0.5.0. | | `classifier` | Jev judges the PR documents; no LLM call. Only a `docs_only` PR whose changed files are all judged PR documents can become GREEN. | | `combined` | Jev judges the PR documents, then the LLM reviews the PR with Jev's flagged units as hints. | | `model_role` | Meaning | |---|---| | `decides` | Default: the findings and doubts of the model hold a PR, e.g. with a frontier model the operator pays for. | | `advises` | With `[path_judge] scope = "all"` and Jev's judgment of every file, the model only advises; see [`[path_judge]`](#path_judge). A failed or timed-out model call is then a hint, not an ERROR. | With `mode = "classifier"` there is no model; with `[path_judge] scope = "all"`, Jev's judgment of every file suffices for GREEN. Both keys are host only. ### `[classifier]` Optional unless `[engine] mode` uses it; when present, every key is required. | Key | Meaning | |---|---| | `endpoint` | `https://api.typesafe.ai/v1/systemone`; https only. | | `model` | `jev-latest`. | | `input_price_usd_per_mtok` | USD per 1M input tokens, maintained by the operator; Jev's output is free. | | `max_state_chars` | A unit with a larger JSON state is not sent; the result is YELLOW "classifier context incomplete". | | `max_units` | Units beyond this number are not sent; the result is YELLOW. | | `timeout_seconds` | Timeout per Jev request. | | `pr_doc_globs` | Globs of the PR documents, e.g. `doc/PR/*.md`; `TEMPLATE*.md` and `README.md` never count. | | `min_confidence` | A choice or score answer below this confidence does not count for the verdict; 0 disables it. | | `scenario_questions`, `section_questions` | Arrays of tables, see below. | Each question has `name`, `type` (`noul`, `choice` or `score`), `instructions`, `red_above` and `yellow_above`. Every question asks for a problem, so a high value always means "bad": - `noul`: the value is the probability of "yes"; optional `criteria = { true = "…", false = "…" }`. - `choice`: `criteria` is a table of options; `problem_options` lists the bad ones, and the value is their summed probability. - `score`: `criteria` is a list of 2 to 10 levels from "no problem" to "worst problem"; the value is Jev's expected level divided by the top level. A value above `red_above` gives ORANGE, a value above `yellow_above` YELLOW. The key `red_above` keeps its name from 0.5.0, when this gave RED. An answer below `min_confidence` is ignored for the verdict, whatever its value: in the calibration on hsadmin-ng such answers came mostly from short sections and named no real problem. A noul has no confidence; its `yellow_above` band covers the uncertain answers. The example configs contain default questions for readability, one aspect per scenario, implementation details, Given/When/Then use and vague outcomes in scenarios, and for readability and slop in sections. The writing rules are no Jev question; `[writing]` checks them deterministically. ### `[gate]` | Key | Default | Meaning | |---|---|---| | `yellow_gate` | `"block"` | `block` (exit 10 for YELLOW, 15 for ORANGE), `pass_with_review_request`, or `approval` (see [Approval mode](#approval-mode)). | | `human_reviewers` | `[]` | Gitea logins whose approval unlocks YELLOW and ORANGE and who get review requests; required unless `block`. | | `assessment_context` | `"ai-review/assessment"` | Context of the informative commit status. | | `require_wip_transition` | `false` | Review only a PR recorded as changed from a "WIP: " title, see [PR workflow and automerge](#pr-workflow-and-automerge). | | `automerge` | `false` | Schedule the merge after GREEN, YELLOW and ORANGE. | | `hints_hold_merge` | `true` | With `automerge`, unresolved line comments of the bot keep a GREEN PR from being merged. | | `max_red_rounds` | `3` | Host only: after this many earlier heads in a row with RED, a new RED stays RED, but requests the human reviewers; 0 never escalates. | | `carry_over_approvals` | `true` | Host only: a human approval of an earlier head stays valid when the new head has the same changed blocks, e.g. after a rebase; see [Approval after a rebase](#approval-after-a-rebase). | The informative status maps GREEN to `success`, YELLOW to `pending` ("human review required"), ORANGE to `warning` ("AI findings; a developer approval after checking them unlocks the merge"), RED to `failure` and ERROR to `error`. Gitea 1.27 supports the state `warning`. The gate itself is the Werkator status of the build (planned context `ai-review/gate`). ### `[codex]` | Key | Default | Meaning | |---|---|---| | `mode` | `"off"` | `off`, `advise` or `gate`: whether the [Codex runner](#codex-runner) reviews the PRs of this repository and holds their merge. | werkrevisor itself only accepts the section; the runner on a workstation reads it at the head of the default branch. `advise` writes a result comment and line comments. `gate` also sets the status `ai-review/codex`: `success` when every Codex line comment is resolved, and for a critical PR or a finding outside the diff only after the approval of a human reviewer for the head. Gitea shows each `pending` status with the same yellow dot, so its description starts with who acts next: `queued ⏳:` and `running ⚙️:` change by themselves; `author ✍️:`, `human 👤:` and `operator 🔧:` wait for a person; `done ✅:` is final, and `done ☑️:` passes a failed Codex review by the approval of a human. The branch protection must require `ai-review/codex` in a `gate` repository; without it, the status holds nothing. werkrevisor never counts a Codex line comment as one of its own hints. ### `[diagrams]` | Key | Default | Meaning | |---|---|---| | `enabled` | `false` | Attach architecture and data-flow diagrams of [PR Lens](https://github.com/coldteadotai/pr-lens) to the result comment. | | `model` | `""` | OpenRouter model that draws them, e.g. `"anthropic/claude-sonnet-5.5"`; empty takes the `[model] id` without `openrouter/`. | | `max_diff_bytes` | `400000` | The diff is cut after this many bytes; the model then draws what it sees. | | `max_output_tokens` | `32768` | Room for the graph document. | | `timeout_seconds` | `300` | Longer runs give no diagrams. | | `max_diagrams` | `5` | Gitea takes at most 5 attachments per comment by default (`[attachment] MAX_FILES`). | | `skip_green` | `true` | No diagrams for a GREEN result: it merges without a human, so nobody would look at them. | Only the host config sets the section, since the diagrams cost a second model call: about 0.21 USD and 35 s for a PR with 36 files and Sonnet 5.5. The model must write a valid graph document. On hsadmin-ng PR#319 and 4 werkrevisor PRs, Sonnet 5.5 and GPT-6 Luna did so each time, Luna with rough boxes only; MiniMax M3 did not on hsadmin-ng PR#319. After the verdict, status and approval, werkrevisor sends the diff to PR Lens, attaches the light SVGs to its result comment and links them under "Diagrams". The diagrams never change the verdict, the status or the exit code; a failure only prints a line. A rerun on the same head links the attached SVGs again and draws no new ones. A possible secret in the diff or `--no-llm` keeps the diff from PR Lens, as from the review model. A dry run with `--diagrams-out ` writes the SVGs to that directory instead; see `tools/review-report`. A model draws them from the diff, so a crafted PR can make them look harmless: they show what a PR touches, not what it does. ## Environment | Variable | Source | Meaning | |---|---|---| | `WERKATOR_PR_NUMBER`, `WERKATOR_PR_HEAD_SHA` | Werkator | PR and head commit to review. | | `WERKATOR_PR_HEAD_REF`, `WERKATOR_PR_BASE_REF`, `WERKATOR_PR_BASE_SHA` | Werkator | Informative. | | `WERKATOR_GITEA_BASE_URL`, `WERKATOR_GITEA_OWNER`, `WERKATOR_GITEA_REPO` | Werkator | Repository; the URL must use https. | | `GITEA_BOT_TOKEN` | host secret | Bot token: read PR, write issue comments, write commit status, request reviewers. | | `OPENROUTER_API_KEY` | host secret | OpenRouter key with a credit limit; not needed with `[engine] mode = "classifier"`. | | `TYPESAFE_API_KEY` | host secret | TypeSafe key for Jev; needed when `[engine] mode` is not `llm`. | | `WERKREVISOR_CONFIG_TOML` | host config | The operator TOML itself; wins over `WERKREVISOR_CONFIG`. | | `WERKREVISOR_CONFIG` | host config | Path to the operator TOML, used when `WERKREVISOR_CONFIG_TOML` is empty. | | `PYTHONNOUSERSITE` | host config | Must be `1`; see [Hardening against a writable HOME](#hardening-against-a-writable-home). | werkrevisor never prints tokens or keys. It redacts known secret values and token patterns from comments, logs and usage lines. HTTP redirects are not followed, so the token cannot reach another host. ## Classifier Jev from TypeSafe is a classifier: it answers typed questions with probabilities, fast and cheap, but writes no findings. werkrevisor calls it directly with the standard library; TLS stays verified, redirects are not followed, the key never appears in output. A request that fails with HTTP 5xx or a timeout is retried once. For each changed PR document werkrevisor reads the new version at the head commit and splits it into units: - each `##` or `###` section with its text, without the scenarios and without `Verified by` blocks; - each Gherkin scenario `#### Scenario#: ` (or `#### Scenario: <title>`) with its steps, its feature and its `#### Background` as context, without the `##### Verified by` block. Headings in code blocks do not count. Jev gets one request per unit with all questions of its kind; up to 4 requests run in parallel. The verdict per question follows the thresholds above; a failed Jev request is ERROR. A clearly bad unit gives ORANGE without an LLM finding, also for the specification questions on Gherkin and PR documents: a classifier can be wrong too, so it never blocks hard, and a developer checks the finding. A leaked secret in the diff keeps the PR away from Jev, as from the LLM. In `combined` mode, werkrevisor appends a hint list to the user prompt after the full files: file, line, scenario id, question and value of each flagged unit, but no text from the PR. An answer ignored for its low confidence is in the list too, marked as uncertain. The LLM checks these units and writes concrete findings. The comment gets a table "Classifier" with the flagged units, the count of passed units and the count of answers ignored for their low confidence. ## Output - One PR comment per head commit: verdict, reasons, up to `max_findings` findings with file and lines, the mechanical files in one line, usage (model, calls, token count before the call, reported tokens, duration, cost), and the reviewed head and base commit. The reasons are sorted from the most to the least severe: ERROR, RED, ORANGE, YELLOW, hint. A reason from a limit names its config key and value, e.g. `[size] max_changed_lines = 800`. A reason that concerns many files is one line with at most 3 examples and the count of the rest. The usage names the config sources, e.g. `config: defaults + host + repository (.werkrevisor.toml at base 3d174cc)`. - The findings in the comment show their severity, e.g. `1. **[high] Possible Bug** at …`. They are sorted from high to low severity, before the cut at `max_findings`. - The commit status `ai-review/assessment` on the head commit. - One JSON line per run on stdout and in `build/werkrevisor/usage.jsonl`; a record that cannot be built or written is logged and never changes the verdict or the exit code; a record that cannot be built is not reported either. - A short summary on stderr. The usage line has timestamp, repository, PR, head and base commit, config sources (`config_source`) and host config (`host_config`: `env` or the file path), verdict, exit code, change kind, count of mechanical files (`mechanical_files`), reasons, model, calls, prompt, completion and cache tokens, cost and cost status, the token count before the call (`counted_input_tokens`, `counted_raw_tokens`, `token_count_factor`, `token_count_method`) and `worst_case_usd`, duration, and the key budget before and after the run. `phase_seconds` holds the seconds of each phase that called a service, e.g. `model_review`, `change_summary`, `context` (full files and excerpt selection) or `path_judge`; the usage line of the comment shows them too. With the classifier it also has the engine mode, the Jev model, units, flagged and ignored answers, calls, input tokens and cost. Reasoning tokens are part of the completion tokens and are not counted twice. The cost is `reported` when PR-Agent could price every call, `estimated` from the operator prices otherwise, `none` with `cost_usd` 0.0 when no model call was made, or `unknown` when a model call reported no usage. The comment then says "no model call, cost 0 USD" or "cost unknown". An ERROR comment has its own marker, so a successful retry can still post its result. werkrevisor posts at most one ERROR comment per head commit. ## Statistics service `statistics/` holds a Kotlin Spring Boot service that collects the usage records of all runs and shows them in a dashboard: costs for 24 hours, 7 and 30 days, per day, week and month, reviews, pull requests and costs per pull request. Per day, week and month, a trend chart shows the cost, the number of pull requests and the cost per pull request, each on its own axis, from the oldest period with reviews or costs on. Below each total, it lists the LLM costs via OpenRouter and the Jev costs via TypeSafe; every amount carries "=" if a provider reported it, or "≈" if werkrevisor calculated it from the configured prices. A table shows the input and output tokens per model and their ratio, which tells whether the input or the output price decides the costs of a model. | Endpoint | Meaning | |---|---| | `POST /api/usage` | Stores the usage record of a run; needs a source token from `WERKREVISOR_STATISTICS_SOURCE_TOKENS`; refuses an LLM cost or a Jev cost above 100 USD, each also as sum of its parts. | | `POST /api/billed-calls` | Stores calls that a provider billed before the first usage record; needs a source token. | | `GET /api/stats` | The statistics, also per repository, model and verdict; a read token from `WERKREVISOR_STATISTICS_READ_TOKENS` adds the recent reviews, summed per pull request, and the names of private repositories. | | `GET /` | A small app that embeds the dashboard and passes the read token to it. | | `GET /werkrevisor-dashboard.js` | The Web Component `<werkrevisor-dashboard api-base="…" token="…">`, plain JavaScript, embeddable in any page. | `WERKREVISOR_STATISTICS_DATABASE` names the SQLite file, `WERKREVISOR_STATISTICS_PORT` and `WERKREVISOR_STATISTICS_ADDRESS` the listener. A run reports its usage record when `WERKREVISOR_STATS_URL` and `WERKREVISOR_STATS_TOKEN` are set; a failed report is logged and never changes the review. The time before the service recorded usage is filled from the activity export of OpenRouter: ```sh WERKREVISOR_STATS_URL=https://… WERKREVISOR_STATS_TOKEN=… tools/import-openrouter-activity openrouter_activity.csv ``` Such billed calls count with "=" in the costs, the tokens and the table per model, but not in the reviews, the pull requests or the table per verdict, since the export names no pull request; the table per repository shows them as "unknown repository". A dry run with `tools/dry-run` reports its usage record as local if `WERKREVISOR_STATS_URL` and `WERKREVISOR_STATS_TOKEN` are set. Local reviews count in the costs and the reviews, not in the pull requests; the table per repository shows them as "local review", so that its rows add up to the total. Calls from 10 minutes before the first usage record on are skipped, so the calls of its run do not count twice. Before the first usage record, the service refuses the import with HTTP 409, since the cut-off is unknown. Build and test the service; Gradle needs a JDK 17 or newer and loads Temurin 25 with Foojay if it is missing: ```sh cd statistics && ./gradlew test bootJar runtime ``` `runtime` jlinks the runtime of the service from Temurin 25 into `build/runtime`, as werkregie ADR 0001 decides. On mih09 the service runs as the systemd user unit `werkrevisor-statistics` of `mih09-werkrevisor`: - the jar is `~/werkrevisor-statistics/werkrevisor-statistics.jar`, the previous one stays as `.jar.prev`; - the runtime is `~/.werkrevisor/jre`, the previous one stays as `~/.werkrevisor/jre.prev`; - `JAVA_OPTS` in `~/werkrevisor-statistics/werkrevisor-statistics.env` holds `--enable-native-access=ALL-UNNAMED`, which sqlite-jdbc needs to load its native library without a warning; - `systemctl --user restart werkrevisor-statistics` starts the new version. ## Build and test The runtime is the pinned image `pragent/pr-agent:0.47.0@sha256:7d98954c…` (Debian 13, Python 3.14, PR-Agent under `/app`). werkrevisor uses only the Python standard library and the packages in that image. For `[diagrams]` the `Dockerfile` adds Debian's `nodejs` and PR Lens under `/opt/pr-lens`, from `tools/pr-lens/package-lock.json` with the integrity of each package; npm stays in a build stage. Run the tests inside the image: ```sh docker run --rm -v "$PWD":/werkrevisor -w /werkrevisor --entrypoint python \ pragent/pr-agent:0.47.0@sha256:7d98954c29289846a08faf00aaea0f1f1375b1676cd8a2ed303e0af07010af7a -m unittest -v ``` The tests need no network, no LLM and no Gitea. One test runs the real PR-Agent reviewer with a fake Gitea provider and a fake model. Build the image and the rootfs archive for hosts without Docker: ```sh tools/build-rootfs ``` This creates the image `werkrevisor:<version>` and `werkrevisor-rootfs-<version>.tar.zst`, and removes all but the two newest archives and image tags; each archive has about 183 MB. Deploy the archive to a Werkator host and switch the listed repositories to it: ```sh tools/deploy-rootfs werkrevisor-rootfs-<version>.tar.zst user@host /path/for/archives repo-a repo-b ``` It backs up each `~/<repo>/.git/werkator/.werkator.yml` and prints only the changed line numbers, since these files hold tokens. Without repositories it updates each machine config on the host that names a werkrevisor archive. On the host it keeps the two newest archives, never one that a `.werkator.yml` still names, and the two newest backups per repository. A Werkator host without a container engine can release each merge itself, with the build `release` of `.werkator.yml`. It assembles the archive from the base archive and the `werkrevisor/` and `config/` of `main`, as the `Dockerfile` does, and installs it like `tools/deploy-rootfs`; a released version is not released again. The base archive is the stage `base` of the `Dockerfile`: the PR-Agent image, Node.js and PR Lens. Its name `werkrevisor-base-<hash>.tar.zst` holds a hash of the base stages and of `tools/pr-lens/package-lock.json`; `python3 tools/release_rootfs.py base-name` prints it. Create it for each new PR-Agent image or PR Lens version, and copy it next to the release archives before the merge: ```sh tools/build-rootfs --base ``` The host part of the build gives the trigger and the native runtime: ```yaml # .git/werkator/.werkator.yml of the werkrevisor repository on the host builds: release: trigger: afterSuccessOf: default branches: ["main"] statusContext: werkator/release werkdock: enabled: false ``` The build runs on the host, outside the sandbox, since it writes the machine configs of all repositories. Before it assembles, the release checks the inline host config `WERKREVISOR_CONFIG_TOML` of each machine config that names a werkrevisor archive against `config/<repo>.toml`, with the parser of the new version. It writes the file into a differing block in the same write that names the new archive, so the new code never runs with a host config that only the old code reads; the backup `.werkator.yml.bak-before-<version>` holds both. A key on the host that the new version no longer knows is dropped. The release fails before any write if a block holds a key that the new version still reads and the file lacks or sets otherwise, e.g. a model switched on the host alone, or if a machine config has no such block, a repository has no `config/<repo>.toml`, or the new version rejects the file. It names the repositories and keys, never a value. Compare the inline host config `WERKREVISOR_CONFIG_TOML` of each machine config with `config/<repo>.toml`, and replace it with `--apply`: ```sh tools/sync-host-config [--apply] user@host repo-a repo-b ``` It prints only the numbers of the differing lines and the names of the differing keys, never a value: keys with another value, keys only in the file, and keys only on the host. A machine config may leave out the keys that the `.werkrevisor.toml` of its repository sets; these keys show as "only in the file", and `--apply` would add them. `--apply` keeps the old file as `.werkator.yml.bak-config-<time>`, and the two newest of these backups. Rotate one secret, e.g. `OPENROUTER_API_KEY`, in each machine config that has it and in `~/.config/ai-coding-agents/secrets.env`: ```sh tools/rotate-secret probe OPENROUTER_API_KEY - # asks the service whether it accepts the new value tools/rotate-secret set user@host OPENROUTER_API_KEY # distributes it; prints the SHA-256 of the old value tools/rotate-secret check user@host OPENROUTER_API_KEY # tells per repository whether it equals secrets.env tools/rotate-secret fetch user@host OPENROUTER_API_KEY # copies the value of the host into a secrets.env without it tools/rotate-secret purge user@host OPENROUTER_API_KEY <sha256> # removes the backups with the old value ``` It reads a value from stdin, hidden on a terminal, and never prints it or passes it as an argument. The probes cost nothing: Gitea `/api/v1/user` and a write to pull request 0, which the scopes of the token decide; OpenRouter `/api/v1/key`, TypeSafe `/v1/models`, and an empty usage record, which the statistics service refuses with HTTP 400 for a known source token. The statistics service takes several source tokens at once, so the old and the new one work during the rotation. The procedure is in [docs/betrieb-git.javagil.de.md](docs/betrieb-git.javagil.de.md), "Secrets austauschen". Count the review rounds per PR since a date, with the findings that stood only in the result comment: ```sh GITEA_TOKEN=… tools/review-rounds https://git.example.org owner 2026-10-09T20:00 repo-a repo-b ``` Evaluate the classifier questions on local PR documents, without Gitea: ```sh tools/jev-eval --units-only ../hsadmin-ng/doc/PR/*.md # extraction only, no network TYPESAFE_API_KEY=… tools/jev-eval --config config/hsadmin-ng.toml --no-repo-config ../hsadmin-ng/doc/PR/*.md TYPESAFE_API_KEY=… tools/jev-eval --config config/werkrevisor.toml docs/prs/*.md ``` It uses the config layers of the review. `--repo-config FILE` puts a repository config on top of the host config, with the same rules as the review; without it, `.werkrevisor.toml` of the current directory is used if present, unless `--no-repo-config` is given. It prints one line per unit and question (file, unit, question, problem value, confidence, verdict) and the totals (verdicts, calls, input tokens, cost). An answer below `min_confidence` shows `IGNORED` with the confidence and counts apart from the verdicts. The archive has no image metadata. Start it with `PATH=/app/.venv/bin:/usr/local/bin:/usr/bin:/bin`, `PYTHONPATH=/werkrevisor:/app`, `PYTHONNOUSERSITE=1` and the command `python -s -m werkrevisor`. Measure real PRs without writing to Gitea and without a model call: ```sh GITEA_BOT_TOKEN=… tools/dry-run https://git.example.org/owner/repo/pulls/302 config/hsadmin-ng.toml ``` With `--prompt-out build/<file>`, the dry run writes the prompt that the model would get to that file, as JSON, and calls no model. Jev and the excerpt selection still run, so it needs the keys of a review. A dry run reviews the current head of a PR; to measure an earlier head, open a PR for it in a bench repository. ## Integration This section is for operators outside the setup of the author. werkrevisor needs no Werkator; any CI system that can start a container for a pull request can run it. ### Supported forges werkrevisor talks only to the Gitea API, with its own client and with the Gitea provider of PR-Agent. Gitea works; Forgejo has the same API but is untested. GitHub, GitLab and Bitbucket are not supported. A port needs a client for the API of the forge in place of `werkrevisor/gitea.py`, the matching `config.git_provider` of PR-Agent, and a parser for the PR URL of the forge. It also needs a replacement for the Gitea features of the gate: the scheduled merge, review requests, and the timeline for `require_wip_transition`. ### What the CI system must do 1. Start a run for each new head commit of an open, ready PR, also after a push to the PR. Skip a PR whose title starts with "WIP: ", Gitea's mark for a draft, and start a run when the title loses it; see [PR workflow and automerge](#pr-workflow-and-automerge). 2. Run the code of the base branch, never the code of the PR. In GitHub terms, this is `pull_request_target`, not `pull_request`. werkrevisor needs no checkout; it reads the PR, the files and `.werkrevisor.toml` through the Gitea API, and writes its usage record to `build/werkrevisor/usage.jsonl` in the working directory. 3. Set the environment of [Environment](#environment). The variables keep the prefix `WERKATOR_`, but any runner can set them. Required are the https base URL of Gitea, owner, repository, PR number and head commit; the base and head refs are informative. 4. Give the secrets to this run only. A build of the PR code must never see `GITEA_BOT_TOKEN`, `OPENROUTER_API_KEY` or `TYPESAFE_API_KEY`. 5. Publish the exit code as a commit status on the PR head, e.g. as the status of the CI job, and make it a required status check in the branch protection after a trial. werkrevisor itself writes the informative status `ai-review/assessment`. 6. Set the job timeout above `timeout_seconds` × (`max_retries` + 1) + 120 seconds. 7. Cancel an older run of the same PR when a new commit arrives, if the CI system can, to save its model call. An older run that is not cancelled sees the moved head, ends with exit 30 and writes nothing. The image of the [Dockerfile](Dockerfile) has the entrypoint `python -s -m werkrevisor` and sets `PYTHONPATH` and `PYTHONNOUSERSITE`. Build it, push it to a registry that the runner can read, and pin its digest: ```sh docker build -t registry.example.org/werkrevisor:0.22.0 . docker push registry.example.org/werkrevisor:0.22.0 ``` The host config comes from the CI system, never from the repository: put the TOML into the secret or variable `WERKREVISOR_CONFIG_TOML`, or copy it into the image and name it in `WERKREVISOR_CONFIG`. Start with one of the examples in [config/](config/) and with `yellow_gate = "block"`; see [Verdicts and exit codes](#verdicts-and-exit-codes) and [Approval mode](#approval-mode). The bot account needs the rights of [Secrets](#secrets). Try the config on past PRs first, without writing to Gitea: `GITEA_BOT_TOKEN=… tools/dry-run https://git.example.org/owner/repo/pulls/302 config.toml`. ### Example: Gitea Actions A sketch for a Gitea version whose Actions support `pull_request_target`. The workflow file comes from the base branch, so a PR cannot change it for its own review. ```yaml # .gitea/workflows/ai-review.yml name: ai-review on: pull_request_target: types: [opened, reopened, synchronize, edited] jobs: gate: if: ${{ !startsWith(github.event.pull_request.title, 'WIP: ') }} runs-on: ubuntu-latest timeout-minutes: 10 container: image: registry.example.org/werkrevisor:0.22.0@sha256:… env: WERKATOR_GITEA_BASE_URL: ${{ github.server_url }} WERKATOR_GITEA_OWNER: ${{ github.repository_owner }} WERKATOR_GITEA_REPO: ${{ github.event.repository.name }} WERKATOR_PR_NUMBER: ${{ github.event.pull_request.number }} WERKATOR_PR_HEAD_SHA: ${{ github.event.pull_request.head.sha }} WERKATOR_PR_BASE_REF: ${{ github.event.pull_request.base.ref }} GITEA_BOT_TOKEN: ${{ secrets.WERKREVISOR_GITEA_TOKEN }} OPENROUTER_API_KEY: ${{ secrets.OPENROUTER_API_KEY }} TYPESAFE_API_KEY: ${{ secrets.TYPESAFE_API_KEY }} WERKREVISOR_CONFIG_TOML: ${{ secrets.WERKREVISOR_CONFIG_TOML }} steps: - run: python -s -m werkrevisor ``` `edited` starts a review when the title loses "WIP: "; the result comment of an already reviewed head is reused without a model call. The status of this job is the merge gate; make its context a required status check. Other CI systems, e.g. Woodpecker, Jenkins or Drone, need the same steps; check that the system gives no secrets to builds of PRs from forks. ## Operations ### Werkator build werkrevisor runs as a Werkator (v1.36.0) build with `trigger.onPullRequestTarget: true`. Such a build is host-only: put it into the machine config `.git/werkator/.werkator.yml` of the checkout, never into a committed file. Werkator checks out the base branch, sets the `WERKATOR_PR_*` and `WERKATOR_GITEA_*` variables, and publishes the exit code under the pinned `statusContext` on the PR head. The werkdock sandbox clears the environment and the rootfs archive keeps no image metadata, so `werkdock.env` sets `PATH` and `PYTHONPATH`. The host config comes from the host, never from the checkout: `WERKREVISOR_CONFIG_TOML` holds the TOML itself, else `WERKREVISOR_CONFIG` names a file in the image (`/werkrevisor/config/`). The repository config comes from the head of the default branch through the Gitea API, never from the PR. ### Hardening against a writable HOME werkdock may bind a home directory as `/root` that ordinary PR builds of the same repository can write. werkrevisor therefore loads no code and no data from HOME: - Start it with `python -s -m werkrevisor` and `PYTHONNOUSERSITE=1`; the image entrypoint does so. - At startup it refuses to run (exit 30) if user site-packages are enabled or a `sys.path` entry lies under HOME. - It removes the working directory, a repository checkout, from `sys.path`. - Before PR-Agent and litellm load, it points `HOME` and the XDG directories to a fresh temporary directory, sets `LITELLM_MODE=PRODUCTION` (no `.env` file is loaded) and `LITELLM_LOCAL_MODEL_COST_MAP=True` (no price map download), and forces `TIKTOKEN_CACHE_DIR` to the tokenizer that litellm ships. ### Machine config mih09, werkdock with the rootfs archive: ```yaml # .git/werkator/.werkator.yml (not committed) builds: ai-review: trigger: onPullRequestTarget: true statusContext: ai-review/gate cleanCommand: "" buildCommand: python -s -m werkrevisor timeout: 10m artifactDirs: [build/werkrevisor] werkdock: enabled: true rootfs: /path/to/werkrevisor-rootfs-0.14.0.tar.zst env: PATH: /app/.venv/bin:/usr/local/bin:/usr/bin:/bin PYTHONPATH: /werkrevisor:/app PYTHONNOUSERSITE: "1" GITEA_BOT_TOKEN: "…" OPENROUTER_API_KEY: "…" # The complete operator config; start from config/werkprobe.toml and change what you need. WERKREVISOR_CONFIG_TOML: | [model] id = "openrouter/anthropic/claude-sonnet-5.5" input_price_usd_per_mtok = 2.00 output_price_usd_per_mtok = 10.00 max_context_tokens = 150000 max_output_tokens = 4000 reasoning_effort = "low" [budget] max_usd_per_review = 0.40 min_key_remaining_usd = 0.50 # … all other sections of config/werkprobe.toml … ``` vm4006, Docker: ```yaml # .git/werkator/.werkator.yml (not committed) builds: ai-review: trigger: onPullRequestTarget: true statusContext: ai-review/gate cleanCommand: "" buildCommand: python -s -m werkrevisor timeout: 10m artifactDirs: [build/werkrevisor] docker: enabled: true image: werkrevisor:0.14.0 env: PYTHONNOUSERSITE: "1" GITEA_BOT_TOKEN: "…" OPENROUTER_API_KEY: "…" WERKREVISOR_CONFIG_TOML: | # the complete content of config/hsadmin-ng.toml, adapted ``` To switch the model or the reasoning level, edit `id`, the two prices, `reasoning_effort` and, if needed, `max_context_tokens` in `WERKREVISOR_CONFIG_TOML` of the machine config; the next review uses them, no new image is needed. The inline TOML replaces the config file completely, so paste the complete host config. A config that lacks a section after the merge with `.werkrevisor.toml` is refused, because a default section would have, for example, no risky paths. Each run logs the config sources on stderr; the usage line has them as `config_source` and `host_config`. A changed `rootfs` path makes werkdock import a new image, so give each version its own archive name. Werkator needs `gitea.baseUrl`, `gitea.owner`, `gitea.repo` and `git.token` for this trigger. Set the build timeout above `timeout_seconds` × (`max_retries` + 1) + 120 seconds; werkrevisor stops itself at that limit. Make `ai-review/gate` a required status in the Gitea branch protection only after the trial. A review approval starts no Werkator build; restart the build by hand to let werkrevisor see a new human approval. ### Secrets - `GITEA_BOT_TOKEN` belongs to a dedicated bot user. It needs repository read access, issue comment write access, commit status write access, and, for `pass_with_review_request` and `approval`, the right to request reviewers. For `approval`, it needs write access to submit official reviews and must be in `approvals_whitelist_username`. - `OPENROUTER_API_KEY` is a dedicated key per host with a credit limit in OpenRouter. - `TYPESAFE_API_KEY` is a TypeSafe key for Jev; it is needed only when `[engine] mode` is not `llm`. - Store them in the machine config `.git/werkator/.werkator.yml` (`werkdock.env` or `docker.env`) with file mode 0600, never in the repository or the operator config. ### Key exchange 1. Create a new key in OpenRouter (or a new Gitea token for the bot) with the same limits. 2. Replace the value in `werkdock.env` or `docker.env` of the machine config. 3. Restart Werkator, so it reads the machine config again. 4. Start a review, e.g. with a new commit on a test PR, and check the usage line. 5. Delete the old key in OpenRouter or Gitea. ### Diagrams of PR Lens PR Lens runs as a child process of werkrevisor: `node werkrevisor/pr_lens.mjs /opt/pr-lens/node_modules`. The child gets only `PATH`, `HOME=/tmp` and `OPENROUTER_API_KEY` in its environment, and it talks only to OpenRouter. It runs as the same user in the same sandbox, though, so it could still read the environment of werkrevisor, including the Gitea token, from `/proc`. The pinned versions and their integrity in the lockfile, the read-only rootfs and `npm audit` in the image build are what keeps foreign code out; a new PR Lens version is a reviewed change of the lockfile. ### Codex runner `tools/codex-runner` reviews the ready PRs of the repositories in `config/codex-runner.toml` with the Codex CLI, after werkrevisor, and exits; only repositories whose `[codex] mode` is `advise` or `gate` take part. It runs on a workstation with a Codex CLI logged in to a subscription, so the login never reaches a shared host. Codex runs read-only in a local worktree of the PR head, without a token, and its sandbox blocks the network of its commands. `bwrap` hides the home directory from it, except its login, the worktree and the program directories on the `PATH`, so it cannot read and quote other secrets of the workstation; without `bwrap` the runner reviews nothing. The runner removes the Gitea token and the values of the Codex login from everything it posts. Before each run, the runner checks that a `bwrap` works inside its jail, as Codex starts its own sandbox there. If AppArmor denies it, the run reviews nothing, counts no attempt, logs the cause and exits with 1, so the systemd unit shows as failed; a review whose `bwrap` fails counts no attempt either. The log line names the next steps: `journalctl -k | grep -E 'unpriv_bwrap|unprivileged_userns'`, and reloading the profile with `sudo apparmor_parser -r --skip-read-cache --write-cache /etc/apparmor.d/bwrap`. - `max_reviews_per_run` bounds the reviews per run, so a run does not review a head that a push already replaced; a failed run is retried twice. - The ready PR that waits longest goes first, across all repositories: the one with the oldest werkrevisor result for its head. A PR whose review failed keeps its place until its last retry. - With the workstation off, a `gate` repository waits, with the status `pending`. - In each listed repository, also with `mode = "off"`, the runner releases a merge that werkrevisor held only for its line comments, once each one is resolved, in the first run that starts afterwards; a run that reviews with Codex ends later, so the release may wait for those reviews. It schedules the merge and withdraws the review request of the human, as a restart of the review would, after it has brought the status `ai-review/codex` up to date, so that a reopened Codex thread still holds the merge. It does so only for the current head, with the approval of werkrevisor and the hidden note of the hold in its result comment, and notes the release in a PR comment. A hint that stands only in the result comment still needs a restart: only a human can judge its answer. The runner cannot read the host config of werkrevisor: after `[gate] automerge` is switched off, it still releases a head that werkrevisor held before, as Gitea still merges a merge scheduled before; removing the repository from `config/codex-runner.toml` stops it. - The units in `tools/systemd/` start it 1 minute after its last run; they read the Gitea token from `~/.config/ai-coding-agents/secrets.env`: ```sh cp tools/systemd/werkrevisor-codex-runner.* ~/.config/systemd/user/ systemctl --user daemon-reload systemctl --user enable --now werkrevisor-codex-runner.timer ``` `tools/codex-eval` runs the same review on single PRs without writing to Gitea, for measurements. `tools/review-report` runs the full werkrevisor dry run and the Codex review in parallel on one PR of any Gitea, e.g. a repository without a bot account, and writes a draft report; check each finding before you post it yourself. With `[diagrams]` on in the config, it puts the SVGs of PR Lens next to the draft and links them; attach them when you post. `tools/review-report --attach` posts the checked draft instead, without a new review: a comment of the PR with the SVGs as attachments, under the account of the token. It refuses a draft that still has its line "> **Draft:** …", is for another head, or is posted already. ### Draft review `tools/draft-review <pr-number | pr-url>` runs `tools/review-report` for a PR that a human reviews, prints the verdict of werkrevisor with its strongest finding, and writes `report.html` next to `report.md` with pandoc; each diagram in it opens on its own with a click. `tools/draft-review <pr> --attach` posts the checked draft, see above. Its local config `~/.config/werkrevisor/draft-review.toml` (or `$WERKREVISOR_DRAFT_REVIEW_CONFIG`) names the host; it holds no secrets, only the names of the token variables: ```toml base_url = "https://git.example.org" repository = "team/app" # a bare PR number means a PR of this repository config = "config/werkrevisor.toml" # the werkrevisor config, relative to this repository secrets = "~/.config/example/secrets.env" # optional KEY=VALUE file with the tokens below read_token_variable = "EXAMPLE_GITEA_READ_TOKEN" # for the review write_token_variable = "EXAMPLE_GITEA_WRITE_TOKEN" # for --attach only browser = "brave-browser" # optional, for --open; default: xdg-open ``` `--open` opens `report.html` in that browser, or with `--attach` the posted comment. `--attach --checked "<reason>"` or `--attach --checked-file <file | ->` confirms the check without an edit of the report: the reason why the remaining points do not hold replaces the line "> **Draft:** …" and stands as "Checked before posting" at the top of the comment; an empty reason is refused. The title of the draft links the PR, e.g. ``## AI review of [PR#343](https://git.example.org/team/app/pulls/343) (`6d1842aa`)``. A URL of a PR on another server is refused, so the tokens never go elsewhere. A link such as `ln -s ~/Projekte/werkrevisor/tools/draft-review ~/bin/<name>` gives it a short name. The dry run then runs in the stage `base` of the `Dockerfile`, with Node.js and PR Lens, and builds it once. ### Stop, restart, remove - Stop: disable the `ai-review` build in the Werkator config and reload Werkator. If `ai-review/gate` is a required status, remove that requirement first, else no PR can merge. In approval mode, also remove the bot from `approvals_whitelist_username`, so humans approve alone. - Restart: enable the build again. A new commit or a re-run of the build reviews the current head again; an existing result comment for that head is reused without a model call. - Remove: stop it, delete the build config with its secrets, the rootfs archive or image, and revoke the OpenRouter key and the bot token.