werkrevisor

werkrevisor is an AI review gate for Gitea pull requests. Simple PRs can merge after an automated review. Risky PRs get human attention.

Werkator starts werkrevisor for each ready (non-draft) PR head commit. werkrevisor runs on a checkout of the base branch and never executes PR files. It reads the PR through the Gitea API, applies deterministic rules, and asks a model through PR-Agent 0.47.0 for a review. The exit code is the merge gate: Werkator publishes it as a commit status on the PR head.

Status: v0.1, trial.

Verdicts and exit codes

Verdict	Label	Exit code	Meaning
GREEN	PASS	0	All files are AI-approvable, no hard rule fired, the model found no relevant issue or every changed file is mechanical.
YELLOW	HUMAN_REVIEW	10	A human must look: hard rule, change kind not AI-approvable, unclear model answer, merge_with_caution, minor model findings, no AI review possible because the PR is too long for the model or its budget.
ORANGE	FINDINGS	15	Soft: a critical path changed, the model reports a finding of high severity or enough findings of medium severity, the classifier rates a PR-document unit as clearly bad, a changed doc violates a writing rule, or the PR is too large for an AI-only approval. A developer checks the PR knowingly; an approval unlocks the merge.
RED	CHANGES_REQUIRED	20	Hard: a deterministic rule is violated, e.g. an open question in the PR document. Only a new commit that fixes it lifts it.
ERROR	review not completed, human action needed	30	Technical error, invalid model answer, timeout, budget, skipped files, superseded head.

The order is GREEN < YELLOW < ORANGE < RED < ERROR; the worst reason decides. ERROR is never GREEN. A run without a complete model review is never GREEN, except a PR of mechanical changes only; see [mechanical].

Only deterministic rules give RED. A model or a classifier can be wrong, so their findings give at most ORANGE and never block hard. On dev.hostsharing.net the repository administrator is an operator, not a developer; an administrator merge override is therefore no way out of a false RED.

With yellow_gate = "block" or "pass_with_review_request", YELLOW and ORANGE can still pass the gate (exit 0):

• A configured human reviewer approved exactly this head commit (Gitea review commit_id).
• yellow_gate = "pass_with_review_request" is set and a review request to a configured human reviewer is pending.

With yellow_gate = "approval", the missing approval is the lock; see Approval mode.

An approval does not unlock RED. Only a new head commit that fixes the violated rule can change RED.

Approval mode

In this mode the branch protection requires one approval, and the bot is one of the allowed approvers. werkrevisor acts as a reviewer:

Verdict	Gitea action	Exit code
GREEN	Submit a review APPROVED with commit_id = head commit.	0
YELLOW	No bot review. Request an official review from human_reviewers, unless a request is pending or a human approved this head.	0; 10 if the request failed
ORANGE	As YELLOW: no bot review, neither REQUEST_CHANGES nor APPROVED; request an official review from human_reviewers. The comment lists the findings.	0; 15 if the request failed
RED	Submit a review REQUEST_CHANGES with commit_id = head commit and up to 3 violated rules.	20
ERROR	No review.	30
Superseded head	No review, no comment.	30

werkrevisor reads the PR again directly before it writes anything. It posts the result comment for every verdict, including GREEN, and links it from the review. A rerun on the same head does not submit a second review. An earlier bot review of the PR stays as it is; Gitea marks a review on an older head as stale. The bot never approves a PR that it opened itself: that is ERROR.

Branch protection of the base branch (Gitea 1.27 API field names):

Field	Value	Why
required_approvals	1	The approval is the lock.
enable_approvals_whitelist	true	Only listed users count.
approvals_whitelist_username	the bot user and human_reviewers	A GREEN bot review or a human approval unlocks the merge.
block_on_rejected_reviews	true	A RED review REQUEST_CHANGES blocks the merge.
block_on_official_review_requests	true	The review request of a YELLOW or ORANGE blocks until the human reviews.
dismiss_stale_approvals	true	A new commit removes the bot approval of the old head.
block_on_outdated_branch	true	Werkator starts no new review when only the base moves.
enable_status_check	true	The Werkator status is required.
status_check_contexts	["ai-review/gate"]	The pinned statusContext of the Werkator build.

The bot user needs write access to the repository to submit official reviews.

How a run works

1. Read the PR. If its head differs from WERKATOR_PR_HEAD_SHA or the PR is not open, stop with exit 30 and write nothing. Read .werkrevisor.toml at the base commit and put it on top of the host config; see Config layers.
2. Look for an own result comment with the marker <!-- werkrevisor:<head_sha> -->. If it exists, reuse its verdict, do not call the model, and do not comment again.
3. Apply the deterministic rules to the changed files and the diff. A changed critical path (see [hard_orange]) or a PR above the [size] limits is ORANGE; the model still reviews it, so its findings help the human reviewer. Check the changed PR documents for open questions; an open question gives RED without a model call. Check the changed lines of the docs against the writing rules; a violation gives ORANGE, and the model still reviews the PR. With full_files = true, read the new version of every changed text file at the head commit.
4. With [engine] mode = "classifier" or "combined", Jev judges the sections and scenarios of the changed PR documents first; see Classifier.
5. Run the PR-Agent reviewer with all settings forced by werkrevisor. It leaves out the mechanical files; a PR of mechanical files only skips Jev and the model. Right before the model call, count the tokens of the prompt that PR-Agent built; see Token count. A count above the effective input limit (see [budget]) gives YELLOW without a model call.
6. Query the OpenRouter key budget. Too little budget gives ERROR without a model call.
7. Check that the model saw every changed file completely.
8. Read the PR again. If the head moved meanwhile, stop with exit 30 and write nothing.
9. Write one PR comment, the commit status ai-review/assessment, and one usage line.

The deterministic rules come only from the host config and the repository config of the base commit. The model can make a verdict worse, never better than the rules allow. If the secret scan fires, the diff is not sent to the model.

Configuration

The operator config is TOML. WERKREVISOR_CONFIG_TOML holds it inline; if that is empty, --config or WERKREVISOR_CONFIG names a file. If both are set, the inline TOML wins. An invalid inline TOML is ERROR (exit 30); the message names the line and column, never the content. Unknown keys are errors, so a typo cannot weaken a rule. Examples: config/werkprobe.toml (cheap) and config/hsadmin-ng.toml (1 USD per review).

Config layers

Three layers make the config; a later layer wins per key:

1. the built-in defaults of werkrevisor,
2. the host config (WERKREVISOR_CONFIG_TOML or WERKREVISOR_CONFIG),
3. the repository config .werkrevisor.toml at the repository root.

werkrevisor reads .werkrevisor.toml through the Gitea API at the base commit of the PR, never at its head, so a PR cannot change its own review. Without the file there is no repository layer. After the merge of both layers, every section except [engine] and [classifier] must be set; else the result is ERROR.

The repository config may set these keys:

Section	Keys	Rule
[size]	all	replaces the host value
[categories]	all	replaces the host value
[hard_orange]	paths	added to the host paths, never replaces them
[hard_yellow]	paths	added to the host paths, never replaces them
[hard_yellow]	secret_scan_exclude	replaces the host value
[hard_yellow]	secret_scan	can only stay on
[pr_doc]	all	replaces the host value
[writing]	all	replaces the host value
[mechanical]	enabled, whitespace_significant	replaces the host value
[mechanical]	imports	replaces the host regexes per file extension
[review]	extra_instructions	appended to the host instructions
[review]	max_findings, orange_min_medium, repo_context_files, full_files	replaces the host value
[review]	max_full_file_chars, max_full_files_chars	only lower or equal
[classifier]	pr_doc_globs, min_confidence, scenario_questions, section_questions	replaces the host value; needs a host [classifier]
[model]	max_context_tokens, max_output_tokens	only lower or equal
[budget]	max_usd_per_review	only lower or equal

Every other key is a host key: the model, its prices and reasoning, the rest of the budget, [gate], [engine], and the classifier endpoint, model and limits. A repository config that sets one of them is ERROR, and the message names the key. Model, money and merge rights stay with the host operator.

.werkrevisor.toml is always a risky path (see [hard_yellow]), so a change needs a human approval. werkrevisor also checks the changed file of the PR against the host config; an invalid file is RED, because after the merge every review would end with ERROR. A section, key or change kind unknown to the installed werkrevisor version gives YELLOW instead, if the rest of the file is valid: a newer version may know it, so deploy that version right after the merge. The change takes effect only after the merge.

config/werkrevisor.toml holds only the host part for werkrevisor itself; the repository part is .werkrevisor.toml.

[model]

Key	Default	Meaning
id	required	LiteLLM model id, e.g. openrouter/google/gemini-3.8-flash. No fallback models are used.
input_price_usd_per_mtok	required	USD per 1M input tokens, maintained by the operator.
output_price_usd_per_mtok	required	USD per 1M output tokens, maintained by the operator.
max_context_tokens	required	Quality cap for the input: long prompts degrade a review even within the context window. About 150000 for frontier models, 80000 for cheap models. Replaces max_input_tokens of 0.6.0.
max_output_tokens	required	Output limit including reasoning tokens.
reasoning_effort	"low"	none, minimal, low, medium, high, xhigh or max.
reasoning_max_tokens	0	OpenRouter reasoning budget; 0 leaves it unset.
context_tokens	0	PR-Agent's window; 0 means twice the input limit plus max_output_tokens, see Token count.
timeout_seconds	120	Timeout per model call.
max_retries	1	0 or 1; PR-Agent retries a failed call at most once.
token_count_factor	1.15	Multiplies the o200k_base count of the prompt; host key. See Token count.

[budget]

Key	Default	Meaning
max_usd_per_review	required	Worst-case cost limit per review; it derives an input limit, see below.
min_key_remaining_usd	0.0	The key must keep at least this budget, else ERROR.
chars_per_token	3.5	Fallback estimate without a tokenizer, see Token count.
prompt_overhead_chars	16000	Fallback estimate: PR-Agent prompt size plus repository context files.
diff_expansion_factor	1.5	Fallback estimate: PR-Agent adds context lines and line numbers to the diff.

The worst case is (counted input × input price + max_output_tokens × output price) × (max_retries + 1). The budget-derived input limit is the largest input whose worst case stays within max_usd_per_review. The effective input limit is the lower of this limit and [model] max_context_tokens; a PR whose token count exceeds it gets no AI review and is YELLOW, and the reason names the count and the limit that applied. Example: Sonnet 5.5 at 2 / 10 USD per 1M tokens, max_output_tokens = 4000, 2 attempts and 0.40 USD give 80000 input tokens. A config whose budget does not even pay for max_output_tokens is refused.

After the token count and before the model call, werkrevisor reads GET https://openrouter.ai/api/v1/key. If limit_remaining is below max_usd_per_review or min_key_remaining_usd, the result is ERROR "budget". A key without a limit passes this check.

Token count

Right before the model call, werkrevisor counts the system and the user prompt that PR-Agent passes to its AI handler: the instructions, the PR description, the diff, the full files, the classifier hints and the repo_context_files. It counts with tiktoken o200k_base; litellm ships its data in the image, so nothing is downloaded. The count times [model] token_count_factor decides about the input limit and the worst-case cost, because Anthropic and other providers count more tokens than o200k_base. Only the host config sets the factor, so a PR cannot lower the count of its own review.

werkrevisor 0.10.0 estimated characters / 3.5 instead, and the estimate was much too low:

PR	Estimate	Prompt tokens reported by the model
hsadmin-ng PR#305	about 168000	268735
werkrevisor PR#7	below 80000	87212 to 88369

Calibrate the factor with the usage records of real reviews: prompt_tokens / counted_raw_tokens. On werkrevisor PR#10 the factor 1.15 gave 49841 tokens, and Anthropic reported 65792 prompt tokens for Sonnet: 1.15 × 65792 / 49841 ≈ 1.52, so the example configs use 1.55.

PR-Agent gets a window of twice the input limit plus max_output_tokens, or [model] context_tokens if set. Thus PR-Agent passes a too long prompt unshortened to the count, and the reason names the exact count. A prompt beyond that window, which PR-Agent prunes or refuses, gets no AI review either, and the reason names the window. If the tokenizer cannot load, werkrevisor uses the estimate from the [budget] keys above and says token count estimated in the reason, the comment and the usage line.

[size]

Key	Default	Meaning
max_changed_lines	800	Changed lines (additions plus deletions) of all files outside the docs category, except mechanical files.
max_files	30	Changed files outside the docs category, except mechanical files.

Above one of these limits, the PR is too large for an AI-only approval: the verdict is at least ORANGE, and a human approves it knowingly. The limits do not stop the model review; if the model can take the PR, its findings help the human. Docs do not count, because a PR document alone has 100 to 200 lines.

This limit is separate from the model limit in [budget]: a PR can be too large for an AI-only approval, too long for the model, or both; then both reasons appear and ORANGE wins.

Changed lines outside the docs per merged PR, measured over one month:

Repository	PRs	Median	p75	p90	Max
Werkator	92	194	489	865	1748
hsadmin-ng	16	441	974	1098	1146

With 800 lines, about every tenth Werkator PR needs a knowing human approval. hsadmin-ng uses 1000 lines: that is about every fourth PR, and it lies just below the hard limit of hsadmin-ng's own tools/pr-quick-check (about 1200 to 1400 lines).

[categories]

Glob lists per category: docs, tests, main, api_spec, harmless. The first match wins in the order api_spec, tests, docs, main, harmless; else the file is other. harmless (default []) lists files, e.g. .editorconfig, that do not spoil an AI-approvable change kind. A pattern without / matches the file name in any directory; ** matches any number of directories.

auto_approvable lists the change kinds that can become GREEN:

Change kind	Condition
docs_only	Only docs.
tests_only_added	Only tests, and the test changes are purely additive.
docs_and_tests	Docs and tests; the test changes are purely additive.
refactoring	Main code, optionally docs; no tests, no API spec.
feature_with_tests	Main code and tests, optionally docs; the test changes are purely additive.
harmless_only	Only files of the harmless category.
mechanical_only	Only mechanical files, see [mechanical]; no model call.

Harmless files may come along with every change kind. Mechanical files do not count for the change kind, e.g. main code with a test file that changed only its imports is a refactoring.

Purely additive test changes add test files, test methods or lines, e.g. a new assert line in an existing test method. They delete no test file and remove or change no line of a test file; blank lines do not count. A removed line that the diff adds again unchanged, e.g. a closing brace after an appended test, also does not count, except a line with @Test, assert, should or expect.

An added line must not disable a test: @Disabled, @Ignore, .skip(, .only(, pytest.mark.skip, pytest.mark.xfail, unittest.skip, skipTest(, xit(, xdescribe( or xtest(.

All other change kinds (tests_modified, feature_with_modified_tests, mixed) are at most YELLOW. The reason names the kinds of the changed files and the AI-approvable change kinds, e.g. with auto_approvable = ["docs_only", "refactoring"]: change kind 'mixed' (main code, other files, docs); AI-approvable are only: docs only, refactoring with unchanged tests. werkrevisor tells the model the change kind, so the model can object if, for example, a refactoring changes behavior.

[hard_orange]

paths lists globs of critical files, e.g. RBAC, authentication, authorization and secrets. A change of such a file gives ORANGE: a developer reviews it deliberately, and the model still reviews the PR. The section is required; paths = [] is allowed.

A renamed file counts with its old and new path. Files that match the docs category globs never match paths. A mechanical change counts too, because a critical path stays critical.

All critical files share one reason line, e.g. critical paths changed, review them deliberately: werkrevisor/gitea.py ([hard_orange] werkrevisor/gitea.py) and 2 more. A file that matches [hard_orange] and a YELLOW path, also a built-in one, appears only in this reason.

[hard_yellow]

paths lists globs of risky files, e.g. migrations, CI and deploy config, dependency manifests. A renamed file counts with its old and new path. Files that match the docs category globs never match paths, so a PR document named after a risky topic, e.g. doc/PR/…-rbac-generator-hardening.md, is no risky file. secret_scan = true scans added lines for private keys, -----BEGIN, AWS keys, generic token|secret|password|api_key = "..." assignments, and tokens with the prefixes hsak_ and sk-or-. secret_scan_exclude lists globs of files with fake secrets, e.g. ["src/test/**"] for test data. There, only real key formats count: private keys (-----BEGIN … PRIVATE KEY), AWS key ids, hsak_<name>. with 64 hex digits and sk-or-v1- with 64 hex digits.

.werkrevisor.toml is a built-in risky path; no layer can remove it. DB schema changes are built-in risky paths too, outside Markdown files: **/*.sql, **/db/changelog/**, **/migrations/**, **/migration/**, **/liquibase/**, **/flyway/**, **/*.changelog.* and **/db.changelog-*. Their reason names them, e.g. DB schema changed: src/main/resources/db/changelog/x.sql (built in). These hard rules give YELLOW; the model cannot override them.

[pr_doc]

Key	Default	Meaning
globs	[]	Globs of the PR documents, e.g. doc/PR/*.md; TEMPLATE*.md and README.md never count.
open_question_headings	["Open Questions", "Offene Fragen"]	Headings of the open-questions section, compared without case and a trailing colon.

A PR never passes while a PR document that it adds or changes has open questions. The section starts at such a heading of any level and ends at the next heading of the same or a higher level; deeper subheadings stay in it. Allowed are an empty section and list items checked as done with [x], e.g. - [x] TODO.spec: …. Every item with [ ] and every list item or paragraph without a checkbox gives RED. Blank lines, comments (<!-- … -->) and continuation lines of a done item do not count. The check is deterministic and runs before any model call; the model cannot lift it, and no model call follows. The reason lists up to 3 open items as file:line: text, then the count of the rest.

[writing]

Key	Default	Meaning
enabled	true	Check the writing rules.
max_line_length	128	Characters per line.
max_sentences_per_paragraph	5	Sentences per paragraph; each list item is a paragraph of its own.
globs	[]	Globs of the checked files; empty means the docs category. The PR documents of [pr_doc] always count.

werkrevisor checks the countable rules of the writing style in AGENTS.md on changed Markdown files: at most one sentence per line, at most max_line_length characters per line, and at most max_sentences_per_paragraph sentences per paragraph. Only lines that the PR adds or changes count, and a paragraph only if the PR changes one of its lines. A table row, a heading, a line with only a link or URL, a fenced code block, an HTML comment and the front matter have no line limit and no sentences.

The sentence split is conservative, so a false finding is rare: a sentence ends at . or ! before a capital letter, but not after an abbreviation such as e.g., z. B. or Nr., an initial or a number. A question mark ends no sentence, because a PR document answers an open question on the same line. Code spans, URLs and link targets do not count.

A violation gives ORANGE: a style rule must not block hard, so a human approves knowingly. The check runs before the model call and does not skip it. The reason lists at most 3 violations as file:line: rule, then the count of the rest.

[mechanical]

Key	Default	Meaning
enabled	true	Find the mechanical files.
whitespace_significant	*.py, *.yml, *.yaml, Makefiles, *.mk, *.md, *.haml, *.pug, *.coffee, *.nim, *.fs	Globs of files in which whitespace has a meaning.
imports	see below	Table from a file extension without the dot to the regexes of lines that an import change may add or remove.

A changed file is mechanical only if one of these rules explains every changed line:

• renamed: Gitea reports the file as renamed with no added or removed line.
• whitespace: the lines differ only in the amount of whitespace, like git diff -b, or are blank lines. A file of whitespace_significant is never mechanical by this rule.
• imports: the line matches a regex of its file extension, or it is part of a multi-line import statement whose lines all appear in the same diff hunk and match together. Blank lines count too.

A file with an added or removed line that no rule explains is not mechanical. Neither is an added, deleted or binary file, a renamed file with changes, or a file whose diff Gitea shortened.

The built-in import regexes:

Extensions	Lines
java, kt, kts	import a.b.C;, import a.b.*;, package a.b;
ts, tsx, js, jsx, mjs	named, default and namespace imports with from; never a side-effect import like import './x'
go	import "x", import y "x" and import ( … ) blocks; never _ "x" or . "x"
rs	use a::b;, pub use a::{b, c};; never mod x;

Python and all other extensions have no import rule. A [mechanical.imports] table in a config layer replaces the regexes of the extensions that it names, e.g. scala = ['^\s*import\s+[\w.]+\s*$']; an empty list switches an extension off. An invalid regex is a config error.

A mechanical file does not go to the model, neither its diff nor its full file; werkrevisor passes it to PR-Agent as ignore.regex, and the coverage check does not miss it. It does not count for the change kind and for the check of purely additive tests. The hard rules apply to it as to every file, e.g. a mechanical change of a risky path gives YELLOW, of a critical path ORANGE. The [size] limits do not count it, so a package move across many files does not make a PR too large.

The comment lists the mechanical files in one line, with at most 3 examples and the count of the rest, e.g. 3 files changed only mechanically and were not reviewed by the model: A.java (imports), b.c (whitespace), c (renamed).

[review]

Key	Default	Meaning
extra_instructions	""	Project review rules for the model.
max_findings	3	Findings requested from the model and shown in the comment.
orange_min_medium	2	This many findings of medium severity give ORANGE; 0 disables it. See Severity of the findings.
repo_context_files	[]	Files PR-Agent reads from the default branch, e.g. AGENTS.md.
full_files	false	Add the complete new version of every changed file to the prompt.
max_full_file_chars	40000	Limit per rendered file; a larger file is left out.
max_full_files_chars	120000	Limit for all rendered files together; files beyond it are left out.

Severity of the findings

The answer schema of PR-Agent 0.47.0 has no severity field. So the extra_instructions of the examples tell the model to start every issue_header with [high], [medium] or [low]:

• [high]: wrong behavior, data loss, a security problem or a broken contract in a realistic scenario.
• [medium]: a defect with limited impact, or a missing test for changed behavior.
• [low]: cosmetics, wording, log noise or style.

werkrevisor reads the prefix case-insensitively and removes it from the title. A missing or unknown prefix counts as medium; the comment says severity unknown, counted as medium. One finding of high severity or orange_min_medium findings of medium severity give ORANGE. Any other finding gives YELLOW with the reason model reports minor findings. merge_recommendation: changes_required alone gives YELLOW; only the findings give ORANGE.

werkrevisor 0.14.0 replaced red_min_findings with orange_min_medium; the old key is a config error.

The full files share the effective input limit with the diff and the prompt overhead. Choose max_full_files_chars well below that limit, at about 3.5 characters per token, so that the diff of a PR at the [size] limit still fits; e.g. 120000 chars are about 34000 of 80000 tokens.

A diff with a few context lines hides constraints stated elsewhere in a file, e.g. in its header comment. PR-Agent 0.47.0 caps the extra context lines per hunk at 10, so werkrevisor adds the full files itself:

• It reads each changed, not deleted file at the head commit through the Gitea API (/raw/{path}?ref=<sha>).
• It skips files that PR-Agent cannot review (binary extensions); the coverage rules already make them YELLOW.
• It skips files of the docs category; they reach the model only as diff and do not count against the limits. The model reviews docs for their wording against the writing style, which the diff shows; full docs would crowd out the code files.
• It appends one section after the diff, in the user prompt, never in the system prompt, because PR content is untrusted: ## Full content of the changed files (new version at <sha>, …), one ### Full file: '<path>' (<n> lines) block per file, and ## End of the full content of the changed files.
• Every content line starts with its line number and |, so file content can never look like a section marker or a ## File: diff header.
• A file that is not UTF-8 text or exceeds a limit is left out. One YELLOW reason per cause names the cause, the count and at most 3 files, e.g. 20 changed files were given to the AI only as diff, not in full, so it could not check them against the rest of their file (all full files together would exceed [review] max_full_files_chars = 120000): README.md, werkrevisor/cli.py, werkrevisor/config.py and 17 more. A review with incomplete context is never GREEN.

The hook wraps get_pr_diff as the PR-Agent reviewer module calls it, so the section becomes part of the prompt variable diff. The operator extra_instructions should tell the model to check the changed lines against constraints stated elsewhere in the full files. They should also tell it not to report the countable writing rules, which [writing] checks deterministically: a model miscounts them, e.g. on PR#8 it called a line of 120 characters too long and a wrap after a comma a violation. For documentation the model reports only bloated or mannered prose, unclear statements, and contradictions between the PR document and the code.

[engine]

Optional; without it the mode is llm.

mode	Meaning
llm	The LLM review alone, as before 0.5.0.
classifier	Jev judges the PR documents; no LLM call. Only a docs_only PR whose changed files are all judged PR documents can become GREEN.
combined	Jev judges the PR documents, then the LLM reviews the PR with Jev's flagged units as hints.

[classifier]

Optional unless [engine] mode uses it; when present, every key is required.

Key	Meaning
endpoint	https://api.typesafe.ai/v1/systemone; https only.
model	jev-latest.
input_price_usd_per_mtok	USD per 1M input tokens, maintained by the operator; Jev's output is free.
max_state_chars	A unit with a larger JSON state is not sent; the result is YELLOW "classifier context incomplete".
max_units	Units beyond this number are not sent; the result is YELLOW.
timeout_seconds	Timeout per Jev request.
pr_doc_globs	Globs of the PR documents, e.g. doc/PR/*.md; TEMPLATE*.md and README.md never count.
min_confidence	A choice or score answer below this confidence does not count for the verdict; 0 disables it.
scenario_questions, section_questions	Arrays of tables, see below.

Each question has name, type (noul, choice or score), instructions, red_above and yellow_above. Every question asks for a problem, so a high value always means "bad":

• noul: the value is the probability of "yes"; optional criteria = { true = "…", false = "…" }.
• choice: criteria is a table of options; problem_options lists the bad ones, and the value is their summed probability.
• score: criteria is a list of 2 to 10 levels from "no problem" to "worst problem"; the value is Jev's expected level divided by the top level.

A value above red_above gives ORANGE, a value above yellow_above YELLOW. The key red_above keeps its name from 0.5.0, when this gave RED.

An answer below min_confidence is ignored for the verdict, whatever its value: in the calibration on hsadmin-ng such answers came mostly from short sections and named no real problem. A noul has no confidence; its yellow_above band covers the uncertain answers.

The example configs contain default questions for readability, one aspect per scenario, implementation details, Given/When/Then use and vague outcomes in scenarios, and for readability and slop in sections. The writing rules are no Jev question; [writing] checks them deterministically.

[gate]

Key	Default	Meaning
yellow_gate	"block"	block (exit 10 for YELLOW, 15 for ORANGE), pass_with_review_request, or approval (see Approval mode).
human_reviewers	[]	Gitea logins whose approval unlocks YELLOW and ORANGE and who get review requests; required unless block.
assessment_context	"ai-review/assessment"	Context of the informative commit status.

The informative status maps GREEN to success, YELLOW to pending ("human review required"), ORANGE to warning ("AI findings; a developer approval after checking them unlocks the merge"), RED to failure and ERROR to error. Gitea 1.27 supports the state warning. The gate itself is the Werkator status of the build (planned context ai-review/gate).

Environment

Variable	Source	Meaning
WERKATOR_PR_NUMBER, WERKATOR_PR_HEAD_SHA	Werkator	PR and head commit to review.
WERKATOR_PR_HEAD_REF, WERKATOR_PR_BASE_REF, WERKATOR_PR_BASE_SHA	Werkator	Informative.
WERKATOR_GITEA_BASE_URL, WERKATOR_GITEA_OWNER, WERKATOR_GITEA_REPO	Werkator	Repository; the URL must use https.
GITEA_BOT_TOKEN	host secret	Bot token: read PR, write issue comments, write commit status, request reviewers.
OPENROUTER_API_KEY	host secret	OpenRouter key with a credit limit; not needed with [engine] mode = "classifier".
TYPESAFE_API_KEY	host secret	TypeSafe key for Jev; needed when [engine] mode is not llm.
WERKREVISOR_CONFIG_TOML	host config	The operator TOML itself; wins over WERKREVISOR_CONFIG.
WERKREVISOR_CONFIG	host config	Path to the operator TOML, used when WERKREVISOR_CONFIG_TOML is empty.
PYTHONNOUSERSITE	host config	Must be 1; see Hardening against a writable HOME.

werkrevisor never prints tokens or keys. It redacts known secret values and token patterns from comments, logs and usage lines. HTTP redirects are not followed, so the token cannot reach another host.

Classifier

Jev from TypeSafe is a classifier: it answers typed questions with probabilities, fast and cheap, but writes no findings. werkrevisor calls it directly with the standard library; TLS stays verified, redirects are not followed, the key never appears in output. A request that fails with HTTP 5xx or a timeout is retried once.

For each changed PR document werkrevisor reads the new version at the head commit and splits it into units:

• each ## or ### section with its text, without the scenarios and without Verified by blocks;
• each Gherkin scenario #### Scenario#<id>: <title> (or #### Scenario: <title>) with its steps, its feature and its #### Background as context, without the ##### Verified by block.

Headings in code blocks do not count. Jev gets one request per unit with all questions of its kind; up to 4 requests run in parallel. The verdict per question follows the thresholds above; a failed Jev request is ERROR. A clearly bad unit gives ORANGE without an LLM finding, also for the specification questions on Gherkin and PR documents: a classifier can be wrong too, so it never blocks hard, and a developer checks the finding. A leaked secret in the diff keeps the PR away from Jev, as from the LLM.

In combined mode, werkrevisor appends a hint list to the user prompt after the full files: file, line, scenario id, question and value of each flagged unit, but no text from the PR. An answer ignored for its low confidence is in the list too, marked as uncertain. The LLM checks these units and writes concrete findings.

The comment gets a table "Classifier" with the flagged units, the count of passed units and the count of answers ignored for their low confidence.

Output

• One PR comment per head commit: verdict, reasons, up to max_findings findings with file and lines, the mechanical files in one line, usage (model, calls, token count before the call, reported tokens, duration, cost), and the reviewed head and base commit. The reasons are sorted from the most to the least severe: ERROR, RED, ORANGE, YELLOW. A reason from a limit names its config key and value, e.g. [size] max_changed_lines = 800. A reason that concerns many files is one line with at most 3 examples and the count of the rest. The usage names the config sources, e.g. config: defaults + host + repository (.werkrevisor.toml at base 3d174cc).
• The findings in the comment show their severity, e.g. 1. **[high] Possible Bug** at …. They are sorted from high to low severity, before the cut at max_findings.
• The commit status ai-review/assessment on the head commit.
• One JSON line per run on stdout and in build/werkrevisor/usage.jsonl.
• A short summary on stderr.

The usage line has timestamp, repository, PR, head and base commit, config sources (config_source) and host config (host_config: env or the file path), verdict, exit code, change kind, count of mechanical files (mechanical_files), reasons, model, calls, prompt, completion and cache tokens, cost and cost status, the token count before the call (counted_input_tokens, counted_raw_tokens, token_count_factor, token_count_method) and worst_case_usd, duration, and the key budget before and after the run. With the classifier it also has the engine mode, the Jev model, units, flagged and ignored answers, calls, input tokens and cost. Reasoning tokens are part of the completion tokens and are not counted twice. The cost is reported when PR-Agent could price every call, estimated from the operator prices otherwise, none with cost_usd 0.0 when no model call was made, or unknown when a model call reported no usage. The comment then says "no model call, cost 0 USD" or "cost unknown".

An ERROR comment has its own marker, so a successful retry can still post its result. werkrevisor posts at most one ERROR comment per head commit.

Build and test

The runtime is the pinned image pragent/pr-agent:0.47.0@sha256:7d98954c… (Debian 13, Python 3.14, PR-Agent under /app). werkrevisor uses only the Python standard library and the packages in that image.

Run the tests inside the image:

docker run --rm -v "$PWD":/werkrevisor -w /werkrevisor --entrypoint python \
    pragent/pr-agent:0.47.0@sha256:7d98954c29289846a08faf00aaea0f1f1375b1676cd8a2ed303e0af07010af7a -m unittest -v

The tests need no network, no LLM and no Gitea. One test runs the real PR-Agent reviewer with a fake Gitea provider and a fake model.

Build the image and the rootfs archive for hosts without Docker:

tools/build-rootfs

This creates the image werkrevisor:<version> and werkrevisor-rootfs-<version>.tar.zst.

Evaluate the classifier questions on local PR documents, without Gitea:

tools/jev-eval --units-only ../hsadmin-ng/doc/PR/*.md                          # extraction only, no network
TYPESAFE_API_KEY=… tools/jev-eval --config config/hsadmin-ng.toml --no-repo-config ../hsadmin-ng/doc/PR/*.md
TYPESAFE_API_KEY=… tools/jev-eval --config config/werkrevisor.toml docs/prs/*.md

It uses the config layers of the review. --repo-config FILE puts a repository config on top of the host config, with the same rules as the review; without it, .werkrevisor.toml of the current directory is used if present, unless --no-repo-config is given.

It prints one line per unit and question (file, unit, question, problem value, confidence, verdict) and the totals (verdicts, calls, input tokens, cost). An answer below min_confidence shows IGNORED with the confidence and counts apart from the verdicts. The archive has no image metadata. Start it with PATH=/app/.venv/bin:/usr/local/bin:/usr/bin:/bin, PYTHONPATH=/werkrevisor:/app, PYTHONNOUSERSITE=1 and the command python -s -m werkrevisor.

Measure real PRs without writing to Gitea and without a model call:

GITEA_BOT_TOKEN=… tools/dry-run https://git.example.org/owner/repo/pulls/302 config/hsadmin-ng.toml

Operations

Werkator build

werkrevisor runs as a Werkator (v1.36.0) build with trigger.onPullRequestTarget: true. Such a build is host-only: put it into the machine config .git/werkator/.werkator.yml of the checkout, never into a committed file. Werkator checks out the base branch, sets the WERKATOR_PR_* and WERKATOR_GITEA_* variables, and publishes the exit code under the pinned statusContext on the PR head. The werkdock sandbox clears the environment and the rootfs archive keeps no image metadata, so werkdock.env sets PATH and PYTHONPATH. The host config comes from the host, never from the checkout: WERKREVISOR_CONFIG_TOML holds the TOML itself, else WERKREVISOR_CONFIG names a file in the image (/werkrevisor/config/). The repository config comes from the base commit through the Gitea API, never from the PR head.

Hardening against a writable HOME

werkdock may bind a home directory as /root that ordinary PR builds of the same repository can write. werkrevisor therefore loads no code and no data from HOME:

• Start it with python -s -m werkrevisor and PYTHONNOUSERSITE=1; the image entrypoint does so.
• At startup it refuses to run (exit 30) if user site-packages are enabled or a sys.path entry lies under HOME.
• It removes the working directory, a repository checkout, from sys.path.
• Before PR-Agent and litellm load, it points HOME and the XDG directories to a fresh temporary directory, sets LITELLM_MODE=PRODUCTION (no .env file is loaded) and LITELLM_LOCAL_MODEL_COST_MAP=True (no price map download), and forces TIKTOKEN_CACHE_DIR to the tokenizer that litellm ships.

Machine config

mih09, werkdock with the rootfs archive:

# .git/werkator/.werkator.yml (not committed)
builds:
  ai-review:
    trigger:
      onPullRequestTarget: true
    statusContext: ai-review/gate
    cleanCommand: ""
    buildCommand: python -s -m werkrevisor
    timeout: 10m
    artifactDirs: [build/werkrevisor]
    werkdock:
      enabled: true
      rootfs: /path/to/werkrevisor-rootfs-0.15.0.tar.zst
      env:
        PATH: /app/.venv/bin:/usr/local/bin:/usr/bin:/bin
        PYTHONPATH: /werkrevisor:/app
        PYTHONNOUSERSITE: "1"
        GITEA_BOT_TOKEN: "…"
        OPENROUTER_API_KEY: "…"
        # The complete operator config; start from config/werkprobe.toml and change what you need.
        WERKREVISOR_CONFIG_TOML: |
          [model]
          id = "openrouter/anthropic/claude-sonnet-5.5"
          input_price_usd_per_mtok = 2.00
          output_price_usd_per_mtok = 10.00
          max_context_tokens = 150000
          max_output_tokens = 4000
          reasoning_effort = "low"

          [budget]
          max_usd_per_review = 0.40
          min_key_remaining_usd = 0.50

          # … all other sections of config/werkprobe.toml …

vm4006, Docker:

# .git/werkator/.werkator.yml (not committed)
builds:
  ai-review:
    trigger:
      onPullRequestTarget: true
    statusContext: ai-review/gate
    cleanCommand: ""
    buildCommand: python -s -m werkrevisor
    timeout: 10m
    artifactDirs: [build/werkrevisor]
    docker:
      enabled: true
      image: werkrevisor:0.15.0
      env:
        PYTHONNOUSERSITE: "1"
        GITEA_BOT_TOKEN: "…"
        OPENROUTER_API_KEY: "…"
        WERKREVISOR_CONFIG_TOML: |
          # the complete content of config/hsadmin-ng.toml, adapted

To switch the model or the reasoning level, edit id, the two prices, reasoning_effort and, if needed, max_context_tokens in WERKREVISOR_CONFIG_TOML of the machine config; the next review uses them, no new image is needed. The inline TOML replaces the config file completely, so paste the complete host config. A config that lacks a section after the merge with .werkrevisor.toml is refused, because a default section would have, for example, no risky paths. Each run logs the config sources on stderr; the usage line has them as config_source and host_config.

A changed rootfs path makes werkdock import a new image, so give each version its own archive name. Werkator needs gitea.baseUrl, gitea.owner, gitea.repo and git.token for this trigger.

Set the build timeout above timeout_seconds × (max_retries + 1) + 120 seconds; werkrevisor stops itself at that limit. Make ai-review/gate a required status in the Gitea branch protection only after the trial. A review approval starts no Werkator build; restart the build by hand to let werkrevisor see a new human approval.

Secrets

• GITEA_BOT_TOKEN belongs to a dedicated bot user. It needs repository read access, issue comment write access, commit status write access, and, for pass_with_review_request and approval, the right to request reviewers. For approval, it needs write access to submit official reviews and must be in approvals_whitelist_username.
• OPENROUTER_API_KEY is a dedicated key per host with a credit limit in OpenRouter.
• TYPESAFE_API_KEY is a TypeSafe key for Jev; it is needed only when [engine] mode is not llm.
• Store them in the machine config .git/werkator/.werkator.yml (werkdock.env or docker.env) with file mode 0600, never in the repository or the operator config.

Key exchange

1. Create a new key in OpenRouter (or a new Gitea token for the bot) with the same limits.
2. Replace the value in werkdock.env or docker.env of the machine config.
3. Restart Werkator, so it reads the machine config again.
4. Start a review, e.g. with a new commit on a test PR, and check the usage line.
5. Delete the old key in OpenRouter or Gitea.

Stop, restart, remove

• Stop: disable the ai-review build in the Werkator config and reload Werkator. If ai-review/gate is a required status, remove that requirement first, else no PR can merge. In approval mode, also remove the bot from approvals_whitelist_username, so humans approve alone.
• Restart: enable the build again. A new commit or a re-run of the build reviews the current head again; an existing result comment for that head is reused without a model call.
• Remove: stop it, delete the build config with its secrets, the rootfs archive or image, and revoke the OpenRouter key and the bot token.