werkrevisor/README.md

werkrevisor

werkrevisor is an AI review gate for Gitea pull requests. Simple PRs can merge after an automated review. Risky PRs get human attention.

Werkator starts werkrevisor for each ready (non-draft) PR head commit. werkrevisor runs on a checkout of the base branch and never executes PR files. It reads the PR through the Gitea API, applies deterministic rules, and asks a model through PR-Agent 0.47.0 for a review. The exit code is the merge gate: Werkator publishes it as a commit status on the PR head.

Status: v0.1, trial.

Verdicts and exit codes

Verdict	Label	Exit code	Meaning
GREEN	PASS	0	All files are AI-approvable, no hard rule fired, the model found no relevant issue.
YELLOW	HUMAN_REVIEW	10	A human must look: hard rule, change kind not AI-approvable, unclear model answer, merge_with_caution, PR too large.
RED	CHANGES_REQUIRED	20	The model reports relevant issues, or the classifier rates a PR-document unit as clearly bad.
ERROR	review not completed, human action needed	30	Technical error, invalid model answer, timeout, budget, clipped context, skipped files, superseded head.

ERROR is never GREEN. A run without a complete model review is never GREEN.

With yellow_gate = "block" or "pass_with_review_request", YELLOW can still pass the gate (exit 0):

• A configured human reviewer approved exactly this head commit (Gitea review commit_id).
• yellow_gate = "pass_with_review_request" is set and a review request to a configured human reviewer is pending.

With yellow_gate = "approval", the missing approval is the lock; see Approval mode.

An approval does not unlock RED. Only a new review of a new head commit can change RED.

Approval mode

In this mode the branch protection requires one approval, and the bot is one of the allowed approvers. werkrevisor acts as a reviewer:

Verdict	Gitea action	Exit code
GREEN	Submit a review APPROVED with commit_id = head commit.	0
YELLOW	No bot review. Request an official review from human_reviewers, unless a request is pending or a human approved this head.	0; 10 if the request failed
RED	Submit a review REQUEST_CHANGES with commit_id = head commit and up to 3 findings.	20
ERROR	No review.	30
Superseded head	No review, no comment.	30

werkrevisor reads the PR again directly before it writes anything. It posts the result comment for every verdict, including GREEN, and links it from the review. A rerun on the same head does not submit a second review. An earlier bot review of the PR stays as it is; Gitea marks a review on an older head as stale. The bot never approves a PR that it opened itself: that is ERROR.

Branch protection of the base branch (Gitea 1.27 API field names):

Field	Value	Why
required_approvals	1	The approval is the lock.
enable_approvals_whitelist	true	Only listed users count.
approvals_whitelist_username	the bot user and human_reviewers	A GREEN bot review or a human approval unlocks the merge.
block_on_rejected_reviews	true	A RED review REQUEST_CHANGES blocks the merge.
block_on_official_review_requests	true	The review request of a YELLOW blocks until the human reviews.
dismiss_stale_approvals	true	A new commit removes the bot approval of the old head.
block_on_outdated_branch	true	Werkator starts no new review when only the base moves.
enable_status_check	true	The Werkator status is required.
status_check_contexts	["ai-review/gate"]	The pinned statusContext of the Werkator build.

The bot user needs write access to the repository to submit official reviews.

How a run works

1. Read the PR. If its head differs from WERKATOR_PR_HEAD_SHA or the PR is not open, stop with exit 30 and write nothing.
2. Look for an own result comment with the marker <!-- werkrevisor:<head_sha> -->. If it exists, reuse its verdict, do not call the model, and do not comment again.
3. Apply the deterministic rules to the changed files and the diff. With full_files = true, read the new version of every changed text file at the head commit.
4. Estimate the input tokens and the worst-case cost, including the full files. Too large or too expensive gives YELLOW without a model call. With [engine] mode = "classifier" or "combined", Jev judges the sections and scenarios of the changed PR documents first; see Classifier.
5. Query the OpenRouter key budget. Too little budget gives ERROR without a model call.
6. Run the PR-Agent reviewer with all settings forced by werkrevisor.
7. Check that the model saw every changed file completely.
8. Read the PR again. If the head moved meanwhile, stop with exit 30 and write nothing.
9. Write one PR comment, the commit status ai-review/assessment, and one usage line.

The deterministic rules come only from the operator config. The model can make a verdict worse, never better than the rules allow. If the secret scan fires, the diff is not sent to the model.

Configuration

The operator config is TOML. WERKREVISOR_CONFIG_TOML holds it inline; if that is empty, --config or WERKREVISOR_CONFIG names a file. If both are set, the inline TOML wins. An invalid inline TOML is ERROR (exit 30); the message names the line and column, never the content. Unknown keys are errors, so a typo cannot weaken a rule. Examples: config/werkprobe.toml (cheap) and config/hsadmin-ng.toml (1 USD per review).

[model]

Key	Default	Meaning
id	required	LiteLLM model id, e.g. openrouter/google/gemini-3.8-flash. No fallback models are used.
input_price_usd_per_mtok	required	USD per 1M input tokens, maintained by the operator.
output_price_usd_per_mtok	required	USD per 1M output tokens, maintained by the operator.
max_input_tokens	required	Larger PRs are YELLOW before any model call.
max_output_tokens	required	Output limit including reasoning tokens.
reasoning_effort	"low"	none, minimal, low, medium, high, xhigh or max.
reasoning_max_tokens	0	OpenRouter reasoning budget; 0 leaves it unset.
context_tokens	0	Context window for models unknown to PR-Agent; 0 means input plus output.
timeout_seconds	120	Timeout per model call.
max_retries	1	0 or 1; PR-Agent retries a failed call at most once.

[budget]

Key	Default	Meaning
max_usd_per_review	required	Worst-case cost limit per review; above it the PR is YELLOW.
min_key_remaining_usd	0.0	The key must keep at least this budget, else ERROR.
chars_per_token	3.5	Token estimate.
prompt_overhead_chars	16000	PR-Agent prompt size plus repository context files.
diff_expansion_factor	1.5	PR-Agent adds context lines and line numbers to the diff.

The worst case is (estimated input × input price + max_output_tokens × output price) × (max_retries + 1). Before the model call, werkrevisor reads GET https://openrouter.ai/api/v1/key. If limit_remaining is below max_usd_per_review or min_key_remaining_usd, the result is ERROR "budget". A key without a limit passes this check.

[size]

max_changed_lines and max_files; exceeding one gives YELLOW without a model call.

[categories]

Glob lists per category: docs, tests, main, api_spec. The first match wins in the order api_spec, tests, docs, main; else the file is other. A pattern without / matches the file name in any directory; ** matches any number of directories.

auto_approvable lists the change kinds that can become GREEN:

Change kind	Condition
docs_only	Only docs.
tests_only_added	Only tests; no deleted test file, no removed line with @Test, assert, should or expect, no added @Disabled, @Ignore or .skip(.
docs_and_tests	Docs and tests with the same test conditions.
refactoring	Main code, optionally docs; no tests, no API spec.

All other change kinds (tests_modified, mixed) are at most YELLOW. werkrevisor tells the model the change kind, so the model can object if, for example, a refactoring changes behavior.

[hard_yellow]

paths lists globs of risky files, e.g. authorization, RBAC, migrations, secrets, CI and deploy config, dependency manifests. A renamed file counts with its old and new path. secret_scan = true scans added lines for private keys, -----BEGIN, AWS keys, generic token|secret|password|api_key = "..." assignments, and tokens with the prefixes hsak_ and sk-or-. Hard rules give YELLOW; the model cannot override them.

[review]

Key	Default	Meaning
extra_instructions	""	Project review rules for the model.
max_findings	3	Findings requested from the model and shown in the comment.
red_min_findings	2	This many findings give RED; 0 disables it. One finding gives YELLOW.
repo_context_files	[]	Files PR-Agent reads from the default branch, e.g. AGENTS.md.
full_files	false	Add the complete new version of every changed file to the prompt.
max_full_file_chars	40000	Limit per rendered file; a larger file is left out.
max_full_files_chars	120000	Limit for all rendered files together; files beyond it are left out.

A diff with a few context lines hides constraints stated elsewhere in a file, e.g. in its header comment. PR-Agent 0.47.0 caps the extra context lines per hunk at 10, so werkrevisor adds the full files itself:

• It reads each changed, not deleted file at the head commit through the Gitea API (/raw/{path}?ref=<sha>).
• It skips files that PR-Agent cannot review (binary extensions); the coverage rules already make them YELLOW.
• It appends one section after the diff, in the user prompt, never in the system prompt, because PR content is untrusted: ## Full content of the changed files (new version at <sha>, …), one ### Full file: '<path>' (<n> lines) block per file, and ## End of the full content of the changed files.
• Every content line starts with its line number and |, so file content can never look like a section marker or a ## File: diff header.
• A file that is not UTF-8 text or exceeds a limit is left out with the YELLOW reason "full-file context incomplete: ". A review with incomplete context is never GREEN.

The hook wraps get_pr_diff as the PR-Agent reviewer module calls it, so the section becomes part of the prompt variable diff. The operator extra_instructions should tell the model to check the changed lines against constraints stated elsewhere in the full files.

[engine]

Optional; without it the mode is llm.

mode	Meaning
llm	The LLM review alone, as before 0.5.0.
classifier	Jev judges the PR documents; no LLM call. Only a docs_only PR whose changed files are all judged PR documents can become GREEN.
combined	Jev judges the PR documents, then the LLM reviews the PR with Jev's flagged units as hints.

[classifier]

Optional unless [engine] mode uses it; when present, every key is required.

Key	Meaning
endpoint	https://api.typesafe.ai/v1/systemone; https only.
model	jev-latest.
input_price_usd_per_mtok	USD per 1M input tokens, maintained by the operator; Jev's output is free.
max_state_chars	A unit with a larger JSON state is not sent; the result is YELLOW "classifier context incomplete".
max_units	Units beyond this number are not sent; the result is YELLOW.
timeout_seconds	Timeout per Jev request.
pr_doc_globs	Globs of the PR documents, e.g. doc/PR/*.md; TEMPLATE*.md and README.md never count.
min_confidence	A choice or score answer below this confidence gives YELLOW; 0 disables it.
scenario_questions, section_questions	Arrays of tables, see below.

Each question has name, type (noul, choice or score), instructions, red_above and yellow_above. Every question asks for a problem, so a high value always means "bad":

• noul: the value is the probability of "yes"; optional criteria = { true = "…", false = "…" }.
• choice: criteria is a table of options; problem_options lists the bad ones, and the value is their summed probability.
• score: criteria is a list of 2 to 10 levels from "no problem" to "worst problem"; the value is Jev's expected level divided by the top level.

A value above red_above with enough confidence gives RED, a value above yellow_above or a low confidence YELLOW. A noul has no confidence; its yellow_above band covers the uncertain answers. The example configs contain default questions for readability, one aspect per scenario, implementation details, Given/When/Then use and vague outcomes in scenarios, and for readability, slop and the writing rules in sections.

[gate]

Key	Default	Meaning
yellow_gate	"block"	block (exit 10), pass_with_review_request, or approval (see Approval mode).
human_reviewers	[]	Gitea logins whose approval unlocks YELLOW and who get review requests; required unless block.
assessment_context	"ai-review/assessment"	Context of the informative commit status.

The informative status maps GREEN to success, YELLOW to pending ("human review required"), RED to failure and ERROR to error. The gate itself is the Werkator status of the build (planned context ai-review/gate).

Environment

Variable	Source	Meaning
WERKATOR_PR_NUMBER, WERKATOR_PR_HEAD_SHA	Werkator	PR and head commit to review.
WERKATOR_PR_HEAD_REF, WERKATOR_PR_BASE_REF, WERKATOR_PR_BASE_SHA	Werkator	Informative.
WERKATOR_GITEA_BASE_URL, WERKATOR_GITEA_OWNER, WERKATOR_GITEA_REPO	Werkator	Repository; the URL must use https.
GITEA_BOT_TOKEN	host secret	Bot token: read PR, write issue comments, write commit status, request reviewers.
OPENROUTER_API_KEY	host secret	OpenRouter key with a credit limit; not needed with [engine] mode = "classifier".
TYPESAFE_API_KEY	host secret	TypeSafe key for Jev; needed when [engine] mode is not llm.
WERKREVISOR_CONFIG_TOML	host config	The operator TOML itself; wins over WERKREVISOR_CONFIG.
WERKREVISOR_CONFIG	host config	Path to the operator TOML, used when WERKREVISOR_CONFIG_TOML is empty.
PYTHONNOUSERSITE	host config	Must be 1; see Hardening against a writable HOME.

werkrevisor never prints tokens or keys. It redacts known secret values and token patterns from comments, logs and usage lines. HTTP redirects are not followed, so the token cannot reach another host.

Classifier

Jev from TypeSafe is a classifier: it answers typed questions with probabilities, fast and cheap, but writes no findings. werkrevisor calls it directly with the standard library; TLS stays verified, redirects are not followed, the key never appears in output. A request that fails with HTTP 5xx or a timeout is retried once.

For each changed PR document werkrevisor reads the new version at the head commit and splits it into units:

• each ## or ### section with its text, without the scenarios and without Verified by blocks;
• each Gherkin scenario #### Scenario#<id>: <title> (or #### Scenario: <title>) with its steps, its feature and its #### Background as context, without the ##### Verified by block.

Headings in code blocks do not count. Jev gets one request per unit with all questions of its kind; up to 4 requests run in parallel. The verdict per question follows the thresholds above; a failed Jev request is ERROR. A RED from the classifier stays RED without an LLM finding, because the PR documents are the specification. A leaked secret in the diff keeps the PR away from Jev, as from the LLM.

In combined mode, werkrevisor appends a hint list to the user prompt after the full files: file, line, scenario id, question and value of each flagged unit, but no text from the PR. The LLM checks these units and writes concrete findings.

The comment gets a table "Classifier" with the flagged units and the count of passed units.

Output

• One PR comment per head commit: verdict, reasons, up to max_findings findings with file and lines, usage (model, calls, tokens, duration, cost), and the reviewed head and base commit.
• The commit status ai-review/assessment on the head commit.
• One JSON line per run on stdout and in build/werkrevisor/usage.jsonl.
• A short summary on stderr.

The usage line has timestamp, repository, PR, head and base commit, verdict, exit code, reasons, model, calls, prompt, completion and cache tokens, cost and cost status, estimated tokens and worst-case cost, duration, and the key budget before and after the run. With the classifier it also has the engine mode, the Jev model, units, flagged answers, calls, input tokens and cost. Reasoning tokens are part of the completion tokens and are not counted twice. The cost is reported when PR-Agent could price every call, estimated from the operator prices otherwise, or unknown.

An ERROR comment has its own marker, so a successful retry can still post its result. werkrevisor posts at most one ERROR comment per head commit.

Build and test

The runtime is the pinned image pragent/pr-agent:0.47.0@sha256:7d98954c… (Debian 13, Python 3.14, PR-Agent under /app). werkrevisor uses only the Python standard library and the packages in that image.

Run the tests inside the image:

docker run --rm -v "$PWD":/werkrevisor -w /werkrevisor --entrypoint python \
    pragent/pr-agent:0.47.0@sha256:7d98954c29289846a08faf00aaea0f1f1375b1676cd8a2ed303e0af07010af7a -m unittest -v

The tests need no network, no LLM and no Gitea. One test runs the real PR-Agent reviewer with a fake Gitea provider and a fake model.

Build the image and the rootfs archive for hosts without Docker:

tools/build-rootfs

This creates the image werkrevisor:<version> and werkrevisor-rootfs-<version>.tar.zst.

Evaluate the classifier questions on local PR documents, without Gitea:

tools/jev-eval --units-only ../hsadmin-ng/doc/PR/*.md                          # extraction only, no network
TYPESAFE_API_KEY=… tools/jev-eval --config config/hsadmin-ng.toml ../hsadmin-ng/doc/PR/*.md

It prints one line per unit and question (file, unit, question, problem value, confidence, verdict) and the totals (calls, input tokens, cost). The archive has no image metadata. Start it with PATH=/app/.venv/bin:/usr/local/bin:/usr/bin:/bin, PYTHONPATH=/werkrevisor:/app, PYTHONNOUSERSITE=1 and the command python -s -m werkrevisor.

Measure real PRs without writing to Gitea and without a model call:

GITEA_BOT_TOKEN=… tools/dry-run https://git.example.org/owner/repo/pulls/302 config/hsadmin-ng.toml

Operations

Werkator build

werkrevisor runs as a Werkator (v1.36.0) build with trigger.onPullRequestTarget: true. Such a build is host-only: put it into the machine config .git/werkator/.werkator.yml of the checkout, never into a committed file. Werkator checks out the base branch, sets the WERKATOR_PR_* and WERKATOR_GITEA_* variables, and publishes the exit code under the pinned statusContext on the PR head. The werkdock sandbox clears the environment and the rootfs archive keeps no image metadata, so werkdock.env sets PATH and PYTHONPATH. The operator config comes from the host, never from the checkout: WERKREVISOR_CONFIG_TOML holds the TOML itself, else WERKREVISOR_CONFIG names a file in the image (/werkrevisor/config/).

Hardening against a writable HOME

werkdock may bind a home directory as /root that ordinary PR builds of the same repository can write. werkrevisor therefore loads no code and no data from HOME:

• Start it with python -s -m werkrevisor and PYTHONNOUSERSITE=1; the image entrypoint does so.
• At startup it refuses to run (exit 30) if user site-packages are enabled or a sys.path entry lies under HOME.
• It removes the working directory, a repository checkout, from sys.path.
• Before PR-Agent and litellm load, it points HOME and the XDG directories to a fresh temporary directory, sets LITELLM_MODE=PRODUCTION (no .env file is loaded) and LITELLM_LOCAL_MODEL_COST_MAP=True (no price map download), and forces TIKTOKEN_CACHE_DIR to the tokenizer that litellm ships.

Machine config

mih09, werkdock with the rootfs archive:

# .git/werkator/.werkator.yml (not committed)
builds:
  ai-review:
    trigger:
      onPullRequestTarget: true
    statusContext: ai-review/gate
    cleanCommand: ""
    buildCommand: python -s -m werkrevisor
    timeout: 10m
    artifactDirs: [build/werkrevisor]
    werkdock:
      enabled: true
      rootfs: /path/to/werkrevisor-rootfs-0.5.0.tar.zst
      env:
        PATH: /app/.venv/bin:/usr/local/bin:/usr/bin:/bin
        PYTHONPATH: /werkrevisor:/app
        PYTHONNOUSERSITE: "1"
        GITEA_BOT_TOKEN: "…"
        OPENROUTER_API_KEY: "…"
        # The complete operator config; start from config/werkprobe.toml and change what you need.
        WERKREVISOR_CONFIG_TOML: |
          [model]
          id = "openrouter/anthropic/claude-sonnet-5.5"
          input_price_usd_per_mtok = 2.00
          output_price_usd_per_mtok = 10.00
          max_input_tokens = 25000
          max_output_tokens = 4000
          reasoning_effort = "low"

          [budget]
          max_usd_per_review = 0.20
          min_key_remaining_usd = 0.50

          # … all other sections of config/werkprobe.toml …

vm4006, Docker:

# .git/werkator/.werkator.yml (not committed)
builds:
  ai-review:
    trigger:
      onPullRequestTarget: true
    statusContext: ai-review/gate
    cleanCommand: ""
    buildCommand: python -s -m werkrevisor
    timeout: 10m
    artifactDirs: [build/werkrevisor]
    docker:
      enabled: true
      image: werkrevisor:0.5.0
      env:
        PYTHONNOUSERSITE: "1"
        GITEA_BOT_TOKEN: "…"
        OPENROUTER_API_KEY: "…"
        WERKREVISOR_CONFIG_TOML: |
          # the complete content of config/hsadmin-ng.toml, adapted

To switch the model or the reasoning level, edit id, the two prices, reasoning_effort and, if needed, the token limits in WERKREVISOR_CONFIG_TOML of the machine config; the next review uses them, no new image is needed. The inline TOML replaces the config file completely, so paste the complete config. A config that lacks a section is refused, because a default section would have, for example, no risky paths. Each run logs the config source on stderr, and the usage line has it as config_source (env or the file path).

A changed rootfs path makes werkdock import a new image, so give each version its own archive name. Werkator needs gitea.baseUrl, gitea.owner, gitea.repo and git.token for this trigger.

Set the build timeout above timeout_seconds × (max_retries + 1) + 120 seconds; werkrevisor stops itself at that limit. Make ai-review/gate a required status in the Gitea branch protection only after the trial. A review approval starts no Werkator build; restart the build by hand to let werkrevisor see a new human approval.

Secrets

• GITEA_BOT_TOKEN belongs to a dedicated bot user. It needs repository read access, issue comment write access, commit status write access, and, for pass_with_review_request and approval, the right to request reviewers. For approval, it needs write access to submit official reviews and must be in approvals_whitelist_username.
• OPENROUTER_API_KEY is a dedicated key per host with a credit limit in OpenRouter.
• TYPESAFE_API_KEY is a TypeSafe key for Jev; it is needed only when [engine] mode is not llm.
• Store them in the machine config .git/werkator/.werkator.yml (werkdock.env or docker.env) with file mode 0600, never in the repository or the operator config.

Key exchange

1. Create a new key in OpenRouter (or a new Gitea token for the bot) with the same limits.
2. Replace the value in werkdock.env or docker.env of the machine config.
3. Restart Werkator, so it reads the machine config again.
4. Start a review, e.g. with a new commit on a test PR, and check the usage line.
5. Delete the old key in OpenRouter or Gitea.

Stop, restart, remove

• Stop: disable the ai-review build in the Werkator config and reload Werkator. If ai-review/gate is a required status, remove that requirement first, else no PR can merge. In approval mode, also remove the bot from approvals_whitelist_username, so humans approve alone.
• Restart: enable the build again. A new commit or a re-run of the build reviews the current head again; an existing result comment for that head is reused without a model call.
• Remove: stop it, delete the build config with its secrets, the rootfs archive or image, and revoke the OpenRouter key and the bot token.