werkrevisor/README.md

# werkrevisor

werkrevisor is an AI review gate for Gitea pull requests.
Simple PRs can merge after an automated review.
Risky PRs get human attention.

Werkator starts werkrevisor for each ready (non-draft) PR head commit.
werkrevisor runs on a checkout of the base branch and never executes PR files.
It reads the PR through the Gitea API, applies deterministic rules,
and asks a model through PR-Agent 0.47.0 for a review.
The exit code is the merge gate: Werkator publishes it as a commit status on the PR head.

Status: v0.1, trial.

## Verdicts and exit codes

| Verdict | Label | Exit code | Meaning |
|---|---|---|---|
| GREEN | PASS | 0 | All files are AI-approvable, no hard rule fired, the model found no relevant issue. |
| YELLOW | HUMAN_REVIEW | 10 | A human must look: hard rule, change kind not AI-approvable, unclear model answer, `merge_with_caution`, PR too large. |
| RED | CHANGES_REQUIRED | 20 | The model reports relevant issues, or the classifier rates a PR-document unit as clearly bad. |
| ERROR | review not completed, human action needed | 30 | Technical error, invalid model answer, timeout, budget, clipped context, skipped files, superseded head. |

ERROR is never GREEN.
A run without a complete model review is never GREEN.

With `yellow_gate = "block"` or `"pass_with_review_request"`, YELLOW can still pass the gate (exit 0):

- A configured human reviewer approved exactly this head commit (Gitea review `commit_id`).
- `yellow_gate = "pass_with_review_request"` is set and a review request to a configured human reviewer is pending.

With `yellow_gate = "approval"`, the missing approval is the lock; see [Approval mode](#approval-mode).

An approval does not unlock RED.
Only a new review of a new head commit can change RED.

## Approval mode

In this mode the branch protection requires one approval, and the bot is one of the allowed approvers.
werkrevisor acts as a reviewer:

| Verdict | Gitea action | Exit code |
|---|---|---|
| GREEN | Submit a review `APPROVED` with `commit_id` = head commit. | 0 |
| YELLOW | No bot review. Request an official review from `human_reviewers`, unless a request is pending or a human approved this head. | 0; 10 if the request failed |
| RED | Submit a review `REQUEST_CHANGES` with `commit_id` = head commit and up to 3 findings. | 20 |
| ERROR | No review. | 30 |
| Superseded head | No review, no comment. | 30 |

werkrevisor reads the PR again directly before it writes anything.
It posts the result comment for every verdict, including GREEN, and links it from the review.
A rerun on the same head does not submit a second review.
An earlier bot review of the PR stays as it is; Gitea marks a review on an older head as stale.
The bot never approves a PR that it opened itself: that is ERROR.

Branch protection of the base branch (Gitea 1.27 API field names):

| Field | Value | Why |
|---|---|---|
| `required_approvals` | `1` | The approval is the lock. |
| `enable_approvals_whitelist` | `true` | Only listed users count. |
| `approvals_whitelist_username` | the bot user and `human_reviewers` | A GREEN bot review or a human approval unlocks the merge. |
| `block_on_rejected_reviews` | `true` | A RED review `REQUEST_CHANGES` blocks the merge. |
| `block_on_official_review_requests` | `true` | The review request of a YELLOW blocks until the human reviews. |
| `dismiss_stale_approvals` | `true` | A new commit removes the bot approval of the old head. |
| `block_on_outdated_branch` | `true` | Werkator starts no new review when only the base moves. |
| `enable_status_check` | `true` | The Werkator status is required. |
| `status_check_contexts` | `["ai-review/gate"]` | The pinned `statusContext` of the Werkator build. |

The bot user needs write access to the repository to submit official reviews.

## How a run works

1. Read the PR.
   If its head differs from `WERKATOR_PR_HEAD_SHA` or the PR is not open, stop with exit 30 and write nothing.
2. Look for an own result comment with the marker `<!-- werkrevisor:<head_sha> -->`.
   If it exists, reuse its verdict, do not call the model, and do not comment again.
3. Apply the deterministic rules to the changed files and the diff.
   With `full_files = true`, read the new version of every changed text file at the head commit.
4. Estimate the input tokens and the worst-case cost, including the full files.
   Too large or too expensive gives YELLOW without a model call.
   With `[engine] mode = "classifier"` or `"combined"`, Jev judges the sections and scenarios of the changed PR documents first;
   see [Classifier](#classifier).
5. Query the OpenRouter key budget.
   Too little budget gives ERROR without a model call.
6. Run the PR-Agent reviewer with all settings forced by werkrevisor.
7. Check that the model saw every changed file completely.
8. Read the PR again.
   If the head moved meanwhile, stop with exit 30 and write nothing.
9. Write one PR comment, the commit status `ai-review/assessment`, and one usage line.

The deterministic rules come only from the operator config.
The model can make a verdict worse, never better than the rules allow.
If the secret scan fires, the diff is not sent to the model.

## Configuration

The operator config is TOML.
`WERKREVISOR_CONFIG_TOML` holds it inline; if that is empty, `--config` or `WERKREVISOR_CONFIG` names a file.
If both are set, the inline TOML wins.
An invalid inline TOML is ERROR (exit 30); the message names the line and column, never the content.
Unknown keys are errors, so a typo cannot weaken a rule.
Examples: [config/werkprobe.toml](config/werkprobe.toml) (cheap) and [config/hsadmin-ng.toml](config/hsadmin-ng.toml) (1 USD per review).

### `[model]`

| Key | Default | Meaning |
|---|---|---|
| `id` | required | LiteLLM model id, e.g. `openrouter/google/gemini-3.8-flash`. No fallback models are used. |
| `input_price_usd_per_mtok` | required | USD per 1M input tokens, maintained by the operator. |
| `output_price_usd_per_mtok` | required | USD per 1M output tokens, maintained by the operator. |
| `max_input_tokens` | required | Larger PRs are YELLOW before any model call. |
| `max_output_tokens` | required | Output limit including reasoning tokens. |
| `reasoning_effort` | `"low"` | `none`, `minimal`, `low`, `medium`, `high`, `xhigh` or `max`. |
| `reasoning_max_tokens` | `0` | OpenRouter reasoning budget; 0 leaves it unset. |
| `context_tokens` | `0` | Context window for models unknown to PR-Agent; 0 means input plus output. |
| `timeout_seconds` | `120` | Timeout per model call. |
| `max_retries` | `1` | 0 or 1; PR-Agent retries a failed call at most once. |

### `[budget]`

| Key | Default | Meaning |
|---|---|---|
| `max_usd_per_review` | required | Worst-case cost limit per review; above it the PR is YELLOW. |
| `min_key_remaining_usd` | `0.0` | The key must keep at least this budget, else ERROR. |
| `chars_per_token` | `3.5` | Token estimate. |
| `prompt_overhead_chars` | `16000` | PR-Agent prompt size plus repository context files. |
| `diff_expansion_factor` | `1.5` | PR-Agent adds context lines and line numbers to the diff. |

The worst case is (estimated input × input price + `max_output_tokens` × output price) × (`max_retries` + 1).
Before the model call, werkrevisor reads `GET https://openrouter.ai/api/v1/key`.
If `limit_remaining` is below `max_usd_per_review` or `min_key_remaining_usd`, the result is ERROR "budget".
A key without a limit passes this check.

### `[size]`

`max_changed_lines` and `max_files`; exceeding one gives YELLOW without a model call.

### `[categories]`

Glob lists per category: `docs`, `tests`, `main`, `api_spec`.
The first match wins in the order `api_spec`, `tests`, `docs`, `main`; else the file is `other`.
A pattern without `/` matches the file name in any directory; `**` matches any number of directories.

`auto_approvable` lists the change kinds that can become GREEN:

| Change kind | Condition |
|---|---|
| `docs_only` | Only docs. |
| `tests_only_added` | Only tests; no deleted test file, no removed line with `@Test`, `assert`, `should` or `expect`, no added `@Disabled`, `@Ignore` or `.skip(`. |
| `docs_and_tests` | Docs and tests with the same test conditions. |
| `refactoring` | Main code, optionally docs; no tests, no API spec. |

All other change kinds (`tests_modified`, `mixed`) are at most YELLOW.
werkrevisor tells the model the change kind, so the model can object if, for example, a refactoring changes behavior.

### `[hard_yellow]`

`paths` lists globs of risky files, e.g. authorization, RBAC, migrations, secrets, CI and deploy config, dependency manifests.
A renamed file counts with its old and new path.
`secret_scan = true` scans added lines for private keys, `-----BEGIN`, AWS keys, generic `token|secret|password|api_key = "..."` assignments,
and tokens with the prefixes `hsak_` and `sk-or-`.
Hard rules give YELLOW; the model cannot override them.

### `[review]`

| Key | Default | Meaning |
|---|---|---|
| `extra_instructions` | `""` | Project review rules for the model. |
| `max_findings` | `3` | Findings requested from the model and shown in the comment. |
| `red_min_findings` | `2` | This many findings give RED; 0 disables it. One finding gives YELLOW. |
| `repo_context_files` | `[]` | Files PR-Agent reads from the default branch, e.g. `AGENTS.md`. |
| `full_files` | `false` | Add the complete new version of every changed file to the prompt. |
| `max_full_file_chars` | `40000` | Limit per rendered file; a larger file is left out. |
| `max_full_files_chars` | `120000` | Limit for all rendered files together; files beyond it are left out. |

A diff with a few context lines hides constraints stated elsewhere in a file, e.g. in its header comment.
PR-Agent 0.47.0 caps the extra context lines per hunk at 10, so werkrevisor adds the full files itself:

- It reads each changed, not deleted file at the head commit through the Gitea API (`/raw/{path}?ref=<sha>`).
- It skips files that PR-Agent cannot review (binary extensions); the coverage rules already make them YELLOW.
- It appends one section after the diff, in the user prompt, never in the system prompt, because PR content is untrusted:
  `## Full content of the changed files (new version at <sha>, …)`, one `### Full file: '<path>' (<n> lines)` block per file,
  and `## End of the full content of the changed files`.
- Every content line starts with its line number and ` | `,
  so file content can never look like a section marker or a `## File:` diff header.
- A file that is not UTF-8 text or exceeds a limit is left out with the YELLOW reason "full-file context incomplete: <file>".
  A review with incomplete context is never GREEN.

The hook wraps `get_pr_diff` as the PR-Agent reviewer module calls it, so the section becomes part of the prompt variable `diff`.
The operator `extra_instructions` should tell the model to check the changed lines against constraints stated elsewhere in the full files.

### `[engine]`

Optional; without it the mode is `llm`.

| `mode` | Meaning |
|---|---|
| `llm` | The LLM review alone, as before 0.5.0. |
| `classifier` | Jev judges the PR documents; no LLM call. Only a `docs_only` PR whose changed files are all judged PR documents can become GREEN. |
| `combined` | Jev judges the PR documents, then the LLM reviews the PR with Jev's flagged units as hints. |

### `[classifier]`

Optional unless `[engine] mode` uses it; when present, every key is required.

| Key | Meaning |
|---|---|
| `endpoint` | `https://api.typesafe.ai/v1/systemone`; https only. |
| `model` | `jev-latest`. |
| `input_price_usd_per_mtok` | USD per 1M input tokens, maintained by the operator; Jev's output is free. |
| `max_state_chars` | A unit with a larger JSON state is not sent; the result is YELLOW "classifier context incomplete". |
| `max_units` | Units beyond this number are not sent; the result is YELLOW. |
| `timeout_seconds` | Timeout per Jev request. |
| `pr_doc_globs` | Globs of the PR documents, e.g. `doc/PR/*.md`; `TEMPLATE*.md` and `README.md` never count. |
| `min_confidence` | A choice or score answer below this confidence gives YELLOW; 0 disables it. |
| `scenario_questions`, `section_questions` | Arrays of tables, see below. |

Each question has `name`, `type` (`noul`, `choice` or `score`), `instructions`, `red_above` and `yellow_above`.
Every question asks for a problem, so a high value always means "bad":

- `noul`: the value is the probability of "yes"; optional `criteria = { true = "…", false = "…" }`.
- `choice`: `criteria` is a table of options; `problem_options` lists the bad ones, and the value is their summed probability.
- `score`: `criteria` is a list of 2 to 10 levels from "no problem" to "worst problem";
  the value is Jev's expected level divided by the top level.

A value above `red_above` with enough confidence gives RED, a value above `yellow_above` or a low confidence YELLOW.
A noul has no confidence; its `yellow_above` band covers the uncertain answers.
The example configs contain default questions for readability, one aspect per scenario, implementation details,
Given/When/Then use and vague outcomes in scenarios, and for readability, slop and the writing rules in sections.

### `[gate]`

| Key | Default | Meaning |
|---|---|---|
| `yellow_gate` | `"block"` | `block` (exit 10), `pass_with_review_request`, or `approval` (see [Approval mode](#approval-mode)). |
| `human_reviewers` | `[]` | Gitea logins whose approval unlocks YELLOW and who get review requests; required unless `block`. |
| `assessment_context` | `"ai-review/assessment"` | Context of the informative commit status. |

The informative status maps GREEN to `success`, YELLOW to `pending` ("human review required"), RED to `failure` and ERROR to `error`.
The gate itself is the Werkator status of the build (planned context `ai-review/gate`).

## Environment

| Variable | Source | Meaning |
|---|---|---|
| `WERKATOR_PR_NUMBER`, `WERKATOR_PR_HEAD_SHA` | Werkator | PR and head commit to review. |
| `WERKATOR_PR_HEAD_REF`, `WERKATOR_PR_BASE_REF`, `WERKATOR_PR_BASE_SHA` | Werkator | Informative. |
| `WERKATOR_GITEA_BASE_URL`, `WERKATOR_GITEA_OWNER`, `WERKATOR_GITEA_REPO` | Werkator | Repository; the URL must use https. |
| `GITEA_BOT_TOKEN` | host secret | Bot token: read PR, write issue comments, write commit status, request reviewers. |
| `OPENROUTER_API_KEY` | host secret | OpenRouter key with a credit limit; not needed with `[engine] mode = "classifier"`. |
| `TYPESAFE_API_KEY` | host secret | TypeSafe key for Jev; needed when `[engine] mode` is not `llm`. |
| `WERKREVISOR_CONFIG_TOML` | host config | The operator TOML itself; wins over `WERKREVISOR_CONFIG`. |
| `WERKREVISOR_CONFIG` | host config | Path to the operator TOML, used when `WERKREVISOR_CONFIG_TOML` is empty. |
| `PYTHONNOUSERSITE` | host config | Must be `1`; see [Hardening against a writable HOME](#hardening-against-a-writable-home). |

werkrevisor never prints tokens or keys.
It redacts known secret values and token patterns from comments, logs and usage lines.
HTTP redirects are not followed, so the token cannot reach another host.

## Classifier

Jev from TypeSafe is a classifier: it answers typed questions with probabilities, fast and cheap, but writes no findings.
werkrevisor calls it directly with the standard library; TLS stays verified, redirects are not followed, the key never appears in output.
A request that fails with HTTP 5xx or a timeout is retried once.

For each changed PR document werkrevisor reads the new version at the head commit and splits it into units:

- each `##` or `###` section with its text, without the scenarios and without `Verified by` blocks;
- each Gherkin scenario `#### Scenario#<id>: <title>` (or `#### Scenario: <title>`) with its steps,
  its feature and its `#### Background` as context, without the `##### Verified by` block.

Headings in code blocks do not count.
Jev gets one request per unit with all questions of its kind; up to 4 requests run in parallel.
The verdict per question follows the thresholds above; a failed Jev request is ERROR.
A RED from the classifier stays RED without an LLM finding, because the PR documents are the specification.
A leaked secret in the diff keeps the PR away from Jev, as from the LLM.

In `combined` mode, werkrevisor appends a hint list to the user prompt after the full files:
file, line, scenario id, question and value of each flagged unit, but no text from the PR.
The LLM checks these units and writes concrete findings.

The comment gets a table "Classifier" with the flagged units and the count of passed units.

## Output

- One PR comment per head commit: verdict, reasons, up to `max_findings` findings with file and lines,
  usage (model, calls, tokens, duration, cost), and the reviewed head and base commit.
- The commit status `ai-review/assessment` on the head commit.
- One JSON line per run on stdout and in `build/werkrevisor/usage.jsonl`.
- A short summary on stderr.

The usage line has timestamp, repository, PR, head and base commit, verdict, exit code, reasons, model, calls,
prompt, completion and cache tokens, cost and cost status, estimated tokens and worst-case cost, duration,
and the key budget before and after the run.
With the classifier it also has the engine mode, the Jev model, units, flagged answers, calls, input tokens and cost.
Reasoning tokens are part of the completion tokens and are not counted twice.
The cost is `reported` when PR-Agent could price every call, `estimated` from the operator prices otherwise, or `unknown`.

An ERROR comment has its own marker, so a successful retry can still post its result.
werkrevisor posts at most one ERROR comment per head commit.

## Build and test

The runtime is the pinned image `pragent/pr-agent:0.47.0@sha256:7d98954c…` (Debian 13, Python 3.14, PR-Agent under `/app`).
werkrevisor uses only the Python standard library and the packages in that image.

Run the tests inside the image:

```sh
docker run --rm -v "$PWD":/werkrevisor -w /werkrevisor --entrypoint python \
    pragent/pr-agent:0.47.0@sha256:7d98954c29289846a08faf00aaea0f1f1375b1676cd8a2ed303e0af07010af7a -m unittest -v
```

The tests need no network, no LLM and no Gitea.
One test runs the real PR-Agent reviewer with a fake Gitea provider and a fake model.

Build the image and the rootfs archive for hosts without Docker:

```sh
tools/build-rootfs
```

This creates the image `werkrevisor:<version>` and `werkrevisor-rootfs-<version>.tar.zst`.

Evaluate the classifier questions on local PR documents, without Gitea:

```sh
tools/jev-eval --units-only ../hsadmin-ng/doc/PR/*.md                          # extraction only, no network
TYPESAFE_API_KEY=… tools/jev-eval --config config/hsadmin-ng.toml ../hsadmin-ng/doc/PR/*.md
```

It prints one line per unit and question (file, unit, question, problem value, confidence, verdict)
and the totals (calls, input tokens, cost).
The archive has no image metadata.
Start it with `PATH=/app/.venv/bin:/usr/local/bin:/usr/bin:/bin`, `PYTHONPATH=/werkrevisor:/app`, `PYTHONNOUSERSITE=1`
and the command `python -s -m werkrevisor`.

Measure real PRs without writing to Gitea and without a model call:

```sh
GITEA_BOT_TOKEN=… tools/dry-run https://git.example.org/owner/repo/pulls/302 config/hsadmin-ng.toml
```

## Operations

### Werkator build

werkrevisor runs as a Werkator (v1.36.0) build with `trigger.onPullRequestTarget: true`.
Such a build is host-only: put it into the machine config `.git/werkator/.werkator.yml` of the checkout, never into a committed file.
Werkator checks out the base branch, sets the `WERKATOR_PR_*` and `WERKATOR_GITEA_*` variables,
and publishes the exit code under the pinned `statusContext` on the PR head.
The werkdock sandbox clears the environment and the rootfs archive keeps no image metadata,
so `werkdock.env` sets `PATH` and `PYTHONPATH`.
The operator config comes from the host, never from the checkout:
`WERKREVISOR_CONFIG_TOML` holds the TOML itself, else `WERKREVISOR_CONFIG` names a file in the image (`/werkrevisor/config/`).

### Hardening against a writable HOME

werkdock may bind a home directory as `/root` that ordinary PR builds of the same repository can write.
werkrevisor therefore loads no code and no data from HOME:

- Start it with `python -s -m werkrevisor` and `PYTHONNOUSERSITE=1`; the image entrypoint does so.
- At startup it refuses to run (exit 30) if user site-packages are enabled or a `sys.path` entry lies under HOME.
- It removes the working directory, a repository checkout, from `sys.path`.
- Before PR-Agent and litellm load, it points `HOME` and the XDG directories to a fresh temporary directory,
  sets `LITELLM_MODE=PRODUCTION` (no `.env` file is loaded) and `LITELLM_LOCAL_MODEL_COST_MAP=True` (no price map download),
  and forces `TIKTOKEN_CACHE_DIR` to the tokenizer that litellm ships.

### Machine config

mih09, werkdock with the rootfs archive:

```yaml
# .git/werkator/.werkator.yml (not committed)
builds:
  ai-review:
    trigger:
      onPullRequestTarget: true
    statusContext: ai-review/gate
    cleanCommand: ""
    buildCommand: python -s -m werkrevisor
    timeout: 10m
    artifactDirs: [build/werkrevisor]
    werkdock:
      enabled: true
      rootfs: /path/to/werkrevisor-rootfs-0.5.0.tar.zst
      env:
        PATH: /app/.venv/bin:/usr/local/bin:/usr/bin:/bin
        PYTHONPATH: /werkrevisor:/app
        PYTHONNOUSERSITE: "1"
        GITEA_BOT_TOKEN: "…"
        OPENROUTER_API_KEY: "…"
        # The complete operator config; start from config/werkprobe.toml and change what you need.
        WERKREVISOR_CONFIG_TOML: |
          [model]
          id = "openrouter/anthropic/claude-sonnet-5.5"
          input_price_usd_per_mtok = 2.00
          output_price_usd_per_mtok = 10.00
          max_input_tokens = 25000
          max_output_tokens = 4000
          reasoning_effort = "low"

          [budget]
          max_usd_per_review = 0.20
          min_key_remaining_usd = 0.50

          # … all other sections of config/werkprobe.toml …
```

vm4006, Docker:

```yaml
# .git/werkator/.werkator.yml (not committed)
builds:
  ai-review:
    trigger:
      onPullRequestTarget: true
    statusContext: ai-review/gate
    cleanCommand: ""
    buildCommand: python -s -m werkrevisor
    timeout: 10m
    artifactDirs: [build/werkrevisor]
    docker:
      enabled: true
      image: werkrevisor:0.5.0
      env:
        PYTHONNOUSERSITE: "1"
        GITEA_BOT_TOKEN: "…"
        OPENROUTER_API_KEY: "…"
        WERKREVISOR_CONFIG_TOML: |
          # the complete content of config/hsadmin-ng.toml, adapted
```

To switch the model or the reasoning level, edit `id`, the two prices, `reasoning_effort` and, if needed, the token limits
in `WERKREVISOR_CONFIG_TOML` of the machine config; the next review uses them, no new image is needed.
The inline TOML replaces the config file completely, so paste the complete config.
A config that lacks a section is refused, because a default section would have, for example, no risky paths.
Each run logs the config source on stderr, and the usage line has it as `config_source` (`env` or the file path).

A changed `rootfs` path makes werkdock import a new image, so give each version its own archive name.
Werkator needs `gitea.baseUrl`, `gitea.owner`, `gitea.repo` and `git.token` for this trigger.

Set the build timeout above `timeout_seconds` × (`max_retries` + 1) + 120 seconds;
werkrevisor stops itself at that limit.
Make `ai-review/gate` a required status in the Gitea branch protection only after the trial.
A review approval starts no Werkator build; restart the build by hand to let werkrevisor see a new human approval.

### Secrets

- `GITEA_BOT_TOKEN` belongs to a dedicated bot user.
  It needs repository read access, issue comment write access, commit status write access,
  and, for `pass_with_review_request` and `approval`, the right to request reviewers.
  For `approval`, it needs write access to submit official reviews and must be in `approvals_whitelist_username`.
- `OPENROUTER_API_KEY` is a dedicated key per host with a credit limit in OpenRouter.
- `TYPESAFE_API_KEY` is a TypeSafe key for Jev; it is needed only when `[engine] mode` is not `llm`.
- Store them in the machine config `.git/werkator/.werkator.yml` (`werkdock.env` or `docker.env`) with file mode 0600,
  never in the repository or the operator config.

### Key exchange

1. Create a new key in OpenRouter (or a new Gitea token for the bot) with the same limits.
2. Replace the value in `werkdock.env` or `docker.env` of the machine config.
3. Restart Werkator, so it reads the machine config again.
4. Start a review, e.g. with a new commit on a test PR, and check the usage line.
5. Delete the old key in OpenRouter or Gitea.

### Stop, restart, remove

- Stop: disable the `ai-review` build in the Werkator config and reload Werkator.
  If `ai-review/gate` is a required status, remove that requirement first, else no PR can merge.
  In approval mode, also remove the bot from `approvals_whitelist_username`, so humans approve alone.
- Restart: enable the build again.
  A new commit or a re-run of the build reviews the current head again;
  an existing result comment for that head is reused without a model call.
- Remove: stop it, delete the build config with its secrets, the rootfs archive or image,
  and revoke the OpenRouter key and the bot token.