72 lines
3.3 KiB
Markdown
72 lines
3.3 KiB
Markdown
# PR#40: MiniMax M3 advises in the reviews on git.javagil.de
|
|
|
|
> **WARNING:** This document describes only the change applied in this PR.
|
|
> It may already be outdated once the next PR is merged.
|
|
> Historic PR documentation is not maintained along with new PRs; treat it as a snapshot.
|
|
|
|
## The Problem
|
|
|
|
Since werkrevisor 0.21.0, Jev decides whether a PR of werkrevisor or Werkator needs a human,
|
|
and the model only advises.
|
|
Sonnet 5.5 still advises there, and costs 0.14 to 0.39 USD per review.
|
|
On 4 heads with 10 known findings, MiniMax M3 with `reasoning_effort = "medium"` found as many as Sonnet 5.5,
|
|
2 of 10, for 0.02 to 0.05 USD per review, but took 181 to 242 s instead of 24 to 63 s.
|
|
|
|
In a first local review of werkrevisor PR#37 with MiniMax M3, both calls hit the timeout of 300 s,
|
|
and the review ended as ERROR, although Jev had judged every file and the model only advises.
|
|
The second review of the same PR took 51 s and cost 0.02 USD.
|
|
|
|
The docs and configs call such a model a "cheap model".
|
|
The user prefers the pair "economy model" and "frontier model".
|
|
|
|
## Non-Goals
|
|
|
|
- No change of `config/hsadmin-ng.toml`: there the model decides, and the team chooses it.
|
|
- No change of the settings in `config/werkprobe.toml`: the tests build on them.
|
|
|
|
## The Scenarios
|
|
|
|
### Feature: an advising model never ends a review
|
|
|
|
#### Scenario#40.01: a failed advising model leaves the verdict to Jev
|
|
|
|
So that a slow economy model does not hold a PR that Jev found harmless.
|
|
|
|
- **Given** `[engine] model_role = "advises"` and `[path_judge] scope = "all"`
|
|
- **When** the model call fails or times out, and Jev `<judged>`
|
|
- **Then** the failure is a reason of verdict `<verdict>`
|
|
|
|
| judged | verdict |
|
|
|---|---|
|
|
| every changed file and found nothing | GREEN, a hint |
|
|
| not every changed file | YELLOW |
|
|
|
|
The reason `model_failed` is YELLOW and becomes a hint only when Jev judged every changed file.
|
|
With `model_role = "decides"`, the failure stays an ERROR.
|
|
With either role, a newer head ends the review.
|
|
|
|
##### Verified by
|
|
|
|
- [test_pathjudge: test_a_failed_advising_model_leaves_the_verdict_to_jev](../../tests/test_pathjudge.py)
|
|
- [test_pathjudge: test_a_superseded_head_still_ends_the_review_when_the_model_only_advises](../../tests/test_pathjudge.py)
|
|
|
|
## The Solution
|
|
|
|
- `config/werkator.toml` and `config/werkrevisor.toml` use `openrouter/minimax/minimax-m3` with `reasoning_effort = "medium"`:
|
|
- prices of 0.30 and 1.20 USD per 1M input and output tokens;
|
|
- `max_output_tokens = 32000`, since MiniMax M3 thinks up to 30000 tokens;
|
|
- `timeout_seconds = 300`, since a call took up to 242 s;
|
|
- `token_count_factor = 1.05`, since MiniMax reported at most 2 % more tokens than o200k_base counted;
|
|
- `max_usd_per_review = 0.20`.
|
|
- With `model_role = "advises"`, a failed or timed-out model call gives the reason `model_failed`,
|
|
which is a hint when Jev judged every changed file, and YELLOW otherwise.
|
|
PR-Agent reports no usage for a failed call, so the usage record holds the remaining key limit after it.
|
|
- Docs, configs and comments say "economy model" instead of "cheap model".
|
|
- At the deployment, the machine configs of werkrevisor and Werkator on mih09 get the same `[model]` and `[budget]`,
|
|
and the Werkator build `ai-review` gets a timeout of 15 minutes for 2 calls of up to 300 s each.
|
|
- werkrevisor 0.21.1.
|
|
|
|
## Prerequisite PRs
|
|
|
|
- PR#39: werkrevisor 0.21.0, Jev decides and the model advises, merged.
|