3.3 KiB
PR#40: MiniMax M3 advises in the reviews on git.javagil.de
WARNING: This document describes only the change applied in this PR. It may already be outdated once the next PR is merged. Historic PR documentation is not maintained along with new PRs; treat it as a snapshot.
The Problem
Since werkrevisor 0.21.0, Jev decides whether a PR of werkrevisor or Werkator needs a human,
and the model only advises.
Sonnet 5.5 still advises there, and costs 0.14 to 0.39 USD per review.
On 4 heads with 10 known findings, MiniMax M3 with reasoning_effort = "medium" found as many as Sonnet 5.5,
2 of 10, for 0.02 to 0.05 USD per review, but took 181 to 242 s instead of 24 to 63 s.
In a first local review of werkrevisor PR#37 with MiniMax M3, both calls hit the timeout of 300 s, and the review ended as ERROR, although Jev had judged every file and the model only advises. The second review of the same PR took 51 s and cost 0.02 USD.
The docs and configs call such a model a "cheap model". The user prefers the pair "economy model" and "frontier model".
Non-Goals
- No change of
config/hsadmin-ng.toml: there the model decides, and the team chooses it. - No change of the settings in
config/werkprobe.toml: the tests build on them.
The Scenarios
Feature: an advising model never ends a review
Scenario#40.01: a failed advising model leaves the verdict to Jev
So that a slow economy model does not hold a PR that Jev found harmless.
- Given
[engine] model_role = "advises"and[path_judge] scope = "all" - When the model call fails or times out, and Jev
<judged> - Then the failure is a reason of verdict
<verdict>
| judged | verdict |
|---|---|
| every changed file and found nothing | GREEN, a hint |
| not every changed file | YELLOW |
The reason model_failed is YELLOW and becomes a hint only when Jev judged every changed file.
With model_role = "decides", the failure stays an ERROR.
With either role, a newer head ends the review.
Verified by
- test_pathjudge: test_a_failed_advising_model_leaves_the_verdict_to_jev
- test_pathjudge: test_a_superseded_head_still_ends_the_review_when_the_model_only_advises
The Solution
config/werkator.tomlandconfig/werkrevisor.tomluseopenrouter/minimax/minimax-m3withreasoning_effort = "medium":- prices of 0.30 and 1.20 USD per 1M input and output tokens;
max_output_tokens = 32000, since MiniMax M3 thinks up to 30000 tokens;timeout_seconds = 300, since a call took up to 242 s;token_count_factor = 1.05, since MiniMax reported at most 2 % more tokens than o200k_base counted;max_usd_per_review = 0.20.
- With
model_role = "advises", a failed or timed-out model call gives the reasonmodel_failed, which is a hint when Jev judged every changed file, and YELLOW otherwise. PR-Agent reports no usage for a failed call, so the usage record holds the remaining key limit after it. - Docs, configs and comments say "economy model" instead of "cheap model".
- At the deployment, the machine configs of werkrevisor and Werkator on mih09 get the same
[model]and[budget], and the Werkator buildai-reviewgets a timeout of 15 minutes for 2 calls of up to 300 s each. - werkrevisor 0.21.1.
Prerequisite PRs
- PR#39: werkrevisor 0.21.0, Jev decides and the model advises, merged.