76 lines
3.4 KiB
Markdown
76 lines
3.4 KiB
Markdown
# PR#56: the operator chooses the OpenRouter providers of the model
|
||
|
||
> **WARNING:** This document describes only the change applied in this PR.
|
||
> It may already be outdated once the next PR is merged.
|
||
> Historic PR documentation is not maintained along with new PRs; treat it as a snapshot.
|
||
|
||
## The Problem
|
||
|
||
OpenRouter routes MiniMax M3 to one of 13 providers, among them two with 4-bit weights
|
||
and some with an uptime of 93 to 97 % on 2026-10-08.
|
||
Some calls hang: on 2026-10-08, two of twelve measured reviews waited more than 700 s for about 1100 output tokens,
|
||
while the others took 75 to 241 s.
|
||
A call that hangs uses up its timeout of 300 s; only the retry delivers, and without a retry the review fails.
|
||
werkrevisor cannot choose the providers, and it does not record which provider answered.
|
||
|
||
## Non-Goals
|
||
|
||
- No `provider_order` for the excerpt selection: it uses GPT-6 Luna, whose calls did not hang in the measurement.
|
||
- No provider in the usage record for the model review: PR-Agent 0.47.0 does not report it.
|
||
|
||
## The Scenarios
|
||
|
||
### Feature: the operator chooses the OpenRouter providers of the model
|
||
|
||
#### Scenario#56.01: the model review asks the providers in the configured order
|
||
|
||
So that the review avoids providers that hang, but still answers when the first one fails.
|
||
|
||
- **Given** `[model] provider_order` names providers
|
||
- **When** werkrevisor configures PR-Agent
|
||
- **Then** PR-Agent asks OpenRouter for these providers in this order, with fallbacks to others
|
||
|
||
##### Verified by
|
||
|
||
- [test_provider_order: test_the_model_review_asks_the_providers_in_the_configured_order](../../tests/test_provider_order.py)
|
||
|
||
#### Scenario#56.02: the change summary asks the same providers and records which one answered
|
||
|
||
So that a slow summary shows its provider in the usage record.
|
||
|
||
- **Given** `[doc_check] provider_order` names providers
|
||
- **When** werkrevisor asks the economy model for the change summary
|
||
- **Then** the request asks OpenRouter for these providers in this order
|
||
- **and** the usage record holds the provider that answered
|
||
|
||
##### Verified by
|
||
|
||
- [test_provider_order: test_the_change_summary_asks_the_same_providers_and_records_which_one_answered](../../tests/test_provider_order.py)
|
||
|
||
## The Solution
|
||
|
||
- `[model] provider_order` and `[doc_check] provider_order`: lists of OpenRouter provider slugs, empty by default;
|
||
`allow_fallbacks` stays on, so that another provider answers when the first ones fail.
|
||
- config/werkrevisor.toml, config/werkator.toml and config/werkcrappit.toml name `minimax`, the maker of the model:
|
||
8-bit weights and 99.86 % uptime on 2026-10-08.
|
||
- The README names both keys; docs/werkrevisor.werkbaum records the measurement of 2026-10-08.
|
||
- werkrevisor 0.27.0.
|
||
|
||
## Measurement
|
||
|
||
On 2026-10-08 werkrevisor reviewed 4 merged PRs (#47, #50, #51, #53) with three settings of MiniMax M3, for about 0.25 USD:
|
||
|
||
| setting | reviews done | model review |
|
||
|---|---|---|
|
||
| `medium`, timeout 300 s, 1 retry | 3 of 4 | 75–118 s; #51 timed out after 735 s |
|
||
| `low`, timeout 300 s, 1 retry | 4 of 4 | 75–171 s; #51 took 540 s with the retry |
|
||
| `medium`, timeout 600 s, no retry | 3 of 4 | 105–241 s; #47 timed out after 728 s |
|
||
|
||
The calls that failed had produced about 1100 output tokens after more than 700 s, so a provider hung.
|
||
In the model comparison of 2026-10-07, `low` failed on 1 of 4 heads too, so neither setting is better:
|
||
both failed on 1 of 8 heads, and the configs keep `medium`.
|
||
|
||
## Prerequisite PRs
|
||
|
||
- PR#55: the diagrams show how a review runs today, merged.
|