85 lines
3.5 KiB
Markdown
85 lines
3.5 KiB
Markdown
# PR#37: the usage record never ends a review, and the statistics bound the cost sums
|
|
|
|
> **WARNING:** This document describes only the change applied in this PR.
|
|
> It may already be outdated once the next PR is merged.
|
|
> Historic PR documentation is not maintained along with new PRs; treat it as a snapshot.
|
|
|
|
## The Problem
|
|
|
|
Two follow-ups of PR#21, the statistics service:
|
|
|
|
- werkrevisor writes the usage record after the review, to stdout and to `build/werkrevisor/usage.jsonl`.
|
|
An error there, e.g. a full disk, ends the finished review with a traceback,
|
|
so the gate check fails although the review itself is done and published.
|
|
- The statistics service bounds each cost value of a usage record at 100 USD, but not the sums it stores:
|
|
the LLM cost plus the excerpt cost, and the four Jev costs.
|
|
A record with values below the bound each can store a review cost of up to 400 USD.
|
|
|
|
## Non-Goals
|
|
|
|
- No change of the bound of 100 USD, which is far above any real review.
|
|
- No retry of a failed write: the usage record is statistics, not part of the review.
|
|
|
|
## The Scenarios
|
|
|
|
### Feature: the usage record never ends a review
|
|
|
|
#### Scenario#37.01: a failed write of the usage record keeps the result of the review
|
|
|
|
So that a full disk or a missing directory does not fail the gate check of a finished review.
|
|
|
|
- **Given** the usage file `build/werkrevisor/usage.jsonl` cannot be written
|
|
- **When** werkrevisor finishes a GREEN review
|
|
- **Then** the review ends with exit code 0
|
|
- **and** werkrevisor still reports the usage record to the statistics service
|
|
|
|
##### Verified by
|
|
|
|
- [test_flow: test_a_failed_write_of_the_usage_record_keeps_the_result](../../tests/test_flow.py)
|
|
|
|
#### Scenario#37.02: a usage record that cannot be built keeps the result of the review
|
|
|
|
So that a defect in the statistics code does not fail the gate check of a finished review.
|
|
|
|
- **Given** the usage record of a run cannot be built
|
|
- **When** werkrevisor finishes a GREEN review
|
|
- **Then** the review ends with exit code 0
|
|
- **and** werkrevisor neither writes nor reports a usage record
|
|
|
|
##### Verified by
|
|
|
|
- [test_flow: test_a_usage_record_that_cannot_be_built_keeps_the_result](../../tests/test_flow.py)
|
|
|
|
### Feature: the statistics service bounds the stored costs
|
|
|
|
#### Scenario#37.03: a record whose costs add up beyond the bound is refused
|
|
|
|
So that one record cannot distort the costs of the dashboard.
|
|
|
|
- **Given** a usage record with `<costs>`
|
|
- **When** werkrevisor reports it to the statistics service
|
|
- **Then** the service refuses it with HTTP 400
|
|
|
|
| costs |
|
|
|---|
|
|
| `cost_usd` 60 and `excerpt_cost_usd` 60 |
|
|
| `classifier_cost_usd`, `finding_judge_cost_usd`, `db_schema_judge_cost_usd` and `path_judge_cost_usd` 30 each |
|
|
|
|
##### Verified by
|
|
|
|
- [UsageApiIntegrationTest: a record whose costs add up beyond the bound is refused (Scenario#37.03)](../../statistics/src/test/kotlin/de/hoennig/werkrevisor/statistics/UsageApiIntegrationTest.kt)
|
|
|
|
## The Solution
|
|
|
|
- werkrevisor builds and writes the usage record inside a guard: an error goes to stderr,
|
|
and the review keeps its verdict and exit code;
|
|
a record that cannot be built is neither written nor reported, a record that cannot be written is still reported.
|
|
- The statistics service checks the sum of `cost_usd` and `excerpt_cost_usd`, and the sum of the four Jev costs,
|
|
against the bound; without `cost_usd`, the review cost stays unknown,
|
|
and `excerpt_cost_usd` is still checked on its own, which it was not before.
|
|
- werkrevisor 0.20.6.
|
|
|
|
## Prerequisite PRs
|
|
|
|
- PR#36: the result comment holds the findings until the line comments exist.
|