94 lines
4.3 KiB
Markdown
94 lines
4.3 KiB
Markdown
# PR#28: the dashboard shows billed calls as "unknown repository" and dry runs as "local review"
|
|
|
|
> **WARNING:** This document describes only the change applied in this PR.
|
|
> It may already be outdated once the next PR is merged.
|
|
> Historic PR documentation is not maintained along with new PRs; treat it as a snapshot.
|
|
|
|
## The Problem
|
|
|
|
The table per repository of werkrevisor 0.19.0 holds only reviews that a CI run reported.
|
|
Two kinds of costs are missing there:
|
|
|
|
- The calls billed by OpenRouter before the service, about 9.76 USD:
|
|
they count in the totals, but the export names no repository, so the rows do not add up to the total.
|
|
- Dry runs with `tools/dry-run`, e.g. the model comparison of 2026-10-06:
|
|
they report nothing, so their costs are missing everywhere.
|
|
|
|
## Non-Goals
|
|
|
|
- No repository for billed calls: the export of OpenRouter does not tell it.
|
|
- No pull requests for dry runs: they review the same pull requests as the CI, so they would count them twice.
|
|
|
|
## The Scenarios
|
|
|
|
### Feature: the rows per repository add up to the total
|
|
|
|
#### Scenario#28.01: billed calls form the row "unknown repository"
|
|
|
|
So that the reader sees where the rest of the total comes from.
|
|
|
|
- **Given** the last 30 days hold billed calls for 9.76 USD
|
|
- **When** someone opens the dashboard
|
|
- **Then** the table per repository shows a row "unknown repository" in grey italics,
|
|
with 9.76 USD, 0 reviews and 0 pull requests
|
|
- **and** the costs of all rows add up to the total of the last 30 days
|
|
|
|
##### Verified by
|
|
|
|
- [UsageApiIntegrationTest: billed calls form the row unknown repository](../../server/src/test/kotlin/de/hoennig/werkrevisor/server/UsageApiIntegrationTest.kt)
|
|
- manual check in a browser, see [Measurement](#measurement)
|
|
|
|
### Feature: dry runs report their costs
|
|
|
|
#### Scenario#28.02: a dry run reports its usage record as local
|
|
|
|
So that the costs of trial runs show up in the statistics.
|
|
|
|
- **Given** `WERKREVISOR_STATS_URL` and `WERKREVISOR_STATS_TOKEN` are set
|
|
- **When** the operator runs `tools/dry-run` for a pull request
|
|
- **Then** the run reports its usage record with `local: true`
|
|
|
|
##### Verified by
|
|
|
|
- [test_stats_client: test_a_dry_run_reports_its_usage_record_as_local](../../tests/test_stats_client.py)
|
|
|
|
#### Scenario#28.03: local reviews count in costs and reviews, not in pull requests
|
|
|
|
So that the cost per pull request stays the one of the CI reviews.
|
|
|
|
- **Given** the CI reviewed one pull request for 0.10 USD
|
|
- **and** a dry run reviewed the same pull request for 0.02 USD
|
|
- **When** someone opens the dashboard
|
|
- **Then** the card shows 2 reviews, 1 pull request and a cost per pull request of 0.10 USD
|
|
- **and** the table per repository shows a row "local review" in grey italics with 0.02 USD, 1 review and 0 pull requests
|
|
|
|
##### Verified by
|
|
|
|
- [UsageApiIntegrationTest: local reviews count in costs and reviews, not in pull requests](../../server/src/test/kotlin/de/hoennig/werkrevisor/server/UsageApiIntegrationTest.kt)
|
|
- manual check in a browser, see [Measurement](#measurement)
|
|
|
|
## The Solution
|
|
|
|
- The usage record holds `local`, true for a run with `--no-publish`;
|
|
`tools/dry-run` passes `WERKREVISOR_STATS_URL` and `WERKREVISOR_STATS_TOKEN` into the container if both are set.
|
|
- The service stores `local`; a local review counts in costs and reviews, not in pull requests,
|
|
and the cost per pull request leaves its costs out.
|
|
- Each group of the table per repository holds `kind`: `repository`, `private`, `unknown` for the billed calls
|
|
or `local` for the dry runs; the Web Component names the last three in grey italics.
|
|
- The cards name the local reviews, e.g. "26 reviews (1 local)", and the recent reviews mark them with "local".
|
|
- No code change for the token: at the deployment, the operator adds one more source token to `WERKREVISOR_SERVER_SOURCE_TOKENS`
|
|
and passes it to dry runs as `WERKREVISOR_STATS_TOKEN`.
|
|
- werkrevisor 0.19.1.
|
|
|
|
## Measurement
|
|
|
|
On 2026-10-06, the service ran locally with a copy of the live database and one constructed local review.
|
|
The table per repository showed "unknown repository" with 9.76 USD, `mi/werkrevisor` with 2.72 USD
|
|
and "local review" with 0.0009 USD; the rows added up to 12.4765 USD, the total of the last 30 days.
|
|
The card of the last 30 days showed "26 reviews (1 local), 6 pull requests".
|
|
|
|
## Prerequisite PRs
|
|
|
|
- PR#24: billed OpenRouter calls.
|
|
- PR#26: private repositories keep their row without name.
|