113 lines
5.2 KiB
Markdown
113 lines
5.2 KiB
Markdown
# PR#24: the dashboard counts the billed OpenRouter calls from before the service
|
|
|
|
> **WARNING:** This document describes only the change applied in this PR.
|
|
> It may already be outdated once the next PR is merged.
|
|
> Historic PR documentation is not maintained along with new PRs; treat it as a snapshot.
|
|
|
|
## The Problem
|
|
|
|
The statistics service of werkrevisor 0.18.0 stores the usage record of each run since 2026-10-06T09:13:41Z.
|
|
The reviews and trial runs before that cost about 10 USD at OpenRouter, but the dashboard does not show them.
|
|
So the costs of the first days and the token ratio of models such as GPT-6 Astra and DeepSeek V4 Pro are missing.
|
|
|
|
OpenRouter exports each call as one CSV row, with an id, the time in UTC, the model, the billed amount and the tokens.
|
|
Such a row has no pull request: the review that caused it cannot be found again.
|
|
|
|
## Non-Goals
|
|
|
|
- No pull requests, reviews or verdicts for billed calls: the CSV does not tell them.
|
|
- No calls from the time that the service already records: they would count twice.
|
|
- No regular comparison with the bill; this import fills the time before the service once.
|
|
|
|
## The Scenarios
|
|
|
|
### Feature: billed calls fill the time before the service
|
|
|
|
#### Scenario#24.01: a billed call counts in the costs and tokens, not in the reviews
|
|
|
|
So that the dashboard shows what werkrevisor cost since its first run.
|
|
|
|
- **Given** OpenRouter billed a call of Claude Sonnet 5.5 for 0.07 USD, with 31000 input and 850 output tokens,
|
|
before the first usage record
|
|
- **When** the operator imports the call
|
|
- **and** someone opens the dashboard with a read token
|
|
- **Then** the card of its period shows "Billed earlier: 1 call = 0.0700 USD" below the total
|
|
- **and** the table per model holds its tokens in the line of Claude Sonnet 5.5
|
|
- **and** the numbers of reviews and pull requests do not change
|
|
|
|
##### Verified by
|
|
|
|
- [UsageApiIntegrationTest: a billed call counts in costs and tokens, not in reviews](../../server/src/test/kotlin/de/hoennig/werkrevisor/server/UsageApiIntegrationTest.kt)
|
|
- manual check in a browser, see [Measurement](#measurement)
|
|
|
|
#### Scenario#24.02: a call from the time of the service is not imported
|
|
|
|
So that no call counts twice.
|
|
|
|
- **Given** the first usage record is from 2026-10-06T09:13:41Z
|
|
- **and** a run writes its usage record at its end
|
|
- **When** the operator imports a call from `<time>`
|
|
- **Then** the call `<result>`
|
|
|
|
| time | result |
|
|
|---|---|
|
|
| 2026-10-06T08:21:43Z, an earlier run | counts |
|
|
| 2026-10-06T09:13:33Z, the run of the first usage record | does not count |
|
|
| 2026-10-06T09:22:29Z, a later run | does not count |
|
|
|
|
##### Verified by
|
|
|
|
- [UsageApiIntegrationTest: calls from the time of the service are skipped](../../server/src/test/kotlin/de/hoennig/werkrevisor/server/UsageApiIntegrationTest.kt)
|
|
|
|
#### Scenario#24.03: an import repeated with the same file changes nothing
|
|
|
|
So that the operator can repeat an import after an error.
|
|
|
|
- **Given** the operator imported a CSV file
|
|
- **When** the operator imports the same file again
|
|
- **Then** each call counts once
|
|
|
|
##### Verified by
|
|
|
|
- [UsageApiIntegrationTest: a billed call imported twice counts once](../../server/src/test/kotlin/de/hoennig/werkrevisor/server/UsageApiIntegrationTest.kt)
|
|
- [test_import_openrouter_activity](../../tests/test_import_openrouter_activity.py)
|
|
|
|
#### Scenario#24.04: the import uses the model names of the usage records
|
|
|
|
So that a model has one line in the table per model.
|
|
|
|
- **Given** the CSV names the model `anthropic/claude-sonnet-5.5-20260928`
|
|
- **When** the operator imports the CSV
|
|
- **Then** the table per model shows the call in the line `anthropic/claude-sonnet-5.5`,
|
|
together with the reviews of this model
|
|
|
|
##### Verified by
|
|
|
|
- [test_import_openrouter_activity](../../tests/test_import_openrouter_activity.py)
|
|
|
|
## The Solution
|
|
|
|
- `POST /api/billed-calls` stores billed calls with a source token in the table `billed_call`, keyed by the provider's id;
|
|
it skips calls from 10 minutes before the first usage record on, since that record is written at the end of its run,
|
|
and answers how many it stored, found again or skipped;
|
|
without a usage record it stores nothing, since the cut-off is then unknown.
|
|
- Each sum holds `billedCostUsd` and `billedCalls`; `costUsd` and the token counts include them,
|
|
the cost per pull request does not, since a billed call has no pull request.
|
|
- The Web Component lists the billed part with "=" below the total, if a period has one;
|
|
the table per model includes billed calls, the tables per repository and per verdict do not.
|
|
- `tools/import-openrouter-activity <csv>` reads the activity export of OpenRouter, removes the date suffix of the model,
|
|
and sends the calls in parts of 500; a cancelled call counts with the amount that OpenRouter billed for it.
|
|
- werkrevisor 0.18.3.
|
|
|
|
## Measurement
|
|
|
|
On 2026-10-06, the service ran locally with a copy of the live database and imported the OpenRouter export of 65 calls.
|
|
It stored 61 calls for 9.76 USD and skipped 4 calls from the first run with a usage record on;
|
|
a second import stored none.
|
|
Each card showed "Billed earlier: 61 calls = 9.76 USD" in one line;
|
|
the table per model showed 4 models, with 24 : 1 for GPT-6 Astra and 88 : 1 for Claude Sonnet 5.5.
|
|
|
|
## Prerequisite PRs
|
|
|
|
- PR#23: input and output tokens per model.
|