81 lines
3.4 KiB
Markdown
81 lines
3.4 KiB
Markdown
# PR#23: the dashboard shows input and output tokens per model
|
|
|
|
> **WARNING:** This document describes only the change applied in this PR.
|
|
> It may already be outdated once the next PR is merged.
|
|
> Historic PR documentation is not maintained along with new PRs; treat it as a snapshot.
|
|
|
|
## The Problem
|
|
|
|
The choice of the LLM depends on how the costs split into input and output.
|
|
On 23 runs of the earlier reviews, Claude Sonnet 5.5 read about 96 input tokens per output token,
|
|
so its input price decided about 94 % of the costs: in the median run, 58400 input tokens at 2 USD per million
|
|
cost about 0.117 USD, 740 output tokens at 10 USD per million about 0.007 USD.
|
|
GPT-6 Astra wrote far more output.
|
|
|
|
The statistics service stores only costs, so this ratio can only be measured by hand from CI artifacts,
|
|
which Werkator prunes after a while.
|
|
The usage record of every run already holds the token counts.
|
|
|
|
## Non-Goals
|
|
|
|
- No cost split into input and output in USD: the service knows no prices, and they change.
|
|
- No change of the command-line tool or of the usage record that it sends; only the service stores more of it.
|
|
|
|
## The Scenarios
|
|
|
|
### Feature: the dashboard shows the tokens per model
|
|
|
|
#### Scenario#23.01: the service keeps the token counts of a run
|
|
|
|
So that the ratio of input and output tokens can be measured over all runs.
|
|
|
|
- **Given** a run reports 58000 prompt tokens, 740 completion tokens and 1200 Jev input tokens
|
|
- **When** someone with a read token asks for the statistics
|
|
- **Then** the sums per model hold 58000 LLM input tokens, 740 LLM output tokens and 1200 Jev input tokens
|
|
|
|
##### Verified by
|
|
|
|
- [UsageApiIntegrationTest: token counts are summed per model](../../server/src/test/kotlin/de/hoennig/werkrevisor/server/UsageApiIntegrationTest.kt)
|
|
|
|
#### Scenario#23.02: the table per model shows the ratio of input to output
|
|
|
|
So that the operator sees at a glance which price decides the costs of a model.
|
|
|
|
- **Given** the reviews of the last 30 days used two models
|
|
- **When** someone opens the dashboard with a read token
|
|
- **Then** the table "Last 30 days by model" shows per model the input tokens, the output tokens and the ratio, e.g. "113 : 1"
|
|
|
|
##### Verified by
|
|
|
|
- manual check in a browser, see [Measurement](#measurement)
|
|
|
|
#### Scenario#23.03: a database of an earlier version gets the new columns
|
|
|
|
So that an update keeps the stored records.
|
|
|
|
- **Given** a database written by werkrevisor 0.18.1
|
|
- **When** the service of this version starts
|
|
- **Then** the stored records stay, with no token counts
|
|
|
|
##### Verified by
|
|
|
|
- [UsageApiIntegrationTest: a database of an earlier version gets the token columns](../../server/src/test/kotlin/de/hoennig/werkrevisor/server/UsageApiIntegrationTest.kt)
|
|
|
|
## The Solution
|
|
|
|
- The service stores `prompt_tokens`, `completion_tokens` and the sum of the three Jev input token counts;
|
|
it adds the columns to an existing database at the start.
|
|
- Each sum holds `llmInputTokens`, `llmOutputTokens` and `jevInputTokens`;
|
|
the Web Component shows them in the table per model, with the ratio.
|
|
- werkrevisor 0.18.2.
|
|
|
|
## Measurement
|
|
|
|
On 2026-10-06, the dashboard ran locally with 187 usage records of earlier runs.
|
|
The table "Last 30 days by model" showed 113 : 1 for Claude Sonnet 5.5, 24 : 1 for GPT-6 Astra,
|
|
19 : 1 for Gemini 3.8 Flash and 1 : 1 for DeepSeek V4 Pro, each in one line without wrapping.
|
|
|
|
## Prerequisite PRs
|
|
|
|
- PR#22: estimated costs marked, LLM and Jev listed apart.
|