9.0 KiB
PR#21: a statistics service and a dashboard Web Component
WARNING: This document describes only the change applied in this PR. It may already be outdated once the next PR is merged. Historic PR documentation is not maintained along with new PRs; treat it as a snapshot.
The Problem
werkrevisor is a command-line tool without state of its own. Each run writes a usage record with its costs, but only as a CI artifact, and Werkator prunes old artifacts: on 2026-10-06, 7 usage files were left of more than 50 reviews. Nobody can tell what the reviews cost per day, week or month, or what one pull request costs on average.
werkrevisor shall become an open-source tool that works with any CI and with Gitea and Forgejo. The long-term target is a service with an API; the command-line tool stays, also for local reviews. The decisions on language, place, store and Web Component are listed under Open Questions. This PR takes the first small step towards it: a service that collects the usage records and shows them.
Non-Goals
- No reviews in the service yet: the command-line tool still runs the review, in the CI or locally.
- No integration into the Werkator GUI yet; the dashboard is a Web Component, so that this can follow later.
- No import of earlier reviews: the statistics start with the first reported run.
- No deployment in this PR; it follows after the merge, see Follow-up PRs.
The Scenarios
Feature: the command-line tool reports each run to the service
Scenario#21.01: a run reports its usage record when a service is configured
So that every review lands in the statistics, wherever it runs.
- Given
WERKREVISOR_STATS_URLandWERKREVISOR_STATS_TOKENare set - When werkrevisor finishes a review
- Then the service has received the usage record of the run
- and the record holds the link to the pull request
Verified by
Scenario#21.02: a failed report never changes the review
So that the statistics can never block or break a review.
- Given the service
<state> - When werkrevisor finishes a review
- Then the verdict and the exit code are the same as without a service
- and the log names the failed report
| state |
|---|
| is not configured |
| cannot be reached |
| answers HTTP 500 |
| refuses the token |
Verified by
- test_stats_client.StatsClientTest.test_failed_report_never_changes_the_review
- test_stats_client.HttpClientTest
Feature: the service collects the usage records
Scenario#21.03: only a known source may report
So that nobody can falsify the statistics.
- Given the service knows the source token of werkprobe
- When a client reports a usage record with
<token> - Then the service answers
<status>
| token | status |
|---|---|
| the source token of werkprobe | 201, record stored |
| an unknown token | 401, nothing stored |
| no token | 401, nothing stored |
Verified by
Scenario#21.04: a record reported twice counts once
So that a retried report does not double the costs.
- Given the service has stored the record of a run
- When a client reports the same record again
- Then the statistics count the run once
A second run on the same commit, e.g. a manual rerun, has its own time; it cost money again, so it counts as another review.
Verified by
Feature: the statistics answer what the reviews cost
Background
- The costs of a run are the model costs plus the costs of the three Jev uses.
- A run with unknown model costs counts as a review, but not in the sum.
Scenario#21.05: totals for the last 24 hours, 7 days and 30 days
So that the operator sees the current spending at a glance.
- Given the service has stored reviews of the last 40 days
- When anyone asks for the statistics
- Then they get for 24 hours, 7 days and 30 days the sum in USD, the number of reviews, the number of pull requests, the costs per pull request and the number of reviews with unknown costs
Verified by
Scenario#21.06: sums per day, week and month
So that the operator sees the development of the costs.
- Given the service has stored reviews of several months
- When anyone asks for the statistics
- Then they get the sums per day of the last 30 days, per week of the last 12 weeks and per month of the last 12 months
Verified by
Scenario#21.07: details only with a read token
So that the names of private repositories and their pull requests stay private.
- Given the service has stored reviews of the repository
team/app - When someone asks for the statistics
<with> - Then the answer
<holds>
| with | holds |
|---|---|
| a read token | the totals, the sums per repository, model and verdict, and the last 20 reviews with links |
| no token | only the totals and the sums per period |
Verified by
Feature: the dashboard is a Web Component
Scenario#21.08: the service delivers the dashboard as a Web Component and an app around it
So that the dashboard can be embedded later in other GUIs, e.g. in Werkator or werkregie.
- Given the service runs
- When a browser opens the start page of the service
- Then the page embeds the element
<werkrevisor-dashboard>from/werkrevisor-dashboard.js- and the page hands a read token, if the user entered one, to the element
Verified by
- UsageApiIntegrationTest: the start page embeds the dashboard Web Component
- manual check in a browser, see Measurement
The Solution
- The service is a Kotlin Spring Boot application in
server/, built with Gradle, as Werkator is; the review core stays Python, because it builds on PR-Agent, and talks to the service only by HTTP and JSON.POST /api/usagestores a usage record in SQLite; a source token from the server config is required. The service refuses implausible values: noowner/namerepository, a link that is no http(s) URL, texts above 200 or links above 500 characters, an unknown verdict or cost status, a missing commit id, and costs that are no number or outside 0 to 100 USD. It refuses a body above 16 KB or without a length, and a token without theBearerscheme.GET /api/statsreturns the statistics; a read token adds the details.GET /delivers the app,GET /werkrevisor-dashboard.jsthe Web Component.
- The Web Component
<werkrevisor-dashboard>is plain JavaScript in one file, a Custom Element with Shadow DOM, without a frontend build, so that Werkator, werkregie or any page can embed it. - The command-line tool posts its usage record to
WERKREVISOR_STATS_URLwithWERKREVISOR_STATS_TOKEN; a failure is logged and changes nothing else; after 8 seconds the run stops waiting for the report. The URL must use https, except on localhost, so that a source token never travels in clear text. - The usage record gets the field
pr_url, so that the dashboard can link the pull request. - The Gradle wrapper is the one of Werkator;
distributionSha256Sumpins the Gradle distribution. - werkrevisor 0.18.0.
Open Questions
- Which language for the service? Kotlin with Spring Boot, as Werkator; no more Python in the backend.
- Where does the service live?
In
server/of this repository, so that a change of the usage record and of the service go into one PR. - Which store? SQLite: a few thousand records per year, a backup is a file copy.
- How is the Web Component built? Plain JavaScript without a build step, as Werkator's own frontend.
Measurement
On 2026-10-06, the service ran locally with the 180 usage records of the earlier dry runs and reviews. Without a token, the dashboard showed the totals and the sums per period; with the read token, also the sums per repository, model and verdict and the 20 recent reviews with their links.
Follow-up PRs
- Deployment of the service on mih09 and the report from the Werkator builds.
- Integration of the Web Component into the Werkator GUI.
- Hit rates and calibration of Jev in the dashboard.
- Reviews in the service, triggered by webhooks, with a read-only Git mirror for the agent.