werkrevisor 0.18.4: the statistics are public, only the recent reviews need a read token #25
No Assignees
Notifications
Due Date
No due date set.
Dependencies
No dependencies set.
Reference: mi/werkrevisor#25
Reference in New Issue
Block a user
See docs/prs/2026-10-06-PR#25-public-statistics.md.
🤖 Generated with Claude Code
🟡 werkrevisor YELLOW for
5f87148, superseded by the result foreb67d95🟡 werkrevisor: YELLOW / HUMAN_REVIEW
Change kind:
mixed.Reasons
Findings
[medium] Weakened test coverage at
server/src/test/kotlin/de/hoennig/werkrevisor/server/UsageApiIntegrationTest.kt:342-360, hint, as a line comment[low] Doc claims manual verification only at
docs/prs/2026-10-06-PR#25-public-statistics.md:33-70, hint, as a line commentClassifier
7 of 7 units passed all questions (1 PR document(s)).
Usage: model `openrouter/anthropic/claude-sonnet-5.5`, LLM done, tokens counted before the call 40349, calls 1, tokens in 39967 / out 576 (reasoning included in out), duration 13.2 s, cost 0.0857 USD (reported), classifier `jev-1.13.0` calls 7, tokens in 5216, cost 0.0002 USD, finding judge `jev-1.13.0` calls 1, tokens in 2149, cost 0.0001 USD, config: defaults + host + repository (.werkrevisor.toml at based7e9afa).Reviewed head `5f87148aefe0f73fa538bee2d7cc61be599b6bdb` against base `d7e9afa73791a535dab3561519983ccc4440c7c6`, werkrevisor 0.18.3.
werkrevisor: YELLOW / HUMAN_REVIEW for
5f87148aefe0f73fa538bee2d7cc61be599b6bdb.The result comment of werkrevisor holds the details.
@@ -0,0 +30,4 @@- **When** someone opens the dashboard without a token- **Then** the dashboard shows the tables "by repository", "by verdict" and "by model"##### Verified by[low] Doc claims manual verification only (werkrevisor)
hint
Scenario#25.01 lists "manual check in a browser" as verification, but the manual check cannot confirm the "by model" token table. The Measurement section says the live-database copy had "10 rows" of recent reviews, while the service returns up to 20 (
RECENT). The figure is plausible but not tied to any stated reason.@@ -340,3 +340,3 @@}test("details only with a read token (Scenario#21.07)") {test("sums per repository, model and verdict without a token (Scenario#25.01)") {[medium] Weakened test coverage (werkrevisor)
hint; Jev: real 0.60, severity 2.0 of 3
The old test "details only with a read token" asserted that
detailswas absent for no token and a wrong token. The split tests now use a non-nulldetailsand checkrecent. That is fine, butrecentis no longer compared withbyRepo,byModelandbyVerdictfor a read token, and the new public-details test never checks that these sums are the same as before. The case with a wrong token is still covered. The test for private data is only a null check onrecent. Nothing asserts that no otherdetailsfield leaks per-PR links. This is a minor reduction in strictness, not a regression in behavior.Assessment of the werkrevisor hints (as of
eb67d95):Fixed:
UsageApiIntegrationTest.kt: the test without a token now asserts that no public value contains a link to a single pull request (/pulls/); Scenario#25.01 states this.Not changed:
server/build.gradle.kts: only the version bump to 0.18.4.🤖 Generated with Claude Code
🟠 werkrevisor ORANGE for
eb67d95, superseded by the result for02276e2🟠 werkrevisor: ORANGE / FINDINGS
Change kind:
mixed.Reasons
Findings
[medium] Public repository names and models at
server/src/main/kotlin/de/hoennig/werkrevisor/server/Statistics.kt:129, ORANGE, as a line comment[medium] Public test checks only one substring at
server/src/test/kotlin/de/hoennig/werkrevisor/server/UsageApiIntegrationTest.kt:346-349, hint, as a line commentClassifier
7 of 7 units passed all questions (1 PR document(s)).
Usage: model `openrouter/anthropic/claude-sonnet-5.5`, LLM done, tokens counted before the call 40719, calls 1, tokens in 40367 / out 589 (reasoning included in out), duration 11.1 s, cost 0.0866 USD (reported), classifier `jev-1.13.0` calls 7, tokens in 5244, cost 0.0002 USD, finding judge `jev-1.13.0` calls 2, tokens in 3282, cost 0.0001 USD, config: defaults + host + repository (.werkrevisor.toml at based7e9afa).Reviewed head `eb67d9586cf31c1af60bb81e7ead33022e43bbfc` against base `d7e9afa73791a535dab3561519983ccc4440c7c6`, werkrevisor 0.18.3.
werkrevisor: ORANGE / FINDINGS for
eb67d9586cf31c1af60bb81e7ead33022e43bbfc.The result comment of werkrevisor holds the details.
@@ -132,2 +128,2 @@byVerdict = groups(lastMonth) { it.verdict ?: "unknown" },recent =Details(byRepo = groups(lastMonth) { it.repo },[medium] Public repository names and models (werkrevisor)
ORANGE; Jev: real 0.64, severity 3.0 of 3
byRepokeys areowner/nameof every reviewed repository and are now returned to anonymous callers. The old comment said repository names are content of possibly private repositories. The PR document states "the repositories are reviewed in public anyway", but this is not enforced in the code. If a private repository is reviewed, its name and its cost leak to anyone who can reach/api/stats. This is an intended design change, but the assumption should be verified or made configurable.@@ -345,3 +346,1 @@stats()["details"].isNull shouldBe truestats("wrong")["details"].isNull shouldBe trueval details = stats("reader")["details"]val public = stats()[medium] Public test checks only one substring (werkrevisor)
hint; Jev: real 0.71, severity 2.1 of 3
The test
sums per repository, model and verdict without a tokenguards privacy only withshouldNotContain "/pulls/". It does not assert thatdetails.recentis absent in the public response, and it depends on the URL shape. A leak ofprnumbers or of aprUrlwithout/pulls/would pass. The old test asserteddetailswas null, so this check is weaker for the privacy contract. The sibling test coversrecentbeing null, which limits the impact.🟠 werkrevisor ORANGE for
02276e2, superseded by the result for38253ea🟠 werkrevisor: ORANGE / FINDINGS
Change kind:
mixed.Reasons
Findings
[medium] Null newest flag ignored in publicRepos at
server/src/main/kotlin/de/hoennig/werkrevisor/server/Statistics.kt:167-173, ORANGE, as a line comment[high] Weakened test: byRepo and recent assertions dropped at
server/src/test/kotlin/de/hoennig/werkrevisor/server/UsageApiIntegrationTest.kt:345-359, hint, still open in an earlier line comment[low] No test for non-boolean Gitea flag at
werkrevisor/cli.py:472-473, hint, as a line commentClassifier
8 of 8 units passed all questions (1 PR document(s)).
Usage: model `openrouter/anthropic/claude-sonnet-5.5`, LLM done, tokens counted before the call 69992, calls 1, tokens in 69136 / out 926 (reasoning included in out), duration 14.2 s, cost 0.1475 USD (reported), classifier `jev-1.13.0` calls 8, tokens in 6325, cost 0.0003 USD, finding judge `jev-1.13.0` calls 2, tokens in 3906, cost 0.0002 USD, config: defaults + host + repository (.werkrevisor.toml at based7e9afa).Reviewed head `02276e2a1d052ea85d0d14fac27328ddf89ff2e3` against base `d7e9afa73791a535dab3561519983ccc4440c7c6`, werkrevisor 0.18.3.
werkrevisor: ORANGE / FINDINGS for
02276e2a1d052ea85d0d14fac27328ddf89ff2e3.The result comment of werkrevisor holds the details.
@@ -163,6 +164,14 @@ class StatisticsService(return dates.map(label).distinct().map { Period(it, Sums.of(byLabel[it].orEmpty(), billedByLabel[it].orEmpty())) }}// The newest report decides, so a repository that became private hides its name again; unknown counts as private.[medium] Null newest flag ignored in publicRepos (werkrevisor)
ORANGE; Jev: real 0.87, severity 2.8 of 3
publicReposdrops records with a nullrepoPrivatebefore picking the newest one. If a repository was reported public by an older run and its newest run has a null flag (for example, a run from an older client version), the old flag still decides and the name is shown. The comment says "unknown counts as private" and the doc says the newest review decides. A repository that became private and is then reported by an old client would leak its name. The impact is narrow, since it needs a mixed-version client setup.@@ -469,6 +469,8 @@ def execute(config, pr, deps, redactor, publish=True, use_llm=True, config_sourcbot_login = ""try:pr_data = _assert_current(gitea, pr, allow_closed and not publish)private = ((pr_data.get("base") or {}).get("repo") or {}).get("private")[low] No test for non-boolean Gitea flag (werkrevisor)
hint
The code maps a missing or non-boolean
privatetoNone, which the server treats as private. No test covers this fallback. Only theTrueandFalsecases are tested.🟡 werkrevisor YELLOW for
38253ea, superseded by the result for83d9cb6🟡 werkrevisor: YELLOW / HUMAN_REVIEW
Change kind:
mixed.Reasons
Findings
[high] Private repository names leak after 30-day window at
server/src/main/kotlin/de/hoennig/werkrevisor/server/Statistics.kt:130, YELLOW, still open in an earlier line comment[medium] Test asserts behavior for
private=Nonethrough a fake that never sets it attests/test_stats_client.py:52-57, hint, as a line comment[low] Document and code disagree on the test link name at
docs/prs/2026-10-06-PR#25-public-statistics.md:82, hint, as a line commentClassifier
8 of 8 units passed all questions (1 PR document(s)).
Usage: model `openrouter/anthropic/claude-sonnet-5.5`, LLM done, tokens counted before the call 70558, calls 1, tokens in 69673 / out 813 (reasoning included in out), duration 13.9 s, cost 0.1475 USD (reported), classifier `jev-1.13.0` calls 8, tokens in 6347, cost 0.0003 USD, finding judge `jev-1.13.0` calls 2, tokens in 2868, cost 0.0001 USD, config: defaults + host + repository (.werkrevisor.toml at based7e9afa).Reviewed head `38253eaba44469cd24020f1034512cf964b73b03` against base `d7e9afa73791a535dab3561519983ccc4440c7c6`, werkrevisor 0.18.3.
werkrevisor: YELLOW / HUMAN_REVIEW for
38253eaba44469cd24020f1034512cf964b73b03.The result comment of werkrevisor holds the details.
@@ -0,0 +79,4 @@##### Verified by- [UsageApiIntegrationTest: private repositories are named only with a read token](../../server/src/test/kotlin/de/hoennig/werkrevisor/server/UsageApiIntegrationTest.kt)- [test_stats_client: a private repository is reported as such](../../tests/test_stats_client.py)[low] Document and code disagree on the test link name (werkrevisor)
hint
Scenario#25.03 links
test_stats_client: a private repository is reported as such. The test is namedtest_a_private_repository_is_reported_as_such, which is close but not exact. The odd-flag testtest_a_missing_or_odd_private_flag_is_reported_as_unknownis not referenced. The0and"false"cases are not mentioned in the PR document either.@@ -43,0 +49,4 @@review(stats, private=True)self.assertIs(stats.records[0]["repo_private"], True)def test_a_missing_or_odd_private_flag_is_reported_as_unknown(self):[medium] Test asserts behavior for
private=Nonethrough a fake that never sets it (werkrevisor)hint; Jev: real 0.78, severity 2.1 of 3
review(stats, private=flag)setsgitea.repo_privatetoNone,"false"or0, soget_prreturns{"private": None}and the key is always present. The case "repo key missing entirely" (for examplebase.repoabsent), which the test name promises, is not tested. A missing key goes through theor {}fallback path incli.execute, which has no test.Assessment of the werkrevisor findings since my last comment (as of
83d9cb6):Fixed:
Statistics.kt(ORANGE, public repository names): as decided by the developer, a private repository is no longer named without a read token. The usage record reports Gitea'sprivateflag, and the newest record of a repository decides. A missing flag counts as private, also after an earlier public record. Scenario#25.03 covers this.UsageApiIntegrationTest.kt, weak privacy test: the test without a token asserts thatrecentis null and that nopr,prUrl,pr_urlor/pulls/value is present.tests/test_stats_client.py: a missing repository, a missing flag and a flag that is not a boolean are tested.Not changed:
Statistics.kt:130, "private repository names leak after 30-day window" (YELLOW, Jev doubts it): not a defect. A repository appears in the 30-day table only with a record from those 30 days, so its newest record is always inside the 12 months thatpublicReposreads.server/build.gradle.kts: only the version bump to 0.18.4.werkrevisor/cli.pygiven only as diff: the change is two lines that read the flag after_assert_current.🤖 Generated with Claude Code
🟡 werkrevisor: YELLOW / HUMAN_REVIEW
Change kind:
mixed.Reasons
Findings
[medium] Public byRepo/byModel reveal private repos via PR counts at
server/src/main/kotlin/de/hoennig/werkrevisor/server/Statistics.kt:167-172, YELLOW, still open in an earlier line comment[medium] Boolean read from SQLite via getBoolean on integer column at
server/src/main/kotlin/de/hoennig/werkrevisor/server/UsageStore.kt:47-77, hint, as a line comment[low] Document claims private repo names stay hidden, but PR counts per group remain public at
docs/prs/2026-10-06-PR#25-public-statistics.md:63-77, hint, as a line commentClassifier
8 of 8 units passed all questions (1 PR document(s)).
Usage: model `openrouter/anthropic/claude-sonnet-5.5`, LLM done, tokens counted before the call 70822, calls 1, tokens in 69946 / out 798 (reasoning included in out), duration 13.4 s, cost 0.1479 USD (reported), classifier `jev-1.13.0` calls 8, tokens in 6347, cost 0.0003 USD, finding judge `jev-1.13.0` calls 2, tokens in 3177, cost 0.0001 USD, config: defaults + host + repository (.werkrevisor.toml at based7e9afa).Reviewed head `83d9cb69f1e5db7542bb371eb016358b28ffb8b1` against base `d7e9afa73791a535dab3561519983ccc4440c7c6`, werkrevisor 0.18.3.
werkrevisor: YELLOW / HUMAN_REVIEW for
83d9cb69f1e5db7542bb371eb016358b28ffb8b1.The result comment of werkrevisor holds the details.
@@ -0,0 +60,4 @@- [UsageApiIntegrationTest: recent reviews only with a read token](../../server/src/test/kotlin/de/hoennig/werkrevisor/server/UsageApiIntegrationTest.kt)#### Scenario#25.03: a private repository is named only with a read token[low] Document claims private repo names stay hidden, but PR counts per group remain public (werkrevisor)
hint
The problem statement says the name of a private repository is content of that repository, yet the public line "private repositories" still shows the aggregated costs, reviews and pull request counts. The document does not state this as accepted. Also, Scenario#25.01 lists "by repository" among public tables without noting the aggregation.
@@ -44,3 +45,3 @@)private val TOKEN_COLUMNS = setOf("llm_input_tokens", "llm_output_tokens", "jev_input_tokens")private val ADDED_COLUMNS = setOf("llm_input_tokens", "llm_output_tokens", "jev_input_tokens", "repo_private")[medium] Boolean read from SQLite via getBoolean on integer column (werkrevisor)
hint; Jev: real 0.47, severity 2.0 of 3
repo_privateis stored with a Boolean parameter in anintegercolumn, and read withrs.getBoolean. Any non-zero value reads as true, which is fine. However, there is no test for the schema migration addingrepo_privateto a database of an earlier version; the existing migration test only checks the token columns.