mi/werkrevisor
1
0
Fork 0
Code Issues Pull Requests 2 Actions Packages Projects Releases Wiki Activity
Files
main
werkrevisor/docs/werkrevisor.werkbaum
T
claude-code 4ed2f1d197
werkator build succeeded after 00:13
Details
werkator/werkrevisor-statistics build succeeded after 00:30
Details
werkator/release build succeeded after 00:14
Details
werkrevisor 0.62.0: draft-review --open shows the report in the browser, and the report names and links its PR (#134) 
Co-authored-by: claude-code <claude-code@javagil.de>
2026-10-23 12:28:32 +02:00

448 lines
27 KiB
Plaintext
Raw Permalink Blame History

%% werkrevisor, planned with Werkbaum (https://werkbaum.javagil.de): what exists, what was dropped, what comes next.
%% Status: [^] live on mih09 or in use, [x] merged but not in use, [~] in work, [ ] planned, [?] idea, [-] dropped.
%% Each PR updates this file; see docs/agentic-workflow.md. The block behind --- explains the nodes with an id.
[~] #wr: werkrevisor https://git.javagil.de/mi/werkrevisor
- [^] #gate: Review gate in Werkator
- [^] Verdicts GREEN to ERROR (S)
" Exit codes and commit status; colors by certainty since PR#16.
- [^] Reviews its own PRs (S)
" PR#3
- [^] Review starts when "WIP: " is removed (S)
- [^] Approval and automerge (M)
" PR#18
- [^] Open hints hold the merge (S)
" PR#32; writing rules too since PR#44.
- [^] The note of a held merge names hints outside the diff (XS)
" PR#95: werkbaum PR#9 waited for a hint on a file that it did not change.
- [^] A hint on a file outside the diff holds no merge (XS)
" PR#97
- [^] Findings as line comments (M)
" PR#20, PR#33, PR#36
- [^] Repository config .werkrevisor.toml (M)
" PR#5; read at the head of the default branch since PR#39.
- [^] Admin override stays (XS)
" PR#19
- [^] #gate.carry: An approval survives a rebase (M)
" PR#73: unchanged blocks keep the approval; Jev judges new blocks first; PR#74: 128-bit fingerprints.
- [ ] #gate.carry.calibrate: Calibrate the thresholds for corrections (S) :#gate.carry
- [^] #gate.after: An approval counts only after the result comment (S)
" PR#73 merged at an approved head whose review then gave ORANGE; PR#75 asks for the approval again.
- [^] #rules: Deterministic rules
- [^] Change kinds (S)
" PR#7, PR#8
- [^] Size limits (XS)
" PR#4
- [^] Hard paths and secret scan (S)
- [^] Open questions in the PR document give RED (S)
" Since 0.6.0; "None." is no question since PR#42.
- [^] Writing rules (S)
" PR#6; line comments since PR#44.
- [^] Generated sections have no writing rules (XS)
" PR#122: the Verified by sections of hsadmin-ng; Codex and the model skip them too.
- [^] Mechanical changes skip Jev and the model (S)
" PR#11
- [x] A package move is mechanical (S)
" PR#112: a renamed file with only package and import changes; werkbaum PR#11 got YELLOW for 46 moved files.
- [^] Raster images are checked by size and signature (S)
" PR#98: werkregie PR#55 got YELLOW only for a screenshot; SVG keeps the normal review.
- [~] #triage: Triage by Jev
- [^] PR document: sections and scenarios (M)
" Answers of low confidence are ignored since PR#10.
- [^] DB changesets (S)
" PR#17
- [^] Build files (S)
" PR#27
- [^] Every changed file: security, weakened checks (M)
" PR#39; found 7 to 8 of 9 critical hs.hsadmin.ng PRs.
- [-] #triage.harm: Harm the tests would not reveal (S)
- [^] #triage.risk: Risk categories from a template
- [^] #triage.risk.jvm: JVM and Spring backend (S)
" PR#60: concurrency asks for missing synchronization; PR#93: error handling for unnoticed failures.
" Chosen by werkator, werkprobe and werkcrappit in their .werkrevisor.toml.
- [x] #triage.risk.ts: TypeScript frontend (S)
" PR#68: measured on 30 codecharta PRs; no repository uses it yet.
- [-] #triage.risk.replace: Categories instead of the security question (S)
- [^] #triage.reversible: Can a mistake corrupt persisted data? (M)
" PR#54; YELLOW from 0.85 on, in the backend template only.
- [^] Categories in each .werkrevisor.toml
" PR#63: choose the template, tune, switch off or add categories.
- [^] werkator (S)
" werkator PR#128: all 16 categories.
- [^] #triage.risk.werkcrappit: werkcrappit (S)
" werkcrappit PR#46: 11 categories off, own category metric_integrity.
- [-] #triage.risk.plugincompat: plugin_compatibility for werkcrappit (S)
- [^] werkprobe (S)
" werkprobe PR#12: 8 categories off.
- [?] Python template (S)
- [?] #triage.context: PR context for each file judgment (M)
- [~] #doc: PR document vs each changed file
- [-] Summary of the diff without the PR document (S)
" PR#47; dropped in PR#117: one Jev call per file instead, no economy model.
- [x] #doc.file: Jev checks each file diff against the PR document (M)
" PR#117: four contradiction questions, the unrelated and the data-outflow question in one call.
- [x] #doc.exfil: Data-outflow question (S)
" PR#117: 6 of 6 backdoors from 0.7 on, 0 false alarms on 109 real files; its own exfil_min.
- [^] Contradictions per file (M)
" PR#47, PR#117; only a conflicting test claim gives ORANGE.
- [^] Resolved hints are not repeated (S)
" PR#50; PR#117 fingerprints the file path, the diff and the PR document.
- [-] #doc.omission: Omissions per document (M)
- [x] #doc.change: One question per change, moved to the file diff (M)
" PR#116 per summarized change; PR#117 per file; needs the conventions of the repository.
- [x] #doc.exfilmin: exfil_min and unrelated_min stay 0.7 (S)
" PR#117: 103 real hs.hsadmin.ng file diffs stay below 0.3, a planted change reached 0.7-0.98.
- [ ] Calibrate hint_min on real PRs; decide whether data-outflow becomes ORANGE (S)
- [^] Calibrate hint_min on real PRs (S)
" PR#66: 0.6 to 0.7; the three hints below 0.7 were noise.
- [-] Keep a summary whose entries lack a statement (XS)
" PR#58, PR#61; moot since PR#117 removed the summary.
- [~] #verdict: Overall verdict
- [^] #verdict.file: RED for risk and finding in one file (M)
" PR#51
- [^] #verdict.cat: RED for risk and finding in one category (M) :#verdict.file
" PR#53; active once a host selects a risk template.
- [^] #verdict.rounds: Limited correction rounds after RED (M)
" PR#62: the fourth RED in a row asks a human.
- [ ] #verdict.pr: PR-wide classification by Jev (L)
- [ ] #verdict.dsl: Review spec as DSL (L)
- [~] #model: Model review
- [^] PR-Agent 0.47 via OpenRouter (M)
- [^] Exact token count and budget (M)
" PR#9, PR#22
- [^] Full files and excerpts (M)
" PR#29, PR#31
- [^] Jev judges findings (M)
" PR#14
- [^] The model only advises (S)
" PR#39; MiniMax M3 as economy model since PR#40.
- [^] Model-dependent limits documented (S)
" PR#49
- [^] #model.output: MiniMax M3 gets 64000 output tokens (XS)
" PR#84; with 32000 it thought until the limit and answered nothing on werkregie PR#26.
- [?] #model.empty: A retry after an empty answer bounds the reasoning (S)
" PR-Agent repeats the same request; whether MiniMax keeps reasoning_max_tokens is not measured.
- [-] #model.bench: Finding benchmarks to choose the model (L)
- [?] Model profiles with defaults (M)
- [ ] #model.reload: Load more context for unclear risks (L)
- [?] #model.external: Findings of an external review system
- [~] #model.codex: Optional review by the Codex CLI with a frontier model in a subscription
- [^] #model.codex.eval: Measure it on 10 hs.hsadmin.ng PRs (S)
" PR#87: tools/codex-eval, gpt-6-astra.
- [^] #model.codex.gate: A runner on the workstation, off, advise or gate per repository (L) :#model.codex.eval
" PR#88: tools/codex-runner; in gate mode the status ai-review/codex.
- [^] #model.codex.on: Switch on advise in the first repositories :#model.codex.gate
- [^] werkrevisor (XS)
" PR#89
- [^] werkator (XS)
" werkator PR#153
- [^] #model.codex.gated: Switch on gate in the first repository :#model.codex.on
- [^] werkrevisor (XS)
" PR#104; it is the gate of all werk repositories and reviews its own PRs.
- [^] First PR through the gate, with a defect on purpose (XS)
" PR#105
- [^] A severe defect on purpose (XS)
" PR#106; werkrevisor did not see it, Codex did.
- [x] #model.codex.weakened: Jev holds a weakened merge gate for the human :#model.codex.gated
- [x] werkrevisor (S)
" PR#107: category gate_weakened from 0.75.
- [x] werkator (S)
" werkator PR#159: category status_misreported from 0.6 and a hard path; gate_weakened missed there.
- [x] #model.codex.report: tools/review-report for a repository without werkrevisor (S)
" PR#90: a draft to check and post by hand, first for hs.hsadmin.ng.
- [x] #model.codex.timer: The runner starts 1 minute after its last run (XS)
" PR#120: 1 review per run; parallel reviews dropped, they save about 1 minute in rare cases.
- [^] #model.codex.order: The longest-waiting ready PR first (S)
" PR#119: by the time of the werkrevisor result, across all repositories.
- [x] #model.codex.sandbox: The runner checks its sandbox before it reviews (S)
" PR#123: an AppArmor fault of the workstation used up the attempts of PR#115 and PR#116.
- [^] #model.codex.fold: A new werkrevisor result no longer folds the Codex comments (XS)
" PR#133: they were labelled "werkrevisor ERROR for ?" on PR#132.
- [~] #model.rounds: Fewer rounds of new small hints
- [-] #model.rounds.human: Rated by the user (S)
- [^] Dry run writes the model prompt (S)
" PR#65: tools/dry-run --prompt-out.
- [^] Dry run and jev-eval also with Podman (XS)
" PR#114: the workstation has Podman, no Docker.
- [^] tools/build-rootfs also with Podman (XS)
" PR#115
- [x] #model.rounds.frontier: Frontier model rates past hints (M)
" 42 hints of PRs #32 and #48 to #64: 14 useful, 28 noise; 4.08 USD.
- [^] #model.rounds.doc: Hints on documentation only in the result comment (S)
" PR#66; [review] doc_hint_line_comments restores the line comments.
- [^] Hints on PR documents are line comments again (XS)
" PR#72: two of six such hints were real errors that reached main.
- [x] #model.rounds.check: Count the rounds again after 10 to 15 PRs (XS)
" 89 PRs until 2026-10-12: 47 one round, 19 three or more; 64 of 111 line comments on documents.
- [?] #model.rounds.jev: Jev question "worth a round?" (M)
- [x] #model.rounds.changed: From round 2 on, hints on documents only on changed lines (S) :#model.rounds.check
" PR#91
- [^] Show the text of a security concern of the model (XS)
" PR#63 got YELLOW for one without its text; PR#67 quotes it in the reason.
- [^] A plus in the text is no added line (XS)
" PR#80: werkregie PR#15 lost three rounds to unchanged optional Werkbaum nodes read as added.
- [^] An unquoted "No" is no security concern (XS)
" PR-Agent loads it as the bool False; werkcrappit PR#52 got YELLOW for "False"; PR#71.
- [^] No classifier section without units (XS)
" PR#64; the comment showed "0 of 0 units passed all questions".
- [~] #diagrams: Diagrams of PR Lens in the result comment
- [x] #diagrams.comment: Light SVGs attached to the result comment, behind [diagrams] enabled (M)
" PR#121: Node.js and the pinned PR Lens in the base archive; the diagrams never change the verdict.
- [^] #diagrams.on: Switch on for werkrevisor, Sonnet, no diagrams for GREEN (S)
" PR#125: MiniMax M3 wrote no valid graph; GPT-6 Luna only rough boxes on 4 werkrevisor PRs.
- [^] #diagrams.hsreview: hs-review puts the SVGs next to its draft (S)
" PR#129: dev.hostsharing.net gets no upload; the human attaches them when posting.
- [^] #diagrams.attach: hs-review --attach posts the checked draft with its diagrams (S)
" PR#130: under the account of the human; refused for an unchecked, outdated or posted draft.
- [^] #diagrams.fold: Diagrams no longer remove the note "replaced by a newer review" (XS)
" PR#127: Codex line comment 6411 on PR#121.
- [^] #diagrams.draft: tools/draft-review from a local config, with an HTML report (S)
" PR#132: replaces ~/bin/hs-review; a click on a diagram opens its SVG.
- [^] #diagrams.open: draft-review --open; the title links the PR (XS)
" PR#134
- [ ] #diagrams.cost: Diagram cost in the usage record and the budget (S)
- [-] #diagrams.same: The review model draws the graph in the same call (M)
" PR-Agent answers in a fixed YAML format; see docs/konzept-pr-lens-diagramme.md.
- [-] #diagrams.port: PR Lens ported to Python, without Node.js (L)
" Each new feature of PR Lens would need a port of its own.
- [~] #stats: Statistics service and dashboard https://werkrevisor.javagil.de
- [x] Package de.javagil.werkrevisor.statistics (XS)
" PR#113, after werkregie PR#86.
- [^] Usage records and dashboard (M)
" PR#21 to PR#26, PR#34
- [^] Cost as lower bound, reviews per PR (S)
" PR#43, PR#45
- [^] #stats.phases: Seconds per phase (S)
" PR#51
- [x] #stats.slow: Find the cause of reviews over 10 minutes (S) :#stats.phases
- [^] #stats.timeout: Keep MiniMax within its timeout (S) :#stats.slow
" PR#56: providers in a configured order.
- [x] #stats.prs: Tables per period count PRs and cost per PR (S)
" PR#57
- [x] #stats.chart: Trend chart per period (M) :#stats.prs
" PR#57; from the oldest period with reviews or costs on.
- [^] #stats.temurin: Temurin 25 and a runtime jlinked from it (S)
" PR#82, werkregie ADR 0001; Kotlin 2.4.21, Gradle 9.8.1, Foojay 1.0.0.
- [~] #ops: Operation
- [^] werkprobe, werkrevisor, werkator, werkcrappit on mih09 (M)
- [~] #ops.werkregie: werkregie on mih09 (S)
" PR#79: config/werkregie.toml; werkregie PR#9 adds its CI.
- [~] #ops.werkbaum: werkbaum on mih09 (S)
" PR#83: config/werkbaum.toml, with automerge.
- [~] #ops.werkkompass: werkkompass on mih09 (S)
" PR#110: config/werkkompass.toml, with automerge; no CI build yet.
- [~] #ops.werkdock: werkdock on mih09 (S)
" PR#126: config/werkdock.toml, with automerge; werkdock PR#15 brings its .werkrevisor.toml.
- [^] Review flow diagram in the README (S)
" Updated with PR#55.
- [^] Diagrams for talks and posts (S)
" PR#58: workflow and inside; PR#59: decisions; light and dark each.
- [^] Werkbaum plan in the repository (XS)
- [^] werkator gets its own .werkrevisor.toml (S)
" werkator PR#128.
- [^] Release tools keep the two newest archives (S)
" PR#69: 51 archives with 9 GB had collected; tools/deploy-rootfs replaces the manual deploy.
- [^] Deploy with two connections and retries (XS)
" PR#70: the host refused the cleanup after eight quick connections.
- [^] #ops.release: Werkator releases each merge to mih09 (M)
" PR#118: build release assembles the archive without a container engine, from a base archive of PR-Agent.
- [^] #ops.releaseconfig: The release installs the host configs with the archive (S)
" PR#124: 0.49.0 rejected the stale host keys of [doc_check]; now code and config switch in one write.
" It stops before any write on a host setting that the file would lose.
- [^] #ops.workflow: One agentic workflow for all werk repositories (S)
" PR#76: docs/agentic-workflow.md, in every werk repository; werkregie holds the master copy since PR#81.
" PR#109 and PR#111: the copy matches the master again.
- [^] #ops.secrets: Secrets of the coding agents in ~/.config/ai-coding-agents/ (S)
" PR#86: the old path ~/.config/pr-agent-test/ is gone.
- [^] #ops.rotate: Replace the token of ai-reviewer that reached a transcript (XS)
" 2026-10-11: new token in the five machine configs on mih09; the old one is revoked.
- [~] #ops.rotate2: Rotate the four secrets of the review builds without a stop
" 2026-10-20: Werkator passed them to werkdock on the command line, and they reached a transcript.
- [x] tools/rotate-secret: set, check, probe and purge one secret (S)
" PR#128: probes without costs; the statistics service takes the old and the new source token.
- [ ] Rotate WERKREVISOR_STATS_TOKEN (XS)
- [ ] Rotate OPENROUTER_API_KEY (XS)
- [ ] Rotate TYPESAFE_API_KEY (XS)
- [ ] Rotate GITEA_BOT_TOKEN (XS)
- [?] #ops.rotate2.auto: Rotation without a human (M)
" The statistics token and OpenRouter by API; Gitea needs the password of ai-reviewer;
" TypeSafe manages keys only in its console.
- [^] #ops.basecommit: Bug: "base commit unknown" on old PRs (S)
" PR#85: a deleted base branch leaves base.sha empty; the merge base stands in.
- [^] #ops.doc: The operations document states the current setup (XS)
" PR#92, PR#96; PR#103: current values, rules instead of a history per version, deploy and sync tools.
- [^] #ops.conflict: A PR in conflict asks for a rebase, not for a human review (S)
" PR#94: werkregie PR#41 asked for three approvals that could not merge.
- [^] #ops.readme: The README and the diagrams describe the current review (S)
" PR#99: steps as in the inside diagram, the GREEN cases without a model, the secret before the shortcut.
- [^] #ops.docglob: The docs category of werkregie is doc/ (XS)
" PR#100: its plan counted as code.
- [^] #ops.sync: tools/sync-host-config compares the host configs on mih09 (S)
" PR#101: prints line numbers and key names only; werkprobe drifted from config/werkprobe.toml.
- [^] #ops.werkprobe: config/werkprobe.toml holds the host config of werkprobe (XS)
" PR#102: the tests read tests/fixtures/test-config.toml instead.
- [^] #ops.slowtls: Gitea requests wait 90 s; hs-review names the cause of a failure (XS)
" PR#131: dev.hostsharing.net took up to 22 s for a TLS handshake.
- [ ] #ops.hs: hs.hsadmin.ng on vm4006 (L)
- [?] Re-check the review after a resolved thread (S)
- [?] Label "info" for hints (XS)
- [ ] #kotlin: Rewrite in Kotlin
- [ ] #kotlin.runtime: Choose the runtime
= [?] Kotlin/Native (M)
= [?] Kotlin on the JVM as GraalVM native image (M)
- [ ] Port module by module, the tests as specification (XL)
- [ ] #kotlin.review: Model review without PR-Agent (L)
- [ ] Token count with jtokkit (S)
- [?] #future: Open source, independent of the CI (XXL)
---
#gate.carry
User request of 2026-10-10: after a rebase or a merge of main, an approval of the same change stays valid.
Fingerprints per changed block in the result comment; only an approval that Gitea dismissed automatically counts.
#gate.carry.calibrate
Measure on PRs whose head changed after an approval, e.g. werkator PR#137, and on planted conflict resolutions.
#triage.harm
PR#39 asked whether a mistake in the file would do harm that the tests would not reveal.
On the 25 merged hs.hsadmin.ng PRs #281 to #316 it did not separate the critical PRs from the others.
#triage.risk
PR#46: 15 categories per ecosystem, on top of the questions about security and weakened checks; PR#54 adds a 16th.
Since 2026-10-09 werkator, werkprobe and werkcrappit choose jvm-spring-backend in their .werkrevisor.toml.
On the 25 hs.hsadmin.ng PRs they found no further critical PR, but they name the risk, e.g. schema migration.
#triage.risk.replace
With the categories instead of the broad security question, 2 of the 9 critical PRs were lost (#281, #288).
#triage.reversible
User request of 2026-10-08: a category that rates whether a change is reversible or can manifest in the data.
PR#54 asks whether a mistake can write wrong data that would need corrections; from 0.85 on it gives YELLOW.
On the 25 hs.hsadmin.ng PRs that adds #302 and #307, the imports of bank data; see the PR document.
Read access is reversible, unless it hands out data of others.
Code that writes wrong data after a bug is riskier: an audit log helps, but many data corrections are tedious.
Research on the number of questions (TypeSafe docs, 2026-10-08):
each question of a request is evaluated independently, and 13 questions in one call gave the same answers as 13 calls.
Jev loses accuracy through a large state with unrelated content, double negatives and questions that need several steps.
More categories still mean more chances that one exceeds its threshold:
on the 25 hs.hsadmin.ng PRs, the categories gave 3 false YELLOW from 0.93 on and 1 from 0.95 on.
#triage.context
Idea of 2026-10-07: each file judgment also gets the list of changed files and the PR document, condensed once.
The PR document is written by the author, so it must not lower the answers about security and weakened checks.
#doc.omission
PR#47: eight questions whether the PR document omits a change, e.g. of the API or the database.
Unchanged documents reached up to 0.89, and removing the solution section raised the values little.
#verdict.cat
PR#53: Jev also assigns each confirmed finding to a category of the template.
Measured on 32 findings of hs.hsadmin.ng: from a confidence of 0.7 on, all 21 assignments outside test code held;
findings in test code do not count, since 3 of 4 wrong assignments were about tests.
#verdict.rounds
The design of 2026-10-08 proposes max_red_mitigation_rounds = 3; after that, the PR escalates to a human.
#verdict.pr
Jev classifies the whole PR from a consolidated report: overall risk, human review required,
mitigation required, interaction of several changes, and whether the evidence suffices.
#verdict.dsl
One place for the review specification, in the style of RbacSpec in hsadmin-ng:
which workflow runs under which conditions, which questions Jev answers, which values give which verdict.
#model.bench
Experiments of 2026-10-06 and 2026-10-07 with Sonnet 5.5, Grok 4.7, MiniMax M3, Mistral Large 4 and GPT-6 Astra.
Each model found 0 to 2 of 10 known findings, each a different one; the list of known findings has no reliable source.
Shelved on 2026-10-08: expensive, little gain; the focus moved to the triage.
#model.reload
For a relevant but unclear risk signal, load dependent methods, access checks, API contracts or tests and judge again.
#model.external
Real reviews that look for concrete issues may come from an external system; its findings feed the overall verdict.
#model.codex
User idea of 2026-10-11: per repository in .werkrevisor.toml; measure it first on the 25 hs.hsadmin.ng PRs.
Needs consent for a new runtime dependency; check whether the subscription allows automated use on mih09.
#stats.slow
On 2026-10-08 a review took 749 s with one model call, a timeout of 300 s and 4.5 s of CPU; its logs were pruned.
The seconds per phase of PR#51 showed the cause in the review of PR#53: model_review took 722.7 s,
the limit of 2 attempts of 300 s plus 120 s, while the other phases took 18 s together.
#stats.timeout
Measured on 2026-10-08 with 4 PRs: the calls that failed had produced only about 1100 output tokens after 700 s,
so some provider hangs; it is not long reasoning. "low" and "medium" both failed on 1 of 8 heads in two measurements;
without a retry a single hang fails the review. PR#56 asks the maker of the model first, with fallbacks to others.
#stats.prs
User request of 2026-10-08: the tables per day, week and month count pull requests instead of reviews,
and show the cost per pull request in each period.
#stats.chart
User request of 2026-10-08: between each heading, e.g. "per day", and its table, a trend chart:
the date on the x axis; the cost per period, the cost per pull request and the number of pull requests on the y axes.
#ops.hs
Waits for the agreement with the team about accounts and the access to vm4006.
#future
Open source, independent of the CI, possibly with a backend and a read-only agent for the whole repository.
#kotlin
User wish of 2026-10-08: werkrevisor in Kotlin instead of Python; the statistics service is Kotlin already.
Only PR-Agent and tiktoken tie werkrevisor to Python; about 6700 lines of code and 590 tests to port.
#kotlin.runtime
Kotlin/Native gives a small binary, but fewer libraries for HTTP and JSON.
A GraalVM native image keeps the JVM libraries of the statistics service, e.g. jtokkit for o200k_base.
#kotlin.review
Since Jev decides and the model only advises, the model review can call OpenRouter directly,
as the excerpt selection does, or stay an external step.
#model.rounds
MiniMax M3 finds a new small hint in the documentation in each round; PR#54 needed 5 rounds.
werkprobe PR#12 needed 5 rounds too, all for hints on the documentation and on comments.
Each hint costs a round of the coding agent, but no time of a human.
#triage.risk.jvm
PR#60: the old question rated every change of concurrent code; on werkator it asked a human for 7 of 30 PRs,
while the security question alone asks for 2.
The new question asks for missing synchronization or idempotency: at most 0.70 on 55 merged PRs,
0.90 to 0.96 on three constructed defects.
PR#93: the old error handling question rated every change of error handling,
e.g. a correct retry in werkator PR#155 at 0.98.
The new question asks for a failure that goes unnoticed: at most 0.69 on 57 merged PRs, 0.87 to 0.98 on four constructed defects.
#triage.risk.werkcrappit
Jev judged the 159 main files of werkcrappit PRs #16 to #45: no template category reached its threshold.
The 11 categories for logins, SQL, transactions, migrations, stored records and REST stayed at 0.32 or below.
metric_integrity asks for the hard invariants of werkcrappit: planted changes 0.80 to 0.91, real files at most 0.66;
YELLOW from 0.75 on.
#triage.risk.plugincompat
It asked whether a change breaks builds or report consumers; a removed schema field gave 0.68,
two compatible schema extensions 0.88 and 0.89.
#model.rounds.human
The user rated a sample of 30 hints: without the context of each PR, one hint took 10 to 15 minutes.
#model.rounds.frontier
anthropic/claude-opus-5.5 rated each answered hint with the prompt that produced it,
rebuilt from a throwaway PR per reviewed head in mi/werkrevisor-bench; 65 heads were rebuilt, 26 rated.
Hints on documentation: 6 useful, 19 noise; on code: 8 useful, 9 noise.
The round, Jev's "real" and the model's severity did not separate them.
#model.rounds.jev
The frontier ratings had a confidence of mostly 0.5 to 0.6; 42 of them are too few to calibrate a Jev question.
#triage.risk.ts
PR#68: on 2077 main files of codecharta PRs #4387 to #4560 only shared_state reached its threshold, once.
Planted defects: xss 0.97, token_handling 0.95, third_party_requests 0.97 are found;
the security question rates xss 0.02 and third_party_requests 0.11, but token_handling 0.96.
unsafe_navigation from 0.85 and critical_user_actions from 0.9 catch a redirect and a dropped confirmation.
shared_state asks for an in-place mutation or a dropped cache invalidation, YELLOW from 0.6: real files at most 0.37.
Reference in New Issue View Git Blame Copy Permalink