84 lines
3.4 KiB
Markdown
84 lines
3.4 KiB
Markdown
# PR#65: A Dry Run Writes the Model Prompt to a File
|
|
|
|
> **WARNING:** This document describes only the change applied in this PR.
|
|
> It may already be outdated once the next PR is merged.
|
|
> Historic PR documentation is not maintained along with new PRs; treat it as a snapshot.
|
|
|
|
## The Problem
|
|
|
|
Many PRs need several review rounds, each for new small hints of the model, e.g. werkprobe PR#12 with 5 rounds.
|
|
To find a rule that keeps the useful hints and drops the others, a frontier model rates the hints of past PRs once.
|
|
For a fair rating, it needs the context in which the economy model wrote each hint:
|
|
the diff, the full files or excerpts, AGENTS.md, the review instructions and Jev's hints.
|
|
werkrevisor does not keep this prompt, and a dry run cannot write it without calling the model.
|
|
|
|
## Non-Goals
|
|
|
|
- No change to the reviews on the review machine.
|
|
- No dry run of an earlier head: PR-Agent reads the current state of a PR through the Gitea API.
|
|
The measurement reviews earlier heads as throwaway PRs in `mi/werkrevisor-bench`.
|
|
- The rating and the rule for the hints come in a later PR.
|
|
|
|
## The Scenarios
|
|
|
|
### Feature: a dry run writes the prompt of the model review
|
|
|
|
#### Scenario#65.01: the prompt goes to a file instead of the model
|
|
|
|
So that a later measurement can give another model the same context.
|
|
|
|
- **Given** the options `--no-publish` and `--prompt-out prompt.json`
|
|
- **When** werkrevisor reviews a PR
|
|
- **Then** `prompt.json` holds the system prompt and the user prompt of the model review as JSON
|
|
- **And** no model is called
|
|
|
|
##### Verified by
|
|
|
|
- [test_prompt_out.PromptOutTest.test_the_prompt_goes_to_the_file_instead_of_the_model](../../tests/test_prompt_out.py)
|
|
|
|
#### Scenario#65.02: the written prompt is the one after the size checks
|
|
|
|
So that the file holds the prompt that the model would get, not one that the budget refuses.
|
|
|
|
- **Given** a dry run with `--prompt-out`
|
|
- **And** the first prompt with the full files is too large
|
|
- **When** werkrevisor shortens the prompt to the excerpts or the diff
|
|
- **Then** the file holds the shortened prompt
|
|
|
|
##### Verified by
|
|
|
|
- [test_prompt_out.PromptOutTest.test_the_file_holds_the_prompt_after_the_size_checks](../../tests/test_prompt_out.py)
|
|
|
|
#### Scenario#65.03: only a dry run writes the prompt
|
|
|
|
So that the option never changes a review on the review machine.
|
|
|
|
- **Given** `--prompt-out` without `--no-publish`
|
|
- **When** werkrevisor starts
|
|
- **Then** it stops with a configuration error
|
|
|
|
##### Verified by
|
|
|
|
- [test_prompt_out.PromptOutTest.test_only_a_dry_run_writes_the_prompt](../../tests/test_prompt_out.py)
|
|
|
|
#### Scenario#65.04: no prompt without the model review
|
|
|
|
So that a run with `--prompt-out` always writes a prompt; with `--no-llm`, werkrevisor never builds one.
|
|
|
|
- **Given** `--prompt-out` together with `--no-llm`
|
|
- **When** werkrevisor starts
|
|
- **Then** it stops with a configuration error
|
|
|
|
##### Verified by
|
|
|
|
- [test_prompt_out.PromptOutTest.test_only_a_dry_run_writes_the_prompt](../../tests/test_prompt_out.py)
|
|
|
|
## The Solution
|
|
|
|
- `werkrevisor --prompt-out FILE` and `tools/dry-run … --prompt-out build/FILE`:
|
|
the prompt that passes the size and budget checks goes to the file, and no model is called.
|
|
Jev and the excerpt selection still run, since their results are part of the prompt.
|
|
- The README describes the option; the Werkbaum plans the rating by a frontier model under `#model.rounds`
|
|
and records that a rating by the user did not work.
|
|
- Version 0.30.2.
|