61 lines
2.5 KiB
Markdown
61 lines
2.5 KiB
Markdown
# PR#71: An Answer "No" Without Quotes Is No Security Concern
|
|
|
|
> **WARNING:** This document describes only the change applied in this PR.
|
|
> It may already be outdated once the next PR is merged.
|
|
> Historic PR documentation is not maintained along with new PRs; treat it as a snapshot.
|
|
|
|
## Related Links
|
|
|
|
- werkcrappit PR#52, head 76d4ee2: YELLOW with the single reason "model reports a security concern: False".
|
|
|
|
## The Problem
|
|
|
|
PR-Agent answers `security_concerns` with "No" or with a text that describes the concern.
|
|
When the model writes `security_concerns: No` without a block scalar or quotes,
|
|
the YAML loader of PR-Agent reads the value as the bool `False`.
|
|
werkrevisor made the text "False" of it and compared only with "no".
|
|
So a review without a security concern became YELLOW, with the reason "model reports a security concern: False".
|
|
|
|
## Non-Goals
|
|
|
|
- No change to the verdict of a real concern: it stays YELLOW and quoted.
|
|
- No interpretation of free text such as "No security concerns found": it stays a concern, as before.
|
|
|
|
## The Scenarios
|
|
|
|
### Feature: an answer without a security concern gives no reason
|
|
|
|
#### Scenario#71.01: every spelling of "no" gives no reason
|
|
|
|
So that the bool of the YAML loader and the case of the answer do not decide the verdict.
|
|
|
|
- **Given** the model answers `security_concerns` with the bool `False`, "False", "false", "no", "No.", "None" or nothing
|
|
- **When** werkrevisor assesses the answer
|
|
- **Then** the verdict is GREEN
|
|
- **And** no reason `model_security` is given
|
|
|
|
##### Verified by
|
|
|
|
- [test_verdict.ModelMappingTest.test_no_security_concern_in_any_spelling_gives_no_reason](../../tests/test_verdict.py)
|
|
|
|
#### Scenario#71.02: the unquoted YAML answer "No" of a real PR-Agent run is no concern
|
|
|
|
So that the fix covers the value as the YAML loader of PR-Agent 0.47.0 gives it.
|
|
|
|
- **Given** the model answer holds the line `security_concerns: No`
|
|
- **When** PR-Agent loads the answer and werkrevisor assesses it
|
|
- **Then** the assessment holds no security concern
|
|
|
|
##### Verified by
|
|
|
|
- [test_pr_agent_run.PrAgentRunTest.test_unquoted_no_is_no_security_concern](../../tests/test_pr_agent_run.py)
|
|
|
|
## The Solution
|
|
|
|
- `parse_model_review` maps the bool `False`, `None` and the texts "no", "false" and "none",
|
|
in any case and with an optional final ".", to an empty concern.
|
|
- `assess_model` gives the reason `model_security` only for a non-empty concern.
|
|
- The bool `True` stays the concern "True": YELLOW, as before.
|
|
- The README says so; the Werkbaum adds the item as done.
|
|
- Version 0.31.5.
|