70 lines
3.7 KiB
Markdown
70 lines
3.7 KiB
Markdown
# PR#140: Executor Integration Specs out of the Mutation Build
|
|
|
|
> **WARNING:** This document describes only the change applied in this PR.
|
|
> It may already be outdated once the next PR is merged.
|
|
> Historic PR-documentation is not maintained along with new PRs — treat it as a snapshot, not as current documentation.
|
|
|
|
## The Problem
|
|
|
|
For each mutant, PIT runs the covering specs fastest first.
|
|
For `BuildExecutor` the two fastest are `PullRequestTargetBuildIntegrationTest` (0.7 s) and `BuildExecutorArtifactIntegrationTest` (1.3 s), both with real git and real processes.
|
|
They wait up to 30 s for a build to end.
|
|
A mutant that never ends a build therefore runs into PIT's timeout in them, before `BuildExecutorUnitTest` could kill it within a second.
|
|
|
|
In a local PIT run of `BuildExecutor` alone, with PR#138 and PR#139, 25 of 194 mutants timed out, and the run took 18.6 min.
|
|
|
|
## Non-Goals
|
|
|
|
- The two specs stay in the normal build: they test the executor with real processes, which no unit test does.
|
|
- No change to the executor itself.
|
|
|
|
## The Scenarios
|
|
|
|
#### Scenario#140.01: A finished build hands its result, its logs and its workspace to the artifact store
|
|
|
|
- **Given** a build that succeeded
|
|
- **When** it is settled
|
|
- **Then** the artifact store gets its result, its staging directory with the logs, and its workspace
|
|
- **So that** the reports of the build can be stored.
|
|
|
|
##### Verified by
|
|
|
|
- [BuildExecutorUnitTest](../../src/test/kotlin/de/hoennig/werkator/build/BuildExecutorUnitTest.kt)
|
|
|
|
#### Scenario#140.02: A build whose artifacts leave the workspace ends failed
|
|
|
|
- **Given** a build whose commands succeed
|
|
- **And** the artifact store refuses its artifacts because they point out of the workspace
|
|
- **When** it is settled
|
|
- **Then** its status is taken back to failed, and its live log says why
|
|
- **So that** a build whose result cannot be collected never counts as green.
|
|
|
|
##### Verified by
|
|
|
|
- [BuildExecutorUnitTest](../../src/test/kotlin/de/hoennig/werkator/build/BuildExecutorUnitTest.kt)
|
|
|
|
Two scenarios of PR#117 get a unit test beside their integration test:
|
|
Scenario#117.03 (a review is started as one and learns its pull request and the forge) and Scenario#117.01 (a head already queued or running is not queued twice).
|
|
|
|
## The Solution
|
|
|
|
- `BuildExecutorUnitTest` gets the four tests above; its fake runner records whether a command was started as a pull-request target build, and its artifact store is a mock it can check.
|
|
- The `pitest` block leaves out `PullRequestTargetBuildIntegrationTest` and `BuildExecutorArtifactIntegrationTest`.
|
|
|
|
A local PIT run of `BuildExecutor` alone, two minions, without history, with PR#138 and PR#139:
|
|
|
|
| Run | Duration | Mutants timed out |
|
|
|-----------------------------------------|---------:|------------------:|
|
|
| With the two specs | 18.6 min | 25 |
|
|
| Without them, before the new unit tests | 9.3 min | 2 |
|
|
| Without them, with the new unit tests | 9.9 min | 2 |
|
|
|
|
Without the two specs, 8 mutants survived that they had killed; the new unit tests kill all of them, and 3 more.
|
|
In `CommandProcess`, `RunningBuild` and `PullRequestTarget`, which the two specs cover as well, a PIT run without them kills every mutant the nightly killed, except two that depend on timing (a removed `Thread.sleep` and `Thread.join`), and the timeouts there drop from 8 to 2.
|
|
|
|
## Additional Changes
|
|
|
|
- `FailFastUnitTest` from PR#139 is left out of the mutation build as well.
|
|
It checks that a normal test run does not fail fast, and PIT's test processes do, so under PIT it fails without a mutant.
|
|
PIT then aborts the whole run already in its coverage phase: the nightly would fail without measuring anything.
|