60 lines
2.3 KiB
Markdown
60 lines
2.3 KiB
Markdown
# PR#139: PIT Specs Stop at Their First Failure
|
|
|
|
> **WARNING:** This document describes only the change applied in this PR.
|
|
> It may already be outdated once the next PR is merged.
|
|
> Historic PR-documentation is not maintained along with new PRs — treat it as a snapshot, not as current documentation.
|
|
|
|
## The Problem
|
|
|
|
For a mutant, PIT runs the whole Kotest spec, every test in it, even after the first test has failed.
|
|
One failure already kills the mutant, so the rest of the spec is wasted time.
|
|
|
|
It is worst where tests wait for something.
|
|
A mutant that never ends a build lets every test of `BuildExecutorUnitTest` wait up to five seconds.
|
|
The spec then runs past PIT's timeout: the mutant ends `TIMED_OUT` after more than 20 s, and PIT starts a new test process.
|
|
|
|
## Non-Goals
|
|
|
|
- No change to normal test runs: they still run every test and show every failure.
|
|
- The integration specs that wait up to 30 s for a build still run into the timeout with their first test; they follow in their own PR.
|
|
|
|
## The Scenarios
|
|
|
|
#### Scenario#139.01: A normal test run runs every test
|
|
|
|
- **Given** a spec with a failing test
|
|
- **When** it runs outside the mutation build
|
|
- **Then** its other tests still run
|
|
- **So that** a developer sees every failure at once.
|
|
|
|
##### Verified by
|
|
|
|
- [FailFastUnitTest](../../src/test/kotlin/de/hoennig/werkator/framework/FailFastUnitTest.kt)
|
|
|
|
#### Scenario#139.02: In the mutation build a spec stops at its first failure
|
|
|
|
- **Given** a mutant that makes a test fail
|
|
- **When** PIT runs the spec for it
|
|
- **Then** the spec skips its remaining tests
|
|
- **So that** the mutant is killed at once and does not wait into PIT's timeout.
|
|
|
|
##### Verified by
|
|
|
|
- The local PIT run below; Kotest's own fail-fast is not tested again here.
|
|
|
|
## The Solution
|
|
|
|
- `ProjectConfig.failfast` is read from the system property `werkator.failfast`.
|
|
- The `pitest` block passes `-Dwerkator.failfast=true` to PIT's test processes only.
|
|
|
|
A local PIT run of `BuildExecutor` alone, two minions, without history, with the tests of PR#138:
|
|
|
|
| Run | Duration | Mutants timed out | Mutants killed |
|
|
|----------------|---------:|------------------:|---------------:|
|
|
| Without the PR | 23.7 min | 35 | 91 |
|
|
| With fail-fast | 18.6 min | 25 | 102 |
|
|
|
|
## Prerequisite PRs
|
|
|
|
- PR#136, which changes the same `ProjectConfig`.
|