werkator/docs/prs/2026-10-10-PR#140-executor-integration-specs-out-of-pit.md

PR#140: Executor Integration Specs out of the Mutation Build

WARNING: This document describes only the change applied in this PR. It may already be outdated once the next PR is merged. Historic PR-documentation is not maintained along with new PRs — treat it as a snapshot, not as current documentation.

The Problem

For each mutant, PIT runs the covering specs fastest first. For BuildExecutor the two fastest are PullRequestTargetBuildIntegrationTest (0.7 s) and BuildExecutorArtifactIntegrationTest (1.3 s), both with real git and real processes. They wait up to 30 s for a build to end. A mutant that never ends a build therefore runs into PIT's timeout in them, before BuildExecutorUnitTest could kill it within a second.

In a local PIT run of BuildExecutor alone, with PR#138 and PR#139, 25 of 194 mutants timed out, and the run took 18.6 min.

Non-Goals

• The two specs stay in the normal build: they test the executor with real processes, which no unit test does.
• No change to the executor itself.

The Scenarios

Scenario#140.01: A finished build hands its result, its logs and its workspace to the artifact store

• Given a build that succeeded
• When it is settled
• Then the artifact store gets its result, its staging directory with the logs, and its workspace
• So that the reports of the build can be stored.

Verified by

• BuildExecutorUnitTest

Scenario#140.02: A build whose artifacts leave the workspace ends failed

• Given a build whose commands succeed
• And the artifact store refuses its artifacts because they point out of the workspace
• When it is settled
• Then its status is taken back to failed, and its live log says why
• So that a build whose result cannot be collected never counts as green.

Verified by

• BuildExecutorUnitTest

Two scenarios of PR#117 get a unit test beside their integration test: Scenario#117.03 (a review is started as one and learns its pull request and the forge) and Scenario#117.01 (a head already queued or running is not queued twice).

The Solution

• BuildExecutorUnitTest gets the four tests above; its fake runner records whether a command was started as a pull-request target build, and its artifact store is a mock it can check.
• The pitest block leaves out PullRequestTargetBuildIntegrationTest and BuildExecutorArtifactIntegrationTest.

A local PIT run of BuildExecutor alone, two minions, without history, with PR#138 and PR#139:

Run	Duration	Mutants timed out
With the two specs	18.6 min	25
Without them, before the new unit tests	9.3 min	2
Without them, with the new unit tests	9.9 min	2

Without the two specs, 8 mutants survived that they had killed; the new unit tests kill all of them, and 3 more. In CommandProcess, RunningBuild and PullRequestTarget, which the two specs cover as well, a PIT run without them kills every mutant the nightly killed, except two that depend on timing (a removed Thread.sleep and Thread.join), and the timeouts there drop from 8 to 2.

Additional Changes

• FailFastUnitTest from PR#139 is left out of the mutation build as well. It checks that a normal test run does not fail fast, and PIT's test processes do, so under PIT it fails without a mutant. PIT then aborts the whole run already in its coverage phase: the nightly would fail without measuring anything.