90 lines
5.0 KiB
Markdown
90 lines
5.0 KiB
Markdown
> **WARNING:** This document describes only the change applied in this PR.
|
|
> It may already be outdated once the next PR is merged.
|
|
> Historic PR-documentation is not maintained along with new PRs — treat it as a snapshot, not as current documentation.
|
|
|
|
## The Problem
|
|
|
|
A build that hangs blocks the machine until somebody notices.
|
|
|
|
On the production instance `main@jvm` of one repository sat at `running` for 68 minutes at a system load of 0.17 — it was not computing anything. It held the single executor slot the whole time, so two builds of another repository waited behind it, and the rollout that finally cleared it was a restart. Nothing in Werkator would have ended it: a build runs until its process exits, is cancelled by hand, or the server shuts down.
|
|
|
|
## Non-Goals
|
|
|
|
- No detection of *why* a build hangs. A timeout is a deadline, not a diagnosis.
|
|
- No per-command limit. Five commands must not each be allowed the full time, so the limit is one deadline for the whole build.
|
|
- No retry. A build stopped by the clock stays stopped; repeating it unasked would burn the slot again.
|
|
|
|
## The Scenarios
|
|
|
|
### Feature: a build that runs too long
|
|
|
|
#### Background
|
|
|
|
- `executor` is an instance-level section and is stripped from a branch's own configuration, so a branch cannot write it.
|
|
|
|
#### Scenario#48.01: Stopped by the clock is its own outcome
|
|
|
|
So that it can be told apart from a failure and from a cancellation.
|
|
|
|
- **Given** a build that exceeds the time it is allowed
|
|
- **Then** its result is `timed_out`, not `failed` — nothing about the work failed, it did not finish — and not `cancelled`, because nobody decided to stop it
|
|
- **and** it is terminal, and unlike `interrupted` it is not re-enqueued on its own
|
|
- **and** it is published to Gitea as `failure`, because nobody will re-run it and a commit whose build never finished is not a green one
|
|
- **and** the page shows it in the watch colour, between a failure and a nothing
|
|
|
|
##### Verified by
|
|
|
|
- [BuildStatusTest](../../src/test/kotlin/de/hoennig/werkator/build/BuildStatusTest.kt)
|
|
|
|
#### Scenario#48.02: The host names a ceiling, a build may ask for less
|
|
|
|
So that a repository can promise to be quick without being able to occupy the machine.
|
|
|
|
- **Given** `executor.buildTimeout: 30m` on the host
|
|
- **When** a build definition sets `timeout: 10m`
|
|
- **Then** the build is allowed ten minutes
|
|
- **When** it sets `timeout: 4h`
|
|
- **Then** it is allowed thirty minutes — a longer wish is capped, not refused
|
|
- **Given** a branch's own committed configuration
|
|
- **Then** it may set `builds.<name>.timeout`, which is a description of its own build
|
|
- **and** it cannot set `executor.buildTimeout`, which is stripped with the rest of the section
|
|
- **Given** neither side setting one
|
|
- **Then** the build may run as long as it likes, which is what an empty ceiling means
|
|
|
|
##### Verified by
|
|
|
|
- [BuildTimeoutsTest](../../src/test/kotlin/de/hoennig/werkator/build/BuildTimeoutsTest.kt)
|
|
- [ConfigLoaderTest](../../src/test/kotlin/de/hoennig/werkator/config/ConfigLoaderTest.kt)
|
|
|
|
#### Scenario#48.03: The slot is free again
|
|
|
|
- **Given** a build sleeping far longer than it is allowed
|
|
- **When** the deadline passes
|
|
- **Then** its process tree is terminated, the live log says why, and the result is `timed_out`
|
|
- **and** the executor slot is released, so the next build starts
|
|
- **Given** a build that finishes in time
|
|
- **Then** nothing about it changes
|
|
|
|
##### Verified by
|
|
|
|
- [BuildExecutorTest](../../src/test/kotlin/de/hoennig/werkator/build/BuildExecutorTest.kt)
|
|
|
|
## The Solution
|
|
|
|
`BuildTimeouts.effective` is the whole rule, in one place: the smaller of the host's ceiling and the build's own wish, either side empty meaning "no opinion", both empty meaning no limit.
|
|
The asymmetry is not enforced by stripping the key — it is enforced by taking the minimum, which is why `builds.<name>.timeout` is one of the few settings a branch may write. A branch is trusted to describe its own build; it is not trusted to occupy the machine because a value in a file said so.
|
|
|
|
The deadline is computed once, where the build's configuration is resolved, and each command then waits bounded by what is left of it. On expiry the process tree is destroyed exactly as a cancellation destroys it, the live log says so, and the command sequence ends.
|
|
|
|
`BuildStatus.TIMED_OUT` is new. It costs a branch in the Gitea mapping, a colour, and one entry in the status vocabulary the page knows — and it buys a build history in which a timeout does not look like a failing test.
|
|
|
|
## Open Questions
|
|
|
|
- The default ceiling of 30 minutes is a guess at what "far too long" means here, taken from build times of a few minutes. A host with a genuinely long build has to raise it, and will notice the first time.
|
|
- Nothing warns when a build is close to its deadline. A build that regularly finishes at 29 minutes under a 30-minute ceiling is a problem waiting for a slow day.
|
|
|
|
## Additional Changes
|
|
|
|
- Version 1.6.0 and its release-notes entry, since this is what gets deployed.
|
|
- The three configuration places are in sync: the data classes, the `init` templates, and `docs/configuration.md`.
|