werkator/docs/prs/2026-09-15-PR#62-a-runtime-the-host-can-start.md

WARNING: This document describes only the change applied in this PR. It may already be outdated once the next PR is merged. Historic PR-documentation is not maintained along with new PRs — treat it as a snapshot, not as current documentation.

Related Links

• RFC 0005: Werkator Deploys Itself — PR#61 shipped its step 1; this PR makes that step work on the production host.
• Self-Deployment — how the deployment is put together.

The Problem

The self-deployment of PR#61 was tried on the production host on 2026-09-15 and could not install anything.

The deploy build failed after four seconds, at the one check that exists for this:

/home/…/.werkator/werkator.new/bin/../jre/bin/java: /lib/x86_64-linux-gnu/libc.so.6:
version `GLIBC_2.38' not found (required by …/werkator.new/jre/bin/../lib/libjli.so)
ERROR: the staged runtime does not run on this host — nothing was swapped

The runtime Werkator installs is not downloaded from anywhere; jlink cuts it out of the Java development kit that Gradle's toolchain selected, and it copies that kit's native libraries along. Those libraries carry the C-library requirement of the machine the kit itself was compiled on. So the toolchain does not only decide what compiles the code — it decides where the resulting runtime is able to start.

Two kits were in play, both Java 21, and that is why the version number gave no warning:

Java development kit	Java	requires glibc
Debian trixie's OpenJDK, in the build sandbox	21	2.38
Adoptium Temurin, on the developer machine	21	2.14
available on the production host (Debian 12)		2.36

A distribution's own kit is built against that distribution. The sandbox runs a newer distribution than the host, so its kit produces a runtime the host cannot start. The developer machine happened to use Temurin, which is built against a deliberately old base, which is why tools/remote … instance-update has worked for months and hid the problem.

Nothing on the host was damaged, and that is the second half of the finding. The staged swap put the new runtime beside the running one, ran its launcher, and stopped there: werkator.new was removed, the installation kept werkator and werkator.prev, the service stayed up on v1.14.0, and the transient restart unit was never scheduled — its journal says "No entries". The build is recorded failed, which is terminal, so the startup recovery does not re-run it.

Non-Goals

• No new build environment. The sandbox keeps its distribution and its rootfs; only which kit Gradle uses inside it changes.
• No second Java version. The kit that compiles is still the kit that ships, which is what keeps one line in charge of the Java version everywhere.
• No change to the deployment itself. tools/self-deploy, the staged swap and the watchdog are untouched; they behaved exactly as designed.
• No pinning of a kit version beyond the language version. Gradle picks the current Temurin 21; reproducing an exact patch level is a different concern.

The Scenarios

Feature: the runtime ships from a portable kit

Background

• The toolchain is the Java development kit Gradle selects for compiling. jlink builds the shipped runtime from it, so it is also the origin of the runtime on the host.
• A kit's glibc requirement comes from the platform it was compiled on, not from its Java version.
• The build sandbox runs Debian trixie (glibc 2.41); the production host runs Debian 12 (glibc 2.36).

Scenario#62.01: The build sandbox produces a runtime the host can start

So that a deployment installs something instead of refusing it.

• Given the build runs in the sandbox, whose distribution is newer than the host's
• When the bundle is built
• Then the toolchain is Temurin, not the distribution's own kit
  • and every native library of the produced runtime asks for glibc 2.15 or less
  • and the host, which has 2.36, starts it

Verified by

• Measured on the produced runtime and then on the production host itself, before the merge: the runtime the sandbox built for this branch starts there and prints its version. Recorded in The glibc requirement of the produced runtime.

Scenario#62.02: Raising Java reaches the host by itself

So that a Java upgrade stays one line and needs no work on the host.

• Given a future release of Java that Werkator should run on
• When languageVersion in build.gradle.kts names it
• Then the build compiles with that version
  • and the runtime installed on the host is that version too
  • and nothing on the host has to be prepared for it, because the vendor's kits keep their old glibc base

Verified by

• Not verifiable today; it is the property the vendor pinning exists for, and the reason a kit baked into the rootfs was rejected — that one would have made every Java upgrade a rootfs upload.

Scenario#62.03: A kit that is not installed is fetched

So that pinning a vendor does not depend on what a machine happens to have.

• Given a build environment without a Temurin kit, such as the sandbox
• When Gradle resolves the toolchain
• Then it downloads the matching Temurin kit and keeps it in the environment's Gradle home
  • and the next build in that environment finds it there

Verified by

• The CI build of this branch, b9203be, green in 144 seconds in exactly such an environment. The runtime it produced reports Temurin-21.0.12.1+1, which the sandbox had to fetch — its rootfs carries Debian's kit and no other.

The Solution

settings.gradle.kts gains the foojay-resolver-convention plugin and the toolchain block gains vendor = JvmVendorSpec.ADOPTIUM.

That is the whole change. Gradle then prefers a Temurin kit everywhere, downloads one where none is installed, and keeps it in that environment's Gradle home — inside the sandbox that home is the persistent toolchain directory the werkdock runner mounts, so the download happens once.

The developer machine already had Temurin, which is why its bundles always worked; the sandbox now uses the same kit, so the two build environments stop differing in the one property that matters here.

A kit baked into the rootfs would also have fixed today's failure. It was rejected because it moves the Java version out of the repository and onto the host: every upgrade would mean rebuilding and uploading roughly a gigabyte of rootfs, which is exactly the manual step the self-deployment exists to remove.

Open Questions

• What is still unproven is the deployment as a whole, not the runtime it installs. deploy has never completed on the host; the attempt that led to this PR stopped at the verification, and this branch is not a follow-up of anything. The first merge after this one will run it, and that run belongs in PR#61's record as its Scenario#61.01.
• The first build in a fresh environment downloads about 190 MB. It is cached in that environment's Gradle home afterwards, but a rootfs rebuilt from scratch pays it again.
• The kit's patch level is whatever the vendor currently offers — the sandbox fetched 21.0.12.1 while the developer machine still has 21.0.5. Both satisfy the language version, and neither the glibc base nor anything else in this PR depends on the patch level; pinning it exactly is a separate question.

Additional Changes

• The release notes for v1.14.1 describe the failure in the terms a reader of the release page needs, without the glibc arithmetic.

Prerequisite PRs

• PR#61 — the self-deployment this PR makes usable. Its tools/self-deploy is unchanged; its native verification is what turned an unbootable runtime into a failed build instead of a broken instance.

Follow-up PRs

• Step 2 of RFC 0005: restartsInstance as a pinned build setting.
• Sample /proc/pressure/io, and record the CPU time a build consumed beside its wall-clock duration — both decided on 2026-09-14, both still open.

Attachments

The glibc requirement of the produced runtime

Built from this branch and measured over all 33 native files of the produced runtime:

measured	value
highest requirement of any native library	GLIBC_2.15
requirement of libjli.so, which failed before	GLIBC_2.14
available on the production host	2.36

For comparison, the same measurement on the kit the sandbox used before this change gives GLIBC_2.38, which is what the host refused.

The runtime the CI build of b9203be produced was then started on the production host directly, out of the build worktree and without installing it:

Werkator v1.14.1
openjdk version "21.0.12.1" 2026-08-18 LTS
OpenJDK Runtime Environment Temurin-21.0.12.1+1 (build 21.0.12.1+1-LTS)
GLIBC_2.14

That is the whole claim of this PR in four lines: the sandbox fetched Temurin, jlink cut the runtime from it, and the host starts what the sandbox built.