Fix Docker builds under rootless daemons and expose git metadata to build containers
Two fixes from the vm4006 rollout (docs/plan/16-git-in-docker-builds.md): Rootless daemons map the host user to container root, so running the build container as --user <host-uid> put it into the subuid range and it could not even create .gradle in a fresh worktree (legacy only worked because its ownership-repair chown had accidentally moved build/ and .gradle/ into subuid ownership in its reused primary checkout). The container now always runs as --user 0: the unprivileged host user under rootless, real root under rootful where the ownership repair still applies; under rootless it degenerates to 0:0. Git now works inside build containers: the primary .git is mounted read-only with .git/gittally/ masked by an empty tmpfs (git.token and the control token stay unreachable, the workspace bind resurfaces only the build's own worktree) and the worktree admin dir mounted read-write for index-refreshing commands. Verified on vm4006: git log/status succeed, the machine config is invisible, ref writes fail on the read-only mount. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
This commit is contained in:
mhoennig
co-authored by
Claude Fable 5
Claude Fable 5
parent
800fcae2f7
commit
8b71d0db8e
@@ -56,7 +56,7 @@ Three places must stay in sync when config keys change: the `GitTallyConfig` dat
|
||||
|
||||
`BuildExecutor` runs builds asynchronously: up to `builds.maxConcurrent` branches concurrently (default 1), but never more than one build per branch at a time. Each branch builds in its own reusable git worktree at `.git/gittally/worktrees/<branchKey>` (`BranchWorkspaces`), checked out detached at the requested commit — the primary checkout is never used for builds. Status transitions are persisted via `BuildResultRepository` (JSON file under `.git/gittally/`), published to Gitea non-fatally, and emitted as `BuildStatusChangedEvent`s. Cancellation addresses a build by artifact key and terminates the whole process tree. Future code (watcher, server, UI) must not assume a single running build.
|
||||
|
||||
The runtime is selected per branch behind the `BuildRunner` interface: `DispatchingBuildRunner` (`@Primary`) routes to native `ProcessBuildRunner` (the default) or to `DockerBuildRunner` when `branches.<name>.docker.enabled`. The Docker runner shells out to the `docker` CLI (no SDK): it (re)builds the configured image when the Dockerfile inputs changed (tracked via the `org.gittally.build-inputs-sha256` image label), maintains a per-repo Gradle cache volume, mounts the worktree and the Docker socket into a labelled (`org.hoennig.gittally`) `--rm --init` container, and repairs workspace ownership in-container after each command. The returned `Process` is the attached `docker run` client, so log streaming and termination work exactly like native builds.
|
||||
The runtime is selected per branch behind the `BuildRunner` interface: `DispatchingBuildRunner` (`@Primary`) routes to native `ProcessBuildRunner` (the default) or to `DockerBuildRunner` when `branches.<name>.docker.enabled`. The Docker runner shells out to the `docker` CLI (no SDK): it (re)builds the configured image when the Dockerfile inputs changed (tracked via the `org.gittally.build-inputs-sha256` image label), maintains a per-repo Gradle cache volume, mounts the worktree and the Docker socket into a labelled (`org.hoennig.gittally`) `--rm --init` container, and repairs workspace ownership in-container after each command (under a rootless daemon the container runs as root, which is the host user, and the repair degenerates to `0:0`). Git works inside the container: the primary `.git` is mounted read-only with `.git/gittally/` masked by an empty tmpfs (credential isolation) and the worktree's admin dir mounted read-write (`gitMetadataMounts`). The returned `Process` is the attached `docker run` client, so log streaming and termination work exactly like native builds.
|
||||
|
||||
## Watcher
|
||||
|
||||
|
||||
@@ -216,6 +216,8 @@ When `dockerfile` is set, the image is (re)built whenever the Dockerfile content
|
||||
Staleness is tracked via the image label `org.gittally.build-inputs-sha256`.
|
||||
A Gradle cache volume `gittally-gradle-<repo-key>` is created per repository and mounted as `GRADLE_USER_HOME`.
|
||||
The build worktree is bind-mounted into the container; after each command the ownership of `build/` and `.gradle/` is repaired to the host user.
|
||||
Git works inside the container: the primary repository's `.git` is mounted read-only (so build steps can run read-only git commands like `git log` or `git describe`), with `.git/gittally/` masked by an empty tmpfs so the build can never read the machine config (`git.token`) or the control token.
|
||||
Note that the rest of `.git` — including `.git/config` — is visible to builds; GitTally never stores credentials there, and neither should you.
|
||||
The Docker socket is mounted into the container and `DOCKER_HOST`/`TESTCONTAINERS_*` variables are set, so Testcontainers-based builds work inside the container.
|
||||
All GitTally containers carry `org.hoennig.gittally` labels; stale build containers of the repository are removed before the first Docker build after a restart.
|
||||
|
||||
|
||||
@@ -0,0 +1,44 @@
|
||||
# Step 16: Git Access Inside Docker Build Containers
|
||||
|
||||
Prerequisites: steps 11, 15.
|
||||
Read `README.md` first.
|
||||
Motivated by the vm4006 rollout (step 15, fourth finding): hs.hsadmin.ng's build calls git in several places; `:prQuickCheck` failed hard with "fatal: not a git repository", other call sites swallowed the failure silently.
|
||||
|
||||
## Problem
|
||||
|
||||
Builds run in git worktrees under `.git/gittally/worktrees/<branchKey>`.
|
||||
A worktree's `.git` is a pointer file into the primary repository's `.git/worktrees/<key>`, and `DockerBuildRunner` bind-mounts only the worktree — so every git call inside the build container fails.
|
||||
The legacy script did not have this problem because it built in the primary checkout with the real `.git` present (read-write, including all secrets stored next to it — full exposure).
|
||||
Hard invariant to preserve: a branch build must never be able to reach credentials; `.git/gittally/.gittally.yml` (`git.token`) and the control token live under `.git`.
|
||||
|
||||
## Considered Options
|
||||
|
||||
- **Read-only `.git` mount with `.git/gittally/` masked (chosen)** — three layered mounts, no config key, no workspace mutation; strictly less privileged than legacy.
|
||||
- Copy minimal git metadata into the workspace (admin dir plus `objects/info/alternates`) — mutates the workspace, still needs the object database mounted, more moving parts.
|
||||
- Document the limitation and require git-free build commands — pushes the problem onto every watched project; hsadmin-ng shows real builds do call git.
|
||||
|
||||
## Design
|
||||
|
||||
`DockerBuildRunner.gitMetadataMounts(workspace, repoDir)` adds three mounts when (and only when) the workspace is a worktree of `repoDir` (detected via the `gitdir:` pointer file, which must resolve into `repoDir/.git`):
|
||||
|
||||
1. `repoDir/.git` → same path, **read-only**: objects, refs, and the worktree admin metadata become resolvable; object and ref writes stay impossible.
|
||||
2. An empty **tmpfs over `repoDir/.git/gittally`**: masks the machine config (`git.token`), the control token, and all GitTally state; the workspace bind (deeper path, Docker nests mounts by target depth) resurfaces only this build's own worktree inside the masked directory.
|
||||
3. `repoDir/.git/worktrees/<key>` → same path, **read-write**: the worktree's admin dir (HEAD, index), so index-refreshing commands like `git status` work.
|
||||
|
||||
No configuration key: the exposure is strictly smaller than the legacy baseline, and a knob would join the pinned sandbox-policy set without a known use case.
|
||||
Remaining, documented exposure: the rest of `.git` — including `.git/config` — is readable by builds; GitTally never stores credentials there (fetch auth uses a secret-free `GIT_ASKPASS` with env-passed credentials).
|
||||
|
||||
## Tests
|
||||
|
||||
- `DockerBuildRunnerTest`: worktree workspace → the three mounts with `:ro` and `--tmpfs`; non-worktree workspace → no git metadata mounts (also keeps the exact-argv test valid).
|
||||
|
||||
## Acceptance Criteria
|
||||
|
||||
- `./gradlew ktlintFormat` then `./gradlew build` is green.
|
||||
- In a real Docker build worktree: `git log`/`git status` succeed inside the container, `.git/gittally/.gittally.yml` and `control-token` are not readable, and a `git push`/ref write fails.
|
||||
- `docs/configuration.md` (docker notes) and the architecture skill describe the mounts.
|
||||
|
||||
## Result (2026-08-10)
|
||||
|
||||
Implemented as designed; verified on vm4006 (see below) and in unit tests.
|
||||
`sh -c 'git log -1 && git status --short && cat .../.git/gittally/.gittally.yml'` inside a build container of the hs.hsadmin.ng worktree: git commands succeed, the machine config read fails with "No such file or directory", `git update-ref` fails on the read-only filesystem.
|
||||
@@ -5,6 +5,7 @@ import de.hoennig.gittally.config.DockerConfig
|
||||
import de.hoennig.gittally.git.GitCommandRunner
|
||||
import org.slf4j.LoggerFactory
|
||||
import org.springframework.stereotype.Component
|
||||
import java.nio.file.Files
|
||||
import java.nio.file.Path
|
||||
|
||||
/**
|
||||
@@ -17,8 +18,12 @@ import java.nio.file.Path
|
||||
* tracked via the [DockerImageInputs.INPUTS_LABEL] image label), and the per-repo
|
||||
* Gradle cache volume. The returned [Process] is the attached `docker run` client
|
||||
* (`--init`, so termination signals reach the build inside the container); the
|
||||
* executor streams and terminates it like a native build. Ownership of `build/`
|
||||
* and `.gradle/` is repaired to the host user inside the same container run.
|
||||
* executor streams and terminates it like a native build. On rootful daemons the
|
||||
* ownership of `build/` and `.gradle/` is repaired to the host user inside the
|
||||
* same container run; under a rootless daemon the container runs as root, which
|
||||
* already is the host user, so the repair chown degenerates to `0:0`.
|
||||
* Git works inside the container: the primary `.git` is mounted read-only with
|
||||
* `.git/gittally/` masked, see [gitMetadataMounts].
|
||||
*/
|
||||
@Component
|
||||
class DockerBuildRunner(
|
||||
@@ -51,8 +56,15 @@ class DockerBuildRunner(
|
||||
ensureImage(docker, repoDir)
|
||||
val uid = id("-u", repoDir)
|
||||
val gid = id("-g", repoDir)
|
||||
val socket = socketLocator.locate(uid)
|
||||
// Under a rootless daemon, container root IS the unprivileged host user (identity mapping),
|
||||
// so files in the bind-mounted worktree stay host-owned and no chown is needed; chowning to
|
||||
// the host ids there would push the files into the subuid range and lock the host user out.
|
||||
val rootless = socket?.rootless == true
|
||||
val ownershipUid = if (rootless) "0" else uid
|
||||
val ownershipGid = if (rootless) "0" else gid
|
||||
val volume = gradleVolumeName(repoKey)
|
||||
prepareGradleVolume(volume, docker.image, uid, gid, repoDir)
|
||||
prepareGradleVolume(volume, docker.image, ownershipUid, ownershipGid, repoDir)
|
||||
val containerName = containerName(repoKey, environment["branch"])
|
||||
removeContainer(containerName, repoDir)
|
||||
val runCommand =
|
||||
@@ -61,11 +73,13 @@ class DockerBuildRunner(
|
||||
workspace = workingDir.toAbsolutePath().normalize(),
|
||||
environment = environment,
|
||||
docker = docker,
|
||||
repoDir = repoDir,
|
||||
repoKey = repoKey,
|
||||
containerName = containerName,
|
||||
volume = volume,
|
||||
uid = uid,
|
||||
gid = gid,
|
||||
socket = socket,
|
||||
uid = ownershipUid,
|
||||
gid = ownershipGid,
|
||||
)
|
||||
return processStarter(runCommand, repoDir)
|
||||
}
|
||||
@@ -207,13 +221,14 @@ class DockerBuildRunner(
|
||||
workspace: Path,
|
||||
environment: Map<String, String>,
|
||||
docker: DockerConfig,
|
||||
repoDir: Path,
|
||||
repoKey: String,
|
||||
containerName: String,
|
||||
volume: String,
|
||||
socket: DockerSocket?,
|
||||
uid: String,
|
||||
gid: String,
|
||||
): List<String> {
|
||||
val socket = socketLocator.locate(uid)
|
||||
val args = mutableListOf("docker", "run", "--rm", "--init", "--name", containerName)
|
||||
args +=
|
||||
listOf(
|
||||
@@ -225,6 +240,7 @@ class DockerBuildRunner(
|
||||
"$GITTALLY_LABEL.role=build",
|
||||
)
|
||||
args += listOf("--workdir", "$workspace", "--volume", "$workspace:$workspace")
|
||||
args += gitMetadataMounts(workspace, repoDir)
|
||||
args += listOf("--volume", "$volume:/gradle-user-home")
|
||||
args += listOf("--env", "HOME=/tmp/docker-home", "--env", "GRADLE_USER_HOME=/gradle-user-home")
|
||||
for ((key, value) in environment) {
|
||||
@@ -244,7 +260,9 @@ class DockerBuildRunner(
|
||||
args += listOf("--group-add", socket.gid.toString())
|
||||
}
|
||||
}
|
||||
args += listOf("--user", if (socket?.rootless == true) uid else "0")
|
||||
// root in the container: the real host user under a rootless daemon (identity mapping),
|
||||
// or actual root under a rootful daemon, where the ownership-repair chown fixes the files.
|
||||
args += listOf("--user", "0")
|
||||
val hostOverride = if (docker.network == "host") "localhost" else "host.docker.internal"
|
||||
args += listOf("--env", "TESTCONTAINERS_HOST_OVERRIDE=$hostOverride")
|
||||
if (docker.network != "host") {
|
||||
@@ -255,6 +273,48 @@ class DockerBuildRunner(
|
||||
return args
|
||||
}
|
||||
|
||||
/**
|
||||
* Makes git work inside the build container without exposing GitTally's secrets.
|
||||
*
|
||||
* The workspace is a git worktree whose `.git` file points into the primary
|
||||
* repository's `.git`, which is not part of the workspace mount — so any git call
|
||||
* in the build would fail. Three layered mounts fix that (Docker nests mounts by
|
||||
* target path): the primary `.git` read-only, an empty tmpfs masking `.git/gittally/`
|
||||
* (machine config with `git.token`, control token, build state — the workspace bind
|
||||
* resurfaces only this build's own worktree inside it), and this worktree's admin
|
||||
* directory read-write, so index-refreshing commands like `git status` keep working.
|
||||
* Object and ref writes stay blocked by the read-only `.git` mount.
|
||||
* No mounts are added when the workspace is not a worktree of [repoDir].
|
||||
*/
|
||||
private fun gitMetadataMounts(
|
||||
workspace: Path,
|
||||
repoDir: Path,
|
||||
): List<String> {
|
||||
val gitDir = repoDir.toAbsolutePath().normalize().resolve(".git")
|
||||
val workspaceGitFile = workspace.resolve(".git")
|
||||
if (!Files.isDirectory(gitDir) || !Files.isRegularFile(workspaceGitFile)) {
|
||||
return emptyList()
|
||||
}
|
||||
val adminDir =
|
||||
Files
|
||||
.readString(workspaceGitFile)
|
||||
.substringAfter("gitdir:", "")
|
||||
.trim()
|
||||
.takeIf { it.isNotEmpty() }
|
||||
?.let { workspace.resolve(it).normalize() }
|
||||
?: return emptyList()
|
||||
if (!adminDir.startsWith(gitDir) || !Files.isDirectory(adminDir)) {
|
||||
return emptyList()
|
||||
}
|
||||
val args = mutableListOf("--volume", "$gitDir:$gitDir:ro")
|
||||
val gittallyDir = gitDir.resolve("gittally")
|
||||
if (Files.isDirectory(gittallyDir)) {
|
||||
args += listOf("--tmpfs", "$gittallyDir")
|
||||
}
|
||||
args += listOf("--volume", "$adminDir:$adminDir")
|
||||
return args
|
||||
}
|
||||
|
||||
private fun id(
|
||||
flag: String,
|
||||
repoDir: Path,
|
||||
|
||||
@@ -8,7 +8,7 @@ import java.nio.file.Paths
|
||||
/** The host-side Docker socket a build container should talk to. */
|
||||
data class DockerSocket(
|
||||
val path: Path,
|
||||
/** True for a rootless daemon socket (`/run/user/<uid>/docker.sock`); the container then runs as the host user. */
|
||||
/** True for a rootless daemon socket (`/run/user/<uid>/docker.sock`); container root then maps to the host user. */
|
||||
val rootless: Boolean,
|
||||
/** Group id owning the socket, for `--group-add` on rootful daemons; null when unavailable. */
|
||||
val gid: Long?,
|
||||
|
||||
@@ -126,7 +126,7 @@ class DockerBuildRunnerTest : FunSpec() {
|
||||
script shouldContain "exit \$build_exit"
|
||||
}
|
||||
|
||||
test("a rootless socket runs the container as the host user without group-add") {
|
||||
test("a rootless socket runs the container as root (the host user) without group-add or host-id chown") {
|
||||
every { socketLocator.locate("1000") } returns
|
||||
DockerSocket(Paths.get("/run/user/1000/docker.sock"), rootless = true, gid = 998L)
|
||||
|
||||
@@ -134,8 +134,42 @@ class DockerBuildRunnerTest : FunSpec() {
|
||||
|
||||
val args = captured.single()
|
||||
args shouldContain "/run/user/1000/docker.sock:/var/run/docker.sock"
|
||||
args[args.indexOf("--user") + 1] shouldBe "1000"
|
||||
args[args.indexOf("--user") + 1] shouldBe "0"
|
||||
args shouldNotContain "--group-add"
|
||||
// container root already is the host user — the repair chown must not target the host ids,
|
||||
// which would push the worktree files into the subuid range
|
||||
args[args.size - 3] shouldBe "0"
|
||||
args[args.size - 2] shouldBe "0"
|
||||
verify {
|
||||
commandRunner.runOrThrow(
|
||||
match { it.take(2) == listOf("docker", "run") && it.takeLast(2) == listOf("0", "0") },
|
||||
repoDir,
|
||||
)
|
||||
}
|
||||
}
|
||||
|
||||
test("exposes git metadata read-only with the gittally dir masked for a worktree workspace") {
|
||||
val gitDir = repoDir.resolve(".git")
|
||||
val adminDir = gitDir.resolve("worktrees/workspace")
|
||||
Files.createDirectories(adminDir)
|
||||
Files.createDirectories(gitDir.resolve("gittally"))
|
||||
Files.createDirectories(workspace)
|
||||
Files.writeString(workspace.resolve(".git"), "gitdir: $adminDir\n")
|
||||
|
||||
runner.start("./gradlew test", workspace, mapOf("branch" to "main"), repoDir, dockerBranchConfig())
|
||||
|
||||
val args = captured.single()
|
||||
args shouldContain "$gitDir:$gitDir:ro"
|
||||
args[args.indexOf("--tmpfs") + 1] shouldBe "${gitDir.resolve("gittally")}"
|
||||
args shouldContain "$adminDir:$adminDir"
|
||||
}
|
||||
|
||||
test("mounts no git metadata when the workspace is not a worktree") {
|
||||
runner.start("./gradlew test", workspace, mapOf("branch" to "main"), repoDir, dockerBranchConfig())
|
||||
|
||||
val args = captured.single()
|
||||
args shouldNotContain "--tmpfs"
|
||||
args.none { it.endsWith(":ro") } shouldBe true
|
||||
}
|
||||
|
||||
test("host network keeps Testcontainers on localhost without add-host") {
|
||||
|
||||
Reference in New Issue
Block a user