werkator build succeeded after 00:06
RFC 0004 had ruled `build` out as "not planned: needs writable rootfs snapshots per step and `pull` for FROM". Both halves expired: `pull` arrived in 0.3.0, and per-step snapshots are what a build *cache* needs, not what a build needs — one writable tree over the base is enough. ## RFC 0006 [docs/rfcs/0006-dockerfile-builds.md](docs/rfcs/0006-dockerfile-builds.md) decides the build model, cuts the one version-dependent interface, and argues the "no layers" question from measurements rather than from the README's shorthand. Measured on a `debian:trixie` rootfs with bwrap 0.11.1 and a `0 <uid> 1` map — every number in the RFC was taken, not reasoned about: - A writable root bind (`--bind` instead of `--ro-bind`) works unprivileged. - `CapEff` is `0000000000000000`: `mount -t tmpfs` and `mknod` are both EPERM, so the sandbox can mount nothing itself and every mount must stand in bwrap's argv. **That** is what "no layers" was ever about. - Plain `apt-get update` dies on apt's `_apt` user (`seteuid 42 failed`, `Method http has died unexpectedly`); `APT::Sandbox::User root` fixes it. - `apt-get install nginx` still fails (`chown: changing ownership of '/var/log/nginx': Invalid argument`, dpkg exit 100). `fakeroot` **alone does not help** — libfakeroot is demonstrably active and its `chown` wrapper still reports the real syscall's EINVAL. `fakeroot` plus `FAKEROOTDONTTRYCHOWN=1` does. - Overlay works unprivileged on bwrap 0.11, `--overlay-src` stacks, and the upper directory is already a layer diff: changed files as themselves, deletions as character devices 0:0 — what OCI writes as `.wh.NAME`. Two snags: the workdir's `work/` is left at `d---------`, and bwrap refuses overlay when installed setuid. - With a writable root, bwrap creates a missing bind mountpoint itself and it stays in the tree — so a build binds only `/etc/resolv.conf`, which base images already have. ## The interface (the point of the RFC) `build.Backend` / `build.Tree` / `build.Step`. The boundary is exactly "how does a writable tree over a base come into being, and how is its result harvested"; no caller ever sets a `RunSpec`'s root fields. Everything else — parser, instructions, COPY, multi-stage, CLI, uid levers, commit — is identical in both worlds. - **Copy backend** (implemented): a real full copy of the base, root bound read-write, commit is the `rename`, no cache. It stays after the Debian 13 upgrade, as the fallback `doctor` picks when bwrap is too old or setuid, and for single-shot builds. - **Overlay backend** (deliberately *not* implemented): designed, with the measurements above as the evidence that it fits. The per-step cache waits for it, because without a union mount a cached step costs a whole tree copy. The warning that belongs with this, in the RFC and as a comment in the code: `store.copyTree` hardlinks so a second tag costs no data — there both copies are read-only. For a build tree it is unusable, because a `RUN` step writing in place through such a hardlink would modify the base image. ## Stage 1, implemented `werkdock build [-f Dockerfile] -t IMAGE CONTEXT` with docker's flags. `FROM`, `RUN`, `COPY` plus the metadata instructions `import -c` already knew, through the same `store.ApplyChange`. Staging and commit reuse the mechanics `import` has had since 0.11.0 (`<image>.tmp`, the per-image `flock`, `rename`), now exported as `store.Staging`. Werkdock sets both uid levers itself so the Dockerfile stays portable; the base image needs `fakeroot` installed, and `build` says so up front when it has not. `engine.RunSpec` gained `WritableRoot`, and a writable-root sandbox gets **no** implicit tmpfs over `/tmp`, `/root`, `/run` — with one, what a `RUN` step writes there would vanish in the next step while docker keeps it in the image. Refused loudly rather than ignored: `EXPOSE`, `HEALTHCHECK`, `STOPSIGNAL`, `ONBUILD`, `MAINTAINER` at parse time; `ARG`, `SHELL`, `ADD` and multi-stage named as stages 2 and 3; BuildKit's `RUN --mount` with the CapEff measurement as the reason; and the dozen `docker build` flags werkdock cannot keep the promise of. ## Verified - `go test ./...`, `go vet ./...`, `gofmt -l .` — green and empty. - A real sandbox integration test builds `FROM`/`WORKDIR`/`COPY`/`RUN`/`CMD` and then *runs* the result; it skips without bwrap/userns like the existing ones. - The apt path, which the levers exist for, is a second integration test pointed at a Debian base (`WERKDOCK_TEST_BASE_ROOTFS`). Run here: `nginx-common` configures, `dpkg-query` in the built image reports `install ok installed`, the apt fragment is gone from the result. - End to end through the CLI: `import` a 143 MB `debian:trixie` base with fakeroot, `build -t probe:1` with `apt-get install -y nginx jq` in the Dockerfile — **6.8 s** — then `run --rm probe:1` prints what its `RUN` steps and its `COPY` produced. Follow-ups the RFC names without deciding: store dedup between image versions, keeping layer blobs as a transport and cache format (the one with the highest running cost today), `doctor IMAGE`, and `ADD` from a URL. 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Co-authored-by: claude-code <claude-code@javagil.de> Co-authored-by: mhoennig <michael@hoennig.de> Reviewed-on: #14
179 lines
5.4 KiB
Go
179 lines
5.4 KiB
Go
package build
|
|
|
|
import (
|
|
"encoding/json"
|
|
"fmt"
|
|
"os"
|
|
"path"
|
|
"path/filepath"
|
|
"sort"
|
|
"strings"
|
|
|
|
"werkdock/internal/rootpath"
|
|
)
|
|
|
|
// copyStep executes one COPY instruction: files from the build context
|
|
// on the host into the work tree. No sandbox is involved — the tree is
|
|
// a plain host directory, and the build is the only writer.
|
|
//
|
|
// The paths are resolved inside the step's writable directory the way
|
|
// the sandbox would see them (rootpath.Resolve), so a symlink in the
|
|
// image cannot make a COPY write outside the tree.
|
|
func (b *builder) copyStep(step Step, instruction Instruction) error {
|
|
sources, dest, err := parseCopy(instruction.Args)
|
|
if err != nil {
|
|
return err
|
|
}
|
|
if !path.IsAbs(dest) {
|
|
workdir := b.config.WorkingDir
|
|
if workdir == "" {
|
|
workdir = "/"
|
|
}
|
|
// Keep a trailing slash: it is what says "this is a directory".
|
|
trailing := strings.HasSuffix(dest, "/")
|
|
dest = path.Join(workdir, dest)
|
|
if trailing {
|
|
dest += "/"
|
|
}
|
|
}
|
|
matched, err := resolveSources(b.opts.Context, sources)
|
|
if err != nil {
|
|
return err
|
|
}
|
|
destIsDir := strings.HasSuffix(dest, "/") || len(matched) > 1
|
|
if !destIsDir && len(matched) == 1 {
|
|
if fi, err := os.Stat(matched[0]); err == nil && fi.IsDir() {
|
|
destIsDir = true
|
|
}
|
|
}
|
|
for _, source := range matched {
|
|
fi, err := os.Stat(source)
|
|
if err != nil {
|
|
return err
|
|
}
|
|
target := dest
|
|
if fi.IsDir() {
|
|
// docker copies a directory's CONTENTS, not the directory.
|
|
host, err := rootpath.Resolve(step.Writable(), target)
|
|
if err != nil {
|
|
return err
|
|
}
|
|
if err := os.MkdirAll(host, 0o755); err != nil {
|
|
return err
|
|
}
|
|
if err := copyTree(source, host); err != nil {
|
|
return err
|
|
}
|
|
continue
|
|
}
|
|
if destIsDir {
|
|
target = path.Join(target, filepath.Base(source))
|
|
}
|
|
host, err := rootpath.Resolve(step.Writable(), target)
|
|
if err != nil {
|
|
return err
|
|
}
|
|
if err := os.MkdirAll(filepath.Dir(host), 0o755); err != nil {
|
|
return err
|
|
}
|
|
if err := copyFile(source, host, fi.Mode().Perm()); err != nil {
|
|
return err
|
|
}
|
|
}
|
|
return nil
|
|
}
|
|
|
|
// parseCopy splits COPY's arguments into sources and destination. The
|
|
// JSON array form (for paths with spaces) is accepted as docker's is.
|
|
// The flags are refused with their reason rather than ignored.
|
|
func parseCopy(args string) ([]string, string, error) {
|
|
fields, err := copyFields(args)
|
|
if err != nil {
|
|
return nil, "", err
|
|
}
|
|
var paths []string
|
|
for _, field := range fields {
|
|
if !strings.HasPrefix(field, "--") {
|
|
paths = append(paths, field)
|
|
continue
|
|
}
|
|
name, _, _ := strings.Cut(strings.TrimPrefix(field, "--"), "=")
|
|
switch name {
|
|
case "from":
|
|
return nil, "", fmt.Errorf("COPY --from is multi-stage, which is stage 3 of RFC 0006")
|
|
case "chown", "chmod":
|
|
return nil, "", fmt.Errorf("COPY --%s is not supported: the sandbox maps one uid, and ownership is "+
|
|
"dropped when an image is unpacked anyway (RFC 0005)", name)
|
|
default:
|
|
return nil, "", fmt.Errorf("COPY --%s is not supported", name)
|
|
}
|
|
}
|
|
if len(paths) < 2 {
|
|
return nil, "", fmt.Errorf("COPY needs at least one source and a destination")
|
|
}
|
|
return paths[:len(paths)-1], paths[len(paths)-1], nil
|
|
}
|
|
|
|
// copyFields reads COPY's arguments: docker's JSON array form, or
|
|
// whitespace-separated words.
|
|
func copyFields(args string) ([]string, error) {
|
|
if !strings.HasPrefix(args, "[") {
|
|
return strings.Fields(args), nil
|
|
}
|
|
var fields []string
|
|
if err := json.Unmarshal([]byte(args), &fields); err != nil {
|
|
return nil, fmt.Errorf("the JSON form of COPY must be an array of strings: %w", err)
|
|
}
|
|
return fields, nil
|
|
}
|
|
|
|
// resolveSources maps COPY's sources — relative to the build context,
|
|
// globs allowed — to host paths, and refuses anything outside the
|
|
// context: a Dockerfile may not read the host beyond what was handed
|
|
// to the build, not even through a symlink in the context, e.g.
|
|
// leak -> ~/.ssh/id_rsa. A symlink within the context counts as its
|
|
// target, as docker copies it.
|
|
func resolveSources(context string, sources []string) ([]string, error) {
|
|
// the walk below follows symlinks, so the check compares the real paths
|
|
realContext, err := filepath.EvalSymlinks(context)
|
|
if err != nil {
|
|
return nil, fmt.Errorf("the build context %s: %w", context, err)
|
|
}
|
|
var result []string
|
|
for _, source := range sources {
|
|
if path.IsAbs(source) {
|
|
return nil, fmt.Errorf("COPY source %s must be relative to the build context", source)
|
|
}
|
|
candidate := filepath.Join(context, filepath.FromSlash(source))
|
|
if !within(context, candidate) {
|
|
return nil, fmt.Errorf("COPY source %s leaves the build context", source)
|
|
}
|
|
matches, err := filepath.Glob(candidate)
|
|
if err != nil {
|
|
return nil, fmt.Errorf("COPY source %s: %w", source, err)
|
|
}
|
|
if len(matches) == 0 {
|
|
return nil, fmt.Errorf("COPY source %s: no such file or directory in the build context", source)
|
|
}
|
|
sort.Strings(matches)
|
|
for _, match := range matches {
|
|
real, err := filepath.EvalSymlinks(match)
|
|
if err != nil {
|
|
return nil, fmt.Errorf("COPY source %s: %w", source, err)
|
|
}
|
|
if !within(context, match) || !within(realContext, real) {
|
|
return nil, fmt.Errorf("COPY source %s leaves the build context", source)
|
|
}
|
|
// the name stays the link's, as docker names the copy
|
|
result = append(result, match)
|
|
}
|
|
}
|
|
return result, nil
|
|
}
|
|
|
|
func within(root, candidate string) bool {
|
|
root = filepath.Clean(root)
|
|
candidate = filepath.Clean(candidate)
|
|
return candidate == root || strings.HasPrefix(candidate, root+string(filepath.Separator))
|
|
}
|