werkdock/docs/rfcs/0005-services-and-testcontainers.md

# RFC 0005: Service Images, Instance Lifecycle, and the Road to Testcontainers

**Status:**
- proposed: 2026-09-10
- accepted: 2026-09-10
- rejected: -

**Proposal:** Werkdock grows from a build sandbox into a runner for service images (PostgreSQL first), in four milestones that each pay for themselves: (1) a non-root uid inside the sandbox and the image's `VOLUME`s, (2) an instance lifecycle (`create`, `start`, `stop`, `rm`, `ps`, `logs`, `exec`) as thin verbs over one internal service with state on disk, (3) that same service on a unix socket speaking the Docker Engine API subset Testcontainers uses, (4) Testcontainers validated against it with `PostgreSQLContainer`. Where Docker and Podman differ, werkdock follows Podman.

## Context and Problem Statement

The goal (2026-09-10): Testcontainers — or a lightly adapted Testcontainers — shall work against werkdock, so that integration tests in the sandboxed builds get a real PostgreSQL. RFC 0002 designed for this as "level 3" and deferred it; `pull` (0.3.0) closed level 2, so the image side is done: `werkdock pull postgres:16` lands with entrypoint, env, and volumes intact.

What is missing is everything a *service* needs and a *build* does not. Checked against what the official `postgres` image does at start:

1. **A non-root uid.** `initdb` and `postgres` refuse `geteuid() == 0`, real or namespace-internal. Werkdock maps the caller to uid 0 by contract (RFC 0002). The image's entrypoint is written for "runs as any non-root uid" and then skips its `chown`/`gosu` dance — so one non-zero uid suffices, which is all an unprivileged user namespace can map anyway.
2. **Writable places the image declares.** `/var/lib/postgresql/data` and `/var/run/postgresql` are `VOLUME`s in the image; docker and podman mount anonymous volumes there. Werkdock's rootfs is read-only; today the caller must know to add `-v` and `--tmpfs`.
3. **A long-lived process.** Only `run --rm` in the foreground exists. A server needs `-d`, and with it instances that outlive the CLI call, a way to stop them, and their output on disk for `logs`.
4. **The API.** Testcontainers never invokes a CLI; it speaks the Engine API over `DOCKER_HOST`. Podman reaches Testcontainers exactly this way (`podman system service`), with a socket that is optional and may exit when idle — a good model for a daemonless tool.

The network stays the host's (RFC 0002): a service binds a host port directly, and two instances wanting the same port collide. That limit is accepted here and worked around per image where it hurts (see Ports below); lifting it needs a network namespace and its own RFC.

## Principles

- **Podman before Docker.** Werkdock is rootless and daemonless, as Podman is. Where the two differ in flags, defaults, or API details, werkdock takes Podman's side and says so in the CHANGELOG: `--userns=keep-id` for "run as myself", fully qualified image names preferred over silent Docker Hub defaults, `--mount type=image` in Podman's spelling, the API server as an optional frontend that may exit. Where Podman serves the Docker Engine API unchanged, that API is the target — it is what Testcontainers speaks.
- **One uid.** An unprivileged user namespace maps the caller to exactly one uid inside. `--user` chooses which; there is no second user, no `setuid`, no `chown` to another owner. Images that need two users (gosu-style entrypoints as root) are out of scope; the official service images support the single-uid mode.
- **Verbs are frontends.** Lifecycle logic lives in one internal service with instance state on disk (RFC 0002). The CLI and the API server call it; neither owns it. This is what keeps milestone 3 small.
- **Every milestone stands alone.** Non-root plus volumes lets a build start PostgreSQL in the foreground beside its tests, without any lifecycle. The lifecycle serves long-running builds and manual use without any API. The API serves Testcontainers.
- **Filesystem-only stays.** Nothing here adds network, resource, or device isolation; refusals stay refusals.

## Milestones

### 1. Non-root uid and image volumes (this branch, first)

- `--user UID[:GID]` (numeric, or a name resolved in the image's `/etc/passwd`) maps the caller to that uid/gid inside; `--userns=keep-id` (Podman) maps the caller to the caller's own uid/gid. Default stays uid 0, as it is for Podman rootless. An image `User` (OCI config) applies when no flag is given, as podman does.
- `HOME` follows the image's passwd entry for the uid when there is one, else `/root`. `/root` stays a tmpfs, so a nameless uid still has a writable home.
- `/run` joins `/tmp` and `/root` as an implicit tmpfs: runtime state by FHS definition, `/var/run`'s target in practically every image, and where services write without declaring a `VOLUME` (`postgres` fails on `/var/run/postgresql` otherwise). A writable instance layer — docker's container layer — is not this milestone's answer: without overlayfs it would be a rootfs copy per run.
- `VOLUME` paths from the image config get an anonymous tmpfs unless a `-v`/`--mount` covers them — docker's anonymous volume, ephemeral like everything under `--rm`. `import -c 'VOLUME /data'` and `USER` join the accepted instructions; `inspect` shows both.
- Proof: `werkdock pull postgres:16-alpine`, then one `run --rm --user postgres postgres:16-alpine sh -c 'initdb … && pg_ctl start && psql -c "select 1" && pg_ctl stop'`, locally and on the webspace. Done 2026-09-10 (0.4.0).
- Finding: `--userns=keep-id` is not enough for PostgreSQL — `initdb` looks its uid up in `/etc/passwd` and the image has no entry for the caller's uid (`could not look up effective user ID 1000`). Podman has the same limit; `--user postgres` is the answer for images that ship the user, an `/etc/passwd` bind for the rest.

### 2. Instance lifecycle

- An instance is a directory under `$WERKDOCK_HOME/instances/<id>/` with `config.json` (the RunSpec plus name, image, created time, `--rm`; `version: 1`), `state.json` (status, supervisor pid, the command's host pid, started, finished, exit code, error), and `stdout`/`stderr` files. Naming follows docker: 64-hex id, `--name`, else the image's basename plus six characters of the id; verbs take an id, a unique prefix, or a name.
- `run -d` and `create`/`start` fork bwrap under a small supervisor process (`werkdock supervise ID`, this binary in a session of its own) that writes state, records the exit code, and removes an `--rm` instance at the end; `ps`, `logs [-f]`, `wait`, `rm`, `inspect` read the files. `stop` is SIGTERM to the command, SIGKILL after `-t` seconds; `kill` any signal. Neither talks to the supervisor: the command's host pid is in `state.json`, and a state that says running while the supervisor is gone (reboot, `kill -9`) is repaired on the next read as exited 255 — the same 255 docker reports.
- The command is pid 1 of the sandbox (`bwrap --as-pid-1`), as in docker: a signal from outside reaches it if it handles the signal, and bwrap reports its host pid on `--info-fd`. bwrap's own init would swallow SIGTERM (measured: SIGTERM to bwrap's pid-1 init is ignored; to the command under `--as-pid-1` it arrives). Consequence, also docker's: an entrypoint without a SIGTERM handler (`sh -c`) stops only by the SIGKILL after the timeout, and orphaned children are the command's to reap.
- `/run` and the anonymous `VOLUME`s are directories of the instance (`run/`, `volumes/<path>`), bound read-write, not tmpfs: so a socket in `/var/run/postgresql` and the data in `/var/lib/postgresql/data` survive `stop`/`start` (docker's anonymous volumes live as long as the container) and are visible to `exec`. `/tmp` and `/root` stay tmpfs per bwrap invocation.
- `exec` is a second bwrap over the same instance: rootfs, mounts (including `/run` and the volumes), uid, and environment, with `-e`/`-w`/`-u` overrides — not the same pid namespace, so `psql` works over the socket or TCP, `kill` and `ps` do not see the service. A detached exec is refused: it would have no instance to record it.
- `--rm` keeps working as today for foreground runs (the instance exists while the command runs, so `ps` and `stop` see it, and no logs are recorded); a foreground run without `--rm` records its output and stays for `ps -a`, `logs`, `start`. The first SIGINT/SIGTERM/SIGHUP a foreground `run` receives is forwarded to the command, a second one or ten seconds without exit kills it; bwrap runs in its own process group so a terminal's Ctrl-C reaches werkdock, not bwrap.
- Proof: `run -d --name pg --user postgres -e PGPORT=… postgres:16-alpine`, `exec pg psql -U postgres` over the unix socket, `stop`, `start`, data still there, `rm`. Done 2026-09-10 (0.5.0), locally and on the webspace.

### 3. API socket

- `werkdock system service [--time D] [unix://PATH]` (Podman's verb) serves, with `net/http`, on `$XDG_RUNTIME_DIR/werkdock.sock`. The endpoints are those a real Testcontainers run sends, measured rather than taken from the API documentation and written down in [docs/testcontainers-api.md](../testcontainers-api.md): `/info`, `/version`, `/_ping`, images list/inspect/create, containers create/start/json/logs/kill/stop/wait/delete/list, exec create/start/inspect, and the three prune calls. Unimplemented endpoints answer 501 with the reason, never 404.
- Findings from the measurement, each of which removed work or risk:
  - `POST /exec/{id}/start` is a plain request with a framed response body, not an HTTP upgrade. No connection hijacking is needed.
  - `logs` and `exec` use docker's frame format: an 8-byte header per chunk, the stream number first, the length big-endian in the last four bytes.
  - A container is created with an empty `HostPort`, meaning "choose a free one", and the answer is read back from `NetworkSettings.Ports`.
  - Testcontainers refuses a server below Docker 1.6.0, so `/version` must report the *Engine* version whose API is served, not werkdock's 0.x. Werkdock's own version goes into `Components` and `/info`'s `OperatingSystem`.
  - The session cleanup lists containers filtered by label, so instances carry labels and the list filters on them.
- Ryuk stays disabled (`TESTCONTAINERS_RYUK_DISABLED=true`): it wants the API socket bind-mounted into a container of its own, which werkdock does not offer. Werkdock's `--rm` and `rm` cover what Ryuk would.
- Done 2026-09-10 (0.6.0).

### 4. Testcontainers

- Validated with `PostgreSQLContainer` (Testcontainers 1.19.3) under `DOCKER_HOST=unix://…/werkdock.sock` and `TESTCONTAINERS_RYUK_DISABLED=true`: the container starts, the log wait strategy passes, JDBC connects and queries, `execInContainer` returns output and an exit code, and the session cleanup finds and removes what it made. The "adapted" Testcontainers turned out to be configuration only — one environment variable — as hoped.
- Done 2026-09-10 (0.6.0), on the developer machine and on the Hostsharing managed webspace, the environment that had to work.

### Ports

Testcontainers asks for the host port mapped to a container port; without a network namespace werkdock can map nothing. Measured on the target, the identity answer is not merely imprecise but wrong: the webspace's own PostgreSQL holds 5432, so a container told to publish it cannot bind at all. The API server therefore picks a free port and passes it *to the service*, through the environment variable the image honours (`PGPORT`, `MARIADB_TCP_PORT`, `MYSQL_TCP_PORT`, `REDIS_PORT`), and reports that port. This per-image knowledge lives in a small table (package api, `quirks`), which also carries the user an image needs when its entrypoint expects to drop privileges itself, and is the deliberate deviation from docker. An image outside the table keeps its own port, which is correct as long as one instance of it runs at a time; a request to move such a port fails with the reason rather than reporting a port nothing listens on.

## Consequences

- RFC 0002's `--user` refusal ends; the "always uid 0" wording in README and usage becomes "uid 0 by default".
- `USER` and `VOLUME` are no longer refused by `import -c`; existing images are unaffected (zero values).
- The instance directory format (milestone 2) is the contract between CLI and API server and is versioned in `config.json`.
- No change to the store, to `pull`/`load`, or to Werkator's build path.

## Considered Alternatives

- **Adapting Testcontainers to call the werkdock CLI**: rejected — Testcontainers core is welded to docker-java and the Engine API; the API subset is smaller than a fork.
- **PostgreSQL only, without a lifecycle** (start it inside the build's own `run`): kept as what milestone 1 already enables, not as the end state — Testcontainers needs the API regardless.
- **Network namespaces now** (pasta/slirp4netns, like rootless Podman): the honest fix for port collisions, but a second isolation dimension with host prerequisites (`/dev/net/tun`, a userspace stack binary) that the webspaces do not offer; own RFC when the identity mapping proves insufficient.

## Decision Outcome

Accepted — all four milestones are implemented (0.4.0, 0.5.0, 0.6.0) and Testcontainers runs against werkdock on the target environment. What remains is breadth, not structure: the socket inside a build sandbox for Werkator's own integration tests.

Since then, `jdbc:tc` URLs — Testcontainers as the JDBC driver, the form most projects use — were validated too, with Testcontainers 1.21.4 and a Spring Boot application in Werkprobe. They needed no endpoint beyond the subset, and a JDBC init script (`TC_INITSCRIPT`) travels over the connection rather than through the archive endpoint, so that gap was narrower than it was written down as. It closed in 0.7.0: `PUT /containers/{id}/archive` unpacks into the instance and binds the result in, rather than writing into the shared read-only image, and `withCopyFileToContainer` works.

The quirk table was checked against the real images in 0.8.0, and three of its four entries were wrong: only `postgres` reads an environment variable for its port. `redis`, `mariadb`, and `mysql` take a command-line flag, which the table can now express — and until then werkdock reported ports that nothing listened on. The lesson is the RFC's own: a deviation from docker has to be measured against the thing it deviates for, or it is a guess wearing a table's clothes.