Files

T

miandmhoennig 5c64391f5f Feature/remote authenticated clone (#24 )

Co-authored-by: mhoennig <michael@hoennig.de>
Reviewed-on: #24

2026-09-05 14:06:58 +02:00

19 KiB

Raw Permalink Blame History

Werkator Deployment

This document describes how to run Werkator as a permanent service. The recommended setup is a systemd user service behind an existing reverse proxy. By default Werkator does not manage nginx or TLS certificates itself; it relies on the host's existing web server and certbot. For hosts without one, an opt-in managed nginx/TLS container is available, see Hosts Without a Reverse Proxy.

Prerequisites

Linux with systemd.
Java runtime (JRE 21) — or none, when using the self-contained runtime bundle, see Hosts Without a Java Runtime.
git CLI on the PATH.
docker CLI on the PATH, only if any branch uses docker.enabled (see configuration.md).
A checked-out working tree of the repository to watch, with a remote named origin.

Jar Location Convention

Build the executable jar once:

./gradlew build
ls build/libs/werkator.jar

Copy the jar to a stable path outside any watched repository, by convention ~/bin/werkator.jar:

mkdir -p ~/bin
cp build/libs/werkator.jar ~/bin/werkator.jar

The systemd unit generated below points at the jar that was used to run init --systemd. So always run it via the stable path, not via build/libs/.

Install the Service

Initialize Werkator in the repository to watch (see bootstrapping.md for details):

cd /path/to/repo
java -jar ~/bin/werkator.jar init
# fill in git.account and git.token in .git/werkator/.werkator.yml
# review .werkator.yml

Generate the systemd user unit:

java -jar ~/bin/werkator.jar init --systemd

This writes .git/werkator/werkator-<repo-name>.service, .git/werkator/werkator.env, and the nightly Docker cleanup units (werkator-docker-prune.service/.timer), and prints the install commands:

ln -sf /path/to/repo/.git/werkator/werkator-<repo-name>.service ~/.config/systemd/user/werkator-<repo-name>.service
ln -sf /path/to/repo/.git/werkator/werkator-docker-prune.service ~/.config/systemd/user/werkator-docker-prune.service
ln -sf /path/to/repo/.git/werkator/werkator-docker-prune.timer ~/.config/systemd/user/werkator-docker-prune.timer
systemctl --user daemon-reload
systemctl --user enable --now werkator-<repo-name>.service
systemctl --user enable --now werkator-docker-prune.timer

The unit runs java -jar ~/bin/werkator.jar server with the repository as working directory and Restart=always. The unit name contains the repository name, so several repositories can be served by one host, each with its own service and port.

Nightly Docker Cleanup

The werkator-docker-prune.timer runs docker system prune -af every night at 02:00 (host time), before the usual auto-build slots. It removes stopped containers, unused images, unused networks, and dangling build cache, so nightly builds start from freshly built images. Unlike the legacy cleanup it does not prune volumes — the per-repository Gradle cache volumes survive. The units are host-global (no repository name): with several Werkator instances on one host, every init --systemd generates the same files and the symlinks coincide. On hosts without a docker CLI the service is skipped, not failed (ExecCondition). Persistent=true catches up a missed run after downtime.

Expect a burst of builds right after the very first start: in a fresh clone every origin branch counts as new, so each branch with commits younger than watcher.newBranchMaxAge (default 5d) is built once — one build per branch, executed serially up to builds.maxConcurrent. Lower watcher.newBranchMaxAge before the first start, or enable requirePullRequest, to limit the initial backlog.

User services stop at logout unless lingering is enabled once per user:

loginctl enable-linger "$USER"

Serving Several Repositories

One instance serves a set of repositories (ADR 0009): one service, one port, one UI, one watcher loop, one control token. Adding a repository is editing a registry entry — never a data migration, because everything repository-specific already lives inside the repository (.git/werkator/: machine config with secrets, build results, auto-build slots, worktrees).

Clone it on the host, next to the ones already served:

git clone https://github.com/<owner>/<repo>.git ~/repos/<repo>

Prepare it like any watched repository — the machine config with the git credentials, and the sandbox policy if the host needs one:
```
cd ~/repos/<repo> && java -jar ~/bin/werkator.jar init
# fill in git.account and git.token in .git/werkator/.werkator.yml
```
Credentials shared by every repository of the same forge can live once in the instance file's defaults block instead (docs/configuration.md); they merge below each repository's own layers, so a repository may still override them. The repository's own .werkator.yml — its builds — is committed and comes with the clone.
Register it in ~/.werkator.yml of the user running the service:
```
repositories:
  - path: ~/repos/werkator
  - path: ~/repos/<repo>
    name: <short-name>      # optional; default is the directory basename
```
The name is the route segment (/repos/<name>/…) and the UI's switcher entry, so it must be unique: a duplicate aborts the start naming this file, and so does an entry that is no git repository. A repository whose configuration Werkator must not read (a version violation) is skipped with an error — the others keep building.

Steps 1 and 2 are mechanical and can be done from the workstation:
```
tools/remote --env-file .env.<instance> werkator repo-add https://github.com/<owner>/<repo>.git [<name>]
```
It clones the repository next to the ones already served, runs init in it, and prints the registry entry. It does not write ~/.werkator.yml: that file is the instance's own — port, global concurrency, possibly shared credentials — and a script editing it in place would rewrite the operator's configuration behind their back. Cloning and initialising is mechanical; registering is a decision. A private https origin authenticates with the shared defaults.git.account/defaults.git.token of ~/.werkator.yml (the token travels via a one-shot GIT_ASKPASS on the host, never in a URL or process list); enter those once before cloning a private repository — without them only public origins clone.
Restart the service; startup recovery re-enqueues what was in flight:
```
systemctl --user restart werkator-<repo-name>.service
```

The service unit keeps the name it was installed with — it is the instance's unit, not one repository's. With one registered repository every URL stays what it was; with several, the pages and the API carry the repository (/repos/<name>/…, /api/repos/<name>/…), the unscoped paths keep meaning the repository the instance was started in, and the navigation shows a switcher. The watcher polls every repository in its own guard: an unreachable origin is that repository's report in the health banner, and the others are polled regardless. executor.maxConcurrent is the global cap across all of them, and builds take slots in enqueue order (FIFO).

Operating the Service

systemctl --user status werkator-<repo-name>.service     # state and last log lines
journalctl --user -u werkator-<repo-name>.service -f     # follow the log
systemctl --user restart werkator-<repo-name>.service    # restart (e.g. after config changes)
systemctl --user stop werkator-<repo-name>.service       # stop

Updating an Existing Installation

A restart is safe at any time: an in-flight build is recorded as INTERRUPTED and re-enqueued by the startup recovery. Waiting for an idle queue is still nicer, because the interrupted build starts over from scratch.

curl -s http://127.0.0.1:18080/api/builds/current      # ideally [] — nothing running

With a jar installation:

./gradlew build                                        # on the dev machine
scp build/libs/werkator.jar <user>@<host>:~/bin/werkator.jar.new
ssh <user>@<host>
  systemctl --user stop werkator-<repo-name>.service
  mv ~/bin/werkator.jar ~/bin/werkator.jar.bak         # rollback copy
  mv ~/bin/werkator.jar.new ~/bin/werkator.jar
  systemctl --user start werkator-<repo-name>.service
  systemctl --user is-active werkator-<repo-name>.service

With a runtime bundle (hosts without Java, see below):

./gradlew runtimeBundle                                # on the dev machine
scp build/distributions/werkator-runtime-linux-x64.tar.gz <user>@<host>:/tmp/werkator-new.tar.gz
ssh <user>@<host>
  systemctl --user stop werkator-<repo-name>.service
  mv ~/opt/werkator ~/opt/werkator.bak                 # rollback copy
  tar xzf /tmp/werkator-new.tar.gz -C /tmp/ && mv /tmp/werkator ~/opt/werkator
  ~/opt/werkator/bin/werkator --version                # expected: the new version
  systemctl --user start werkator-<repo-name>.service
  systemctl --user is-active werkator-<repo-name>.service
  rm -f /tmp/werkator-new.tar.gz

The tarball unpacks to a werkator/ directory, so it must not be extracted over ~/opt directly — unpack it in /tmp and move it into place, as above. Rollback is the reverse: stop, remove the new directory (or jar), move .bak back, start.

tools/remote --env-file .env.<instance> werkator instance-update does the same sequence for any host, not only the webspace layout it was written for. Three optional keys in the env file name what differs (see the script's header): WERKATOR_REPO_DIR (directory of the watched repository, which also names the systemd unit), WERKATOR_INSTALL_DIR (where the runtime bundle is unpacked), and WERKATOR_SANDBOX (werkdock, the default, or docker — a Docker host has no werkdock binary and no rootfs archive to upload). Their defaults are the layout instance-install creates, so an env file that names none of them behaves exactly as before. The upload happens before the service is stopped and every artifact is checksum-verified after the transfer, so a dropped connection costs the transfer and not the running service.

Then check https://<public-url>/ for the new version in the footer, and journalctl --user -u werkator-<repo-name>.service -n 50 for a clean start. Config file changes are not needed for an update; new keys take their defaults.

Control Token

Viewing is public by design: build states, logs and artifacts are readable without any login, so they can be linked from Gitea, chats or tickets. That is safe as long as the builds themselves handle no real secrets — Werkator has no per-endpoint gating, so an installation whose build output could contain credentials must stay off the public internet (reverse proxy with access control, or server.bindAddress: 127.0.0.1). Only the three mutating actions — restart, cancel, delete — require the control token from .git/werkator/control-token, a random secret the server generates on first start (mode 0600; delete the file to rotate it).

The token is never embedded in a page. The first time you press one of the control buttons, the browser asks for it once and keeps it in localStorage for that browser; a rejected token is dropped and asked for again. Read it on the host:

cat ~/<repo>/.git/werkator/control-token

For scripts, pass it as a header — it is not accepted as a query parameter, because URLs end up in access logs and browser history:

curl -X POST -H "X-werkator-Token: $(cat .git/werkator/control-token)" \
  "https://ci.example.org/api/builds/restart?branch=main"

Environment File

.git/werkator/werkator.env is loaded by the unit as EnvironmentFile. It only tunes the JVM process, e.g. JAVA_OPTS=-Xmx256m. All Werkator configuration lives in the YAML files described in configuration.md, not in environment variables. init --systemd never overwrites an existing environment file.

Reverse Proxy (nginx)

Bind Werkator to localhost — the default since v0.9.9 — and set the public URL in .werkator.yml:

server:
  bindAddress: 127.0.0.1
  port: 18080
  publicBaseUrl: "https://ci.example.org/"

publicBaseUrl is used for all links posted to Gitea, so it must be the externally reachable URL.

Add a server block to the host's nginx:

server {
    listen 443 ssl;
    server_name ci.example.org;

    ssl_certificate     /etc/letsencrypt/live/ci.example.org/fullchain.pem;
    ssl_certificate_key /etc/letsencrypt/live/ci.example.org/privkey.pem;

    location / {
        proxy_pass http://127.0.0.1:18080;
        proxy_set_header Host $host;
        proxy_set_header X-Forwarded-Proto $scheme;
        proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
    }
}

Obtain and renew the certificate with the host's existing certbot, e.g.:

sudo certbot --nginx -d ci.example.org

This replaces the legacy script's managed nginx/Let's Encrypt Docker container for hosts that have their own web server.

Hosts Without a Java Runtime (Runtime Bundle)

Some hosts provide git and Docker but no Java runtime and no way to install one, e.g. Hostsharing container servers. For these, Werkator ships as a self-contained runtime bundle: a jlink-trimmed JRE, werkator.jar, and a launcher script in one tarball (ADR 0006).

Build the bundle on a Linux x86_64 machine whose glibc is not newer than the target's:

./gradlew runtimeBundle
ls build/distributions/werkator-runtime-linux-x64.tar.gz

Copy and unpack it on the target host, by convention to ~/opt/werkator:

scp build/distributions/werkator-runtime-linux-x64.tar.gz user@host:/tmp/
ssh user@host 'mkdir -p ~/opt && tar -xzf /tmp/werkator-runtime-linux-x64.tar.gz -C ~/opt'

Then use ~/opt/werkator/bin/werkator wherever this document says java -jar ~/bin/werkator.jar:

cd /path/to/repo
~/opt/werkator/bin/werkator init
~/opt/werkator/bin/werkator init --systemd

init --systemd detects the bundle automatically: the generated unit's ExecStart points at the bundle's jre/bin/java and lib/werkator.jar, so the install commands printed by init --systemd work unchanged. JAVA_OPTS from the environment file applies as usual.

To update Werkator, stop the service, unpack the new bundle over ~/opt/werkator, and restart the service.

Hosts Without a Reverse Proxy (Managed nginx/TLS)

Some hosts provide Docker but no root access and no host web server, e.g. Hostsharing managed container environments. For these, Werkator can manage its own nginx+certbot Docker container (ADR 0005). This is opt-in; where a host web server exists, prefer the reverse-proxy setup above.

Enable it in the server section of the configuration:

server:
  port: 18080
  nginx:
    enabled: true
    serverName: ci.example.org
    httpPort: 8080
    httpsPort: 8443
    letsencryptEmail: admin@example.org

On server start, Werkator writes the nginx configuration, starts a labelled nginx container publishing httpPort and httpsPort, obtains a Let's Encrypt certificate via a certbot container (webroot mode), and restarts nginx with the full HTTPS configuration. A renewal check runs daily; certificates and nginx state persist in server.nginx.stateDir across restarts. On shutdown the container is removed. All nginx and certificate failures are non-fatal warnings — the plain HTTP server keeps running without the proxy.

serverName must be a public DNS name pointing at the host, reachable from the internet on port 80/443 (directly or via a port forward to httpPort/httpsPort), otherwise the ACME challenge fails. The nginx container cannot reach localhost of the host, so the proxy upstream defaults to serverName; set server.nginx.upstreamHost if the host is reachable under a different name from inside containers. With the managed nginx, set server.bindAddress: 0.0.0.0 explicitly (or an address reachable from the Docker network) — the default 127.0.0.1 makes Werkator unreachable for the proxy container. See configuration.md for all server.nginx.* keys.

Hostsharing Managed Webspace

The third deployment variant (plan step 21, verified live on a real webspace): no root, no Docker daemon, no own reverse proxy. Werkator runs as a systemd user service on the assigned localhost port ("eigener Serverdienst"), the platform's managed Apache terminates TLS and proxies via .htaccess, and builds run in the bubblewrap sandbox executed by the werkdock CLI (ADR 0008, step 21 session C).

Werkator is never built on the webspace: the runtime bundle and the werkdock binary are built locally and uploaded (ADR 0006). All steps are driven by tools/remote; commands name their role — instance-* manages the installed Werkator, repo-* the repository it watches. Each instance is a pair of files (step 23): a transport env file selected with --env-file (default .env), and a YAML fragment in the configuration schema, named by its WERKATOR_INIT_CONFIG key and installed remotely via werkator init --apply — e.g. .env.mih34 + .env.mih34.yml, both gitignored. The fragment carries the Werkator configuration (server.port, publicBaseUrl, systemd limits, builds.default.werkdock.*); the env file only says where and how to reach the host.

tools/remote --env-file .env.mih34 werkator check-prerequisites  # uploads werkdock, runs its doctor
tools/remote --env-file .env.mih34 werkator instance-install     # upload + unpack the runtime bundle and werkdock
tools/remote --env-file .env.mih34 werkator repo-init            # clone the watched repo, rootfs archive, init --apply
tools/remote --env-file .env.mih34 werkator instance-start       # init --apply --systemd, place .htaccess, enable the unit
tools/remote --env-file .env.mih34 port-forward start            # browser tunnel while no public domain is set up

Layout on the host: the watched repository at $WERKATOR_PATH/werkator/, the unpacked runtime at $WERKATOR_PATH/.werkator/werkator/, the werkdock binary at $WERKATOR_PATH/.werkator/bin/werkdock. That is the default, not a requirement: WERKATOR_REPO_DIR, WERKATOR_INSTALL_DIR and WERKATOR_SANDBOX bend it to an installation that predates the script, see Updating an Existing Installation. The rootfs archive is loaded once per source into werkdock's image store (~/.werkdock), shared by every repository of the user. Fill git.account/git.token in the machine config when the origin is private, and make the user's services survive logout with loginctl enable-linger.

Updates are one command, refused while a build runs (FORCE=1 overrides):

tools/remote --env-file .env.mih34 werkator instance-update

The previous runtime stays as .werkator/werkator.prev for one deployment as the rollback asset. For the few seconds instance-update restarts the service, nothing listens on its port; the generated .htaccess maps that refused connection (ErrorDocument 502/503/504) to a static werkator-maintenance.html placed next to it by instance-start, so the page reads "Werkator is restarting" instead of Apache's default error page or a hung request.

The /system page's disk metric is quota-aware (PR#16): on a Managed Webspace the binding limit is usually the package's group quota, not the free space of the shared host volume, so diskTotalGib there is the quota's soft limit — the info line names it (group quota <package>, hard limit … GiB) instead of showing the host's full disk size.

19 KiB Raw Permalink Blame History