miandmhoennig
dbb6d9e4be
`tools/remote repo-add` installed Werkator's own `builds.deploy` (restartsInstance, unsandboxed) into werkcrappit on mih09. Its `main@deploy` ran on the host with werkcrappit's build command and would have restarted the instance on every green main. - Registry entries get `selfDeploys`; with a registry, `restartsInstance` is refused in every repository whose entry lacks it (repository not served, logged). At most one entry may say it. - `init --apply` repeats; fragments are merged in order. - `tools/remote`: `WERKATOR_SELF_DEPLOY_CONFIG` for the watched repository alone; `repo-add` refuses a shared fragment with `restartsInstance`; a failed fragment upload now aborts. - v1.26.0 **Before merging:** add `selfDeploys: true` to the `werkator` entry of `~/.werkator.yml` on mih09, or the self-deployed v1.26.0 will not serve `werkator`. PR-doc: `docs/prs/2026-10-01-PR#87-self-deploy-only-from-its-repository.md` 🤖 Generated with [Claude Code](https://claude.com/claude-code) Co-authored-by: mhoennig <michael@hoennig.de> Reviewed-on: #87
709 lines
33 KiB
Bash
Executable File
709 lines
33 KiB
Bash
Executable File
#!/usr/bin/env bash
|
|
#
|
|
# Central control script for remote Werkator operations (same pattern as the
|
|
# `remote` scripts in the other repos): the first argument is the repo selector,
|
|
# the second the command. All connection and deployment values come from the
|
|
# `.env` file in the repository root — never as command line parameters.
|
|
#
|
|
# Commands name their role (step 21 session D): `instance-*` manages the
|
|
# BUILDER — the installed Werkator instance and its werkdock sandbox tool —
|
|
# while `repo-*` acts on the BUILT, the repository the instance watches.
|
|
# Werkator is never built on the target: the instance is installed from the
|
|
# locally built runtime bundle (ADR 0006), and builds of the watched
|
|
# repository are the running instance's job (or `bin/werkator build` on the
|
|
# host — the werkator CLI, not this script).
|
|
#
|
|
# Usage (step 23: `--env-file` selects the target instance, default `.env`):
|
|
# tools/remote [--env-file FILE] werkator check-prerequisites werkdock doctor on the host
|
|
# tools/remote [--env-file FILE] werkator instance-install first-time: upload + unpack bundle and werkdock
|
|
# tools/remote [--env-file FILE] werkator instance-update redeploy bundle + werkdock, restart the service
|
|
# tools/remote [--env-file FILE] werkator instance-start apply fragment, Apache proxy, systemd unit
|
|
# tools/remote [--env-file FILE] werkator repo-init clone the watched repo, init --apply, rootfs
|
|
# tools/remote [--env-file FILE] werkator repo-add URL [NAME] clone and init ANOTHER repository for the
|
|
# registry, then print the entry to add to ~/.werkator.yml
|
|
# tools/remote [--env-file FILE] werkator control-token
|
|
# tools/remote [--env-file FILE] port-forward start background tunnel to the Werkator UI
|
|
# tools/remote [--env-file FILE] port-forward stop
|
|
#
|
|
# The env file carries TRANSPORT values only; everything that is Werkator
|
|
# configuration travels as a YAML fragment in the config schema, named by
|
|
# WERKATOR_INIT_CONFIG and installed remotely via `werkator init --apply`
|
|
# (docs/plan/23-init-owns-the-files.md). Pair the files per instance, e.g.
|
|
# `.env.mih34` + `.env.mih34.yml` (both gitignored). A self-deploying instance
|
|
# keeps its deployment block in a second fragment, WERKATOR_SELF_DEPLOY_CONFIG,
|
|
# which only the watched repository gets — never one added by repo-add (PR#87).
|
|
#
|
|
# Required in the env file:
|
|
# WERKATOR_REMOTE user@host to operate on, e.g. mih34-werkator@mih34.hostsharing.net
|
|
# WERKATOR_PATH target directory on that host, e.g. /home/storage/mih34/users/werkator
|
|
#
|
|
# Required for `instance-start`:
|
|
# WERKATOR_DOMAIN the domain served by the managed Apache (docroot location for the
|
|
# generated .htaccess); the port lives in the fragment (server.port)
|
|
#
|
|
# Required for `port-forward`:
|
|
# WERKATOR_LOCAL_PORT the local port the browser uses
|
|
# Optional in the env file:
|
|
# WERKATOR_INIT_CONFIG the init fragment every repository gets (repo-init, instance-start,
|
|
# repo-add): the sandbox policy, never a restartsInstance build
|
|
# WERKATOR_SELF_DEPLOY_CONFIG
|
|
# the fragment applied on top for the watched repository alone
|
|
# (repo-init, instance-start): the build that deploys the instance;
|
|
# its registry entry in ~/.werkator.yml needs selfDeploys: true
|
|
# WERKATOR_REPO_URL https clone URL of the watched repository
|
|
# (default: https://git.javagil.de/mi/werkator.git)
|
|
# WERKATOR_REPO_DIR directory of the watched repository, absolute or relative to
|
|
# WERKATOR_PATH (default: werkator); it also names the systemd
|
|
# unit, exactly as `init --systemd` derives it
|
|
# WERKATOR_INSTALL_DIR directory holding the unpacked runtime bundle, absolute or
|
|
# relative to WERKATOR_PATH (default: .werkator)
|
|
# WERKATOR_SANDBOX build runtime of the host: werkdock (default, the bubblewrap
|
|
# sandbox; `bwrap` is accepted as its former name) or docker —
|
|
# a docker host needs neither the werkdock binary nor a rootfs
|
|
# WERKDOCK_REPO checkout of the werkdock repository, whose binary the
|
|
# instance runs (default: <repo>/../werkdock); its binary is
|
|
# rebuilt whenever a source there is newer than it, so what
|
|
# is uploaded matches what is checked out
|
|
# WERKDOCK_BINARY the built werkdock binary (default: $WERKDOCK_REPO/dist/werkdock);
|
|
# pointed anywhere else it is taken as given and never rebuilt
|
|
# WERKATOR_ROOTFS rootfs archive path for repo-init
|
|
# (default: <repo>/build/werkator-buildenv-trixie-java-go-node.tar.zst)
|
|
#
|
|
# Install layout on the host — the default, which the three keys above bend to an
|
|
# installation that predates this script (e.g. the docker host vm4006: the watched
|
|
# repository is ~/hs.hsadmin.ng, the runtime lives in ~/opt, there is no werkdock):
|
|
# $WERKATOR_PATH/werkator/ the watched repository (clone)
|
|
# $WERKATOR_PATH/.werkator/werkator/ the unpacked runtime bundle
|
|
# $WERKATOR_PATH/.werkator/bin/ the werkdock binary and werkator-self-deploy
|
|
# $WERKATOR_PATH/.werkator/*.tar.* uploaded bundle and rootfs archives
|
|
#
|
|
|
|
set -euo pipefail
|
|
|
|
die() { echo "ERROR: $*" >&2; exit 1; }
|
|
|
|
ENV_FILE=""
|
|
if [ "${1:-}" = "--env-file" ]; then
|
|
ENV_FILE="${2:?missing value for --env-file}"
|
|
shift 2
|
|
fi
|
|
|
|
REPO="${1:-}"
|
|
COMMAND="${2:-}"
|
|
|
|
REPO_ROOT="$(cd "$(dirname "${BASH_SOURCE[0]}")/.." && pwd)"
|
|
RUNTIME_BUNDLE="$REPO_ROOT/build/distributions/werkator-runtime-linux-x64.tar.gz"
|
|
# Werkdock lives in its own repository since step 21 session E. Its binary is
|
|
# built there, not here: WERKDOCK_REPO names the checkout (default: a sibling
|
|
# of this repository), WERKDOCK_BINARY the built binary within it — kept in step
|
|
# with the checkout by ensure_werkdock_binary.
|
|
WERKDOCK_REPO="${WERKDOCK_REPO:-$REPO_ROOT/../werkdock}"
|
|
WERKDOCK_BINARY="${WERKDOCK_BINARY:-$WERKDOCK_REPO/dist/werkdock}"
|
|
PID_FILE="/tmp/werkator-port-forward-$(id -u).pid"
|
|
LOG_FILE="/tmp/werkator-port-forward-$(id -u).log"
|
|
|
|
usage() {
|
|
awk 'NR > 2 && !/^#/ { exit } NR > 2 { sub(/^# ?/, ""); print }' "${BASH_SOURCE[0]}"
|
|
exit 2
|
|
}
|
|
|
|
require_env() {
|
|
local missing=0
|
|
for name in "$@"; do
|
|
if [ -z "${!name:-}" ]; then
|
|
echo "ERROR: $name is not set — define it in $ENV_FILE" >&2
|
|
missing=1
|
|
fi
|
|
done
|
|
[ "$missing" -eq 0 ] || exit 1
|
|
}
|
|
|
|
[ -n "$REPO" ] && [ -n "$COMMAND" ] || usage
|
|
|
|
# Load the transport values. The selected env file (default: the .env in the
|
|
# repository root) wins over the environment — `set -a; source` assigns
|
|
# unconditionally; pick the target with --env-file rather than by exporting
|
|
# single values.
|
|
ENV_FILE="${ENV_FILE:-$REPO_ROOT/.env}"
|
|
set -a
|
|
[ -f "$ENV_FILE" ] && source "$ENV_FILE"
|
|
set +a
|
|
|
|
require_env WERKATOR_REMOTE WERKATOR_PATH
|
|
HOST="$WERKATOR_REMOTE"
|
|
TARGET_DIR="$WERKATOR_PATH"
|
|
ROOTFS="${WERKATOR_ROOTFS:-$REPO_ROOT/build/werkator-buildenv-trixie-java-go-node.tar.zst}"
|
|
REPO_URL="${WERKATOR_REPO_URL:-https://git.javagil.de/mi/werkator.git}"
|
|
|
|
# The host layout is three values, not one convention: an installation that grew
|
|
# before this script existed puts them elsewhere, and the defaults are exactly what
|
|
# `instance-install` creates, so an env file that names none of them behaves as before.
|
|
# Both directories may be absolute; a bare name is taken relative to WERKATOR_PATH.
|
|
resolve_dir() {
|
|
case "$1" in
|
|
/*) echo "$1" ;;
|
|
*) echo "$TARGET_DIR/$1" ;;
|
|
esac
|
|
}
|
|
REPO_DIR="$(resolve_dir "${WERKATOR_REPO_DIR:-werkator}")"
|
|
INSTALL_DIR="$(resolve_dir "${WERKATOR_INSTALL_DIR:-.werkator}")"
|
|
# where `repo-add` puts a further repository of the registry: beside the watched one
|
|
SIBLING_DIR="$(dirname "$REPO_DIR")"
|
|
SANDBOX="${WERKATOR_SANDBOX:-werkdock}"
|
|
# `bwrap` was the name of the config section until Werkator v1.2.0; accepted so an env
|
|
# file written for the older script keeps working, normalised so only one name is used.
|
|
[ "$SANDBOX" = "bwrap" ] && SANDBOX="werkdock"
|
|
case "$SANDBOX" in
|
|
werkdock|docker) ;;
|
|
*) die "WERKATOR_SANDBOX is 'werkdock' or 'docker', not '$SANDBOX'" ;;
|
|
esac
|
|
|
|
MACHINE_CONFIG="$REPO_DIR/.git/werkator/.werkator.yml"
|
|
WERKATOR_BIN="$INSTALL_DIR/werkator/bin/werkator"
|
|
# mirrors SystemdServiceFiles.unitName: the repository's directory name, every
|
|
# character outside [A-Za-z0-9_.-] replaced by a dash — the unit `init --systemd`
|
|
# writes, which is the one this script may stop and start.
|
|
UNIT="werkator-$(basename "$REPO_DIR" | sed 's/[^A-Za-z0-9_.-]/-/g').service"
|
|
|
|
# One TCP connection per run: every ssh/scp below goes through a shared control
|
|
# master (opened by the first call, kept for ten minutes so a follow-up command
|
|
# reuses it). A Hostsharing host refuses rapid successive SSH connections from one
|
|
# IP ("Connection refused" halfway through instance-update, seen on mih09 on
|
|
# 2026-09-08) — a limit in its firewall, not ours to change — and a plain run of
|
|
# this script opened about ten of them in a few seconds. The control socket lives
|
|
# in the user's runtime dir; %C hashes user, host and port into a short name.
|
|
SSH_MUX_DIR="${XDG_RUNTIME_DIR:-${TMPDIR:-/tmp}}"
|
|
SSH_MUX_OPTS=(-o ControlMaster=auto -o "ControlPath=$SSH_MUX_DIR/werkator-remote-%C" -o ControlPersist=10m)
|
|
ssh() { command ssh "${SSH_MUX_OPTS[@]}" "$@"; }
|
|
scp() { command scp "${SSH_MUX_OPTS[@]}" "$@"; }
|
|
|
|
ssh_present() {
|
|
ssh -o BatchMode=yes -o ConnectTimeout=10 "$HOST" true 2>/dev/null
|
|
}
|
|
|
|
ensure_ssh() {
|
|
if ssh_present; then
|
|
echo "==> SSH access to $HOST: ok"
|
|
else
|
|
echo "==> No key-based SSH access yet; running ssh-copy-id (password prompt expected)"
|
|
ssh-copy-id "$HOST"
|
|
ssh_present || die "SSH access still not working after ssh-copy-id"
|
|
fi
|
|
}
|
|
|
|
# Werkdock owns the host checks (`werkdock doctor` ports the old prerequisites
|
|
# script); the binary is uploaded first, so the check works pre-install.
|
|
# On a docker host there is no werkdock and no sandbox to check: the build runtime
|
|
# is the docker daemon, so the check is that the daemon answers this user.
|
|
check_prerequisites() {
|
|
if [ "$SANDBOX" = "docker" ]; then
|
|
echo "==> Checking the docker build runtime on $HOST (WERKATOR_SANDBOX=docker)"
|
|
ssh "$HOST" "docker info >/dev/null" || die "docker is not usable by this user on $HOST"
|
|
ssh "$HOST" "docker --version"
|
|
return 0
|
|
fi
|
|
ensure_werkdock_binary
|
|
echo "==> Uploading werkdock and running its doctor on $HOST (target dir: $TARGET_DIR)"
|
|
ssh "$HOST" "mkdir -p '$INSTALL_DIR/bin'"
|
|
scp -q "$WERKDOCK_BINARY" "$HOST:$INSTALL_DIR/bin/werkdock.new"
|
|
ssh "$HOST" "mv '$INSTALL_DIR/bin/werkdock.new' '$INSTALL_DIR/bin/werkdock' && chmod 755 '$INSTALL_DIR/bin/werkdock'"
|
|
if ! ssh "$HOST" "'$INSTALL_DIR/bin/werkdock' doctor '$TARGET_DIR'"; then
|
|
die "werkdock doctor failed on $HOST — install aborted"
|
|
fi
|
|
}
|
|
|
|
# Makes sure the binary about to be uploaded was built from the sources that are
|
|
# checked out now (PR#50). It used to build only when the file was missing, so a
|
|
# binary built weeks ago survived every `git pull` in the werkdock checkout and was
|
|
# shipped in silence — that is how an instance came to run werkdock 0.1.0-dev while
|
|
# its checkout had long been at 0.2.0.
|
|
#
|
|
# Rebuilt when a source file is newer than the binary. Only the default path inside
|
|
# the checkout is ever written: a WERKDOCK_BINARY pointed somewhere else is a
|
|
# deliberate choice of a prebuilt binary, and overwriting it would be a surprise.
|
|
# Without a checkout at all there is nothing to build from, and the given binary is
|
|
# used as it is. Either way the version is printed, so what goes to the host is on
|
|
# the screen before it gets there.
|
|
ensure_werkdock_binary() {
|
|
local default_binary="$WERKDOCK_REPO/dist/werkdock"
|
|
if [ -f "$WERKDOCK_REPO/go.mod" ] && [ "$WERKDOCK_BINARY" = "$default_binary" ]; then
|
|
if werkdock_binary_is_stale; then
|
|
echo "==> Building werkdock in $WERKDOCK_REPO (go build)"
|
|
(cd "$WERKDOCK_REPO" && CGO_ENABLED=0 go build -o dist/werkdock .) \
|
|
|| die "go build failed in $WERKDOCK_REPO"
|
|
fi
|
|
fi
|
|
[ -f "$WERKDOCK_BINARY" ] || die \
|
|
"werkdock binary missing: $WERKDOCK_BINARY — clone https://git.javagil.de/mi/werkdock.git \
|
|
next to this repository, or point WERKDOCK_REPO/WERKDOCK_BINARY at your checkout"
|
|
echo "==> Using $("$WERKDOCK_BINARY" --version 2>/dev/null || echo "werkdock (version unknown)") from $WERKDOCK_BINARY"
|
|
}
|
|
|
|
# True when the binary is missing or older than any Go source of the checkout.
|
|
# Timestamps, not hashes: a `git pull`, a branch switch and an edit all touch the
|
|
# files they change, which is exactly the set that would produce a different binary.
|
|
werkdock_binary_is_stale() {
|
|
[ -f "$WERKDOCK_BINARY" ] || return 0
|
|
[ -n "$(find "$WERKDOCK_REPO" -name '.git' -prune -o -newer "$WERKDOCK_BINARY" \
|
|
\( -name '*.go' -o -name 'go.mod' -o -name 'go.sum' \) -print -quit)" ]
|
|
}
|
|
|
|
# Uploads one init fragment and echoes its `--apply` option; empty when no
|
|
# fragment is named.
|
|
upload_fragment() {
|
|
local fragment="${1:-}"
|
|
[ -n "$fragment" ] || { echo ""; return 0; }
|
|
[ -f "$fragment" ] || die "init fragment missing: $fragment"
|
|
local remote="$INSTALL_DIR/$(basename "$fragment")"
|
|
scp -q "$fragment" "$HOST:$remote"
|
|
echo "--apply '$remote'"
|
|
}
|
|
|
|
# The `--apply` options for the watched repository: the fragment every
|
|
# repository gets, then the self-deployment on top (init merges them in order).
|
|
watched_repo_fragments() {
|
|
local shared self_deploy
|
|
# command substitutions do not inherit set -e, so a failed upload must exit by hand
|
|
shared="$(upload_fragment "${WERKATOR_INIT_CONFIG:-}")" || exit 1
|
|
self_deploy="$(upload_fragment "${WERKATOR_SELF_DEPLOY_CONFIG:-}")" || exit 1
|
|
echo "$shared $self_deploy"
|
|
}
|
|
|
|
# The `--apply` option for a repository added by repo-add. The shared fragment
|
|
# must not carry a build that restarts the instance: it would run in a foreign
|
|
# repository, on that repository's build command, and restart the instance on
|
|
# its green main (2026-10-01, werkcrappit). Werkator refuses such a repository
|
|
# anyway unless its registry entry says selfDeploys; this stops it before.
|
|
added_repo_fragment() {
|
|
if [ -n "${WERKATOR_INIT_CONFIG:-}" ] &&
|
|
grep -Eq '^[[:space:]]*restartsInstance:[[:space:]]*true' "$WERKATOR_INIT_CONFIG" 2>/dev/null; then
|
|
die "$WERKATOR_INIT_CONFIG sets restartsInstance: true, which repo-add would install into a foreign repository — move that build into WERKATOR_SELF_DEPLOY_CONFIG"
|
|
fi
|
|
upload_fragment "${WERKATOR_INIT_CONFIG:-}"
|
|
}
|
|
|
|
# The instance artifacts are built locally (ADR 0006): the runtime bundle via
|
|
# Gradle, the werkdock binary via the Go toolchain. Both are rebuilt when
|
|
# missing, never on the target.
|
|
ensure_instance_artifacts() {
|
|
if [ ! -f "$RUNTIME_BUNDLE" ]; then
|
|
echo "==> Runtime bundle not found; building it locally (./gradlew runtimeBundle)"
|
|
(cd "$REPO_ROOT" && ./gradlew runtimeBundle --console=plain -q)
|
|
fi
|
|
[ -f "$RUNTIME_BUNDLE" ] || die "runtime bundle missing: $RUNTIME_BUNDLE"
|
|
[ "$SANDBOX" = "docker" ] || ensure_werkdock_binary
|
|
}
|
|
|
|
# Uploads one file and verifies it arrived whole: a transfer that dies mid-way
|
|
# (scp: Connection closed) otherwise leaves a truncated archive that unpacks into
|
|
# a broken runtime. Retries twice, because a dropped WAN connection is not a reason
|
|
# to abort a deployment.
|
|
upload_verified() {
|
|
local src="$1" dest="$2"
|
|
local local_sha remote_sha attempt
|
|
local_sha="$(sha256sum "$src" | cut -d' ' -f1)"
|
|
remote_sha="$(ssh "$HOST" "sha256sum '$dest' 2>/dev/null | cut -d' ' -f1" || true)"
|
|
if [ "$local_sha" = "$remote_sha" ]; then
|
|
echo " $(basename "$src"): already on the host, skipping"
|
|
return 0
|
|
fi
|
|
for attempt in 1 2 3; do
|
|
if scp -q "$src" "$HOST:$dest.part"; then
|
|
remote_sha="$(ssh "$HOST" "sha256sum '$dest.part' 2>/dev/null | cut -d' ' -f1" || true)"
|
|
if [ "$local_sha" = "$remote_sha" ]; then
|
|
ssh "$HOST" "mv '$dest.part' '$dest'"
|
|
return 0
|
|
fi
|
|
echo " checksum mismatch after transfer $attempt of $(basename "$src")" >&2
|
|
else
|
|
echo " transfer $attempt of $(basename "$src") failed" >&2
|
|
fi
|
|
done
|
|
ssh "$HOST" "rm -f '$dest.part'" || true
|
|
die "cannot upload $src to $HOST:$dest — three attempts failed"
|
|
}
|
|
|
|
# Uploads the instance artifacts, without touching the installed runtime: the
|
|
# service keeps running until swap_instance_runtime replaces it, so a failed
|
|
# transfer costs nothing but the transfer.
|
|
upload_instance_artifacts() {
|
|
if [ "$SANDBOX" = "docker" ]; then
|
|
echo "==> Uploading runtime bundle"
|
|
else
|
|
echo "==> Uploading runtime bundle and werkdock binary"
|
|
fi
|
|
ssh "$HOST" "mkdir -p '$INSTALL_DIR/bin'"
|
|
upload_verified "$RUNTIME_BUNDLE" "$INSTALL_DIR/$(basename "$RUNTIME_BUNDLE")"
|
|
# the swap itself, so that this script and a self-deploying instance install a
|
|
# bundle exactly alike (RFC 0005); it is made executable by swap_instance_runtime
|
|
upload_verified "$REPO_ROOT/tools/self-deploy" "$INSTALL_DIR/bin/werkator-self-deploy"
|
|
if [ "$SANDBOX" != "docker" ]; then
|
|
upload_verified "$WERKDOCK_BINARY" "$INSTALL_DIR/bin/werkdock.new"
|
|
fi
|
|
}
|
|
|
|
# Swaps in the uploaded artifacts. The runtime swap is tools/self-deploy — the
|
|
# same script an instance runs when it deploys itself (RFC 0005), so the two
|
|
# cannot drift apart at the first change of the layout: it unpacks beside the
|
|
# running runtime, proves the new launcher starts on the host, and only then
|
|
# takes its place, leaving the replaced one as werkator.prev. Given both an
|
|
# installation directory and a bundle it does that and nothing else — no restart
|
|
# is scheduled, because this script stops and starts the service itself.
|
|
swap_instance_runtime() {
|
|
echo "==> Unpacking"
|
|
ssh "$HOST" "set -e
|
|
cd '$INSTALL_DIR'
|
|
[ ! -f bin/werkdock.new ] || { mv bin/werkdock.new bin/werkdock && chmod 755 bin/werkdock; }
|
|
chmod 755 bin/werkator-self-deploy
|
|
./bin/werkator-self-deploy --no-restart --install-dir '$INSTALL_DIR' --bundle '$INSTALL_DIR/$(basename "$RUNTIME_BUNDLE")'
|
|
[ ! -x bin/werkdock ] || './bin/werkdock' version"
|
|
}
|
|
|
|
instance_install() {
|
|
ensure_ssh
|
|
check_prerequisites
|
|
ensure_instance_artifacts
|
|
upload_instance_artifacts
|
|
swap_instance_runtime
|
|
echo
|
|
echo "==> Instance installed."
|
|
echo " Runtime: $WERKATOR_BIN"
|
|
[ "$SANDBOX" = "docker" ] || echo " werkdock: $INSTALL_DIR/bin/werkdock"
|
|
echo " Next: tools/remote werkator repo-init, then instance-start"
|
|
}
|
|
|
|
# Refuse to swap the runtime under a running build; FORCE=1 overrides.
|
|
require_idle() {
|
|
local port
|
|
port="$(ssh "$HOST" "cd '$REPO_DIR' 2>/dev/null && '$WERKATOR_BIN' config:print 2>/dev/null" | awk '/^server:/{f=1;next} f && /^ port:/{print $2; exit}' | tr -d '"' || true)"
|
|
[ -n "$port" ] || return 0
|
|
local current
|
|
current="$(ssh "$HOST" "curl -s --max-time 5 http://127.0.0.1:$port/api/builds/current" || true)"
|
|
if [ -n "$current" ] && [ "$current" != "[]" ]; then
|
|
[ "${FORCE:-}" = "1" ] || die "a build is running on $HOST — retry when idle, or FORCE=1 to override"
|
|
echo "==> WARNING: deploying although a build is running (FORCE=1)"
|
|
fi
|
|
}
|
|
|
|
instance_update() {
|
|
ensure_ssh
|
|
ensure_instance_artifacts
|
|
require_idle
|
|
# upload first, stop second: a transfer that fails must not leave the host
|
|
# without a running service (measured on vm4006, 2026-09-03)
|
|
upload_instance_artifacts
|
|
local was_active=0
|
|
if ssh "$HOST" "XDG_RUNTIME_DIR=/run/user/\$(id -u) systemctl --user is-active --quiet '$UNIT'"; then
|
|
was_active=1
|
|
fi
|
|
if [ "$was_active" = "1" ]; then
|
|
echo "==> Stopping $UNIT"
|
|
ssh "$HOST" "XDG_RUNTIME_DIR=/run/user/\$(id -u) systemctl --user stop '$UNIT'"
|
|
fi
|
|
swap_instance_runtime
|
|
if [ "$was_active" = "1" ]; then
|
|
echo "==> Starting $UNIT"
|
|
ssh "$HOST" "XDG_RUNTIME_DIR=/run/user/\$(id -u) systemctl --user start '$UNIT' && sleep 3 && systemctl --user is-active '$UNIT'"
|
|
else
|
|
echo "==> Service was not running; not started (use instance-start for the first start)"
|
|
fi
|
|
echo "==> Instance updated."
|
|
}
|
|
|
|
# Clones one URL into one directory on the host.
|
|
# A private https origin authenticates with the shared `defaults.git.account` /
|
|
# `defaults.git.token` of `~/.werkator.yml` (ADR 0009). The whole authenticated
|
|
# clone runs in one remote python script: the token is read from the instance
|
|
# file on the host and passed to git via a one-shot GIT_ASKPASS script, so it
|
|
# appears in neither the local process list nor a repository config.
|
|
# Public origins (or an instance without shared credentials) clone anonymously.
|
|
clone_repo() {
|
|
local url="$1" dest="$2"
|
|
if ssh "$HOST" "test -d '$dest/.git'"; then
|
|
echo " (already cloned, skipping)"
|
|
return 0
|
|
fi
|
|
case "$url" in
|
|
https://*)
|
|
# The python helper travels base64-encoded: the clone command itself
|
|
# stays a plain `ssh` line, so no `$` inside the script is ever
|
|
# expanded by the local shell, and the token never leaves the host.
|
|
local helper_b64
|
|
helper_b64="$(python3 -c 'import base64,sys; print(base64.b64encode(sys.stdin.read().encode()).decode())' <<'PYEOF_CLONE'
|
|
import os, stat, subprocess, sys, tempfile, urllib.parse
|
|
url, dest = sys.argv[1], sys.argv[2]
|
|
try:
|
|
import yaml
|
|
cfg = yaml.safe_load(open(os.path.expanduser("~/.werkator.yml"))) or {}
|
|
except (FileNotFoundError, ImportError):
|
|
cfg = {}
|
|
d = (cfg.get("defaults") or {}).get("git") or {}
|
|
account, token = d.get("account"), d.get("token")
|
|
env = dict(os.environ, GIT_TERMINAL_PROMPT="0")
|
|
ask = None
|
|
if account and token:
|
|
parts = urllib.parse.urlsplit(url)
|
|
host = parts.netloc.rsplit("@", 1)[-1]
|
|
url = urllib.parse.urlunsplit(parts._replace(netloc=account + "@" + host))
|
|
ask = tempfile.NamedTemporaryFile(mode="w", prefix="werkator-clone-askpass-",
|
|
suffix=".sh", delete=False)
|
|
ask.write("#!/bin/sh\nexec echo \"$WERKATOR_CLONE_TOKEN\"\n")
|
|
ask.close()
|
|
os.chmod(ask.name, stat.S_IRWXU)
|
|
env.update(GIT_ASKPASS=ask.name, WERKATOR_CLONE_TOKEN=token)
|
|
try:
|
|
subprocess.run(["git", "clone", url, dest], env=env, check=True)
|
|
finally:
|
|
if ask is not None:
|
|
os.unlink(ask.name)
|
|
PYEOF_CLONE
|
|
)"
|
|
ssh "$HOST" "echo '$helper_b64' | base64 -d | python3 - '$url' '$dest'"
|
|
;;
|
|
*)
|
|
ssh "$HOST" "git clone '$url' '$dest'"
|
|
;;
|
|
esac
|
|
}
|
|
|
|
# Sets up the WATCHED repository: an https clone (a private origin
|
|
# authenticates with the shared `defaults.git.*` credentials of
|
|
# `~/.werkator.yml`; see `clone_repo`), the werkator init with the instance
|
|
# applied, and the rootfs archive for the sandbox builds. All configuration
|
|
# writing is init's — this script transports and invokes (step 23).
|
|
repo_init() {
|
|
ensure_ssh
|
|
[ "$SANDBOX" = "docker" ] || [ -f "$ROOTFS" ] ||
|
|
die "rootfs archive missing: $ROOTFS — build it with tools/build-bwrap-rootfs.sh or set WERKATOR_ROOTFS"
|
|
ssh "$HOST" "test -x '$WERKATOR_BIN'" || die "no instance on $HOST — run instance-install first"
|
|
|
|
echo "==> Cloning the watched repository"
|
|
clone_repo "$REPO_URL" "$REPO_DIR"
|
|
|
|
if [ "$SANDBOX" = "docker" ]; then
|
|
echo "==> No rootfs needed (WERKATOR_SANDBOX=docker) — the build image is the repository's own Dockerfile"
|
|
else
|
|
echo "==> Uploading the rootfs archive (skipped when unchanged)"
|
|
local rootfs_remote="$INSTALL_DIR/$(basename "$ROOTFS")"
|
|
local local_sha remote_sha
|
|
local_sha="$(sha256sum "$ROOTFS" | cut -d' ' -f1)"
|
|
remote_sha="$(ssh "$HOST" "sha256sum '$rootfs_remote' 2>/dev/null | cut -d' ' -f1" || true)"
|
|
if [ "$local_sha" = "$remote_sha" ]; then
|
|
echo " (already on the host, skipping)"
|
|
else
|
|
scp -q "$ROOTFS" "$HOST:$rootfs_remote"
|
|
remote_sha="$(ssh "$HOST" "sha256sum '$rootfs_remote' | cut -d' ' -f1")"
|
|
[ "$local_sha" = "$remote_sha" ] || die "rootfs upload checksum mismatch"
|
|
fi
|
|
fi
|
|
|
|
echo "==> Running werkator init${WERKATOR_INIT_CONFIG:+ --apply $(basename "${WERKATOR_INIT_CONFIG}")}${WERKATOR_SELF_DEPLOY_CONFIG:+ --apply $(basename "${WERKATOR_SELF_DEPLOY_CONFIG}")}"
|
|
local apply_options
|
|
apply_options="$(watched_repo_fragments)"
|
|
ssh "$HOST" "cd '$REPO_DIR' && '$WERKATOR_BIN' init $apply_options"
|
|
|
|
echo "==> Verifying the effective configuration"
|
|
ssh "$HOST" "cd '$REPO_DIR' && '$WERKATOR_BIN' config:print 2>/dev/null | grep -A4 '$SANDBOX:' | head -5"
|
|
|
|
echo
|
|
echo "==> Repository ready."
|
|
echo " Repo: $REPO_DIR"
|
|
echo " Next: fill git.account/git.token in $MACHINE_CONFIG if the origin is private,"
|
|
if [ -n "${WERKATOR_SELF_DEPLOY_CONFIG:-}" ]; then
|
|
echo " mark this repository's entry in ~/.werkator.yml with selfDeploys: true (if there is a registry),"
|
|
fi
|
|
echo " then tools/remote werkator instance-start"
|
|
}
|
|
|
|
# Prepare a SECOND (third, …) repository for the registry of an installed
|
|
# instance (ADR 0009): clone it next to the others and run `init` in it, so it
|
|
# has its own machine config. The registry entry itself is only PRINTED, never
|
|
# written: `~/.werkator.yml` is the instance's own file — it carries the port,
|
|
# the global concurrency and possibly shared credentials, and a script that
|
|
# edits it in place would rewrite the operator's own configuration behind their
|
|
# back. Cloning and initialising is mechanical, registering is a decision.
|
|
repo_add() {
|
|
local url="${1:-}"
|
|
[ -n "$url" ] || die "usage: tools/remote [--env-file FILE] werkator repo-add <clone-url> [name]"
|
|
local name="${2:-$(basename "$url" .git)}"
|
|
case "$name" in
|
|
*/*|"") die "the repository name is one path segment (it becomes the route segment /repos/<name>)" ;;
|
|
esac
|
|
|
|
ensure_ssh
|
|
ssh "$HOST" "test -x '$WERKATOR_BIN'" || die "no instance on $HOST — run instance-install first"
|
|
|
|
echo "==> Cloning $url as '$name'"
|
|
clone_repo "$url" "$SIBLING_DIR/$name"
|
|
|
|
# The instance fragment carries the sandbox policy (bwrap rootfs and werkdock
|
|
# binary). Without it a watched repository builds on the bare host, where the
|
|
# toolchains are not installed — the same --apply repo-init does.
|
|
# WERKATOR_SELF_DEPLOY_CONFIG is the watched repository's alone (PR#87).
|
|
echo "==> Running werkator init in $name${WERKATOR_INIT_CONFIG:+ --apply $(basename "${WERKATOR_INIT_CONFIG}")}"
|
|
local apply_option
|
|
apply_option="$(added_repo_fragment)"
|
|
ssh "$HOST" "cd '$SIBLING_DIR/$name' && '$WERKATOR_BIN' init $apply_option"
|
|
|
|
echo "==> Checking the registry"
|
|
# Grepped locally: the entry may name the path absolute or as ~/<name>, and
|
|
# matching both is easier without a second layer of remote shell quoting.
|
|
if ssh "$HOST" "cat ~/.werkator.yml 2>/dev/null" |
|
|
grep -qE "path: *(~|$SIBLING_DIR)/$name[[:space:]]*$"; then
|
|
echo " (~/.werkator.yml already names this path)"
|
|
else
|
|
echo " not registered yet — add this entry to ~/.werkator.yml on $HOST:"
|
|
echo
|
|
echo " repositories:"
|
|
echo " - path: $SIBLING_DIR/$name"
|
|
echo " name: $name"
|
|
echo
|
|
fi
|
|
|
|
echo "==> Repository prepared."
|
|
echo " Repo: $SIBLING_DIR/$name"
|
|
echo " Next: fill git.account/git.token in $SIBLING_DIR/$name/.git/werkator/.werkator.yml if the origin is private"
|
|
echo " (or once for all repositories in the 'defaults' block of ~/.werkator.yml),"
|
|
echo " then restart the service — the registry is read at start."
|
|
}
|
|
|
|
# Start the server as a systemd user unit behind the managed Apache. All
|
|
# configuration comes from the instance fragment (server.port, publicBaseUrl,
|
|
# systemd limits); init generates the units AND the .htaccess — this script
|
|
# only places and activates them (step 23).
|
|
instance_start() {
|
|
ensure_ssh
|
|
require_env WERKATOR_DOMAIN
|
|
|
|
echo "==> Applying the instance fragment and generating the host integration (init --systemd)"
|
|
local apply_options
|
|
apply_options="$(watched_repo_fragments)"
|
|
ssh "$HOST" "cd '$REPO_DIR' && '$WERKATOR_BIN' init $apply_options --systemd"
|
|
|
|
local htaccess_src="$REPO_DIR/.git/werkator/werkator.htaccess"
|
|
local htaccess="$TARGET_DIR/doms/$WERKATOR_DOMAIN/subs/www/.htaccess"
|
|
local maintenance_src="$REPO_DIR/.git/werkator/werkator-maintenance.html"
|
|
local maintenance="$TARGET_DIR/doms/$WERKATOR_DOMAIN/subs/www/werkator-maintenance.html"
|
|
if ssh "$HOST" "test -f '$htaccess_src'"; then
|
|
echo "==> Placing the generated Apache reverse proxy at $htaccess"
|
|
ssh "$HOST" "mkdir -p '$TARGET_DIR/doms/$WERKATOR_DOMAIN/subs/www' && cp '$htaccess_src' '$htaccess' && cp '$maintenance_src' '$maintenance'"
|
|
else
|
|
echo "==> No werkator.htaccess generated (no server.publicBaseUrl configured) — skipping the Apache proxy"
|
|
fi
|
|
|
|
echo "==> Linking the units into ~/.config/systemd/user and enabling the service"
|
|
ssh "$HOST" "mkdir -p ~/.config/systemd/user && \
|
|
ln -sf '$REPO_DIR/.git/werkator/$UNIT' ~/.config/systemd/user/ && \
|
|
ln -sf '$REPO_DIR/.git/werkator/werkator-docker-prune.service' ~/.config/systemd/user/ && \
|
|
ln -sf '$REPO_DIR/.git/werkator/werkator-docker-prune.timer' ~/.config/systemd/user/ && \
|
|
XDG_RUNTIME_DIR=/run/user/\$(id -u) systemctl --user daemon-reload && \
|
|
XDG_RUNTIME_DIR=/run/user/\$(id -u) systemctl --user restart '$UNIT' && \
|
|
XDG_RUNTIME_DIR=/run/user/\$(id -u) systemctl --user status '$UNIT' --no-pager -l | head -12"
|
|
|
|
echo
|
|
echo "==> Server started. Verify: https://$WERKATOR_DOMAIN/"
|
|
echo " Logs: ssh $HOST -- systemctl --user status '$UNIT'"
|
|
}
|
|
|
|
# Background SSH tunnel to the Werkator server, so the browser reaches the UI
|
|
# at http://localhost:<WERKATOR_LOCAL_PORT> without keeping a terminal busy.
|
|
# `start` runs ssh -N -L detached with a pid file; `stop` kills it.
|
|
port_forward() {
|
|
require_env WERKATOR_LOCAL_PORT
|
|
# the effective port, wherever it is configured (machine config or applied
|
|
# fragment) — config:print is the single answer, not this script's parser
|
|
local remote_port
|
|
remote_port="$(ssh "$HOST" "cd '$REPO_DIR' && '$WERKATOR_BIN' config:print 2>/dev/null" | awk '/^server:/{f=1;next} f && /^ port:/{print $2; exit}' | tr -d '"')"
|
|
[ -n "$remote_port" ] || die "no server.port configured — run 'tools/remote werkator instance-start' first"
|
|
|
|
case "$COMMAND" in
|
|
start)
|
|
if [ -f "$PID_FILE" ] && kill -0 "$(cat "$PID_FILE")" 2>/dev/null; then
|
|
echo "==> Port-forward already running (pid $(cat "$PID_FILE")) — http://localhost:$WERKATOR_LOCAL_PORT"
|
|
exit 0
|
|
fi
|
|
nohup ssh -N -L "$WERKATOR_LOCAL_PORT:127.0.0.1:$remote_port" "$HOST" \
|
|
>"$LOG_FILE" 2>&1 &
|
|
echo $! > "$PID_FILE"
|
|
sleep 1
|
|
if kill -0 "$(cat "$PID_FILE")" 2>/dev/null; then
|
|
echo "==> Forwarding http://localhost:$WERKATOR_LOCAL_PORT -> $HOST:127.0.0.1:$remote_port (pid $(cat "$PID_FILE"))"
|
|
else
|
|
echo "ERROR: port-forward failed to start — see $LOG_FILE" >&2
|
|
rm -f "$PID_FILE"
|
|
exit 1
|
|
fi
|
|
;;
|
|
stop)
|
|
if [ -f "$PID_FILE" ] && kill -0 "$(cat "$PID_FILE")" 2>/dev/null; then
|
|
kill "$(cat "$PID_FILE")"
|
|
rm -f "$PID_FILE"
|
|
echo "==> Port-forward stopped"
|
|
else
|
|
rm -f "$PID_FILE"
|
|
echo "==> Port-forward is not running"
|
|
fi
|
|
;;
|
|
*)
|
|
echo "ERROR: unknown port-forward command: $COMMAND (use start or stop)" >&2
|
|
exit 2
|
|
;;
|
|
esac
|
|
}
|
|
|
|
# Print the control token guarding the mutating build endpoints — the werkator
|
|
# CLI owns creation and format (step 23), this script only invokes it.
|
|
control_token() {
|
|
ensure_ssh
|
|
ssh "$HOST" "cd '$REPO_DIR' && '$WERKATOR_BIN' control-token"
|
|
}
|
|
|
|
case "$REPO" in
|
|
port-forward)
|
|
port_forward
|
|
;;
|
|
werkator)
|
|
case "$COMMAND" in
|
|
check-prerequisites)
|
|
ensure_ssh
|
|
check_prerequisites
|
|
;;
|
|
instance-install)
|
|
instance_install
|
|
;;
|
|
instance-update)
|
|
instance_update
|
|
;;
|
|
instance-start)
|
|
instance_start
|
|
;;
|
|
repo-init)
|
|
repo_init
|
|
;;
|
|
repo-add)
|
|
repo_add "${3:-}" "${4:-}"
|
|
;;
|
|
control-token)
|
|
control_token
|
|
;;
|
|
install)
|
|
die "'install' was the self-build prototype; use instance-install + repo-init (step 21 session D)"
|
|
;;
|
|
build)
|
|
die "'build' (the self-build) is retired; the instance builds pushes itself, or run '$WERKATOR_BIN build <branch>' on the host"
|
|
;;
|
|
start)
|
|
die "'start' is now 'instance-start' — commands name their role (builder vs built)"
|
|
;;
|
|
*)
|
|
echo "ERROR: unknown command: $COMMAND" >&2
|
|
usage
|
|
;;
|
|
esac
|
|
;;
|
|
*)
|
|
echo "ERROR: unknown repo selector: $REPO" >&2
|
|
usage
|
|
;;
|
|
esac
|