Feature/remote authenticated clone (#24)
Co-authored-by: mhoennig <michael@hoennig.de> Reviewed-on: #24
This commit was merged in pull request #24.
This commit is contained in:
mhoennig
@@ -8,6 +8,28 @@ Lightweight, declarative and highly opinionated software build system (CI/CD).
|
||||
- [docs/bootstrapping.md](docs/bootstrapping.md) — initializing a repository with `init`
|
||||
- [docs/deployment.md](docs/deployment.md) — running Werkator as a systemd service behind a reverse proxy
|
||||
|
||||
## Adding a Gitea Repository
|
||||
|
||||
One instance serves several repositories (`docs/deployment.md`, ADR 0009).
|
||||
From the workstation, clone and initialise, then register:
|
||||
|
||||
```bash
|
||||
tools/remote --env-file .env.<instance> werkator repo-add https://gitea.example.org/<owner>/<repo>.git [<name>]
|
||||
```
|
||||
|
||||
It prints the registry entry: add it to `~/.werkator.yml` under `repositories:`, then restart the service.
|
||||
The optional `[<name>]` overrides the directory basename: it becomes the route segment (`/repos/<name>/…`) and the UI switcher entry, so it must be unique.
|
||||
Needed only when the clone directory name is wrong or collides — e.g. `michael.hoennig.de.git` checked out as `michael.hoennig.de`, or two forges serving a repo of the same name.
|
||||
A **public** repository needs nothing else: the clone runs anonymously.
|
||||
A **private** repository needs shared credentials once on the host, in `~/.werkator.yml` of the service user, before cloning:
|
||||
|
||||
```yaml
|
||||
defaults:
|
||||
git:
|
||||
account: <gitea-user>
|
||||
token: <token> # Gitea → Settings → Applications → Generate Token, scope read:repository
|
||||
```
|
||||
|
||||
## Developer Setup
|
||||
|
||||
Source `.envrc` to add `tools/` to your `PATH`, or install [direnv](#direnv) to have this done automatically on `cd`:
|
||||
|
||||
+4
-3
@@ -120,9 +120,10 @@ Adding a repository is editing a registry entry — never a data migration, beca
|
||||
tools/remote --env-file .env.<instance> werkator repo-add https://github.com/<owner>/<repo>.git [<name>]
|
||||
```
|
||||
|
||||
It clones the repository next to the ones already served, runs `init` in it, and **prints** the registry entry.
|
||||
It does not write `~/.werkator.yml`: that file is the instance's own — port, global concurrency, possibly shared credentials — and a script editing it in place would rewrite the operator's configuration behind their back.
|
||||
Cloning and initialising is mechanical; registering is a decision.
|
||||
It clones the repository next to the ones already served, runs `init` in it, and **prints** the registry entry.
|
||||
It does not write `~/.werkator.yml`: that file is the instance's own — port, global concurrency, possibly shared credentials — and a script editing it in place would rewrite the operator's configuration behind their back.
|
||||
Cloning and initialising is mechanical; registering is a decision.
|
||||
A private `https` origin authenticates with the shared `defaults.git.account`/`defaults.git.token` of `~/.werkator.yml` (the token travels via a one-shot `GIT_ASKPASS` on the host, never in a URL or process list); enter those once before cloning a private repository — without them only public origins clone.
|
||||
|
||||
4. **Restart** the service; startup recovery re-enqueues what was in flight:
|
||||
|
||||
|
||||
@@ -0,0 +1,92 @@
|
||||
> **WARNING:** This document describes only the change applied in this PR.
|
||||
> It may already be outdated once the next PR is merged.
|
||||
> Historic PR-documentation is not maintained along with new PRs — treat it as a snapshot, not as current documentation.
|
||||
|
||||
## Related Links
|
||||
|
||||
- ADR 0009 — multi-repo instance: the `defaults` block carries shared repository-level keys, read by `ConfigLoader`, never directly by consumers.
|
||||
- `docs/deployment.md` — registry setup: `repo-add` clones, initialises, and prints the registry entry.
|
||||
|
||||
## The Problem
|
||||
|
||||
`tools/remote werkator repo-add <private-https-url>` fails on the host with `could not read Username`.
|
||||
The clone runs anonymously, but the credentials exist only in the instance file's `defaults.git` block — which the script never consults.
|
||||
Registering a private repository therefore needs a manual SSH session today.
|
||||
|
||||
## Non-Goals
|
||||
|
||||
The script still does not write `~/.werkator.yml`: registering stays the operator's decision.
|
||||
No SSH-URL support: the forge is reached over `https` from the host.
|
||||
No new config keys: `defaults.git.account`/`defaults.git.token` already exist.
|
||||
|
||||
## The Scenarios
|
||||
|
||||
### Feature: authenticated clone for private origins
|
||||
|
||||
#### Background
|
||||
|
||||
- The shared credentials live in `~/.werkator.yml` under `defaults.git` (ADR 0009).
|
||||
- Public origins and instances without shared credentials must keep cloning anonymously.
|
||||
|
||||
#### Scenario#000.01: Private https origin clones with shared credentials
|
||||
|
||||
- **Given** `defaults.git.account`/`defaults.git.token` in `~/.werkator.yml` on the host
|
||||
- **When** `tools/remote werkator repo-add <private-https-url>` runs
|
||||
- **Then** the clone authenticates with those credentials and succeeds.
|
||||
|
||||
##### Verified by
|
||||
|
||||
- Manual stub-`git` test: URL carries `account@`, askpass answers the token, `GIT_TERMINAL_PROMPT=0`.
|
||||
|
||||
#### Scenario#000.02: Public origin clones anonymously
|
||||
|
||||
- **Given** no shared credentials (or a public repository)
|
||||
- **When** `repo-add` or `repo-init` runs
|
||||
- **Then** the clone runs exactly as before, without authentication.
|
||||
|
||||
##### Verified by
|
||||
|
||||
- Local helper test against a `file://` origin with and without an instance file.
|
||||
|
||||
#### Scenario#000.03: Token never leaks locally
|
||||
|
||||
- **Given** an authenticated clone
|
||||
- **When** the command runs from the workstation
|
||||
- **Then** the token appears in neither the local process list nor a repository config.
|
||||
|
||||
##### Verified by
|
||||
|
||||
- Code inspection: the token is read on the host and passed via a one-shot `GIT_ASKPASS` script.
|
||||
|
||||
## The Solution
|
||||
|
||||
`tools/remote` gained a `clone_repo` helper used by both `repo-init` and `repo-add`.
|
||||
For `https://` URLs it ships a small Python helper (base64-encoded, so no `$` is expanded locally) to the host.
|
||||
The helper reads `defaults.git.account`/`defaults.git.token` from `~/.werkator.yml`, puts the account into the URL, and hands the token to git via a one-shot `0700` `GIT_ASKPASS` script deleted in `finally`.
|
||||
Without credentials it falls back to the plain anonymous clone; non-`https` URLs clone unchanged.
|
||||
`docs/deployment.md` documents that the shared credentials must exist before cloning a private repository.
|
||||
|
||||
## Open Questions
|
||||
|
||||
- None.
|
||||
|
||||
## Attachments
|
||||
|
||||
### Adding a Gitea repository — example
|
||||
|
||||
From the workstation, clone and initialise, then register:
|
||||
|
||||
```bash
|
||||
tools/remote --env-file .env.<instance> werkator repo-add https://gitea.example.org/<owner>/<repo>.git [<name>]
|
||||
```
|
||||
|
||||
Add the printed entry to `~/.werkator.yml` under `repositories:`, then restart the service.
|
||||
A public repository needs nothing else: the clone runs anonymously.
|
||||
A private repository needs shared credentials once on the host, in `~/.werkator.yml` of the service user, before cloning (token: Gitea → Settings → Applications → Generate Token, scope `read:repository`):
|
||||
|
||||
```yaml
|
||||
defaults:
|
||||
git:
|
||||
account: <gitea-user>
|
||||
token: <token>
|
||||
```
|
||||
+61
-13
@@ -332,9 +332,65 @@ instance_update() {
|
||||
echo "==> Instance updated."
|
||||
}
|
||||
|
||||
# Sets up the WATCHED repository: an anonymous https clone (a private origin
|
||||
# gets its credentials via git.account/git.token in the machine config that
|
||||
# `werkator init` creates), the werkator init with the instance fragment
|
||||
# Clones one URL into one directory on the host.
|
||||
# A private https origin authenticates with the shared `defaults.git.account` /
|
||||
# `defaults.git.token` of `~/.werkator.yml` (ADR 0009). The whole authenticated
|
||||
# clone runs in one remote python script: the token is read from the instance
|
||||
# file on the host and passed to git via a one-shot GIT_ASKPASS script, so it
|
||||
# appears in neither the local process list nor a repository config.
|
||||
# Public origins (or an instance without shared credentials) clone anonymously.
|
||||
clone_repo() {
|
||||
local url="$1" dest="$2"
|
||||
if ssh "$HOST" "test -d '$dest/.git'"; then
|
||||
echo " (already cloned, skipping)"
|
||||
return 0
|
||||
fi
|
||||
case "$url" in
|
||||
https://*)
|
||||
# The python helper travels base64-encoded: the clone command itself
|
||||
# stays a plain `ssh` line, so no `$` inside the script is ever
|
||||
# expanded by the local shell, and the token never leaves the host.
|
||||
local helper_b64
|
||||
helper_b64="$(python3 -c 'import base64,sys; print(base64.b64encode(sys.stdin.read().encode()).decode())' <<'PYEOF_CLONE'
|
||||
import os, stat, subprocess, sys, tempfile, urllib.parse
|
||||
url, dest = sys.argv[1], sys.argv[2]
|
||||
try:
|
||||
import yaml
|
||||
cfg = yaml.safe_load(open(os.path.expanduser("~/.werkator.yml"))) or {}
|
||||
except (FileNotFoundError, ImportError):
|
||||
cfg = {}
|
||||
d = (cfg.get("defaults") or {}).get("git") or {}
|
||||
account, token = d.get("account"), d.get("token")
|
||||
env = dict(os.environ, GIT_TERMINAL_PROMPT="0")
|
||||
ask = None
|
||||
if account and token:
|
||||
parts = urllib.parse.urlsplit(url)
|
||||
host = parts.netloc.rsplit("@", 1)[-1]
|
||||
url = urllib.parse.urlunsplit(parts._replace(netloc=account + "@" + host))
|
||||
ask = tempfile.NamedTemporaryFile(mode="w", prefix="werkator-clone-askpass-",
|
||||
suffix=".sh", delete=False)
|
||||
ask.write("#!/bin/sh\nexec echo \"$WERKATOR_CLONE_TOKEN\"\n")
|
||||
ask.close()
|
||||
os.chmod(ask.name, stat.S_IRWXU)
|
||||
env.update(GIT_ASKPASS=ask.name, WERKATOR_CLONE_TOKEN=token)
|
||||
try:
|
||||
subprocess.run(["git", "clone", url, dest], env=env, check=True)
|
||||
finally:
|
||||
if ask is not None:
|
||||
os.unlink(ask.name)
|
||||
PYEOF_CLONE
|
||||
)"
|
||||
ssh "$HOST" "echo '$helper_b64' | base64 -d | python3 - '$url' '$dest'"
|
||||
;;
|
||||
*)
|
||||
ssh "$HOST" "git clone '$url' '$dest'"
|
||||
;;
|
||||
esac
|
||||
}
|
||||
|
||||
# Sets up the WATCHED repository: an https clone (a private origin
|
||||
# authenticates with the shared `defaults.git.*` credentials of
|
||||
# `~/.werkator.yml`; see `clone_repo`), the werkator init with the instance
|
||||
# applied, and the rootfs archive for the sandbox builds. All configuration
|
||||
# writing is init's — this script transports and invokes (step 23).
|
||||
repo_init() {
|
||||
@@ -344,11 +400,7 @@ repo_init() {
|
||||
ssh "$HOST" "test -x '$WERKATOR_BIN'" || die "no instance on $HOST — run instance-install first"
|
||||
|
||||
echo "==> Cloning the watched repository"
|
||||
if ssh "$HOST" "test -d '$REPO_DIR/.git'"; then
|
||||
echo " (already cloned, skipping)"
|
||||
else
|
||||
ssh "$HOST" "git clone '$REPO_URL' '$REPO_DIR'"
|
||||
fi
|
||||
clone_repo "$REPO_URL" "$REPO_DIR"
|
||||
|
||||
if [ "$SANDBOX" = "docker" ]; then
|
||||
echo "==> No rootfs needed (WERKATOR_SANDBOX=docker) — the build image is the repository's own Dockerfile"
|
||||
@@ -401,11 +453,7 @@ repo_add() {
|
||||
ssh "$HOST" "test -x '$WERKATOR_BIN'" || die "no instance on $HOST — run instance-install first"
|
||||
|
||||
echo "==> Cloning $url as '$name'"
|
||||
if ssh "$HOST" "test -d '$SIBLING_DIR/$name/.git'"; then
|
||||
echo " (already cloned, skipping)"
|
||||
else
|
||||
ssh "$HOST" "git clone '$url' '$SIBLING_DIR/$name'"
|
||||
fi
|
||||
clone_repo "$url" "$SIBLING_DIR/$name"
|
||||
|
||||
# The instance fragment carries the sandbox policy (bwrap rootfs and werkdock
|
||||
# binary). Without it a watched repository builds on the bare host, where the
|
||||
|
||||
Reference in New Issue
Block a user