werkjournal/docs/RUNBOOK.md

Werkjournal operations

This is the current operator entry point. Historical implementation notes describe earlier releases and are not installation instructions. Commands below run from the repository on the operator's computer unless explicitly labelled as server commands. Use tools/remote when direnv has not added tools/ to PATH. Space SSH operations out: Hostsharing rate-limits repeated connections.

Installation and private configuration

Item	Established value
Application account	mih04-werkjournal@mih04.hostsharing.net
Account home	/home/pacs/mih04/users/werkjournal
Application	~/opt/werkjournal
JDK	~/opt/jdk25 (pinned Temurin 25)
H2 database	~/opt/werkjournal/data/werkjournal.mv.db
systemd user unit	werkjournal-backend.service
Environment	~/.config/werkjournal/environment, private mode 0600
HTTPS document root	~/doms/werkjournal.javagil.de/htdocs-ssl
HTTP document root	~/doms/werkjournal.javagil.de/htdocs
Public URL	https://werkjournal.javagil.de, without www
Backend	127.0.0.1:18090
CI account	mih09-werkator@mih09.hostsharing.net
CI dashboard	https://werkator.javagil.de/repos/werkjournal

Copy .env.example to the ignored .env and retain the supported layout. These are local transport settings, not the server's systemd environment. scripts/lib-env.sh rejects unsupported deployment directory/port overrides. The application and static proxy are owned by the same target user.

For a genuinely empty target, provision the domain and HTTPS in Hostsharing, configure key-based SSH, then run:

remote backend install-jdk
remote backend deploy-prototype --artifact /path/to/verified-ci-release
remote backend info

The artifact directory must contain the matching werkjournal.jar and manifest.json produced by CI. Provisioning writes the unit, maintenance document and proxy rules; existing service/releases/database cause it to refuse. Never use first-deployment or guest-preview commands to upgrade the current installation.

TLS terminates at Hostsharing's web server. The HTTPS .htaccess proxies to the loopback port and supplies forwarded host, port and protocol. HTTP redirects to HTTPS. A stray index.html must not replace the app; provisioning removes the initial placeholder and disables DirectoryIndex. Keep the backend bound to loopback. The unit trusts the reverse-proxy headers and supplies the forwarded-host setting. remote backend setup refreshes the versioned unit and reloads systemd without restarting or replacing secrets.

Create or edit the protected server environment privately, preserving existing entries. Relevant settings are:

• WERKJOURNAL_GUEST_MODE=false for invitation-only private accounts plus the explicit shared guest login.
• WERKJOURNAL_AUTH_BOOTSTRAP_CODE for the initial administrator only; configure a high-entropy setup code before first private-account setup. Enter it under Setup / recovery, together with the email address, and complete the emailed proof. The database permanently consumes initial setup; retaining the old value does not reopen registration.
• WERKJOURNAL_MAIL_SENDER=werkjournal@javagil.de and WERKJOURNAL_SENDMAIL=/usr/sbin/sendmail use local Postfix. Replies are not monitored. No SMTP password or OIDC configuration is needed.
• Push settings are described below. Do not override the database path: the deployment preflight requires the known H2 file location so its backup is complete.

Restart through remote backend restart after changing runtime configuration. Login codes and links last two hours from issuance, are single-use and can be entered on another device. Re-requesting replaces the earlier proof; repeated wrong codes are limited. Invitations last seven days. Account deletion requires its own email proof and a further explicit confirmation.

Routine releases and inspection

tools/ci-jvm runs operations tests, JVM/Chromium tests and production packaging, frontend tests, Firefox acceptance and the production-JAR Gherkin smoke. It creates the checksummed release manifest only after these gates. The second Gradle stage excludes an already completed JVM test task. tools/ci-deploy runs only with WERKJOURNAL_CI_DEPLOY=1 and branch=main, checks clean tracked source and matching commit, then invokes the same remote deployment command used manually.

remote backend info
remote backend status
remote backend log --lines 100
remote backend log --since '1 hour ago'
remote backend deploy --artifact /path/to/verified-ci-release

info checks internal and public readiness and the source/version identity. CI retains build/reports/deployment/result.log and the test reports under build/reports/. A failed deployment remains a failed CI job even after a successful rollback. Inspect the actual terminal CI result; a running or pending build is not a successful release.

remote backend upload --artifact DIRECTORY stages an immutable candidate only. Restarting still selects the current verified release. deploy validates checksums at both ends, locks deployment, enables public maintenance, stops the process, copies the closed database with its matching application, migrates/starts the candidate and checks readiness before publication. It never rebuilds during deployment. A stale queued candidate is rejected using recorded Git ancestry.

Successful updates retain one previous successful application and its matching rollback state. Each attempt takes a fresh backup, so two failed candidates do not replace the known-good target or discard bookings saved since the earlier successful deployment. Do not manually repoint current, delete journals or replace a live H2 file.

start, stop, restart, enable and disable use the common deployment safeguards. enable/disable change autostart only. Existing operator maintenance remains active until the operator resolves its cause.

Interrupted deployment and incidents

Start with status, recent logs and the CI failure. The application directory may contain pending-deployment.json or pending-restore.json; they record unfinished operations. Keep their associated snapshots. Under htdocs-ssl, maintenance.flag makes public requests return the maintenance page with HTTP 503. An open client shows a reconnection message and retains its work draft.

Situation	Action
Interrupted deployment	remote backend recover
Interrupted, previously confirmed operator restore	remote backend restore --recover
Failed rollback/readiness	Keep maintenance active; inspect the journal and logs before recovery. Do not bypass readiness.
Running service but public placeholder/error	Compare remote backend info, .htaccess, domain document root and maintenance state.
Email not arriving	Check the configured sender/local sendmail and application logs; inspect the recipient's spam folder. Do not log tokens or queue additional test mail without authorization.
No reminders	Check device opt-in, work-tracker activation, timezone, work/day-off entry and the 20:00–22:00 weekday window; inspect runtime/maintenance state.

Automatic rollback is safe only before publication: once access has reopened, subsequent writes must be preserved. Recovery uses the journal's publication intent to decide whether to finish forward or restore the previous pair. Never force an older snapshot over an already reopened application merely because a later cleanup step failed. Details: deployment and operator restore.

Consistent operator backup and restore

remote backend backup
JAVA_HOME=/path/to/jdk25 tools/probe-restore --backup backups/selected-backup.tgz
remote backend restore --backup backups/selected-backup.tgz
remote backend info

Choose the actual backup filename returned by the first command. Backups contain private records and login state plus the matching JAR/manifest. Store them outside Git, public document roots and CI artifacts. The backup briefly stops the application, copies the closed database and restores the previous service state. The probe restores only into temporary local directories and disables external mail/push.

The restore command verifies a frozen local copy before connecting and asks for RESTORE <full-commit>. There is no automatic-confirmation bypass. It replaces the entire account/tracking/access state with the chosen snapshot; records created afterward will be lost and old access state can return. It takes a private pre-restore snapshot before changing data. Confirm only the selected target and timestamp. Preserve restores/<id>/source, prior and displaced data until the restored state is accepted. If interrupted, use restore --recover, not a second unrelated restore.

Afterward verify application/Flow readiness and representative account, work, habit and historical-limit data using the account owner. The automated isolated probe has passed for an actual operator backup; production has not been restored as a test.

Recovery from Hostsharing's own backups

Hostsharing documents requesting restored files from its service team with the package, requested paths and backup date; the recovered data is staged below /home/restore/. Use the exact location supplied by the service team. Request package mih04 and the complete Werkjournal application directory together with its private configuration and user unit, not just an unrelated JAR or database file. See Hostsharing's recovery instructions.

Keep the recovered source read-only and first work on a private copy. An existing verified Werkjournal backup archive can use the probe/restore procedure above. A provider filesystem snapshot is not automatically a Werkjournal archive: the current restore tool intentionally rejects arbitrary trees. H2 consistency, the matching release and the snapshot's deployment-journal state must be established before importing such a source. Do not claim a raw H2 file is a closed-file backup merely because it came from the provider.

The representative provider-backup retrieval, isolated consistency test and an approved import path for a raw restored tree remain open under restore-test / WJ-024. This runbook does not authorize sending a support email or replacing production data. Until that acceptance is complete, use the already tested consistent operator-backup format for a planned application restore.

Recover the last administrator

remote backend prepare-admin-recovery

This refuses while any administrator exists, including a disabled one. It performs a short maintenance operation against the existing database, without running migrations or sending mail. Read the generated code privately from ~/.config/werkjournal/admin-recovery-code on the server; do not paste it in tickets or logs. It expires after 24 hours and a new recovery request replaces it.

Open Setup / recovery, enter the code and email address, then complete the ordinary email proof. An existing account can be promoted while retaining its data. Remove the plaintext server code file after successful use. The consumed original setup code cannot be reused. If interrupted, inspect maintenance/service and pending journals before restarting; follow the detailed administrator recovery procedure.

Push operation

remote backend setup-push creates a VAPID pair in the private environment file, preserving an existing complete pair. It does not enable sending or restart the service. Activation requires WERKJOURNAL_PUSH_ENABLED=true, the corresponding public/private keys and a controlled restart. Retain existing keys; rotation would invalidate subscriptions.

Each device opts in separately in Settings. The last reported timezone determines its fixed weekday reminder window. A work entry or day-off marker suppresses the day's reminder; catch-up ends at 22:00. Global logout revokes the device sessions and associated push subscriptions. The maintenance flag suppresses delivery. HTTP acceptance by a push provider does not prove display on Android; real-device delivery remains a separate acceptance check. See push operation and implementation.

Managed host change: Bookworm to Trixie

Coordinate the operating-system change with Hostsharing; the application user does not run a distribution upgrade. Treat the following as a planned maintenance checklist, not evidence that the upgrade has already been tested:

1. Pause automatic application deployment for the change window (disable WERKJOURNAL_CI_DEPLOY in the deployment-enabled job configuration). Record current remote backend info, the unit/environment locations and the installed JDK version. Make a consistent operator backup and pass its isolated restore probe.
2. Agree with Hostsharing on the maintenance window and host rollback route. Preserve the account home, permissions, service unit, private environment, VAPID keys, domain configuration, JDK and all application state. Do not combine the OS change with a schema or application upgrade.
3. After the host change, inspect the OS/architecture and run ~/opt/jdk25/bin/java -version in one SSH session. Check systemctl --user status werkjournal-backend.service and its journal. If the pinned JDK needs reinstalling, use remote backend install-jdk --force; do not substitute the system Java silently.
4. Use remote backend info to verify the same source/version, internal Flow initialization and public HTTPS. Check email login and representative existing data with the owner, then the installed Android PWA and device notification state. Send test mail/push only by agreement. Recheck user-service autostart after a coordinated restart.
5. Re-enable automatic deployment only after these checks pass. If the OS change breaks the runtime, use the agreed host rollback while preserving any new writes. An application/database restore is not an operating-system rollback. Retain the pre-change backup until acceptance is recorded.

The target stays x86-64 and the application stays a JVM JAR. GraalVM remains optional and deferred. Future native-build compatibility would require its own verification.

Evidence and remaining release gates

Implementation evidence lives in the acceptance map. Chromium and Firefox examples, the production smoke and the operator-backup probe do not replace the still-open physical Android/PWA/push and provider-backup exercises. Record their actual results before marking those Werkbaum nodes or the final release complete.