135 lines
14 KiB
Markdown
135 lines
14 KiB
Markdown
# Werkjournal operations
|
||
|
||
This is the current operator entry point. Historical implementation notes describe earlier releases and are not installation instructions. Commands below run from the repository on the operator's computer unless explicitly labelled as server commands. Use `tools/remote` when `direnv` has not added `tools/` to PATH. Space SSH operations out: Hostsharing rate-limits repeated connections.
|
||
|
||
## Installation and private configuration
|
||
|
||
| Item | Established value |
|
||
| --- | --- |
|
||
| Application account | `mih04-werkjournal@mih04.hostsharing.net` |
|
||
| Account home | `/home/pacs/mih04/users/werkjournal` |
|
||
| Application | `~/opt/werkjournal` |
|
||
| JDK | `~/opt/jdk25` (pinned Temurin 25) |
|
||
| H2 database | `~/opt/werkjournal/data/werkjournal.mv.db` |
|
||
| systemd user unit | `werkjournal-backend.service` |
|
||
| Environment | `~/.config/werkjournal/environment`, private mode 0600 |
|
||
| HTTPS document root | `~/doms/werkjournal.javagil.de/htdocs-ssl` |
|
||
| HTTP document root | `~/doms/werkjournal.javagil.de/htdocs` |
|
||
| Public URL | `https://werkjournal.javagil.de`, without `www` |
|
||
| Backend | `127.0.0.1:18090` |
|
||
| CI account | `mih09-werkator@mih09.hostsharing.net` |
|
||
| CI dashboard | <https://werkator.javagil.de/repos/werkjournal> |
|
||
|
||
Copy `.env.example` to the ignored `.env` and retain the supported layout. These are local transport settings, not the server's systemd environment. `scripts/lib-env.sh` rejects unsupported deployment directory/port overrides. The application and static proxy are owned by the same target user.
|
||
|
||
For a genuinely empty target, provision the domain and HTTPS in Hostsharing, configure key-based SSH, then run:
|
||
|
||
```sh
|
||
remote backend install-jdk
|
||
remote backend deploy-prototype --artifact /path/to/verified-ci-release
|
||
remote backend info
|
||
```
|
||
|
||
The artifact directory must contain the matching `werkjournal.jar` and `manifest.json` produced by CI. Provisioning writes the unit, maintenance document and proxy rules; existing service/releases/database cause it to refuse. Never use first-deployment or guest-preview commands to upgrade the current installation.
|
||
|
||
TLS terminates at Hostsharing's web server. The HTTPS `.htaccess` proxies to the loopback port and supplies forwarded host, port and protocol. HTTP redirects to HTTPS. A stray `index.html` must not replace the app; provisioning removes the initial placeholder and disables DirectoryIndex. Keep the backend bound to loopback. The unit trusts the reverse-proxy headers and supplies the forwarded-host setting. `remote backend setup` refreshes the versioned unit and reloads systemd without restarting or replacing secrets.
|
||
|
||
Create or edit the protected server environment privately, preserving existing entries. Relevant settings are:
|
||
|
||
- `WERKJOURNAL_GUEST_MODE=false` for invitation-only private accounts plus the explicit shared guest login.
|
||
- `WERKJOURNAL_AUTH_BOOTSTRAP_CODE` for the initial administrator only; configure a high-entropy setup code before first private-account setup. Enter it under **Setup / recovery**, together with the email address, and complete the emailed proof. The database permanently consumes initial setup; retaining the old value does not reopen registration.
|
||
- `WERKJOURNAL_MAIL_SENDER=werkjournal@javagil.de` and `WERKJOURNAL_SENDMAIL=/usr/sbin/sendmail` use local Postfix. Replies are not monitored. No SMTP password or OIDC configuration is needed.
|
||
- Push settings are described below. Do not override the database path: the deployment preflight requires the known H2 file location so its backup is complete.
|
||
|
||
Restart through `remote backend restart` after changing runtime configuration. Login codes and links last two hours from issuance, are single-use and can be entered on another device. Re-requesting replaces the earlier proof; repeated wrong codes are limited. Invitations last seven days. Account deletion requires its own email proof and a further explicit confirmation.
|
||
|
||
## Routine releases and inspection
|
||
|
||
`tools/ci-jvm` runs operations tests, JVM/Chromium tests and production packaging, frontend tests, Firefox acceptance and the production-JAR Gherkin smoke. It creates the checksummed release manifest only after these gates. The second Gradle stage excludes an already completed JVM test task. `tools/ci-deploy` runs only with `WERKJOURNAL_CI_DEPLOY=1` and `branch=main`, checks clean tracked source and matching commit, then invokes the same remote deployment command used manually.
|
||
|
||
```sh
|
||
remote backend info
|
||
remote backend status
|
||
remote backend log --lines 100
|
||
remote backend log --since '1 hour ago'
|
||
remote backend deploy --artifact /path/to/verified-ci-release
|
||
```
|
||
|
||
`info` checks internal and public readiness and the source/version identity. CI retains `build/reports/deployment/result.log` and the test reports under `build/reports/`. A failed deployment remains a failed CI job even after a successful rollback. Inspect the actual terminal CI result; a running or pending build is not a successful release.
|
||
|
||
`remote backend upload --artifact DIRECTORY` stages an immutable candidate only. Restarting still selects the current verified release. `deploy` validates checksums at both ends, locks deployment, enables public maintenance, stops the process, copies the closed database with its matching application, migrates/starts the candidate and checks readiness before publication. It never rebuilds during deployment. A stale queued candidate is rejected using recorded Git ancestry.
|
||
|
||
Successful updates retain one previous successful application and its matching rollback state. Each attempt takes a fresh backup, so two failed candidates do not replace the known-good target or discard bookings saved since the earlier successful deployment. Do not manually repoint `current`, delete journals or replace a live H2 file.
|
||
|
||
`start`, `stop`, `restart`, `enable` and `disable` use the common deployment safeguards. `enable`/`disable` change autostart only. Existing operator maintenance remains active until the operator resolves its cause.
|
||
|
||
## Interrupted deployment and incidents
|
||
|
||
Start with status, recent logs and the CI failure. The application directory may contain `pending-deployment.json` or `pending-restore.json`; they record unfinished operations. Keep their associated snapshots. Under `htdocs-ssl`, `maintenance.flag` makes public requests return the maintenance page with HTTP 503. An open client shows a reconnection message and retains its work draft.
|
||
|
||
| Situation | Action |
|
||
| --- | --- |
|
||
| Interrupted deployment | `remote backend recover` |
|
||
| Interrupted, previously confirmed operator restore | `remote backend restore --recover` |
|
||
| Failed rollback/readiness | Keep maintenance active; inspect the journal and logs before recovery. Do not bypass readiness. |
|
||
| Running service but public placeholder/error | Compare `remote backend info`, `.htaccess`, domain document root and maintenance state. |
|
||
| Email not arriving | Check the configured sender/local sendmail and application logs; inspect the recipient's spam folder. Do not log tokens or queue additional test mail without authorization. |
|
||
| No reminders | Check device opt-in, work-tracker activation, timezone, work/day-off entry and the 20:00–22:00 weekday window; inspect runtime/maintenance state. |
|
||
|
||
Automatic rollback is safe only before publication: once access has reopened, subsequent writes must be preserved. Recovery uses the journal's publication intent to decide whether to finish forward or restore the previous pair. Never force an older snapshot over an already reopened application merely because a later cleanup step failed. Details: [deployment](implementation/deployment.md) and [operator restore](implementation/operator-restore.md).
|
||
|
||
## Consistent operator backup and restore
|
||
|
||
```sh
|
||
remote backend backup
|
||
JAVA_HOME=/path/to/jdk25 tools/probe-restore --backup backups/selected-backup.tgz
|
||
remote backend restore --backup backups/selected-backup.tgz
|
||
remote backend info
|
||
```
|
||
|
||
Choose the actual backup filename returned by the first command. Backups contain private records and login state plus the matching JAR/manifest. Store them outside Git, public document roots and CI artifacts. The backup briefly stops the application, copies the closed database and restores the previous service state. The probe restores only into temporary local directories and disables external mail/push.
|
||
|
||
The restore command verifies a frozen local copy before connecting and asks for `RESTORE <full-commit>`. There is no automatic-confirmation bypass. It replaces the entire account/tracking/access state with the chosen snapshot; records created afterward will be lost and old access state can return. It takes a private pre-restore snapshot before changing data. Confirm only the selected target and timestamp. Preserve `restores/<id>/source`, `prior` and displaced data until the restored state is accepted. If interrupted, use `restore --recover`, not a second unrelated restore.
|
||
|
||
Afterward verify application/Flow readiness and representative account, work, habit and historical-limit data using the account owner. The automated isolated probe has passed for an actual operator backup; production has not been restored as a test.
|
||
|
||
## Recovery from Hostsharing's own backups
|
||
|
||
Hostsharing documents requesting restored files from its service team with the package, requested paths and backup date; the recovered data is staged below `/home/restore/`. Use the exact location supplied by the service team. Request package `mih04` and the complete Werkjournal application directory together with its private configuration and user unit, not just an unrelated JAR or database file. See [Hostsharing's recovery instructions](https://www.hostsharing.net/doc/managed-operations-platform/backup/).
|
||
|
||
Keep the recovered source read-only and first work on a private copy. An existing verified Werkjournal backup archive can use the probe/restore procedure above. A provider filesystem snapshot is not automatically a Werkjournal archive: the current restore tool intentionally rejects arbitrary trees. H2 consistency, the matching release and the snapshot's deployment-journal state must be established before importing such a source. Do not claim a raw H2 file is a closed-file backup merely because it came from the provider.
|
||
|
||
The representative provider-backup retrieval, isolated consistency test and an approved import path for a raw restored tree remain open under `restore-test` / WJ-024. This runbook does not authorize sending a support email or replacing production data. Until that acceptance is complete, use the already tested consistent operator-backup format for a planned application restore.
|
||
|
||
## Recover the last administrator
|
||
|
||
```sh
|
||
remote backend prepare-admin-recovery
|
||
```
|
||
|
||
This refuses while any administrator exists, including a disabled one. It performs a short maintenance operation against the existing database, without running migrations or sending mail. Read the generated code privately from `~/.config/werkjournal/admin-recovery-code` on the server; do not paste it in tickets or logs. It expires after 24 hours and a new recovery request replaces it.
|
||
|
||
Open **Setup / recovery**, enter the code and email address, then complete the ordinary email proof. An existing account can be promoted while retaining its data. Remove the plaintext server code file after successful use. The consumed original setup code cannot be reused. If interrupted, inspect maintenance/service and pending journals before restarting; follow the detailed [administrator recovery procedure](implementation/admin-recovery.md).
|
||
|
||
## Push operation
|
||
|
||
`remote backend setup-push` creates a VAPID pair in the private environment file, preserving an existing complete pair. It does not enable sending or restart the service. Activation requires `WERKJOURNAL_PUSH_ENABLED=true`, the corresponding public/private keys and a controlled restart. Retain existing keys; rotation would invalidate subscriptions.
|
||
|
||
Each device opts in separately in Settings. The last reported timezone determines its fixed weekday reminder window. A work entry or day-off marker suppresses the day's reminder; catch-up ends at 22:00. Global logout revokes the device sessions and associated push subscriptions. The maintenance flag suppresses delivery. HTTP acceptance by a push provider does not prove display on Android; real-device delivery remains a separate acceptance check. See [push operation and implementation](implementation/web-push-transport.md).
|
||
|
||
## Managed host change: Bookworm to Trixie
|
||
|
||
Coordinate the operating-system change with Hostsharing; the application user does not run a distribution upgrade. Treat the following as a planned maintenance checklist, not evidence that the upgrade has already been tested:
|
||
|
||
1. Pause automatic application deployment for the change window (disable `WERKJOURNAL_CI_DEPLOY` in the deployment-enabled job configuration). Record current `remote backend info`, the unit/environment locations and the installed JDK version. Make a consistent operator backup and pass its isolated restore probe.
|
||
2. Agree with Hostsharing on the maintenance window and host rollback route. Preserve the account home, permissions, service unit, private environment, VAPID keys, domain configuration, JDK and all application state. Do not combine the OS change with a schema or application upgrade.
|
||
3. After the host change, inspect the OS/architecture and run `~/opt/jdk25/bin/java -version` in one SSH session. Check `systemctl --user status werkjournal-backend.service` and its journal. If the pinned JDK needs reinstalling, use `remote backend install-jdk --force`; do not substitute the system Java silently.
|
||
4. Use `remote backend info` to verify the same source/version, internal Flow initialization and public HTTPS. Check email login and representative existing data with the owner, then the installed Android PWA and device notification state. Send test mail/push only by agreement. Recheck user-service autostart after a coordinated restart.
|
||
5. Re-enable automatic deployment only after these checks pass. If the OS change breaks the runtime, use the agreed host rollback while preserving any new writes. An application/database restore is not an operating-system rollback. Retain the pre-change backup until acceptance is recorded.
|
||
|
||
The target stays x86-64 and the application stays a JVM JAR. GraalVM remains optional and deferred. Future native-build compatibility would require its own verification.
|
||
|
||
## Evidence and remaining release gates
|
||
|
||
Implementation evidence lives in [the acceptance map](implementation/acceptance.md). Chromium and Firefox examples, the production smoke and the operator-backup probe do not replace the still-open physical Android/PWA/push and provider-backup exercises. Record their actual results before marking those Werkbaum nodes or the final release complete.
|