4.6 KiB
Confirmed operator restore
remote backend restore --backup backups/werkjournal-....tgz
remote backend restore --recover
The first command freezes a private local copy and verifies the entire backup before asking for confirmation. It displays the target, backup creation time and application commit, explains that the snapshot replaces current account/tracking/access state, and requires the exact phrase RESTORE <full-commit>. Cancellation opens no SSH connection. There is no -y bypass. The frozen bytes are transferred after confirmation and independently verified again on the server.
The server uses the deployment lock and refuses an outstanding deployment/restore or inconsistent current manifest. It extracts only the already verified regular members into a private ~/opt/werkjournal/restores/<id>/source directory, then records pending-restore.json durably. All mutating operations using the shared deployment preflight refuse to proceed while this journal exists. Read-only status/log remain available.
Under maintenance the service is stopped and its process exit checked. Before changing live data, the command copies the current database and matching application into restores/<id>/prior with a checksummed backup.json. The incoming database is copied to a separate directory, flushed and renamed into place by the same replace_database core used for automatic deployment rollback; displaced directories remain in the private restore bundle. The matching JAR is selected, the application and Flow initialization checked, and deployed.json updated. A previously stopped service is stopped again after the readiness check. Existing operator maintenance is preserved.
The durable phases are:
| Phase | Recovery behavior |
|---|---|
prepared |
No live database replacement has begun. Restore the previous service/release state without copying an incomplete prior snapshot. |
backed_up, installed |
Verify the complete prior snapshot, restore its database and previous application, and check readiness when it was previously running. |
published |
The restored state may already have accepted writes. Verify the selected release and finish bookkeeping without reapplying any database snapshot. |
reverted |
Rollback may already have accepted writes. Finish bookkeeping without reapplying the prior snapshot. |
Publication intent is durable before maintenance is removed. A failure before publication attempts to return to the prior pair but still reports the restore as failed. If rollback also fails, the service is stopped and maintenance/journal retained for inspection. restore --recover resumes that already confirmed operation; it does not select another backup or authorize a new restore. Check remote backend status and remote backend log first. Do not delete either journal or manually remove maintenance to bypass a failed verification.
The prior/source/displaced files remain private for operator recovery; no restore bundle is automatically swept. Retain the bundle until the restored state has been accepted, then archive or remove obsolete bundles deliberately. Ordinary deployment continues to manage its own previous/rollback pointers; the explicit pre-restore pair is recorded in the restore bundle. Server environment secrets and the installed JDK remain unchanged and must already be configured for the restored application.
Repeatable isolated verification
JAVA_HOME=/path/to/jdk25 tools/probe-restore --backup backups/werkjournal-....tgz
This optional operator check uses temporary directories under ignored work/, the JAR and H2 library contained in the archive, a loopback-only JVM and disabled mail/push. It starts the application, adds a synthetic marker table to the temporary working database, invokes the actual restore implementation, checks application/Flow readiness, and then confirms the restored database lacks the marker while the prior snapshot retains it. Temporary copies and JVM processes are cleaned up; the source archive and production service are untouched. Do not upload private backups or temporary databases as CI artifacts.
Validation on 2026-09-09: all 58 operations tests passed, including eleven restore tests covering successful replacement, retained prior pair, cancellation, frozen transfer, failed startup and rollback, interrupted directory replacement, published-write retention, corrupted-prior refusal and retained stopped/maintenance states. The genuine operator backup werkjournal-20260909T121309Z-17b48c0d.tgz also passed the isolated JAR/H2/Flow probe at 0.1.17-SNAPSHOT; production was not restored. A representative restore from Hostsharing's own backup system remains a separate acceptance item.