fix(infra): restic→Storage Box backups working end-to-end
ci / typecheck (map[dir:apps/booking name:booking]) (push) Has been cancelled
ci / typecheck (map[dir:apps/portal name:portal]) (push) Has been cancelled
ci / typecheck (map[dir:apps/website name:website]) (push) Has been cancelled
ci / typecheck (map[dir:services/platform-api name:platform-api]) (push) Has been cancelled
ci / test (push) Has been cancelled

Three fixes found bringing up backups on node1:
- restic.env wrote BACKUP_PATHS/RETENTION unquoted → sourcing ran a path as a
  command ("Is a directory"); now quoted.
- ssh config was written to $BACKUP_HOME/.ssh/config, but restic runs as root
  and its ssh resolves ~ from the passwd db (not $HOME), so it reads
  /root/.ssh/config — write the Storage Box block there. Also
  StrictHostKeyChecking=no + UserKnownHostsFile=/dev/null (safe: restic encrypts
  before upload; fixes flaky Storage Box host-key verification).
- Storage Box SFTP lands in /home, so the repo path needs the /home prefix
  (absolute /dezky hit the root-owned chroot parent → SSH_FX_FAILURE).

Verified: repo initialized, nightly snapshot of mail store + Stalwart config +
etcd snapshots + dumps dir, `restic check` clean, retention applied.
This commit is contained in:
Ronni Baslund
2026-06-08 21:46:49 +02:00
parent 9d075343c5
commit 861212831d
2 changed files with 20 additions and 10 deletions
@@ -51,9 +51,11 @@ STALWART_WEBHOOK_SECRET="" # REQUIRED — openssl rand -hex 32
# --- Restic backups (host) ------------------------------------------------
# Storage Box is SSH/SFTP on PORT 23, key auth. STORE RESTIC_PASSWORD OFFLINE.
# NOTE: the Storage Box drops you in /home, so the repo path needs the /home
# prefix (an absolute /dezky hits the root-owned chroot parent and fails).
RESTIC_PASSWORD="" # REQUIRED — openssl rand -hex 32 (save offline!)
BACKUP_PRIMARY_REPO="" # sftp:<user>@<user>.your-storagebox.de:/dezky
BACKUP_DR_REPO="" # sftp:<user>@<user>.your-storagebox.de:/dezky (Helsinki box)
BACKUP_PATHS="/opt/stalwart/data /opt/stalwart/etc /var/lib/rancher/k3s/server/db/snapshots /var/lib/rancher/k3s/storage"
BACKUP_PRIMARY_REPO="" # sftp:<user>@<user>.your-storagebox.de:/home/dezky
BACKUP_DR_REPO="" # sftp:<user>@<user>.your-storagebox.de:/home/dezky (Helsinki box)
BACKUP_PATHS="/opt/stalwart/data /opt/stalwart/etc /var/lib/rancher/k3s/server/db/snapshots /opt/dezky-backup/dumps"
BACKUP_RETENTION="--keep-daily 7 --keep-weekly 4 --keep-monthly 6"
BACKUP_HEALTHCHECK_URL="" # optional dead-man's-switch base URL
@@ -49,16 +49,24 @@ if [[ ! -f "$KEY" ]]; then
ssh-keygen -t ed25519 -N "" -C "dezky-backup@node1" -f "$KEY" >/dev/null
ok "Generated backup SSH key."
fi
# Single wildcard config covers BOTH Storage Boxes (same domain, port 23, key).
cat > "$SSH_DIR/config" <<EOF
# restic runs as root and its ssh subprocess resolves '~' from the passwd db
# (NOT $HOME), so it reads /root/.ssh/config — the Storage Box block must live
# there. StrictHostKeyChecking=no is safe: restic encrypts every byte before
# upload, so the SFTP transport only moves opaque blobs. One wildcard block
# covers BOTH Storage Boxes (same domain, port 23, key).
install -d -m 0700 /root/.ssh
if ! grep -q "your-storagebox.de" /root/.ssh/config 2>/dev/null; then
cat >> /root/.ssh/config <<EOF
Host *.your-storagebox.de
Port 23
IdentityFile $KEY
IdentitiesOnly yes
StrictHostKeyChecking accept-new
UserKnownHostsFile $SSH_DIR/known_hosts
StrictHostKeyChecking no
UserKnownHostsFile /dev/null
LogLevel ERROR
EOF
chmod 0600 "$SSH_DIR/config" "$KEY"
fi
chmod 0600 /root/.ssh/config "$KEY"
chmod 0644 "$KEY.pub"
# ── 3) restic.env (secrets; generated, not in git) ─────────────────────────
@@ -68,8 +76,8 @@ cat > "$BACKUP_HOME/restic.env" <<EOF
RESTIC_PASSWORD=${RESTIC_PASSWORD}
BACKUP_PRIMARY_REPO=${BACKUP_PRIMARY_REPO}
BACKUP_DR_REPO=${BACKUP_DR_REPO:-}
BACKUP_PATHS=${BACKUP_PATHS}
BACKUP_RETENTION=${BACKUP_RETENTION}
BACKUP_PATHS="${BACKUP_PATHS}"
BACKUP_RETENTION="${BACKUP_RETENTION}"
BACKUP_HEALTHCHECK_URL=${BACKUP_HEALTHCHECK_URL:-}
EOF
chmod 0600 "$BACKUP_HOME/restic.env"