feat(infra): production host bootstrap and bare-metal Stalwart scaffolding
Host provisioning for the single-server production target: SSH + firewall hardening (nftables allowlist), k3s node registration, bare-metal Stalwart install with systemd units and TLS cert-sync from the cluster secret, and Restic encrypted backup/restore (primary + DR) with timer units. Host-specific secrets live in config.env (gitignored); config.env.example is the template. Also gitignores MemPalace per-project files.
This commit is contained in:
Ronni Baslund
@@ -0,0 +1,227 @@
|
||||
# Dezky production — host layer
|
||||
|
||||
OS baseline + firewall for the bare-metal **Hetzner AX41** that runs the k3s
|
||||
node. This layer is everything that lives on the *host* (outside Kubernetes):
|
||||
hardening, the k3s-safe firewall, and — added next — k3s registration, Stalwart
|
||||
mail, and Restic backups.
|
||||
|
||||
Managed by **Fleet/Rancher** once k3s is up; this host layer is the part Fleet
|
||||
*can't* do, so it runs over SSH from reviewed scripts.
|
||||
|
||||
## Files
|
||||
|
||||
| File | Purpose |
|
||||
|------|---------|
|
||||
| `config.env.example` | Template for host-specific values |
|
||||
| `config.env` | **Real values — gitignored.** Source of truth lives only on your machine/box |
|
||||
| `bootstrap.sh` | One-shot OS hardening: user, SSH, sysctl, swap, fail2ban, auto-updates, firewall |
|
||||
| `firewall/firewall.sh` | Renders + applies the k3s-safe nftables ruleset (idempotent) |
|
||||
| `firewall/dezky-firewall.service` | systemd unit; reapplies our table on boot, never flushes globally |
|
||||
| `k3s/register.sh` | Registers the node into Rancher (Custom k3s cluster); secrets from `config.env` |
|
||||
| `stalwart/install.sh` | Installs Stalwart as a hardened host service (binary, units, secrets, bootstrap cert) |
|
||||
| `stalwart/config.toml` | Production Stalwart config (mail ports on host, JMAP on internal 8080) |
|
||||
| `stalwart/stalwart-mail.service` | systemd unit; non-root + `CAP_NET_BIND_SERVICE` for low ports |
|
||||
| `stalwart/cert-sync.sh` + `*.service`/`*.timer` | Pulls the cert-manager mail cert into Stalwart, reloads on change |
|
||||
| `restic/install.sh` | Sets up Restic, the backup SSH key/config, env, and the nightly timer |
|
||||
| `restic/backup.sh` | Backup → primary Storage Box, retention, then `copy` → Helsinki DR |
|
||||
| `restic/restore.sh` | List/restore snapshots (run drills!) |
|
||||
| `restic/dezky-backup.service` + `.timer` | Nightly 03:20 UTC backup |
|
||||
|
||||
## The firewall model (read this)
|
||||
|
||||
k3s, kube-proxy and flannel manage their **own** nftables tables (`ip`/`ip6`:
|
||||
`filter`, `nat`, `mangle`). The classic mistake is running `ufw`/`firewalld` or
|
||||
`nft flush ruleset`, which wipes or fights those rules and breaks pod networking.
|
||||
|
||||
So instead:
|
||||
|
||||
- We own a single dedicated table — **`inet dezky_fw`** — with only an INPUT
|
||||
chain (default `drop`). Separate tables coexist; a packet is dropped if *any*
|
||||
base chain drops it, so our default-drop INPUT gates host-bound traffic while
|
||||
k3s keeps owning FORWARD/NAT untouched.
|
||||
- We explicitly **accept the pod (`10.42.0.0/16`) and service (`10.43.0.0/16`)
|
||||
CIDRs and the CNI interfaces** (`cni0`, `flannel.1`) so cluster↔host traffic
|
||||
(API server, kubelet, CoreDNS) is never dropped.
|
||||
- We **never** `flush ruleset`. The systemd unit's `ExecStop` removes only our
|
||||
table.
|
||||
|
||||
### Access policy
|
||||
|
||||
| Surface | Ports | Who |
|
||||
|---------|-------|-----|
|
||||
| Web + ACME | 80, 443 | **World** (customers) |
|
||||
| Mail | 25, 465, 587, 143, 993, 4190 | **World** |
|
||||
| SSH | 22 | **`MGMT_ALLOW_V4/V6` only** |
|
||||
| k3s API | 6443 | **`MGMT_ALLOW_V4/V6` only** |
|
||||
|
||||
Current management allowlist: **home `46.32.144.38`**, **office `46.32.144.45`**.
|
||||
|
||||
The Rancher plane (`91.99.122.153`) needs **no inbound rule** — the cluster
|
||||
agent dials *out* to Rancher over 443, so replies ride the established/related
|
||||
fast-path.
|
||||
|
||||
## Apply order
|
||||
|
||||
> Prereqs: AX41 provisioned with **Debian 12 (bookworm)**, reachable as `root`.
|
||||
> `config.env` filled in — in particular `ADMIN_SSH_PUBKEY` and
|
||||
> `SERVER_PUBLIC_IPV4` (still TODO until the box exists).
|
||||
|
||||
```bash
|
||||
# From your laptop:
|
||||
scp -r infrastructure/production/host root@<server-ip>:/opt/dezky-host
|
||||
|
||||
# On the server:
|
||||
ssh root@<server-ip>
|
||||
cd /opt/dezky-host
|
||||
# config.env is gitignored, so copy it up separately or recreate it here:
|
||||
# cp config.env.example config.env && nano config.env
|
||||
./bootstrap.sh
|
||||
```
|
||||
|
||||
`bootstrap.sh` creates your admin user and installs your key **before** it
|
||||
disables root/password SSH, so the order is lockout-safe. It's idempotent —
|
||||
re-run anytime.
|
||||
|
||||
To touch only the firewall later:
|
||||
|
||||
```bash
|
||||
sudo ./firewall/firewall.sh --dry-run # preview the ruleset
|
||||
sudo ./firewall/firewall.sh # render, validate, apply, install unit
|
||||
```
|
||||
|
||||
### Then register into Rancher
|
||||
|
||||
Once the host is hardened, register the node as a **Custom k3s cluster**
|
||||
(create the cluster in Rancher first, choosing the **K3s** distribution, then
|
||||
paste its token/checksum into `config.env`):
|
||||
|
||||
```bash
|
||||
sudo ./k3s/register.sh # downloads agent installer, joins cluster
|
||||
journalctl -u rancher-system-agent -f # follow provisioning
|
||||
```
|
||||
|
||||
Rancher is currently reached by IP, so the installer is fetched with
|
||||
`--insecure`; the agent's ongoing link is still verified via `--ca-checksum`.
|
||||
Give Rancher a real hostname + cert later to drop the insecure fetch.
|
||||
|
||||
### Then install Stalwart (mail)
|
||||
|
||||
```bash
|
||||
sudo ./stalwart/install.sh # binary + systemd + bootstrap cert
|
||||
systemctl status stalwart-mail
|
||||
```
|
||||
|
||||
Requires `STALWART_ADMIN_PASSWORD` + `STALWART_WEBHOOK_SECRET` in `config.env`
|
||||
(`openssl rand -hex 24` / `-hex 32`). See the mail topology below.
|
||||
|
||||
## Mail (Stalwart) topology
|
||||
|
||||
Stalwart runs on the **host**, not in k3s — mail must keep flowing regardless of
|
||||
cluster state, and SMTP/IMAP want the real public IP for reputation. The single
|
||||
public IP forces a deliberate split with Traefik:
|
||||
|
||||
| Concern | Owner | Detail |
|
||||
|---------|-------|--------|
|
||||
| Mail protocol ports (25/465/587/143/993/4190) | **Stalwart (host)** | Bound on the public IP; opened to the world by the firewall |
|
||||
| Web/JMAP for `mail.dezky.eu:443` | **Traefik (k3s)** | Terminates TLS, reverse-proxies to Stalwart's internal `:8080` |
|
||||
| ACME / TLS issuance | **cert-manager (k3s)** | Issues `mail.dezky.eu` via HTTP-01; Stalwart runs no ACME (80/443 are Traefik's) |
|
||||
| Cert delivery to mail ports | **`cert-sync.sh` (host)** | Reads the cluster TLS secret via local kubeconfig, reloads Stalwart on change |
|
||||
| Storage | **RocksDB on host disk** | Intentionally independent of the in-cluster Postgres |
|
||||
| Domain/DKIM provisioning | **platform-api (k3s)** | JMAP management API at `http://<node>:8080/jmap`, Basic auth |
|
||||
| Audit webhook | **Stalwart → platform-api** | POSTs to `https://api.dezky.eu/ingest/...`, HMAC-signed |
|
||||
|
||||
**platform-api Fleet env** (must match the host's `config.env`):
|
||||
|
||||
```
|
||||
STALWART_API_URL=http://<node-internal-ip>:8080
|
||||
STALWART_ADMIN_USER=admin
|
||||
STALWART_ADMIN_PASSWORD=<same as host STALWART_ADMIN_PASSWORD>
|
||||
STALWART_WEBHOOK_SECRET=<same as host STALWART_WEBHOOK_SECRET>
|
||||
STALWART_PROVISIONING_ENABLED=true
|
||||
```
|
||||
|
||||
The firewall already lets the k3s pod CIDR reach host `:8080` while blocking the
|
||||
world, so no extra rule is needed.
|
||||
|
||||
> **Forward dependency:** `cert-sync.sh` needs the fleet layer to create the
|
||||
> `mail/mail-tls` cert secret. Until then Stalwart serves the self-signed
|
||||
> bootstrap cert `install.sh` generated; the timer swaps in the real cert
|
||||
> automatically once it exists.
|
||||
|
||||
### Finally, backups
|
||||
|
||||
```bash
|
||||
sudo ./restic/install.sh # restic + key + nightly timer
|
||||
# upload the printed public key to BOTH Storage Boxes (port 23), then:
|
||||
sudo ./restic/install.sh # re-run to init the repos
|
||||
sudo /opt/dezky-backup/backup.sh # first backup (or wait for 03:20 UTC)
|
||||
```
|
||||
|
||||
Needs `RESTIC_PASSWORD` + `BACKUP_PRIMARY_REPO` (+ `BACKUP_DR_REPO`) in
|
||||
`config.env`. See backups below.
|
||||
|
||||
## Backups (Restic)
|
||||
|
||||
Nightly at **03:20 UTC**: back up to the **primary Storage Box**, apply
|
||||
retention, `restic check`, then a dedup-aware **`copy` to the Helsinki DR box**.
|
||||
|
||||
| What | Why |
|
||||
|------|-----|
|
||||
| `/opt/stalwart/data` + `/etc` | Mail store (RocksDB) + config — the crown jewels |
|
||||
| `/var/lib/rancher/k3s/server/db/snapshots` | k3s **etcd snapshots** (cluster state) |
|
||||
| `/var/lib/rancher/k3s/storage` | local-path PVCs — incl. where fleet `pg_dump`/`mongodump` CronJobs land |
|
||||
|
||||
- **Retention:** 7 daily · 4 weekly · 6 monthly (tunable via `BACKUP_RETENTION`).
|
||||
- **Storage Box quirk:** SSH/SFTP on **port 23**, key auth. A single ssh-config
|
||||
wildcard covers both boxes, so one key + `restic copy` mirrors primary → DR.
|
||||
- **Encryption:** repos are Restic-encrypted with `RESTIC_PASSWORD`. **Store it
|
||||
offline** — losing it makes every backup unrecoverable.
|
||||
- **Alerting:** set `BACKUP_HEALTHCHECK_URL` (e.g. healthchecks.io) for a
|
||||
dead-man's switch — get paged when a nightly run is missed, not when you need
|
||||
to restore.
|
||||
|
||||
> **Database consistency:** live DB files in PVCs are crash-consistent at best.
|
||||
> The reliable path is logical dumps — the **fleet layer** adds `pg_dump` /
|
||||
> `mongodump` CronJobs that write into a backup PVC under
|
||||
> `/var/lib/rancher/k3s/storage`, which Restic then captures. Restore those
|
||||
> dumps, not the raw data dirs.
|
||||
|
||||
**Run restore drills.** A backup you've never restored isn't a backup:
|
||||
|
||||
```bash
|
||||
sudo /opt/dezky-backup/restore.sh snapshots
|
||||
sudo /opt/dezky-backup/restore.sh restore latest /tmp/restore-test
|
||||
```
|
||||
|
||||
## ⚠️ Lockout safety
|
||||
|
||||
- **Always** open a second SSH session and confirm access **before** closing the
|
||||
one you ran bootstrap in.
|
||||
- Management is pinned to home + office IPs. **Residential IPs can change** — if
|
||||
yours does, you'll be locked out of SSH/6443 (public services stay up).
|
||||
- **Break-glass:** Hetzner's **KVM/LARA** console (Robot panel) is out-of-band
|
||||
and bypasses the firewall entirely. From there you can edit
|
||||
`/etc/nftables.d/dezky-fw.nft` or update `config.env` + re-run `firewall.sh`.
|
||||
- If your IP changes often, widen `MGMT_ALLOW_V4` to a small prefix, or we add a
|
||||
WireGuard bastion later.
|
||||
|
||||
## Verifying after apply
|
||||
|
||||
```bash
|
||||
sudo nft list table inet dezky_fw # our rules
|
||||
sudo nft list ruleset | grep -c KUBE # k3s rules still present (>0 once k3s runs)
|
||||
sudo systemctl status dezky-firewall # enabled + active (exited)
|
||||
sudo fail2ban-client status sshd # jail active
|
||||
# From a NON-allowlisted network, `ssh` should hang/timeout; 443 should work.
|
||||
```
|
||||
|
||||
## Host layer status
|
||||
|
||||
**Complete:** hardening ✅ · firewall ✅ · k3s registration ✅ · Stalwart ✅ ·
|
||||
backups ✅.
|
||||
|
||||
Next is the **Fleet/GitOps layer** (`infrastructure/production/fleet/`):
|
||||
cert-manager + `ClusterIssuer`, ingress, the data tier (Postgres/Mongo/Redis),
|
||||
Authentik, OCIS + Collabora, and portal + platform-api — plus the
|
||||
`mail/mail-tls` cert and the DB-dump CronJobs this layer's `cert-sync` and
|
||||
backups depend on.
|
||||
Reference in New Issue
Block a user