A root-run z-push-admin (kubectl exec defaults to root) left a root-owned 'users' file on the state PVC; Apache runs as www-data, so every request 500'd with 'Not possible to write to the configured state directory'. An initContainer now normalizes ownership on every start (state is disposable, ownership isn't precious), and the docs say to exec z-push-admin as www-data.
dezky production — fleet (k3s app tier)
Pipeline verified end-to-end 2026-06-10: push to main → build → deploy.
k3s manifests for the dezky application tier that runs in-cluster on the
Hetzner AX41 node (see ../host/README.md for the host layer). This layer
deploys the first-party apps:
| App | Image | Public host | Internal Service |
|---|---|---|---|
| platform-api | git.lastcloud.io/ronnibaslund/dezky/platform-api |
api.dezky.eu |
platform-api.dezky-apps:3001 |
| portal | git.lastcloud.io/ronnibaslund/dezky/portal |
app.dezky.eu |
portal.dezky-apps:3000 |
| booking | git.lastcloud.io/ronnibaslund/dezky/booking |
booking.dezky.eu |
booking.dezky-apps:3000 |
| operator | git.lastcloud.io/ronnibaslund/dezky/operator |
operator.dezky.eu |
operator.dezky-apps:3000 |
All of them live in the dezky-apps namespace. The data tier (Postgres/Mongo/
Redis), Authentik and OCIS are added by other parts of the fleet layer and live
in their own namespaces; these manifests reference them by cluster DNS only.
Files
apps/
├── kustomization.yaml # bundles the non-secret resources
├── namespace.yaml # dezky-apps namespace
├── redirect-middleware.yaml # per-router HTTP→HTTPS redirect (ACME-safe)
├── platform-api.yaml # Deployment + Service + Ingress (api.dezky.eu)
├── platform-api-config.yaml # non-secret ConfigMap (Stalwart URL, toggles)
├── portal.yaml # Deployment + Service + Ingress (app.dezky.eu)
├── booking.yaml # Deployment + Service + Ingress (booking.dezky.eu)
├── operator.yaml # Deployment + Service + Ingress (operator.dezky.eu)
└── secrets.example.yaml # SECRET TEMPLATE — never commit real values
ci/
├── gitea-runner.yaml # in-cluster Gitea Actions runner (+ dind)
└── ci-deployer.yaml # ServiceAccount/RBAC the CI deploy job uses
Prerequisites (other fleet layers)
These manifests assume the cluster already has:
- Traefik ingress controller (ships with k3s) —
ingressClassName: traefik. - cert-manager with a
ClusterIssuernamedletsencrypt-prod(HTTP-01). The Ingresses request TLS certs via thecert-manager.io/cluster-issuerannotation; cert-manager fills the named*-tlsSecrets. - MongoDB reachable at
mongo.dezky-data.svc.cluster.local:27017. - Authentik reachable publicly at
https://auth.dezky.euwith OIDC clients provisioned for the portal. - Stalwart on the host, reachable from the pod CIDR at its node-internal IP
on
:8080(JMAP management). Update the placeholder IP inplatform-api-config.yaml.
DNS
Point these A/AAAA records at the AX41 public IP:
api.dezky.eu → <AX41 IP>
app.dezky.eu → <AX41 IP>
booking.dezky.eu → <AX41 IP>
Deploy
Push to main = release. CI (.gitea/workflows/ci.yml, runner in-cluster —
see ci/gitea-runner.yaml) typechecks + tests, builds each app image tagged
:latest and the commit SHA, pushes to the Gitea registry, then the deploy job
pins the kustomization to that SHA (kustomize edit set image), runs
kubectl apply -k apps/ and waits for the rollouts. No GitOps controller, no
bot commits — the pipeline that built the image deploys it.
One-time bootstrap for the deploy job's cluster access:
# 1) ServiceAccount + RBAC (admin scoped to dezky-apps).
kubectl apply -f ci/ci-deployer.yaml
# 2) Mint a kubeconfig from its token and store it as the Gitea repo secret
# KUBECONFIG_B64 (repo Settings → Actions → Secrets). API server address is
# the in-cluster service IP — the runner's jobs run inside the cluster.
TOKEN=$(kubectl -n dezky-apps get secret ci-deployer-token -o jsonpath='{.data.token}' | base64 -d)
CA=$(kubectl -n dezky-apps get secret ci-deployer-token -o jsonpath='{.data.ca\.crt}')
cat <<EOF | base64 | pbcopy # paste into the KUBECONFIG_B64 secret
apiVersion: v1
kind: Config
clusters:
- name: dezky
cluster:
server: https://10.43.0.1:443
certificate-authority-data: $CA
users:
- name: ci-deployer
user:
token: $TOKEN
contexts:
- name: dezky
context: {cluster: dezky, user: ci-deployer, namespace: dezky-apps}
current-context: dezky
EOF
Manual / break-glass deploy (first boot, runner down, secrets changed):
# Real Secrets are applied out-of-band (NOT from git) and the Deployments
# won't start without them. Copy the template, fill in values, apply.
cp apps/secrets.example.yaml /tmp/dezky-secrets.yaml
$EDITOR /tmp/dezky-secrets.yaml # fill every REPLACE / PASSWORD
kubectl apply -f /tmp/dezky-secrets.yaml
rm /tmp/dezky-secrets.yaml
kubectl apply -k apps/
kubectl -n dezky-apps rollout status deploy/platform-api
kubectl -n dezky-apps get ingress,certificate
Required env / secrets
Non-secret config lives in platform-api-config.yaml (ConfigMap) and inline
env: in each Deployment. Secrets are defined in secrets.example.yaml and
must be supplied at deploy time:
platform-api (platform-api-secrets)
| Key | Purpose | How to get it |
|---|---|---|
MONGODB_URI |
Portal/app database connection | From the in-cluster Mongo credentials |
SCHEDULING_CREDENTIAL_KEY |
AES key encrypting stored scheduling creds | openssl rand -hex 32 — back up |
STALWART_ADMIN_PASSWORD |
JMAP management auth | Same value as the host config.env |
STALWART_WEBHOOK_SECRET |
Audit webhook HMAC | Same value as the host config.env |
AUTHENTIK_API_TOKEN |
Authentik admin API (provisioning) | dezky-auth/authentik-secret.AUTHENTIK_BOOTSTRAP_TOKEN |
AUDIT_SIGNING_KEY |
Audit hash-chain signing key | openssl rand -hex 32 — back up |
AUDIT_COLD_ACCESS_KEY / AUDIT_COLD_SECRET_KEY |
Hetzner Object Storage IAM | placeholder until the bucket exists |
OCIS_SVC_PASSWORD |
OCIS service user (files tier) | placeholder until OCIS is deployed |
Non-secret runtime config (Authentik issuer/audience/JWKS, Stalwart URL, OCIS
URL, cold-storage endpoint, feature toggles) lives in
platform-api-config.yaml. PORT and DEZKY_ENV are set inline.
portal (portal-secrets)
| Key | Purpose |
|---|---|
NUXT_OIDC_CLIENT_ID / NUXT_OIDC_CLIENT_SECRET |
Authentik OIDC client |
NUXT_OIDC_REDIRECT_URI |
https://app.dezky.eu/auth/callback |
NUXT_OIDC_SESSION_SECRET |
Session encryption (openssl rand -hex 32) |
NUXT_PUBLIC_AUTH_URL |
Public Authentik URL (login + full sign-out) |
PLATFORM_API_INTERNAL_URL / NUXT_API_BASE / NUXT_PUBLIC_PORTAL_URL are set
inline in portal.yaml.
booking (booking-secrets)
| Key | Purpose |
|---|---|
NUXT_PUBLIC_TURNSTILE_SITE_KEY |
Cloudflare Turnstile site key (public, env-injected) |
PLATFORM_API_INTERNAL_URL / NUXT_PUBLIC_SITE_URL are set inline in
booking.yaml.
Never commit real secret values.
secrets.example.yamlis a template and is deliberately excluded fromkustomization.yaml. Manage real secrets via sealed-secrets / SOPS / the Rancher secret store.