Operator sign-out hardcoded the dev Authentik end-session URL, so prod
logout landed on auth.dezky.local. Mirror the portal's env-driven pattern
(NUXT_PUBLIC_AUTH_URL/NUXT_PUBLIC_OPERATOR_URL with .local fallbacks).
Expose authUrl/operatorUrl via public runtimeConfig and use them for the
Authentik admin links and the cosmetic host labels (sidebar, eyebrows,
auth-page hints). Portal: signed-out + webmail copy now derive their hosts
from runtime config (new public.mailUrl, NUXT_PUBLIC_MAIL_URL in prod).
nuxt-oidc-auth persists sessions via useStorage('oidc'), whose default
mount is per-pod memory — broken at >1 replica (random 401s) and every
deploy logged all users out. A nitro plugin now mounts 'oidc' on the
dezky-data Redis (db 1, app-prefixed keys, 14d TTL) when SESSION_REDIS_URL
is set; dev keeps the memory driver with no Redis required. Replicas back
to 2 for both apps.
nuxt-oidc-auth stores sessions in per-pod memory. With 2 replicas, any
request balanced to the pod that didn't handle the login 401s — in practice
roughly half of all operator API calls failed after sign-in. One replica
until sessions move to shared storage (nitro storage on the dezky-data
Redis), then scale back up. Already scaled live; this pins the manifests so
the next deploy doesn't undo it.
CI builds the Nuxt images with no env, so nuxt.config bakes empty OIDC
client creds and .local Authentik URLs into runtimeConfig — sign-in
dead-ended on the app's own /auth/login. Nitro env overrides only apply
when the var name matches the runtimeConfig path
(oidc.providers.oidc.* -> NUXT_OIDC_PROVIDERS_OIDC_*), so production
secrets need that second set of names; the plain NUXT_OIDC_* ones only
work in dev. Also pin NUXT_OIDC_TOKEN_KEY/AUTH_SESSION_SECRET so sessions
survive pod restarts. Live secrets patched on the cluster accordingly.
gitea/runner can only bind-mount a UNIX-socket docker host into job
containers — the old tcp://localhost:2376 + TLS daemon address cannot be
mounted, so build jobs still had no docker API. Share dind's
/var/run/docker.sock with the runner via a /var/run emptyDir and drop the
DOCKER_HOST/TLS env; the runner auto-finds the socket and the bind path
resolves inside dind where the socket lives.
gitea/runner 1.x no longer auto-mounts the docker daemon into job
containers (act_runner 0.2.x did), so 'docker build' in the build jobs
failed with 'cannot connect to /var/run/docker.sock'. container.docker_host
"" restores find-and-mount.
Gitea 1.26 never marked finished jobs complete with the deprecated
act_runner 0.2.11: the runner ran the job, logged 'Job succeeded' and freed
its slot, but Gitea kept the job 'Running' forever, so dependent jobs
(build -> deploy) were never dispatched. gitea/runner is the successor
project; config, env vars and the .runner registration file are unchanged.
Push to main = release: after build, a deploy job pins each app image to the
commit SHA (kustomize edit set image), kubectl-applies fleet/apps and waits
for the rollouts. The runner already runs in-cluster, so it reaches the API
server on the in-cluster service IP with a kubeconfig for the new ci-deployer
ServiceAccount (namespace-scoped admin, KUBECONFIG_B64 repo secret).
The drafted Flux sync/image-automation layer is removed — a GitOps controller
plus bot tag-bump commits is more machinery than a single-node cluster needs.
Sortable image tags and $imagepolicy markers go with it.
Also: per-router ACME-safe HTTP->HTTPS redirects for the app ingresses,
platform-api prod config completed (Authentik JWT/JWKS + admin API, Stalwart
via the cni0 gateway IP, OCIS/cold-storage placeholders until those tiers
exist) and the secrets template/README updated to match.
- Dockerfile for the operator app (same pattern as portal/booking).
- Env-driven auth/app base URLs in nuxt.config so one build serves
dev (.local) and production (.eu).
- Deployment + Service + Ingress on operator.dezky.eu.
- Add operator to the typecheck matrix.
- Pin the helm-controller chart version (unset = silent latest upgrades) and
move the image tag under global.image per the 2026.5 chart layout.
- Authentik 2026.5 enforces a per-provider grant_types allowlist; empty list
rejected every authorize request. Allow authorization_code + refresh_token
for portal and operator providers.
- Fix the portal redirect URI to the nuxt-oidc-auth callback path.
- Serve the auth ingress on :80 with a per-router HTTPS redirect so the
cert-manager HTTP-01 solver keeps working.
Add an act_runner config.yaml (ConfigMap, CONFIG_FILE env): capacity 4 so the
typecheck matrix + image builds run in parallel instead of one-at-a-time, and
cache.enabled: false (we removed the setup-node cache; the cache server isn't
reachable from the DinD job containers anyway).
Self-registering act_runner on node1 with a privileged docker:dind sidecar so
workflow jobs can build + push app images (k3s has containerd only, no Docker
daemon). Labels ubuntu-latest + docker; state persisted on a Longhorn PVC. The
registration token is applied out-of-band as the gitea-runner-token Secret
(not in git). Verified: runner declared successfully, dind API up.
pg_dumpall (all Postgres DBs + roles) and mongodump (all Mongo DBs) write
gzipped dumps to the hostPath /opt/dezky-backup/dumps at 02:50/02:52 UTC, which
the host Restic job (03:20) ships to the Storage Box. Each keeps the last 7
local dumps; Restic holds the real off-box retention.
- pods run as root (hostPath dir is root-owned, as is the host Restic reader)
- mongo job uses bash (mongo:7 /bin/sh is dash → no pipefail)
- creds from postgres-secret / mongo-secret via secretKeyRef
Verified: both jobs Complete, dumps present on the host
(postgres-all ~2.2MB w/ Authentik data, mongo archive).
Brand CSS only reaches the flow shadow DOM via CSS vars (colors), not the
logo/favicon (deeper shadow root) or the "Powered by authentik" footer (light
DOM). So, dev-style: serve real dezky assets + sed the bundle.
- web-assets/: dezky-logo.svg, dezky-favicon.svg, dezky-bg.svg (carbon).
- server-rebrand.py: patches the authentik-server Deployment with an
initContainer that copies /web/dist to an emptyDir, drops the svgs into
assets/icons, and seds "Powered by authentik" -> "Powered by Dezky".
- brand.yaml: branding_logo / branding_favicon / branding_default_flow_background
point at the served svgs; auth-flow title "Welcome to Dezky"; signal-green CSS.
Verified live: login now matches dev (logo, title, carbon bg, green button,
favicon, Powered by Dezky). Durability caveat documented (reverts on helm
upgrade).
branding_logo / branding_default_flow_background are file-path fields (reject
data URIs), so the dezky logo + carbon background are injected via the brand's
custom CSS (data URIs allowed there): logo replaces the authentik wordmark,
background overrides the forest. Auth-flow title -> "Welcome to Dezky".
Signal-green primary button retained.
Mirror the dev Authentik config in prod via blueprints, applied & successful on
node1:
- brand.yaml: dezky branding on the default brand (title + signal-green custom
CSS) — login page now in dezky colors.
- portal-application.yaml / operator-application.yaml: dezky-portal &
dezky-operator OIDC apps/providers (prod redirect URLs) + the
dezky-platform-admins group & operator access policy.
Two 2026.5 gotchas handled + documented in README:
- invalidation_flow is now REQUIRED on OAuth2 providers (added via !Find).
- ConfigMap mounts are symlinks (discovery can't read them) → worker uses an
initContainer that copies them to an emptyDir as real files. (chart
worker.volumes didn't apply on this version; patch reverts on helm upgrade —
noted as a durability TODO.)
Client secrets (PORTAL/OPERATOR_OIDC_CLIENT_SECRET) live in authentik-secret;
the apps must reuse them.
Adds the production cluster foundation (authored + applied live on node1):
- cert-manager via the k3s HelmChart controller + letsencrypt staging/prod
ClusterIssuers (HTTP-01 / Traefik).
- Longhorn config for single-node (values: replica=1, default StorageClass,
Retain) + backup-to-Hetzner-Object-Storage credential template.
- In-cluster data tier (dezky-data): Postgres 16 (with Authentik+OCIS DB init),
MongoDB 7, Redis 7 as StatefulSets on Longhorn, + secret template.
- bootstrap.sh: install open-iscsi/nfs-common + enable iscsid (Longhorn prereq).
- RUNBOOK.md: full reproducible node1 build order.
Real secrets are generated on-box and kept in Bitwarden — never in git.
Ronni Baslund