Identifying the company tenant by slug in env was fragile — every
purge/recreate changed the slug (or id) and the apex guard chased reality
through three config flips in one day. The identity now lives ON the
tenant document: isPlatformTenant, operator-set from the tenant page
(single holder — setting it clears the flag everywhere else), guarded so
tenant admins can't set it on themselves through the shared PATCH route.
The dezky.eu apex guard reads the flag; PLATFORM_TENANT_SLUG is gone.
Dev seed flags its seeded tenant. config-rev 5 rolls platform-api.
The recreated company tenant got slug dezky-aps (wizard auto-derives from
the display name 'Dezky ApS'), so the dezky.eu apex guard 409'd it while
the config still said 'dezky'. Also SEED_ENABLED=false in prod — the
seeder resurrected a ghost 'dezky' tenant on every platform-api boot,
which is how the slug landscape kept shifting. config-rev 4 rolls the
pods.
The minimal create modal silently dropped adminName/adminEmail — the invite
only existed in the partner wizard's server path. Operator now gets the
same 5-step wizard UX (organization, domain, first admin, plan with live
price catalog, review) composed client-side: POST /tenants creates +
provisions, then POST /users/invite-tenant-admin (new, operator-only —
lives in UsersModule because UsersModule already imports TenantsModule and
the reverse would be circular) runs the same inviteTenantAdmin flow the
partner gets, and the result view hands over the single-use recovery link
or temp password. Tenant detail page gains an Invite admin action for
retries/successors. PLATFORM_TENANT_SLUG back to 'dezky' (the recreated
company tenant) + config-rev bump to roll platform-api.
The company tenant ended up as slug dezky-aps (the seeded 'dezky' tenant was
deleted), so the hardcoded apex allowance for slug 'dezky' would have
rejected adding dezky.eu to the real tenant. PLATFORM_TENANT_SLUG env
(default 'dezky') now names the only tenant allowed to claim the
PLATFORM_TENANT_DOMAIN apex.
The v0.16 config migration silently dropped the fallback admin — the live
server had ZERO accounts, so every platform-api JMAP call 401'd and tenant
mail provisioning was dead. Bootstrapped via recovery mode on node1
(STALWART_RECOVERY_ADMIN): created the dezky.eu domain + an admin account
with the Admin role and the existing STALWART_ADMIN_PASSWORD.
v0.16 logins use the full address, so STALWART_ADMIN_USER becomes
admin@dezky.eu; config-rev annotation bump rolls platform-api so it picks
up the new env. install.sh follow-ups now document the recovery-mode
bootstrap for rebuilds instead of the defunct fallback-admin promise.
dezky.eu doubles as the platform's infrastructure domain AND the company's
own employee mail domain (added to the dezky tenant via the normal Domains
flow). Guard rails in DomainsService.add:
- a domain already used by ANY other workspace is rejected — Stalwart's
idempotent ensureDomain would otherwise silently share one mail domain
(and its mailboxes) between tenants
- the PLATFORM_TENANT_DOMAIN apex is claimable only by the dezky tenant;
everything under it (per-tenant service domains, auth/api/mail/* infra
hosts) is reserved outright
Set PLATFORM_TENANT_DOMAIN=dezky.eu in the prod ConfigMap (was unset, so
prod service domains would have been {slug}.dezky.local) and align the
seeded dezky tenant's display domain with the environment.
The operator infrastructure page probed docker-compose hostnames
(stalwart/postgres/redis/traefik…) which don't resolve in k3s — 7 of 9
services showed down. Probe targets now come from HEALTH_* env vars with
the compose names as dev defaults; platform-api-config.yaml sets the
in-cluster/host addresses. 'disabled' omits a service from the report —
used for OCIS/Collabora until the files tier is deployed.
Operator sign-out hardcoded the dev Authentik end-session URL, so prod
logout landed on auth.dezky.local. Mirror the portal's env-driven pattern
(NUXT_PUBLIC_AUTH_URL/NUXT_PUBLIC_OPERATOR_URL with .local fallbacks).
Expose authUrl/operatorUrl via public runtimeConfig and use them for the
Authentik admin links and the cosmetic host labels (sidebar, eyebrows,
auth-page hints). Portal: signed-out + webmail copy now derive their hosts
from runtime config (new public.mailUrl, NUXT_PUBLIC_MAIL_URL in prod).
nuxt-oidc-auth persists sessions via useStorage('oidc'), whose default
mount is per-pod memory — broken at >1 replica (random 401s) and every
deploy logged all users out. A nitro plugin now mounts 'oidc' on the
dezky-data Redis (db 1, app-prefixed keys, 14d TTL) when SESSION_REDIS_URL
is set; dev keeps the memory driver with no Redis required. Replicas back
to 2 for both apps.
nuxt-oidc-auth stores sessions in per-pod memory. With 2 replicas, any
request balanced to the pod that didn't handle the login 401s — in practice
roughly half of all operator API calls failed after sign-in. One replica
until sessions move to shared storage (nitro storage on the dezky-data
Redis), then scale back up. Already scaled live; this pins the manifests so
the next deploy doesn't undo it.
CI builds the Nuxt images with no env, so nuxt.config bakes empty OIDC
client creds and .local Authentik URLs into runtimeConfig — sign-in
dead-ended on the app's own /auth/login. Nitro env overrides only apply
when the var name matches the runtimeConfig path
(oidc.providers.oidc.* -> NUXT_OIDC_PROVIDERS_OIDC_*), so production
secrets need that second set of names; the plain NUXT_OIDC_* ones only
work in dev. Also pin NUXT_OIDC_TOKEN_KEY/AUTH_SESSION_SECRET so sessions
survive pod restarts. Live secrets patched on the cluster accordingly.
gitea/runner can only bind-mount a UNIX-socket docker host into job
containers — the old tcp://localhost:2376 + TLS daemon address cannot be
mounted, so build jobs still had no docker API. Share dind's
/var/run/docker.sock with the runner via a /var/run emptyDir and drop the
DOCKER_HOST/TLS env; the runner auto-finds the socket and the bind path
resolves inside dind where the socket lives.
gitea/runner 1.x no longer auto-mounts the docker daemon into job
containers (act_runner 0.2.x did), so 'docker build' in the build jobs
failed with 'cannot connect to /var/run/docker.sock'. container.docker_host
"" restores find-and-mount.
Gitea 1.26 never marked finished jobs complete with the deprecated
act_runner 0.2.11: the runner ran the job, logged 'Job succeeded' and freed
its slot, but Gitea kept the job 'Running' forever, so dependent jobs
(build -> deploy) were never dispatched. gitea/runner is the successor
project; config, env vars and the .runner registration file are unchanged.
Push to main = release: after build, a deploy job pins each app image to the
commit SHA (kustomize edit set image), kubectl-applies fleet/apps and waits
for the rollouts. The runner already runs in-cluster, so it reaches the API
server on the in-cluster service IP with a kubeconfig for the new ci-deployer
ServiceAccount (namespace-scoped admin, KUBECONFIG_B64 repo secret).
The drafted Flux sync/image-automation layer is removed — a GitOps controller
plus bot tag-bump commits is more machinery than a single-node cluster needs.
Sortable image tags and $imagepolicy markers go with it.
Also: per-router ACME-safe HTTP->HTTPS redirects for the app ingresses,
platform-api prod config completed (Authentik JWT/JWKS + admin API, Stalwart
via the cni0 gateway IP, OCIS/cold-storage placeholders until those tiers
exist) and the secrets template/README updated to match.
- Dockerfile for the operator app (same pattern as portal/booking).
- Env-driven auth/app base URLs in nuxt.config so one build serves
dev (.local) and production (.eu).
- Deployment + Service + Ingress on operator.dezky.eu.
- Add operator to the typecheck matrix.
- Pin the helm-controller chart version (unset = silent latest upgrades) and
move the image tag under global.image per the 2026.5 chart layout.
- Authentik 2026.5 enforces a per-provider grant_types allowlist; empty list
rejected every authorize request. Allow authorization_code + refresh_token
for portal and operator providers.
- Fix the portal redirect URI to the nuxt-oidc-auth callback path.
- Serve the auth ingress on :80 with a per-router HTTPS redirect so the
cert-manager HTTP-01 solver keeps working.
Add an act_runner config.yaml (ConfigMap, CONFIG_FILE env): capacity 4 so the
typecheck matrix + image builds run in parallel instead of one-at-a-time, and
cache.enabled: false (we removed the setup-node cache; the cache server isn't
reachable from the DinD job containers anyway).
Self-registering act_runner on node1 with a privileged docker:dind sidecar so
workflow jobs can build + push app images (k3s has containerd only, no Docker
daemon). Labels ubuntu-latest + docker; state persisted on a Longhorn PVC. The
registration token is applied out-of-band as the gitea-runner-token Secret
(not in git). Verified: runner declared successfully, dind API up.
pg_dumpall (all Postgres DBs + roles) and mongodump (all Mongo DBs) write
gzipped dumps to the hostPath /opt/dezky-backup/dumps at 02:50/02:52 UTC, which
the host Restic job (03:20) ships to the Storage Box. Each keeps the last 7
local dumps; Restic holds the real off-box retention.
- pods run as root (hostPath dir is root-owned, as is the host Restic reader)
- mongo job uses bash (mongo:7 /bin/sh is dash → no pipefail)
- creds from postgres-secret / mongo-secret via secretKeyRef
Verified: both jobs Complete, dumps present on the host
(postgres-all ~2.2MB w/ Authentik data, mongo archive).
Brand CSS only reaches the flow shadow DOM via CSS vars (colors), not the
logo/favicon (deeper shadow root) or the "Powered by authentik" footer (light
DOM). So, dev-style: serve real dezky assets + sed the bundle.
- web-assets/: dezky-logo.svg, dezky-favicon.svg, dezky-bg.svg (carbon).
- server-rebrand.py: patches the authentik-server Deployment with an
initContainer that copies /web/dist to an emptyDir, drops the svgs into
assets/icons, and seds "Powered by authentik" -> "Powered by Dezky".
- brand.yaml: branding_logo / branding_favicon / branding_default_flow_background
point at the served svgs; auth-flow title "Welcome to Dezky"; signal-green CSS.
Verified live: login now matches dev (logo, title, carbon bg, green button,
favicon, Powered by Dezky). Durability caveat documented (reverts on helm
upgrade).
branding_logo / branding_default_flow_background are file-path fields (reject
data URIs), so the dezky logo + carbon background are injected via the brand's
custom CSS (data URIs allowed there): logo replaces the authentik wordmark,
background overrides the forest. Auth-flow title -> "Welcome to Dezky".
Signal-green primary button retained.
Mirror the dev Authentik config in prod via blueprints, applied & successful on
node1:
- brand.yaml: dezky branding on the default brand (title + signal-green custom
CSS) — login page now in dezky colors.
- portal-application.yaml / operator-application.yaml: dezky-portal &
dezky-operator OIDC apps/providers (prod redirect URLs) + the
dezky-platform-admins group & operator access policy.
Two 2026.5 gotchas handled + documented in README:
- invalidation_flow is now REQUIRED on OAuth2 providers (added via !Find).
- ConfigMap mounts are symlinks (discovery can't read them) → worker uses an
initContainer that copies them to an emptyDir as real files. (chart
worker.volumes didn't apply on this version; patch reverts on helm upgrade —
noted as a durability TODO.)
Client secrets (PORTAL/OPERATOR_OIDC_CLIENT_SECRET) live in authentik-secret;
the apps must reuse them.
Adds the production cluster foundation (authored + applied live on node1):
- cert-manager via the k3s HelmChart controller + letsencrypt staging/prod
ClusterIssuers (HTTP-01 / Traefik).
- Longhorn config for single-node (values: replica=1, default StorageClass,
Retain) + backup-to-Hetzner-Object-Storage credential template.
- In-cluster data tier (dezky-data): Postgres 16 (with Authentik+OCIS DB init),
MongoDB 7, Redis 7 as StatefulSets on Longhorn, + secret template.
- bootstrap.sh: install open-iscsi/nfs-common + enable iscsid (Longhorn prereq).
- RUNBOOK.md: full reproducible node1 build order.
Real secrets are generated on-box and kept in Bitwarden — never in git.
Ronni Baslund