05 — Tenant lifecycle

State machine
REQUESTED → PROVISIONING → ACTIVE ⇄ SUSPENDED → OFFBOARDING → ERASED
↕
PROVISIONING_FAILED (retry resumes at the failed step)
| State | Meaning | Allowed transitions |
|---|---|---|
| REQUESTED | Row exists, nothing built | → PROVISIONING |
| PROVISIONING | Saga running (one job, serialized per tenant) | → ACTIVE, → PROVISIONING_FAILED |
| PROVISIONING_FAILED | Needs retry or operator cleanup | → PROVISIONING, → OFFBOARDING |
| ACTIVE | Serving | → SUSPENDED, → OFFBOARDING |
| SUSPENDED | Org deactivated, stacks stopped, data retained | → ACTIVE, → OFFBOARDING |
| OFFBOARDING | Teardown + erase running | → ERASED |
| ERASED | All data and keys destroyed; slug quarantined 90 days | — |
Provisioning saga
Input: { name, slug, tier, modules[], adminEmail, region }.
Every step is create-or-get, records its result in provisioning_runs, and has
a compensation (right-hand column of the diagram).
- Reserve slug, choose shard (
tenant_count < max_tenants, status ACTIVE; if none, enqueueshard.addand wait), insert tenant row. - Zitadel: create org; grant the Fortspace project with the roles tenants may delegate; create applications (go-meet PKCE client with exact redirect and post-logout URIs, back-channel logout URI; Tuwunel OIDC client); create the stack service account and key.
- Infisical: folder
/tenants/<slug>/{meet,chat}, machine identity with read-only access to its folder, write all generated secrets. - RustFS: bucket
tenant-<slug>, default encryption, lifecycle rules, scoped access key (policy limited to that bucket), key stored in Infisical. - Shard:
CREATE DATABASE meet_<slug>+ role,CONNECTonly for that role; run go-meet migrations (cmd/migrate up). - Route: insert
room_routes(slug → internal go-meet URL) and a per-tenant webhook signing secret (stored in Infisical, ref in table). - Coolify: project
tenant-<slug>, environment, deploy the tenant Compose template (go-meet, redis, tuwunel). Only the Infisical bootstrap identity and non-secret config are Coolify env vars. - Domains:
meet.<slug>.<platform-domain>,chat.<slug>…via wildcard record + Traefik routers. - Probes: go-meet health, OIDC discovery, Tuwunel health, and one real token exchange against the broker for a synthetic room.
- Activate: invite the first tenant admin by email (Zitadel invitation), state ACTIVE, audit event.
Idempotency and concurrency
- Job key = tenant id → at most one mutating job per tenant (also prevents Pulumi stack lock errors).
- Steps check for existing resources by deterministic name before creating.
- A job crash resumes from the last completed step; compensations only run on failure of a forward run or on explicit offboarding.
Suspension
- Zitadel: deactivate org (blocks new logins; existing ID tokens remain valid until expiry).
- go-meet: control plane calls a signed
POST /internal/admin/suspend→ flush Redis sessions, deny new room tokens (broker also refuses to sign). - Coolify: stop tenant stack (data volumes kept).
Resume reverses the order.
Offboarding and erasure
- Export (optional): tenant admin downloads recordings/transcripts (presigned), chat export via Matrix admin tooling.
- Stop stack, delete Coolify project (compose + volumes).
DROP DATABASE/DROP ROLEafter terminating sessions; decrement shard.- Delete bucket and its access key (after emptying); delete the RustFS SSE key if per-tenant keys are in use.
- Delete Infisical folder + identity.
- Zitadel: delete org (users, apps, service accounts).
- Destroy the tenant backup key; older encrypted backups become unreadable.
- Remove routes/DNS; Pulumi
destroy, then remove stack. - State ERASED, slug quarantined, audit entry (ids + timestamps only).
A tenant can ask for an erasure certificate: the audit trail of steps with timestamps and resource ids.
Shard management
| Operation | Trigger | How |
|---|---|---|
| Add shard | No ACTIVE shard under capacity | Create Postgres + PgBouncer as a new Coolify app in tenant-data-pool, register in db_shards |
| Mark FULL | tenant_count >= max_tenants | Allocator skips it |
| Relocate tenant | Noisy neighbor, tier change | Set tenant read-only → pg_dump → restore on target → switch db_shard_id + env → redeploy → verify → drop source |
| Migrate schema | go-meet release | Parallel workers (bounded, e.g. 10), schema_version per tenant tracked in registry |
Defaults from the discussions: max_tenants=50, go-meet pool max_open_conns=3,
PgBouncer in transaction mode with small default pool; identical Postgres major
versions across core and shards.
Upgrades
- Tenant stacks pin image tags per module in
tenant_modules.version. - Rollouts: canary tenant(s) → ring of 10% → all. Each rollout is a job that
runs migrations, redeploys, probes, and can roll back via Coolify
(
rollback). - Zitadel and LiveKit upgrades are platform changes handled outside tenant jobs, with their own maintenance windows.