Skip to main content

05 — Tenant lifecycle

Provisioning saga

State machine​

REQUESTED → PROVISIONING → ACTIVE ⇄ SUSPENDED → OFFBOARDING → ERASED
↕
PROVISIONING_FAILED (retry resumes at the failed step)
StateMeaningAllowed transitions
REQUESTEDRow exists, nothing built→ PROVISIONING
PROVISIONINGSaga running (one job, serialized per tenant)→ ACTIVE, → PROVISIONING_FAILED
PROVISIONING_FAILEDNeeds retry or operator cleanup→ PROVISIONING, → OFFBOARDING
ACTIVEServing→ SUSPENDED, → OFFBOARDING
SUSPENDEDOrg deactivated, stacks stopped, data retained→ ACTIVE, → OFFBOARDING
OFFBOARDINGTeardown + erase running→ ERASED
ERASEDAll data and keys destroyed; slug quarantined 90 days—

Provisioning saga​

Input: { name, slug, tier, modules[], adminEmail, region }. Every step is create-or-get, records its result in provisioning_runs, and has a compensation (right-hand column of the diagram).

  1. Reserve slug, choose shard (tenant_count < max_tenants, status ACTIVE; if none, enqueue shard.add and wait), insert tenant row.
  2. Zitadel: create org; grant the Fortspace project with the roles tenants may delegate; create applications (go-meet PKCE client with exact redirect and post-logout URIs, back-channel logout URI; Tuwunel OIDC client); create the stack service account and key.
  3. Infisical: folder /tenants/<slug>/{meet,chat}, machine identity with read-only access to its folder, write all generated secrets.
  4. RustFS: bucket tenant-<slug>, default encryption, lifecycle rules, scoped access key (policy limited to that bucket), key stored in Infisical.
  5. Shard: CREATE DATABASE meet_<slug> + role, CONNECT only for that role; run go-meet migrations (cmd/migrate up).
  6. Route: insert room_routes (slug → internal go-meet URL) and a per-tenant webhook signing secret (stored in Infisical, ref in table).
  7. Coolify: project tenant-<slug>, environment, deploy the tenant Compose template (go-meet, redis, tuwunel). Only the Infisical bootstrap identity and non-secret config are Coolify env vars.
  8. Domains: meet.<slug>.<platform-domain>, chat.<slug>… via wildcard record + Traefik routers.
  9. Probes: go-meet health, OIDC discovery, Tuwunel health, and one real token exchange against the broker for a synthetic room.
  10. Activate: invite the first tenant admin by email (Zitadel invitation), state ACTIVE, audit event.

Idempotency and concurrency​

  • Job key = tenant id → at most one mutating job per tenant (also prevents Pulumi stack lock errors).
  • Steps check for existing resources by deterministic name before creating.
  • A job crash resumes from the last completed step; compensations only run on failure of a forward run or on explicit offboarding.

Suspension​

  1. Zitadel: deactivate org (blocks new logins; existing ID tokens remain valid until expiry).
  2. go-meet: control plane calls a signed POST /internal/admin/suspend → flush Redis sessions, deny new room tokens (broker also refuses to sign).
  3. Coolify: stop tenant stack (data volumes kept).

Resume reverses the order.

Offboarding and erasure​

  1. Export (optional): tenant admin downloads recordings/transcripts (presigned), chat export via Matrix admin tooling.
  2. Stop stack, delete Coolify project (compose + volumes).
  3. DROP DATABASE / DROP ROLE after terminating sessions; decrement shard.
  4. Delete bucket and its access key (after emptying); delete the RustFS SSE key if per-tenant keys are in use.
  5. Delete Infisical folder + identity.
  6. Zitadel: delete org (users, apps, service accounts).
  7. Destroy the tenant backup key; older encrypted backups become unreadable.
  8. Remove routes/DNS; Pulumi destroy, then remove stack.
  9. State ERASED, slug quarantined, audit entry (ids + timestamps only).

A tenant can ask for an erasure certificate: the audit trail of steps with timestamps and resource ids.

Shard management​

OperationTriggerHow
Add shardNo ACTIVE shard under capacityCreate Postgres + PgBouncer as a new Coolify app in tenant-data-pool, register in db_shards
Mark FULLtenant_count >= max_tenantsAllocator skips it
Relocate tenantNoisy neighbor, tier changeSet tenant read-only → pg_dump → restore on target → switch db_shard_id + env → redeploy → verify → drop source
Migrate schemago-meet releaseParallel workers (bounded, e.g. 10), schema_version per tenant tracked in registry

Defaults from the discussions: max_tenants=50, go-meet pool max_open_conns=3, PgBouncer in transaction mode with small default pool; identical Postgres major versions across core and shards.

Upgrades​

  • Tenant stacks pin image tags per module in tenant_modules.version.
  • Rollouts: canary tenant(s) → ring of 10% → all. Each rollout is a job that runs migrations, redeploys, probes, and can roll back via Coolify (rollback).
  • Zitadel and LiveKit upgrades are platform changes handled outside tenant jobs, with their own maintenance windows.