04 — Control plane (TypeScript / Node)

The control plane (CP) is the only component that knows about all tenants. It owns the tenant registry, runs provisioning, signs LiveKit tokens, and routes LiveKit events. It was a Go design in the discussions; the decision is now TypeScript on Node, which also matches the Pulumi Automation API (TypeScript is a first-class language there) and the admin console.
Stack
| Concern | Choice | Notes |
|---|---|---|
| Runtime | Node LTS, TypeScript strict | One repo, two entrypoints: api and worker (same image) |
| HTTP | Fastify (or Hono) + zod schemas → OpenAPI | Schemas shared with the console |
| Admin console | TanStack Start (React) | Authorization inside every server function/route — never only beforeLoad |
| DB access | Kysely or Drizzle on pg | SQL migrations in migrations/ (node-pg-migrate) |
| Queue | pg-boss (core Postgres) | No extra infra; singletonKey = tenant id for per-tenant serialization |
| IaC | Pulumi Automation API (@pulumi/pulumi/automation) | Drives the Pulumi CLI at runtime; state in Postgres DIY backend |
| Auth | jose + Zitadel JWKS | Verify issuer, audience, org id, role on every request |
| LiveKit | livekit-server-sdk | AccessToken + VideoGrant, WebhookReceiver |
| Secrets | Infisical SDK | Machine identity for the CP; short-lived reads |
| S3 | @aws-sdk/client-s3 | Bucket, lifecycle, presign; RustFS admin API for keys |
| Observability | pino logs, OpenTelemetry traces, Prometheus metrics | tenant_id label on every signal |
Pulumi Automation API drives the CLI, so the worker image must ship the Pulumi CLI and the provider plugins it needs; pin versions.
Process model
api process stateless, N replicas verifies JWT, validates, enqueues jobs,
serves admin/tenant/internal APIs
worker process M replicas, queue consumers runs sagas, Pulumi, adapters, reconciler
broker path part of api, latency-critical token signing + webhook router
(deployable as its own service so it scales and fails independently)
The broker/router must not queue behind provisioning work: it runs in the
api process (or a dedicated broker deployment) with no dependency on
workers, only on the DB and Infisical (cached).
API surfaces
| Surface | Audience | AuthN | Examples |
|---|---|---|---|
/admin/v1 | Fortspace operators | Zitadel login, platform org, role operator | POST /tenants, POST /tenants/{id}/suspend, GET /shards, POST /tenants/{id}/relocate |
/tenant/v1 | Tenant admins | Zitadel login, role tenant_admin; tenant derived from token | PUT /sso, POST /domains, GET /usage, POST /modules/{name} |
/internal/v1 | Tenant stacks, LiveKit | Service-account JWT; LiveKit webhook signature | POST /livekit/token, POST /livekit/webhook, POST /tenants/self/heartbeat |
Rules: no tenant id in a URL accepted from a tenant-level caller (the token
supplies it); every mutating endpoint is idempotent via Idempotency-Key; all
errors use RFC 9457 problem JSON.
Data model (core Postgres, schema control_plane)
CREATE TABLE tenants (
id uuid PRIMARY KEY,
slug text NOT NULL UNIQUE CHECK (slug ~ '^[a-z][a-z0-9-]{1,30}$'),
name text NOT NULL,
tier text NOT NULL DEFAULT 'standard', -- standard | dedicated_data | enterprise
status text NOT NULL, -- REQUESTED | PROVISIONING | PROVISIONING_FAILED | ACTIVE | SUSPENDED | OFFBOARDING | ERASED
zitadel_org_id text,
orchestrator_type text NOT NULL DEFAULT 'coolify', -- never store vendor-specific columns
orchestrator_resource_id text, -- project uuid
db_shard_id uuid REFERENCES db_shards(id),
db_name text,
bucket text,
region text NOT NULL DEFAULT 'eu',
created_at timestamptz NOT NULL DEFAULT now(),
erased_at timestamptz
);
CREATE TABLE tenant_domains (
tenant_id uuid REFERENCES tenants(id) ON DELETE CASCADE,
domain text PRIMARY KEY, -- verified email/login domain (unique across tenants)
verified_at timestamptz
);
CREATE TABLE tenant_modules (
tenant_id uuid REFERENCES tenants(id) ON DELETE CASCADE,
module text NOT NULL, -- meet | chat | mail | files ...
version text NOT NULL,
state text NOT NULL,
PRIMARY KEY (tenant_id, module)
);
CREATE TABLE db_shards (
id uuid PRIMARY KEY, name text UNIQUE, host text, port int DEFAULT 6432,
tenant_count int DEFAULT 0, max_tenants int DEFAULT 50, status text DEFAULT 'ACTIVE'
);
CREATE TABLE room_routes ( -- derived; used by the event router
tenant_id uuid REFERENCES tenants(id) ON DELETE CASCADE PRIMARY KEY,
slug text UNIQUE, webhook_url text NOT NULL, webhook_secret_ref text NOT NULL
);
CREATE TABLE provisioning_runs (
id uuid PRIMARY KEY, tenant_id uuid, action text, step text, state text,
attempt int, error text, started_at timestamptz, finished_at timestamptz
);
CREATE TABLE audit_log (
id bigserial PRIMARY KEY, at timestamptz DEFAULT now(), actor text, actor_org text,
tenant_id uuid, action text, target text, detail jsonb -- ids only, no content
);
Secrets are never columns: only Infisical paths/refs (*_ref) are stored.
Provisioner
- Triggered by
POST /admin/v1/tenants; writes the tenant row, enqueuestenant.provisionwithsingletonKey = tenantId. - The job loads (or creates) the Pulumi stack
fortspace/tenant/<slug>viaLocalWorkspace.createOrSelectStack, passes config, callsup. - Each step is a Pulumi resource implemented as a dynamic provider wrapping
an adapter call (Zitadel org, Infisical folder, RustFS bucket, shard DB,
Coolify project, DNS).
createis create-or-get;deleteis the compensation;readsupportspulumi refreshfor drift detection. - On failure: job is retried with backoff (3×), then state
PROVISIONING_FAILEDand an operator alert;provisioning_runskeeps the step trace. - Detail and diagram: 05.
Why Pulumi and adapters
Coolify has no maintained Pulumi/Terraform provider, and mutating Coolify's runtime state behind its back splits authority. The compromise: Pulumi owns the desired set of tenant resources and their lifecycle; every mutation goes through Coolify's own API via the adapter, so Coolify stays the runtime source of truth. Base infrastructure (VPS, DNS, firewall, Coolify install) is regular Pulumi with real providers.
Reconciler
A scheduled job per tenant (nightly + on demand):
pulumi refresh(dynamic providersread()call the adapters).- Health probes: go-meet
/healthz, tuwunel, OIDC discovery. - Compare registry ↔ reality: orphan buckets/DBs/Coolify projects (alert, do
not auto-delete), missing resources (auto-heal via
up). - Emit drift metric + audit entry.
Auto-heal is limited to idempotent re-creation; destructive actions always need an operator.
Ports and adapters
interface Orchestrator { createStack(t: TenantSpec): Promise<StackRef>;
redeploy(ref: StackRef): Promise<void>; destroy(ref: StackRef): Promise<void>;
health(ref: StackRef): Promise<Health>; }
// Coolify, Dokploy, K8s adapters implement it
interface IdentityPort { createOrg; grantProject; addIdp; createServiceAccount; deactivateOrg }
interface SecretsPort { putFolder; createIdentity; rotate; deleteFolder }
interface ObjectStorePort{ createBucket; setLifecycle; createScopedKey; deleteBucket }
interface DatabasePort { createDatabase; migrate; dropDatabase; relocate }
interface DnsPort { upsert; remove }
Rule: no vendor type crosses a port. The tenant row keeps
orchestrator_type + orchestrator_resource_id, so moving to Dokploy (or
Kubernetes) means writing one adapter and migrating tenants one at a time.
Admin console
TanStack Start app talking to the CP API (same repo, shared zod types). Operator views: tenants, shards, provisioning runs, drift, audit. Tenant-admin views: SSO connection wizard, domains, modules, usage, data export. Server functions re-check the caller's role and tenant on every call.
Failure and availability
| Scenario | Behavior |
|---|---|
| Worker down | Provisioning delayed; calls unaffected |
| API down | Console and new joins fail → run ≥ 2 API replicas |
| Core Postgres down | CP down; running calls continue (tokens already issued, tenant stacks self-sufficient) |
| Infisical down | CP caches broker key in memory for its lifetime; new tenant stacks cannot start (existing keep running) |
| Zitadel down | New logins fail; existing sessions and calls continue |