Tenant onboarding stops working when you assume each step completes in order.

Consider what happens when five integration calls succeed but the sixth one hangs. The database schema is ready. The identity provider has the organization. Billing has the customer. DNS times out. The worker process crashes before it can record that any of this happened.

Now restart the process. Should everything run again? If billing creates a second customer record, the finance team will notice. If DNS retries and the record already exists, some providers return errors. If the worker read from its in-memory cache instead of the database, it might miss that the schema is already live.

Most teams solve this with retries. The queue redelivers the event. The worker runs the same function again. Hope it works this time.

This breaks down the moment any external system changes. A support engineer manually resets a tenant’s DNS. The billing provider decides to rename the customer. Someone deletes the object-storage namespace by accident. A retry queue doesn’t notice these things. It only knows whether its last command succeeded.

Commands versus desired state

A command-based workflow looks like a recipe:

1. Create identity account.
2. Create billing customer.
3. Create schema.
4. Create DNS record.

Each step depends on the previous one. Each step is usually wrapped in a try/catch with a retry. This works while every step succeeds on the first attempt. It starts to wobble when step three succeeds but step four fails. The system has no record of what is already done. It retries from the top or, worse, skips the steps it thinks are complete.

Now look at the desired-state model. Instead of issuing commands, the platform records what the tenant should look like:

{
  "tenantId": "tenant-314",
  "status": "active",
  "identity": { "enabled": true },
  "billing": { "plan": "pro" },
  "database": { "isolation": "schema" },
  "domain": { "hostname": "tenant-314.example.com" }
}

This document is durable. It lives in your database. It survives worker crashes, queue restarts, and deployment rollbacks. Events and queues are only mechanisms for scheduling reconciliation — for telling the system the document changed.

The reconciler’s job is not to replay a workflow. It is to make the real world match the document.

The minimum data model

You need three separate fields to move a tenant from “requested” to “ready” without losing track.

spec stores what the user wants. It does not change unless the user makes a change.

status stores what the platform has observed from external systems. If the DNS provider says the record exists, status reflects that. If the billing provider returned an error, status records the error.

generation tracks which version of the desired state the system has processed. When a user updates their plan from pro to enterprise, you write a new generation number. The reconciler checks: is this the latest generation? If not, it stops. Observers might still be writing status for an older generation. You do not want newer status overwritten by older results.

Recommended fields on the tenant row:

Field	Purpose
tenant_id	Primary key
desired_generation	Which spec version we want
observed_generation	Which spec version we’ve processed
provisioning_status	pending, reconciling, ready, degraded, failed
identity_external_id	Link back to the provider’s resource
billing_external_id	Same for billing
database_schema	Schema name or reference
dns_record_id	Provider-specific record identifier
last_reconciled_at	Timestamp of last successful reconciliation
next_reconcile_at	When the next sweep should run
failure_count	Number of consecutive reconciliation failures
last_error_code	Category of the most recent error

Persist external resource identifiers the moment you create the resource. Do not try to rediscover them later by querying the identity provider for “the tenant-314 organization.” Names collide. Resources get renamed. The ID you stored when you created the resource is the only link you can trust.

Anatomy of one reconciliation cycle

The reconciler runs a loop. Each loop is small. It does one unit of meaningful progress and then exits. The scheduler brings it back later.

The algorithm:

1. Lock the tenant row.
2. Load the spec and the last observed status.
3. If desired_generation == observed_generation and the spec is fully satisfied, mark ready.
4. Otherwise, calculate the difference: what is missing or drifted?
5. Pick one bounded action from the plan.
6. Execute the action with an idempotency key.
7. Record the observation.
8. Update next_reconcile_at based on backoff.
9. Release the lock.

Here’s what this looks like in code:

async function reconcileTenant(tenantId: string): Promise<ReconcileResult> {
  const tenant = await repository.lockTenant(tenantId);

  if (!tenant) {
    return { outcome: "complete" };
  }

  const plan = await planner.calculatePlan(tenant);

  if (plan.isComplete) {
    await repository.markReady(tenant.id, tenant.desiredGeneration);
    return { outcome: "complete" };
  }

  const action = plan.nextAction;
  const result = await executor.execute(action, {
    idempotencyKey: `${tenant.id}:${tenant.desiredGeneration}:${action.type}`,
  });

  await repository.recordObservation(tenant.id, result);

  return {
    outcome: "requeue",
    delayMs: calculateBackoff(tenant.failureCount),
  };
}

This is a pull model. The scheduler pulls tenants into the loop. The loop decides what to do. External events only trigger the loop sooner. They do not decide what the loop does.

The bounded progress rule matters here. A single reconciliation should perform at most one or a few external calls. If the plan is “create identity, create billing, create schema, create DNS,” the reconciler does not run all four in one pass. It does one, records the result, and returns. The next pass picks up where it left off. This keeps each attempt small, recoverable, and easy to retry.

Designing for convergence and idempotency

Convergence is the property that repeating reconciliation with the same desired state moves the system closer to the same final state without creating additional resources. If you run reconcileTenant("tenant-314") fifty times in a row, you should still have exactly one identity organization, one billing customer, one schema, and one DNS record.

This sounds obvious. A lot of systems fail at it.

Provider-side idempotency keys

Every external call should carry an idempotency key. The key is deterministic. For tenant-314’s identity organization, the key is always something like provision-tenant-314:identity. The billing provider sees the same key on the first call and the hundredth call. It returns the same customer ID every time.

Without this, a duplicate event or a manual rerun creates duplicates. Duplicates break convergence.

Compare-and-set database updates

Never overwrite the observed generation with an older value. When you write status, check that desired_generation still matches the tenant’s current spec version. If someone updated the spec between your reads, your observation is stale. Abort and let the next reconciliation cycle pick up the real current state.

await repository.tryUpdateStatus(tenant.id, {
  ...newStatus,
  observedGeneration: tenant.desiredGeneration,
}, {
  where: {
    id: tenant.id,
    desiredGeneration: tenant.desiredGeneration, // must match current spec
  }
});

This is optimistic concurrency in its simplest form.

Deterministic external resource names

When naming resources in external systems, use deterministic names derived from the tenant ID. tenant-314-db is better than tenant-314-db-uuid-abc123. Deterministic names make recovery obvious. If the reconciler finds a resource with the right name, it is likely the right resource. Random names force you to track IDs.

Resource fingerprints

Some providers do not guarantee unique naming. In those cases, tag resources on creation with the tenant ID and generation. Reconcile by querying for your tag and comparing the result against the spec. The tag acts as a fingerprint.

Safe repeated updates

Mark all update operations as idempotent in your provider client. If the provider returns “already exists,” treat that as a possible success, not always an error. A 409 during creation can mean “the resource exists and it is exactly what we need” or “the resource exists but it is incomplete.” Record the external ID, fetch the resource, and decide.

Separating retryable and terminal failures

Not all failures deserve eternal retries. A DNS timeout is retryable. A misconfigured billing plan that the user must fix is terminal. Retrying a terminal failure wastes cycles and fills logs.

Create two error classes:

class RetryableError extends Error {
  constructor(message: string) {
    super(message);
    this.name = "RetryableError";
  }
}

class TerminalError extends Error {
  constructor(message: string, public readonly userAction: string) {
    super(message);
    this.name = "TerminalError";
  }
}

A TerminalError halts the reconciliation loop for that tenant and sets the status to failed. The user fixes the problem. The spec changes. A new generation triggers a fresh reconciliation.

Watches, scheduled sweeps, and missed events

Event-driven reconciliation is fast. When the spec changes, the event queue wakes the reconciler. But events are not the full truth.

External systems drift. A support engineer deletes a DNS record through a console. The identity provider rotates an organization. Billing applies a late fee. These changes do not emit events to your queue.

Kubernetes controllers solve this with a combination of watches and polling. Crossplane, which applies the controller pattern to cloud resources, uses watches where the external API supports webhooks, and falls back to periodic polling when it does not. The polling interval is a trade-off: faster repair versus more external API load.

Your system should do the same. Run a scheduled sweep — perhaps every five minutes for critical resources, every hour for stable ones. The sweep scans tenants where next_reconcile_at <= now() and invokes the reconciler. A past-due sweep acts as a safety net. It catches drift, missed events, and workers that died mid-reconciliation.

Events accelerate convergence. Sweeps guarantee correctness.

Stale observations and reconciliation thrashing

Two advanced problems appear once your system enters production.

Stale observations

The reconciler reads a cached status entry. It decides the tenant needs a DNS update. It calls the provider. But the cache was written by a previous reconciliation cycle that was itself outdated. The reconciler repeats an unnecessary write.

This is not just inefficiency. Some providers treat writes as full replacements. An unnecessary update can reset fields you did not intend to touch.

Mitigations:

• Read the authoritative database record before destructive operations.
• Store the observed_generation with every status write. Reject updates based on older generations.
• Use resource versions or ETags if the provider supports them.
• Apply freshness thresholds for cached observations. If the cache is older than your sweep interval, re-read the live resource.

Thrashing

Two controllers reconcile the same tenant at the same time. Or a single transient failure produces a tight reconciliation loop: fail, requeue with 100ms backoff, fail, requeue. The system spins without making progress.

Kubernetes v1.36 specifically addressed stale controller-cache visibility to reduce exactly this kind of thrashing. Crossplane documents a token-bucket circuit breaker that stops a resource from entering a tight reconciliation loop.

Mitigations for your implementation:

• Single owner per tenant. One reconciler instance owns a tenant at a time. Use a distributed lock or a database advisory lock.
• Exponential backoff with jitter. delayMs = min(base * 2^failures + jitter, maxDelay). This spaces out attempts after repeated failures.
• Per-resource token buckets. Limit how many reconciliation attempts a tenant can receive in a time window.
• Circuit breaker. If a tenant fails N times in M minutes, halt reconciliation for a cooldown period. Surface the failure to the user.
• Terminal-error state. Transition to failed when the error is clearly not recoverable without user input.

Deletion and finalization

Deleting a tenant is not a database row removal.

If you simply delete the row, the external resources you created — identity providers, DNS records, schemas, storage buckets — become orphans. They may continue to consume budget. Some providers keep billing for resources until you explicitly release them.

Use a terminal lifecycle:

Active -> Deleting -> Deleted

When the user requests deletion:

1. Set provisioning_status = deleting. Stop accepting writes.
2. Disable authentication access. Revoke tokens, disable SSO.
3. Stop billable services. Pause subscriptions. Contact the billing provider to end the current cycle cleanly.
4. Archive or delete data according to policy. Some regulated industries require 90 days of retention before deletion.
5. Remove external resources in reverse dependency order. DNS first, because it points users to the service. Billing last, because you need it for audit history.
6. Record compliance evidence: deletion timestamps, provider response IDs, retention certificate.
7. Remove the database row.

Some external resources cannot be deleted immediately. A billing provider might enforce a 30-day termination window. The identity provider might delay org deletion. Record these in the status as pending_deletion. The reconciler checks them on each sweep. When the provider confirms deletion, remove the reference.

Observability for reconciliation systems

You cannot debug a reconciliation system with generic “success” and “error” metrics.

Track these specifically:

• Reconciliation attempts by outcome. How many complete, how many requeue, how many hit circuit breakers.
• Time to convergence. From spec creation to Ready status. P50, P95, P99.
• Age of oldest drift. How long has a tenant been out of convergence? This surfaces bottlenecks in provider APIs.
• Resources stuck by condition. Count tenants where IdentityReady = true but DNSReady = false. Isolates the failing provider.
• External calls per reconciliation. Should average close to one. Higher means your cycle is doing too much.
• Consecutive failure count. Per tenant. Spikes indicate a widespread issue or a misconfigured spec.
• Requeue delay. How far back the schedule pushes after failures. Verify backoff is working.
• Desired-versus-observed generation gap. A growing gap means reconciliation is falling behind.
• Circuit-breaker activations. Shows which tenants enter cooldown and why.

Logs should include at least:

tenant_id, desired_generation, observed_generation, 
action, provider, idempotency_key, external_resource_id, 
outcome, next_reconcile_at

These fields are enough to replay a tenant’s full onboarding history without guessing.

Failure-injection demonstration

Here is a concrete scenario. Run it mentally or in a test environment.

Step 1. Tenant request arrives. Generation = 2.

Step 2. Reconciler creates identity organization. Provider returns org-ext-abc. Status: IdentityReady = true.

Step 3. Reconciler creates billing customer. Provider returns cust-ext-xyz. Status: BillingReady = true.

Step 4. Reconciler creates database schema tenant-314. Status: DatabaseReady = true.

Step 5. Reconciler calls DNS provider. Timeout after 30 seconds. The reconciler catches the RetryableError. It increments failure_count to 1. next_reconcile_at = now + 2 seconds. The reconciler returns. The tenant is in reconciling state.

Step 6. The worker process receives a SIGTERM (OOM or deployment). It never gets to persist the observation for step 5. The database still shows failure_count = 0. The lock is released on process exit.

Step 7. The queue delivers the event again — or the scheduled sweep picks up the tenant because its next_reconcile_at has passed — or a new worker instance starts and reads the database.

Step 8. Reconciler loads the tenant. desired_generation is 2. observed_generation is 2 (identity, billing, database created successfully in prior cycles). DNSReady is false. The planner selects “create DNS record.”

Step 9. Reconciler calls DNS provider with idempotency key tenant-314:2:dns. The provider checks its idempotency table. No prior matching key. It creates the record and returns dns-ext-123.

Step 10. Status updates. DNSReady = true. provisioning_status = ready. Generation is now fully observed.

The zombie is dead. The tenant is live.

Now run the extended test:

Step 11. A QA test deletes the DNS record through the provider’s console.

Step 12. The periodic sweep runs. It checks every tenant. It sees that DNSReady = true in the spec, but the actual DNS state is missing.

Step 13. Reconciler calls the DNS provider to verify the record exists. It does not. The provider returns 404.

Step 14. Planner calculates the gap: the record is missing. It selects “create DNS record.”

Step 15. Reconciler recreates the record with the same idempotency key. Provider returns a new record ID or the same ID, depending on its idempotency behavior.

Step 16. Tenant is Ready again.

Without the sweep, the tenant would have drifted silently. Users would see 404s on their custom domain and blame the network.

When not to use this pattern

Reconciliation is powerful, but it is not free. It adds database tables, scheduler processes, lock contention, and operational complexity.

Skip it when:

• The operation is a single, atomic database transaction. A transfer between two accounts does not need a reconciler.
• The work is fire-and-forget. Sending a welcome email does not need convergence logic.
• There is no durable desired state. If you cannot write down “what the system should look like,” reconciliation has nothing to compare against.
• Automatically reversing manual changes would be dangerous. Production infrastructure often has manual break-glass procedures. A reconciler that enforces a spec on top of a manual repair can undo the fix.

Use it when the cost of a broken process is high, the number of integration points is large, and the system must survive its own failure modes.

Summary

A retry queue asks “did the previous command finish?” A reconciler asks “does the system currently match what the user requested?”

The second question produces systems that recover after crashes, repair drift, survive duplicate messages, and resume work without reconstructing the exact failed execution path. It replaces workflow replay with continuous observation. It is the difference between hoping the installer ran correctly and knowing the tenant is actually ready.