By Appropri8 Team

Event-Choreography Pattern: A Modern Alternative to Orchestration in Distributed Systems

event-drivenmicroservicesdistributed-systemsarchitectureserverlessdomain-driven-design

Introduction

For years, orchestration has been the go-to pattern for managing complex workflows in distributed systems. You’d have a central coordinator telling each service what to do and when to do it. It worked, but it created tight coupling and single points of failure.

Now we’re seeing a shift. Modern distributed systems, especially those built for serverless and cloud-native environments, are moving toward event choreography. Instead of a conductor directing an orchestra, you have dancers who know their moves and react to the music.

This isn’t just a trendy pattern. Event choreography fits perfectly with domain-driven design and the autonomous nature of microservices. When each service owns its domain and makes its own decisions, the system becomes more resilient and scalable.

What is the Event-Choreography Pattern

Event choreography is about services reacting to domain events rather than following commands from a central coordinator. Each service publishes events when something important happens in its domain, and other services subscribe to events they care about.

Think of it this way: in orchestration, you have a workflow engine that says “first call the payment service, then call the inventory service, then send a notification.” In choreography, the payment service publishes a “PaymentCompleted” event, the inventory service listens for that event and updates stock, and the notification service listens for the same event and sends an email.

The key difference is control. Orchestration gives you sequence control—you know exactly what happens when. Choreography gives you reaction control—services respond to events as they see fit.

When to Use Event Choreography

Event choreography works best in certain scenarios. If you’re building event-driven domains where business processes naturally flow through events, choreography makes sense. E-commerce is a perfect example—orders, payments, inventory updates, and notifications all happen as reactions to business events.

High scalability requirements also favor choreography. When you need to handle thousands of events per second, having services react independently scales better than routing everything through a central orchestrator.

Decoupled teams benefit too. Each team can work on their service’s event handling without coordinating with a central workflow team. They just need to agree on the event schema.

But choreography isn’t always the right choice. If you need strict process enforcement or critical ordering guarantees, orchestration might be better. Some business processes require exact sequences that are hard to guarantee with events alone.

Design Implementation

Event Bus Architecture

The foundation of event choreography is a reliable event bus. You have several options here. Apache Kafka is popular for high-throughput scenarios. AWS SNS/SQS works well in cloud environments. Azure Service Bus provides enterprise features. The choice depends on your specific needs.

The event bus needs to handle several concerns. Message ordering matters for some events. Delivery guarantees are crucial—you need to know if an event was delivered. And you need to handle failures gracefully.

Event Versioning and Schema Evolution

Events change over time. Your payment service might add new fields to the PaymentCompleted event. Other services need to handle both old and new versions gracefully.

Schema evolution strategies help here. You can use backward-compatible changes—adding optional fields, for example. Or you can version your events explicitly. The key is planning for change from the start.

Idempotency and Compensating Transactions

In choreography, services might receive the same event multiple times. Network issues, retries, or system failures can cause duplicates. Your event handlers need to be idempotent—they should produce the same result whether they run once or multiple times.

Sometimes things go wrong. A payment succeeds but inventory update fails. You need compensating transactions to undo the payment. This is more complex in choreography than orchestration, but it’s manageable with proper design.

Event Flow Visualization

Here’s how the event choreography pattern works in our order processing example:

sequenceDiagram
    participant Customer
    participant OrderService
    participant EventBus
    participant PaymentService
    participant InventoryService
    participant NotificationService

    Customer->>OrderService: Create Order
    OrderService->>OrderService: Save Order
    OrderService->>EventBus: Publish OrderCreated Event
    
    EventBus->>PaymentService: OrderCreated Event
    EventBus->>InventoryService: OrderCreated Event
    
    PaymentService->>PaymentService: Process Payment
    PaymentService->>EventBus: Publish PaymentProcessed Event
    
    InventoryService->>InventoryService: Reserve Inventory
    InventoryService->>EventBus: Publish InventoryUpdated Event
    
    EventBus->>OrderService: PaymentProcessed Event
    OrderService->>EventBus: Publish OrderCompleted Event
    
    EventBus->>NotificationService: OrderCompleted Event
    NotificationService->>Customer: Send Confirmation Email

Code Samples

Let’s look at a real example. We’ll build an order processing system using event choreography. The flow is: order created → payment processed → inventory updated → notification sent.

Event Definitions

First, let’s define our events using a simple schema:

// Event schemas
interface OrderCreatedEvent {
  eventType: 'OrderCreated';
  orderId: string;
  customerId: string;
  items: Array<{
    productId: string;
    quantity: number;
    price: number;
  }>;
  totalAmount: number;
  timestamp: string;
}

interface PaymentProcessedEvent {
  eventType: 'PaymentProcessed';
  orderId: string;
  paymentId: string;
  amount: number;
  status: 'success' | 'failed';
  timestamp: string;
}

interface InventoryUpdatedEvent {
  eventType: 'InventoryUpdated';
  orderId: string;
  items: Array<{
    productId: string;
    quantityReserved: number;
  }>;
  timestamp: string;
}

interface OrderCompletedEvent {
  eventType: 'OrderCompleted';
  orderId: string;
  customerId: string;
  totalAmount: number;
  timestamp: string;
}

Order Service Implementation

Here’s how the order service publishes events:

import { EventBus } from './event-bus';
import { v4 as uuidv4 } from 'uuid';

export class OrderService {
  constructor(private eventBus: EventBus) {}

  async createOrder(customerId: string, items: any[]) {
    const orderId = uuidv4();
    const totalAmount = items.reduce((sum, item) => sum + (item.price * item.quantity), 0);

    // Save order to database
    const order = {
      id: orderId,
      customerId,
      items,
      totalAmount,
      status: 'created',
      createdAt: new Date()
    };
    
    await this.saveOrder(order);

    // Publish event
    const event: OrderCreatedEvent = {
      eventType: 'OrderCreated',
      orderId,
      customerId,
      items,
      totalAmount,
      timestamp: new Date().toISOString()
    };

    await this.eventBus.publish('order.created', event);
    
    return order;
  }

  async handlePaymentProcessed(event: PaymentProcessedEvent) {
    if (event.status === 'success') {
      await this.updateOrderStatus(event.orderId, 'payment_completed');
      
      // Publish order completed event
      const orderCompletedEvent: OrderCompletedEvent = {
        eventType: 'OrderCompleted',
        orderId: event.orderId,
        customerId: '', // Would fetch from order
        totalAmount: event.amount,
        timestamp: new Date().toISOString()
      };

      await this.eventBus.publish('order.completed', orderCompletedEvent);
    } else {
      await this.updateOrderStatus(event.orderId, 'payment_failed');
    }
  }

  private async saveOrder(order: any) {
    // Database save logic
  }

  private async updateOrderStatus(orderId: string, status: string) {
    // Database update logic
  }
}

Payment Service Implementation

The payment service listens for order events and publishes payment events:

export class PaymentService {
  constructor(private eventBus: EventBus) {}

  async start() {
    // Subscribe to order created events
    await this.eventBus.subscribe('order.created', this.handleOrderCreated.bind(this));
  }

  private async handleOrderCreated(event: OrderCreatedEvent) {
    try {
      // Process payment
      const paymentResult = await this.processPayment({
        orderId: event.orderId,
        amount: event.totalAmount,
        customerId: event.customerId
      });

      // Publish payment processed event
      const paymentEvent: PaymentProcessedEvent = {
        eventType: 'PaymentProcessed',
        orderId: event.orderId,
        paymentId: paymentResult.paymentId,
        amount: event.totalAmount,
        status: paymentResult.success ? 'success' : 'failed',
        timestamp: new Date().toISOString()
      };

      await this.eventBus.publish('payment.processed', paymentEvent);
    } catch (error) {
      // Publish failed payment event
      const paymentEvent: PaymentProcessedEvent = {
        eventType: 'PaymentProcessed',
        orderId: event.orderId,
        paymentId: '',
        amount: event.totalAmount,
        status: 'failed',
        timestamp: new Date().toISOString()
      };

      await this.eventBus.publish('payment.processed', paymentEvent);
    }
  }

  private async processPayment(paymentRequest: any) {
    // Payment processing logic
    // This would integrate with payment providers
    return {
      paymentId: uuidv4(),
      success: Math.random() > 0.1 // 90% success rate for demo
    };
  }
}

Inventory Service Implementation

The inventory service reserves stock when orders are created:

export class InventoryService {
  constructor(private eventBus: EventBus) {}

  async start() {
    await this.eventBus.subscribe('order.created', this.handleOrderCreated.bind(this));
    await this.eventBus.subscribe('payment.processed', this.handlePaymentProcessed.bind(this));
  }

  private async handleOrderCreated(event: OrderCreatedEvent) {
    // Reserve inventory
    const reservedItems = await this.reserveInventory(event.items);

    const inventoryEvent: InventoryUpdatedEvent = {
      eventType: 'InventoryUpdated',
      orderId: event.orderId,
      items: reservedItems,
      timestamp: new Date().toISOString()
    };

    await this.eventBus.publish('inventory.updated', inventoryEvent);
  }

  private async handlePaymentProcessed(event: PaymentProcessedEvent) {
    if (event.status === 'failed') {
      // Release reserved inventory
      await this.releaseInventory(event.orderId);
    }
  }

  private async reserveInventory(items: any[]) {
    // Inventory reservation logic
    return items.map(item => ({
      productId: item.productId,
      quantityReserved: item.quantity
    }));
  }

  private async releaseInventory(orderId: string) {
    // Inventory release logic
  }
}

Notification Service Implementation

Finally, the notification service sends emails when orders complete:

export class NotificationService {
  constructor(private eventBus: EventBus) {}

  async start() {
    await this.eventBus.subscribe('order.completed', this.handleOrderCompleted.bind(this));
  }

  private async handleOrderCompleted(event: OrderCompletedEvent) {
    // Send notification
    await this.sendOrderConfirmation({
      customerId: event.customerId,
      orderId: event.orderId,
      totalAmount: event.totalAmount
    });
  }

  private async sendOrderConfirmation(notification: any) {
    // Email sending logic
    console.log(`Sending confirmation for order ${notification.orderId}`);
  }
}

C# Implementation Example

Here’s the same pattern implemented in C# using .NET and a message broker:

using System;
using System.Threading.Tasks;
using Microsoft.Extensions.Hosting;
using Microsoft.Extensions.DependencyInjection;
using Microsoft.Extensions.Logging;

// Event definitions
public record OrderCreatedEvent(
    string OrderId,
    string CustomerId,
    OrderItem[] Items,
    decimal TotalAmount,
    DateTime Timestamp
);

public record OrderItem(
    string ProductId,
    int Quantity,
    decimal Price
);

public record PaymentProcessedEvent(
    string OrderId,
    string PaymentId,
    decimal Amount,
    PaymentStatus Status,
    DateTime Timestamp
);

public enum PaymentStatus
{
    Success,
    Failed
}

// Order Service
public class OrderService
{
    private readonly IEventBus _eventBus;
    private readonly ILogger<OrderService> _logger;

    public OrderService(IEventBus eventBus, ILogger<OrderService> logger)
    {
        _eventBus = eventBus;
        _logger = logger;
    }

    public async Task<Order> CreateOrderAsync(string customerId, OrderItem[] items)
    {
        var orderId = Guid.NewGuid().ToString();
        var totalAmount = items.Sum(item => item.Price * item.Quantity);

        var order = new Order
        {
            Id = orderId,
            CustomerId = customerId,
            Items = items,
            TotalAmount = totalAmount,
            Status = OrderStatus.Created,
            CreatedAt = DateTime.UtcNow
        };

        // Save to database
        await SaveOrderAsync(order);

        // Publish event
        var orderCreatedEvent = new OrderCreatedEvent(
            orderId,
            customerId,
            items,
            totalAmount,
            DateTime.UtcNow
        );

        await _eventBus.PublishAsync("order.created", orderCreatedEvent);
        _logger.LogInformation("Order {OrderId} created and event published", orderId);

        return order;
    }

    public async Task HandlePaymentProcessedAsync(PaymentProcessedEvent paymentEvent)
    {
        if (paymentEvent.Status == PaymentStatus.Success)
        {
            await UpdateOrderStatusAsync(paymentEvent.OrderId, OrderStatus.PaymentCompleted);
            _logger.LogInformation("Order {OrderId} payment completed", paymentEvent.OrderId);
        }
        else
        {
            await UpdateOrderStatusAsync(paymentEvent.OrderId, OrderStatus.PaymentFailed);
            _logger.LogWarning("Order {OrderId} payment failed", paymentEvent.OrderId);
        }
    }

    private async Task SaveOrderAsync(Order order)
    {
        // Database save implementation
        await Task.Delay(10); // Simulate database operation
    }

    private async Task UpdateOrderStatusAsync(string orderId, OrderStatus status)
    {
        // Database update implementation
        await Task.Delay(10); // Simulate database operation
    }
}

// Payment Service
public class PaymentService : BackgroundService
{
    private readonly IEventBus _eventBus;
    private readonly ILogger<PaymentService> _logger;

    public PaymentService(IEventBus eventBus, ILogger<PaymentService> logger)
    {
        _eventBus = eventBus;
        _logger = logger;
    }

    protected override async Task ExecuteAsync(CancellationToken stoppingToken)
    {
        await _eventBus.SubscribeAsync<OrderCreatedEvent>("order.created", HandleOrderCreatedAsync);
    }

    private async Task HandleOrderCreatedAsync(OrderCreatedEvent orderEvent)
    {
        try
        {
            _logger.LogInformation("Processing payment for order {OrderId}", orderEvent.OrderId);

            var paymentResult = await ProcessPaymentAsync(new PaymentRequest
            {
                OrderId = orderEvent.OrderId,
                Amount = orderEvent.TotalAmount,
                CustomerId = orderEvent.CustomerId
            });

            var paymentProcessedEvent = new PaymentProcessedEvent(
                orderEvent.OrderId,
                paymentResult.PaymentId,
                orderEvent.TotalAmount,
                paymentResult.Success ? PaymentStatus.Success : PaymentStatus.Failed,
                DateTime.UtcNow
            );

            await _eventBus.PublishAsync("payment.processed", paymentProcessedEvent);
            _logger.LogInformation("Payment processed for order {OrderId}: {Status}", 
                orderEvent.OrderId, paymentResult.Success ? "Success" : "Failed");
        }
        catch (Exception ex)
        {
            _logger.LogError(ex, "Error processing payment for order {OrderId}", orderEvent.OrderId);
            
            var failedPaymentEvent = new PaymentProcessedEvent(
                orderEvent.OrderId,
                string.Empty,
                orderEvent.TotalAmount,
                PaymentStatus.Failed,
                DateTime.UtcNow
            );

            await _eventBus.PublishAsync("payment.processed", failedPaymentEvent);
        }
    }

    private async Task<PaymentResult> ProcessPaymentAsync(PaymentRequest request)
    {
        // Simulate payment processing
        await Task.Delay(100);
        
        return new PaymentResult
        {
            PaymentId = Guid.NewGuid().ToString(),
            Success = Random.Shared.NextDouble() > 0.1 // 90% success rate
        };
    }
}

// Event Bus Interface
public interface IEventBus
{
    Task PublishAsync<T>(string topic, T eventData);
    Task SubscribeAsync<T>(string topic, Func<T, Task> handler);
}

// Supporting classes
public class Order
{
    public string Id { get; set; }
    public string CustomerId { get; set; }
    public OrderItem[] Items { get; set; }
    public decimal TotalAmount { get; set; }
    public OrderStatus Status { get; set; }
    public DateTime CreatedAt { get; set; }
}

public enum OrderStatus
{
    Created,
    PaymentCompleted,
    PaymentFailed,
    Completed
}

public class PaymentRequest
{
    public string OrderId { get; set; }
    public decimal Amount { get; set; }
    public string CustomerId { get; set; }
}

public class PaymentResult
{
    public string PaymentId { get; set; }
    public bool Success { get; set; }
}

Testing and Observability

Contract Testing

With event choreography, services need to agree on event schemas. Contract testing helps ensure compatibility. You can use tools like Pact to verify that event producers and consumers stay in sync.

// Contract test example
describe('OrderCreated Event Contract', () => {
  it('should match the expected schema', () => {
    const event = {
      eventType: 'OrderCreated',
      orderId: '123',
      customerId: '456',
      items: [{ productId: '789', quantity: 2, price: 10.00 }],
      totalAmount: 20.00,
      timestamp: '2025-10-11T10:00:00Z'
    };

    expect(event).toMatchSchema(orderCreatedSchema);
  });
});

Event Tracing

Tracing events across services is crucial for debugging. OpenTelemetry with correlation IDs helps track the flow:

import { trace, context } from '@opentelemetry/api';

export class EventBus {
  async publish(topic: string, event: any) {
    const tracer = trace.getTracer('event-bus');
    const span = tracer.startSpan(`publish.${topic}`);
    
    try {
      // Add correlation ID to event
      event.correlationId = span.spanContext().traceId;
      
      // Publish to actual message broker
      await this.messageBroker.publish(topic, event);
      
      span.setStatus({ code: SpanStatusCode.OK });
    } catch (error) {
      span.setStatus({ code: SpanStatusCode.ERROR, message: error.message });
      throw error;
    } finally {
      span.end();
    }
  }
}

Pitfalls and Best Practices

Event Storms

One common problem is event storms—when a single event triggers a cascade of other events, creating a flood. You can mitigate this with event filtering, batching, or circuit breakers.

Eventual Consistency

Event choreography means eventual consistency. Your system might be in an inconsistent state temporarily. Design your UI and business logic to handle this gracefully.

Retry Handling

Events can fail to process. You need retry logic with exponential backoff. Dead letter queues help handle events that keep failing.

Best Practices

  • Keep events small and focused
  • Use idempotent handlers
  • Plan for schema evolution
  • Monitor event processing latency
  • Use correlation IDs for tracing
  • Implement proper error handling

Hybrid Approaches

You don’t have to choose between orchestration and choreography. Many systems use both. Use orchestration for critical business processes that need strict ordering. Use choreography for reactive, scalable scenarios.

For example, you might orchestrate the core order-to-payment flow but use choreography for notifications and analytics. This gives you the benefits of both patterns.

The Future

Looking ahead, we’ll see more AI-assisted event routing. Systems will automatically determine the best path for events based on current conditions. Observability tools will get better at tracking complex event flows.

Event choreography is becoming the standard for modern distributed systems. It fits naturally with serverless architectures, domain-driven design, and autonomous teams. While it requires more upfront planning than orchestration, the benefits in scalability and resilience make it worth the effort.

The key is starting simple. Begin with a few events and services. Learn the patterns. Then expand gradually. Event choreography isn’t just a technical pattern—it’s a way of thinking about how systems should interact in a distributed world.

Join the Discussion

Have thoughts on this article? Share your insights and engage with the community.