By Appropri8 Team

Event-Sourced Integration Patterns for Hybrid Cloud Architectures

event-sourcinghybrid-cloudmicroservicesintegrationkafkaawsazure

Most companies run their systems across multiple clouds these days. AWS here, Azure there, maybe some on-premise stuff too. The problem? Getting all these systems to talk to each other without creating a mess.

Traditional integration approaches like REST APIs and ETL jobs work fine for simple cases. But when you have dozens of services across different clouds, things get complicated fast. You end up with point-to-point connections everywhere, data inconsistencies, and no clear audit trail of what happened when.

Event sourcing changes this. Instead of trying to sync data between systems, you capture what happened as events. These events become your single source of truth, flowing through a central event hub that connects all your clouds.

What Makes Event-Sourced Integration Different

Event sourcing isn’t just another way to move data around. It’s a fundamental shift in how you think about integration.

With traditional approaches, you’re always trying to keep data in sync. System A updates a record, then you need to tell System B about the change. If something goes wrong, you’re stuck trying to figure out what the “correct” state should be.

Event sourcing flips this. Instead of syncing state, you capture the events that led to that state. The events are immutable - they never change. If you need to know the current state, you replay the events from the beginning.

This gives you three big advantages:

Replayability: You can always rebuild any system’s state by replaying events. Need to test something? Replay events from last week. Want to create a new service? It can consume the same events.

Temporal queries: You can ask questions like “what was the state at 3 PM yesterday?” or “show me all changes made by user X last month.” This is impossible with traditional sync approaches.

Auditability: Every change is captured as an event with a timestamp, user, and context. You have a complete audit trail without building it separately.

The Hybrid Cloud Challenge

Running event sourcing across multiple clouds adds complexity. Each cloud has its own message broker with different capabilities and limitations.

AWS EventBridge is great for serverless event routing but has message size limits. Azure Service Bus handles large messages well but doesn’t have the same routing flexibility. Kafka gives you the most control but requires more operational overhead.

Then there’s the latency problem. Events flowing from AWS to Azure will always have some delay. You need to design your consumers to handle this gracefully.

Schema drift is another issue. When you have services in different clouds, they might evolve at different rates. Your event schemas need to be backward compatible, or you’ll break consumers that haven’t updated yet.

A Reference Architecture

Here’s how we structure event-sourced integration across hybrid clouds:

Event-Sourced Integration Architecture

The event hub sits in the center, usually running Kafka for its durability and replay capabilities. This becomes your integration spine - all events flow through it.

Each cloud connects to the hub through adapters. AWS services publish to EventBridge, which forwards events to Kafka. Azure services use Service Bus, which also forwards to Kafka. Your on-premise systems can connect directly to Kafka.

Microservices in each cloud follow the outbox pattern. When they update their local database, they also write events to an outbox table. A separate process reads from the outbox and publishes events to the event hub.

This gives you eventual consistency with strong durability guarantees. If a service crashes after updating its database but before publishing events, the outbox process will catch up when the service restarts.

Implementation Example

Let’s look at how this works in practice with a C# example. Here’s a basic outbox implementation using Entity Framework Core:

public class OrderService
{
    private readonly DbContext _dbContext;
    private readonly IEventBus _eventBus;
    
    public async Task SaveAndPublishAsync<T>(T aggregate) where T : AggregateRoot
    {
        using var transaction = await _dbContext.Database.BeginTransactionAsync();
        
        try
        {
            // Save the aggregate changes
            await _dbContext.SaveChangesAsync();
            
            // Get uncommitted events
            var events = aggregate.GetUncommittedEvents();
            
            // Write events to outbox
            foreach (var evt in events)
            {
                var outboxEvent = new OutboxEvent
                {
                    Id = Guid.NewGuid(),
                    AggregateId = aggregate.Id,
                    EventType = evt.GetType().Name,
                    EventData = JsonSerializer.Serialize(evt),
                    CreatedAt = DateTime.UtcNow,
                    Processed = false
                };
                
                _dbContext.Set<OutboxEvent>().Add(outboxEvent);
            }
            
            await _dbContext.SaveChangesAsync();
            await transaction.CommitAsync();
            
            // Publish events (this happens after the transaction commits)
            foreach (var evt in events)
            {
                await _eventBus.PublishAsync(evt);
            }
            
            aggregate.MarkEventsAsCommitted();
        }
        catch
        {
            await transaction.RollbackAsync();
            throw;
        }
    }
}

The key here is the two-phase approach. First, we save both the business data and the events in a single transaction. This ensures we never lose events. Then, after the transaction commits, we publish the events to the event bus.

If publishing fails, we have the events in the outbox table. A background process can retry publishing them later.

Bridging Cloud Providers

Getting events from AWS EventBridge to Azure Service Bus (or vice versa) requires some translation. Kafka Connect makes this easier.

Here’s a basic Kafka Connect configuration for bridging EventBridge to Service Bus:

{
  "name": "eventbridge-to-servicebus",
  "config": {
    "connector.class": "io.confluent.connect.aws.eventbridge.EventBridgeSinkConnector",
    "topics": "user-events,order-events,payment-events",
    "aws.eventbridge.region": "us-east-1",
    "aws.eventbridge.eventbus": "production-events",
    "transforms": "routeToServiceBus",
    "transforms.routeToServiceBus.type": "org.apache.kafka.connect.transforms.RegexRouter",
    "transforms.routeToServiceBus.regex": "(.+)-events",
    "transforms.routeToServiceBus.replacement": "azure-servicebus-$1"
  }
}

This connector reads events from Kafka topics and publishes them to EventBridge. You’d run a similar connector in the other direction to get events from Service Bus back into Kafka.

The routing transform ensures events go to the right EventBridge rules, which can then trigger Lambda functions or other AWS services.

Advanced Patterns

Once you have basic event flow working, you can add more sophisticated patterns.

Idempotent consumers are essential when dealing with network issues and retries. Each event should have a unique ID, and consumers should track which events they’ve already processed:

public class IdempotentEventConsumer
{
    private readonly HashSet<string> _processedEvents = new();
    
    public async Task HandleAsync(OrderCreatedEvent evt)
    {
        if (_processedEvents.Contains(evt.EventId))
        {
            return; // Already processed
        }
        
        // Process the event
        await ProcessOrderCreated(evt);
        
        // Remember we processed it
        _processedEvents.Add(evt.EventId);
    }
}

Saga orchestration helps coordinate long-running processes across multiple services and clouds. Instead of trying to use distributed transactions (which don’t work well across clouds), you use events to coordinate:

public class OrderSaga
{
    public async Task HandleAsync(OrderCreatedEvent evt)
    {
        // Start the saga
        await _eventBus.PublishAsync(new ReserveInventoryCommand 
        { 
            OrderId = evt.OrderId,
            Items = evt.Items 
        });
    }
    
    public async Task HandleAsync(InventoryReservedEvent evt)
    {
        if (evt.Success)
        {
            await _eventBus.PublishAsync(new ProcessPaymentCommand 
            { 
                OrderId = evt.OrderId,
                Amount = evt.TotalAmount 
            });
        }
        else
        {
            await _eventBus.PublishAsync(new CancelOrderCommand 
            { 
                OrderId = evt.OrderId,
                Reason = "Insufficient inventory" 
            });
        }
    }
}

Event versioning is crucial when you have services evolving at different rates. Use a version field in your events and handle multiple versions in your consumers:

public class OrderCreatedEventV1
{
    public string OrderId { get; set; }
    public string CustomerId { get; set; }
    public decimal TotalAmount { get; set; }
    public DateTime CreatedAt { get; set; }
}

public class OrderCreatedEventV2 : OrderCreatedEventV1
{
    public string CustomerEmail { get; set; }
    public string ShippingAddress { get; set; }
}

public class OrderEventHandler
{
    public async Task HandleAsync(object evt)
    {
        switch (evt)
        {
            case OrderCreatedEventV1 v1:
                await HandleV1(v1);
                break;
            case OrderCreatedEventV2 v2:
                await HandleV2(v2);
                break;
        }
    }
}

Observability and Governance

With events flowing across multiple clouds, you need good observability. You want to trace an event from its origin through all the systems that process it.

Set up event tracing with correlation IDs. Each event gets a unique ID that flows through all processing steps. This lets you build dashboards showing event flow and identify bottlenecks.

Schema registry management becomes critical. You need a central place to store and version your event schemas. This helps prevent breaking changes and makes it easier for teams to understand what events are available.

Data lineage tools help you understand how data flows through your system. When someone asks “where did this data come from?” you can trace it back through the event chain.

For governance, you need policies around event retention, access control, and compliance. Some events might need to be kept for years for audit purposes. Others might contain sensitive data that needs special handling.

Making It Work

Event sourcing across hybrid clouds isn’t easy. You’re dealing with network latency, different broker capabilities, and the complexity of distributed systems.

But when it works, it’s powerful. You get a clear audit trail, the ability to replay any system’s state, and loose coupling between your services. Changes in one cloud don’t break systems in another cloud.

Start simple. Pick one or two critical business processes and model them as events. Get the basic event flow working between your clouds. Then gradually add more processes and more sophisticated patterns.

The key is treating events as first-class citizens in your architecture. They’re not just a way to move data around - they’re the foundation of how your systems communicate and maintain consistency.

With the right patterns and tools, event sourcing can turn the complexity of hybrid cloud integration into architectural clarity. Your systems become more resilient, more auditable, and easier to evolve over time.

Join the Discussion

Have thoughts on this article? Share your insights and engage with the community.