Event-Driven Architecture at Scale: Lessons from Production

Beyond Request-Response

Event-driven architecture decouples producers from consumers, enabling systems that scale horizontally and evolve independently. But at scale, new challenges emerge.

The Event Backbone

// Event schema with versioning
interface OrderEvent {
  eventId: string;
  eventType: "order.created" 
"order.updated"
"order.cancelled";
  version: "2.0";
  timestamp: string;
  data: {
    orderId: string;
    customerId: string;
    items: OrderItem[];
    total: number;
  };
  metadata: {
    correlationId: string;
    causationId: string;
    source: string;
  };
}

Guarantees and Trade-offs

At-least-once delivery — The practical choice. Design consumers to be idempotent.

Ordering — Partition by entity ID for per-entity ordering guarantees.

Schema evolution — Use schema registries with backward compatibility rules.

The Outbox Pattern

Ensure atomicity between database writes and event publication:

BEGIN;
  INSERT INTO orders (id, customer_id, total) VALUES (...);
  INSERT INTO outbox (event_type, payload) VALUES ('order.created', '...');
COMMIT;

A separate process reads the outbox and publishes events, guaranteeing consistency between state and notifications. This is production-grade event-driven architecture.