Kafka for Event-Driven Systems

Introduction

Kafka is useful when systems need to react to events independently. Instead of tightly connecting services through direct calls, Kafka allows producers to publish events and consumers to process them at their own pace.

This makes Kafka powerful for telemetry pipelines, audit logs, background processing, activity streams, analytics, crash detection, notification systems, and event-driven architectures where multiple services need to respond to the same system activity.

The Problem

Direct service-to-service communication works at small scale, but it becomes harder to manage when systems grow. One service may become slow, another may fail, and tightly coupled workflows can make the entire system fragile.

The challenge is to design event flow in a way that keeps services independent while still making events reliable, traceable, versioned, and easy to process.

System Design / Approach

The approach is to treat events as system contracts. Producers should publish meaningful events, topics should separate event categories, and consumers should own their own processing logic.

Producer Service
    ↓
Kafka Topic
    ↓
Consumer Group
    ↓
Worker / Processor
    ↓
Database / Cache / Alerting
    ↓
Monitoring and Replay

Implementation

Step 1: Publish Events to a Topic

A producer sends an event when something important happens in the system. The producer does not need to know which services will consume that event.

await producer.send({
  topic: "aegis.telemetry.crashes",
  messages: [
    {
      key: event.service,
      value: JSON.stringify({
        eventId: event.id,
        eventType: "service.crashed",
        version: 1,
        service: event.service,
        severity: event.severity,
        timestamp: new Date().toISOString(),
      }),
    },
  ],
});

Topics create durable streams of system activity that different consumers can read and process independently.

Step 2: Consume Events Independently

Consumers subscribe to topics and process events separately. This allows analytics, alerting, auditing, and remediation workflows to evolve without changing the producer.

await consumer.run({
  eachMessage: async ({ topic, partition, message }) => {
    const event = JSON.parse(message.value!.toString());

    await handleEvent({
      topic,
      partition,
      offset: message.offset,
      event,
    });
  },
});

Consumers can scale, fail, restart, and evolve independently because they are not directly blocking the producer.

Step 3: Design Event Payloads

Events should include enough context for downstream consumers to understand what happened, where it happened, and when it happened.

{
  "eventId": "evt_01HZX",
  "eventType": "service.crashed",
  "version": 1,
  "service": "demo-api",
  "type": "oom",
  "severity": "critical",
  "timestamp": "2026-06-08T10:00:00Z",
  "metadata": {
    "containerId": "demo-api-01",
    "host": "local-aegis-node"
  }
}

Good event payloads reduce extra service calls and make auditing, debugging, replay, and recovery easier.

Step 4: Use Keys for Ordering

Kafka ordering is guaranteed within a partition. Using a stable key helps related events go to the same partition so consumers can process them in order for that entity.

await producer.send({
  topic: "aegis.telemetry.crashes",
  messages: [
    {
      key: crashEvent.service,
      value: JSON.stringify(crashEvent),
    },
  ],
});

Using the service name as a key helps keep events for the same service ordered within a partition.

Step 5: Handle Consumer Failures

Consumers should handle failures carefully. A malformed event or temporary database issue should not silently stop the entire processing pipeline.

try {
  await processCrashEvent(event);
} catch (error) {
  await deadLetterProducer.send({
    topic: "aegis.telemetry.crashes.dlq",
    messages: [
      {
        key: event.eventId,
        value: JSON.stringify({
          event,
          reason: error.message,
          failedAt: new Date().toISOString(),
        }),
      },
    ],
  });
}

Dead-letter topics keep failed events visible instead of losing them or blocking the full stream.

Step 6: Make Consumers Idempotent

Kafka consumers may process the same event more than once in some failure scenarios. Consumers should be designed so duplicate processing does not create duplicate side effects.

const alreadyProcessed = await db.processedEvent.findUnique({
  where: {
    eventId: event.eventId,
  },
});

if (alreadyProcessed) {
  return;
}

await handleEvent(event);

await db.processedEvent.create({
  data: {
    eventId: event.eventId,
    processedAt: new Date(),
  },
});

Idempotent consumers keep retries and reprocessing safe.

Step 7: Add Event Schema Validation

Since events are contracts, malformed payloads should be rejected or routed into a failure stream. Schema validation helps protect consumers from unexpected event shapes.

const CrashEventSchema = z.object({
  eventId: z.string(),
  eventType: z.literal("service.crashed"),
  version: z.number(),
  service: z.string(),
  severity: z.enum(["low", "medium", "high", "critical"]),
  timestamp: z.string(),
});

const event = CrashEventSchema.parse(rawEvent);

Validation makes event-driven systems safer as producers and consumers evolve independently.

Step 8: Monitor Consumer Lag

Consumer lag shows whether consumers are keeping up with the event stream. High lag usually means processing is too slow, consumers are failing, or traffic has increased.

console.info("Kafka consumer processed event", {
  topic,
  partition,
  offset: message.offset,
  eventId: event.eventId,
  processedAt: new Date().toISOString(),
});

Lag monitoring helps detect bottlenecks before downstream systems fall behind.

Trade-offs

Real-World Impact

What I Learned

• Kafka is useful when multiple services need to react to the same event independently.
• Topics should represent meaningful event streams, not random message buckets.
• Event payloads should be explicit because they become contracts between services.
• Partition keys affect ordering, load distribution, and processing behavior.
• Consumers must be idempotent because events may be processed more than once.
• Dead-letter topics are important for debugging failed event processing.
• Consumer lag is a key signal for detecting processing bottlenecks.

Conclusion

Kafka makes systems more flexible by separating event production from event processing. Producers publish events once, while multiple consumers can process those events independently.

A strong Kafka architecture depends on clear topics, explicit event payloads, stable partition keys, idempotent consumers, schema validation, dead-letter topics, and monitoring for consumer lag.

The key lesson is simple: Kafka is not only a messaging tool. It is an event backbone that helps distributed systems stay decoupled, replayable, and resilient under real-world conditions.