What is Apache Kafka?

What is Apache Kafka?

Apache Kafka was created at LinkedIn to handle their activity stream — tracking every click, view, and interaction across the platform. It was open-sourced in 2011 and became an Apache top-level project in 2012. Today it processes trillions of events daily across the world's largest data platforms.

Unlike traditional message queues that delete messages after delivery, Kafka retains all events in an append-only log. This makes it both a messaging system and a storage system — consumers can read live events or replay historical ones from any point in time.

A Kafka cluster has three core actors. Producers are applications that write events to topics; they're fully decoupled from consumers. Brokers are the Kafka servers that store topic partitions and serve reads/writes — a production cluster has multiple brokers for fault tolerance. Consumers subscribe to topics and read events; consumer groups share partitions for parallel processing, and each group tracks its own offset independently.

Why does Kafka matter?

• Fully decoupled — producers and consumers evolve independently with zero coordination
• Consumers fail and restart without losing events because offsets track position in the log
• Replay any historical window from the retained log — invaluable for backfills and bug fixes
• New consumers added with zero producer changes — one topic feeds many downstream systems
• Real-time pipelines process events within milliseconds, end to end
• Horizontal scaling — add brokers and partitions to handle millions of events per second

How does Kafka work?

Events flow from producer → broker (partitioned topic) → consumer group → downstream system. Each topic is split into one or more partitions, and each partition is an ordered, immutable log on disk. Producers choose the partition (by key, round-robin, or custom) when they write. Consumers in a group split partitions among themselves — one consumer per partition at most — and commit their offset after processing.

Modern Kafka clusters run in KRaft mode, which removes the long-standing ZooKeeper dependency and lets Kafka manage its own metadata via the Raft consensus protocol. This simplifies operations significantly versus older ZooKeeper-based deployments.

Producing and consuming events with the Python kafka-python client:

from kafka import KafkaProducer, KafkaConsumer
import json

# Producer: publish events to a topic
producer = KafkaProducer(
    bootstrap_servers=['localhost:9092'],
    value_serializer=lambda v: json.dumps(v).encode(),
)
producer.send('user-events', {'user_id': 42, 'action': 'purchase'})

# Consumer: read events from a topic
consumer = KafkaConsumer(
    'user-events',
    bootstrap_servers=['localhost:9092'],
    group_id='analytics-service',
    auto_offset_reset='earliest',  # replay from beginning
)
for msg in consumer:
    process_event(json.loads(msg.value))

Kafka vs RabbitMQ vs Redis Pub/Sub

Use Kafka when you need durable event storage, replay, or millions of events per second. Use RabbitMQ for task queues, job processing, and low-volume async messaging with flexible routing. Use Redis Pub/Sub for ephemeral, fire-and-forget notifications where message loss is acceptable.

What can you build with Kafka?

• Real-time event streaming — clickstream, user activity, IoT sensors, and application logs at millisecond latency
• Change data capture (CDC) — stream Postgres, MySQL, or MongoDB changes via Debezium and propagate to lakes and search indexes
• Microservices decoupling — replace synchronous API calls with async event publishing so services evolve independently
• Real-time analytics — feed OLAP stores (ClickHouse, Druid, Pinot) and warehouses with live event streams for sub-second dashboards
• Log aggregation — collect logs from thousands of services into one searchable stream
• Stream processing pipelines — connect Kafka to Flink, Spark Streaming, or Kafka Streams for joins, aggregations, fraud detection, and ML inference

Common mistakes (and what to do instead)

• Too few partitions — partitions are the unit of parallelism. One partition means one consumer in a group can do work. Over-partition rather than under-partition; you can't add partitions to a keyed topic later without rebalancing keys.
• Wrong retention for the use case — the default is 7 days. Event sourcing or audit logs may need indefinite retention; ephemeral routing may want hours. Set log.retention.hours explicitly per topic.
• Using Kafka as a database — Kafka is not a query engine. No ad-hoc queries, joins, or aggregations. Feed real databases, warehouses, or stream processors (Flink, ksqlDB) for query workloads.
• Committing offsets before processing — a crash will skip the message permanently. Commit after successful processing, never before.
• One topic for everything — mixing event types in a single topic destroys schema evolution, access control, and consumer filtering. Use one topic per event type or domain entity.
• Skipping a schema registry — without a Schema Registry (Confluent, Apicurio), producer changes silently break consumers. Pair Kafka with Avro or Protobuf schemas from day one.

Who is Kafka for?

Kafka is built for data engineers, platform engineers, and backend engineers building event-driven systems. If you need durable, replayable event streams at any non-trivial scale, Kafka is the default choice.

Teams that benefit most:

• Streaming teams ingesting clickstream, IoT, or transactional events at millions per second
• Platform teams running CDC pipelines from operational databases to lakes and search
• Backend teams decoupling microservices and moving to event-driven architecture
• ML teams feeding real-time features and inference pipelines with sub-second freshness