Kafka vs RabbitMQ: What's the Difference?
Kafka is a distributed event log — events are persisted on disk and replayable by multiple independent consumers. RabbitMQ is a message broker — messages are routed to queues and deleted after acknowledgement. The key difference: Kafka retains events; RabbitMQ consumes them.
Side-by-Side Comparison
Apache Kafka
- • Log-based: events stored on disk, retained by policy
- • Consumer groups independently track their offset
- • Replay any historical window at any time
- • Handles millions of events/sec horizontally
- • Pull-based: consumers poll at their own pace
- • Ordering guaranteed per partition
RabbitMQ
- • Queue-based: messages deleted after acknowledgement
- • Flexible routing via exchanges (direct, fanout, topic, headers)
- • No replay — once consumed, messages are gone
- • Best for lower-volume task processing
- • Push-based: broker pushes messages to consumers
- • Supports complex routing logic out of the box
Mental Model
Think of Kafka as a newspaper printing press — every edition is printed and archived. Any reader can pick up today's paper, last week's, or last year's at any time. Think of RabbitMQ as a postal service — letters are delivered to specific addresses and discarded. You can't re-read a delivered letter, but you can route letters anywhere with flexible addressing rules.
When to Use Each
Choose Kafka when:
- • Multiple services need to consume the same events
- • You need event replay for reprocessing or new consumers
- • Processing millions of events per second
- • Building event sourcing or audit log systems
- • Feeding real-time analytics pipelines
Choose RabbitMQ when:
- • Distributing work items to a pool of workers
- • Request/reply async patterns between services
- • Complex routing (fan-out, topic-based, header matching)
- • Low to medium volume (<100K msgs/sec)
- • Simple setup with minimal infrastructure overhead
How They Work Together
Many architectures use both: Kafka as the high-throughput event backbone, RabbitMQ for internal service task distribution. A Kafka consumer can publish processed results to RabbitMQ for downstream task routing:
# Kafka consumer → RabbitMQ producer pattern
for msg in kafka_consumer:
event = json.loads(msg.value)
if event['type'] == 'order_placed':
# Route to fulfillment workers via RabbitMQ
rabbit_channel.basic_publish(
exchange='orders',
routing_key='fulfillment',
body=json.dumps(event)
)
Common Mistakes
Using RabbitMQ for event sourcing
RabbitMQ deletes messages after delivery — you can't rebuild state or replay events. Event sourcing requires a durable log like Kafka.
Using Kafka for simple task queues
Kafka's partition model is overkill for distributing 10 jobs across 3 workers. RabbitMQ's work queue pattern is simpler and more appropriate.
Assuming they're interchangeable
Kafka and RabbitMQ have fundamentally different delivery semantics. Migrating between them requires rethinking your consumer model, not just changing connection strings.