How to Create a Kafka Topic and Produce Your First Event

# docker-compose.yml
services:
  kafka:
    image: bitnami/kafka:3.6
    environment:
      KAFKA_CFG_NODE_ID: 0
      KAFKA_CFG_PROCESS_ROLES: controller,broker
      KAFKA_CFG_LISTENERS: PLAINTEXT://:9092,CONTROLLER://:9093
      KAFKA_CFG_ADVERTISED_LISTENERS: PLAINTEXT://localhost:9092
    ports:
      - '9092:9092'

docker-compose up -d

This uses KRaft mode (no ZooKeeper required). Kafka will be available at localhost:9092.

# Create a topic with 3 partitions
docker exec kafka kafka-topics.sh \
  --create \
  --topic user-events \
  --partitions 3 \
  --replication-factor 1 \
  --bootstrap-server localhost:9092

# List all topics
docker exec kafka kafka-topics.sh \
  --list --bootstrap-server localhost:9092

Use --replication-factor 1 for local development. In production, use at least 3 for fault tolerance.

pip install kafka-python

from kafka import KafkaProducer
import json, time

producer = KafkaProducer(
    bootstrap_servers=['localhost:9092'],
    value_serializer=lambda v: json.dumps(v).encode()
)

for i in range(10):
    producer.send('user-events', {
        'user_id': i, 'action': 'view', 'ts': time.time()
    })

producer.flush()  # ensure all messages are sent

Always call producer.flush() before exiting — otherwise buffered messages may be lost.

from kafka import KafkaConsumer
import json

consumer = KafkaConsumer(
    'user-events',
    bootstrap_servers=['localhost:9092'],
    group_id='analytics-pipeline',  # consumer group
    auto_offset_reset='earliest',  # read from beginning
    value_deserializer=lambda v: json.loads(v)
)

for msg in consumer:
    print(f'partition={msg.partition} offset={msg.offset} value={msg.value}')

The group_id identifies the consumer group. Each group tracks its own offset — a new group always starts from the beginning when auto_offset_reset='earliest'.

# Check consumer group lag
docker exec kafka kafka-consumer-groups.sh \
  --describe \
  --group analytics-pipeline \
  --bootstrap-server localhost:9092

# Output: PARTITION | CURRENT-OFFSET | LOG-END-OFFSET | LAG
#            0      |       10       |       10       |  0  <- no lag

Consumer lag = LOG-END-OFFSET - CURRENT-OFFSET. Non-zero lag means the consumer is falling behind. Alert when lag exceeds your SLA threshold.