Apache Kafka is a distributed commit log used as the central nervous system for streaming data. Producers append records to topics; consumers read them at their own pace; the data stays on disk, replicated and replayable, for as long as the retention policy allows. Originally built at LinkedIn, Kafka now underpins event-driven architectures, log aggregation, metrics pipelines, and the ingestion side of most real-time analytics stacks.

The core abstraction: an append-only log

A Kafka topic is split into partitions, and each partition is an ordered, immutable sequence of records. A record's position in its partition is its offset. Writes only ever append to the end; reads are sequential scans from an offset the consumer controls. This single design choice explains most of Kafka's properties: appends and sequential reads are fast on commodity disks, ordering is guaranteed within a partition, and because consumers track their own offset, the same data can be read by many independent consumers and re-read after a failure.

Partitions, keys, and ordering

Partitions are the unit of parallelism. A topic with 12 partitions can be consumed by up to 12 instances of a consumer group working in parallel. The producer decides a record's partition, usually by hashing a key: all records with the same key (say, a user ID) land in the same partition and therefore stay ordered relative to each other. Kafka guarantees order within a partition, never across partitions - so the key you choose is also your ordering and parallelism boundary. Picking a key with poor cardinality (or none) creates hot partitions that bottleneck the whole topic.

Consumer groups and offset management

Consumers join a group; Kafka assigns each partition to exactly one member, and rebalances when members join or leave. The group commits its read offsets back to Kafka, so on restart it resumes where it left off. Two groups reading the same topic are independent - one can be a real-time dashboard at the log's head while another is a batch loader catching up from yesterday.

# Inspect lag — the single most important streaming health metric
kafka-consumer-groups.sh --bootstrap-server broker:9092 \
  --describe --group analytics-loader
# CURRENT-OFFSET vs LOG-END-OFFSET per partition; the gap is LAG

Consumer lag - how far behind the head a group is - is the number to alarm on. Rising lag means consumers can't keep up with producers and downstream data is going stale.

Durability: replication and acks

Each partition has a leader and a set of follower replicas. Producers can choose how durable a write must be:

  • acks=0 - fire and forget; fastest, can lose data.
  • acks=1 - leader has written it; lost if the leader dies before replication.
  • acks=all - every in-sync replica has it; safe, higher latency.

Pair acks=all with min.insync.replicas=2 on a topic replicated 3× to survive a single broker loss with no data loss. This is the standard durable configuration for pipelines that must not drop events.

Delivery semantics

By default Kafka gives at-least-once delivery: a consumer can reprocess records after a crash, so downstream logic should be idempotent. For end-to-end exactly-once within Kafka, enable the idempotent producer and transactions, which let a consume-process-produce loop commit input offsets and output records atomically. Exactly-once costs throughput and only holds inside Kafka - a sink to an external system still needs idempotent writes.

A minimal producer and consumer

from kafka import KafkaProducer, KafkaConsumer
import json

producer = KafkaProducer(
    bootstrap_servers="broker:9092",
    acks="all",
    value_serializer=lambda v: json.dumps(v).encode(),
)
producer.send("events", key=b"user-42", value={"action": "click"})
producer.flush()

consumer = KafkaConsumer(
    "events", bootstrap_servers="broker:9092",
    group_id="analytics-loader", enable_auto_commit=False,
)
for msg in consumer:
    process(msg.value)
    consumer.commit()   # commit after successful processing, not before

Where it fits in a data stack

  • Ingestion buffer - decouples bursty producers from slower sinks so a warehouse load or a Flink job can fall behind without dropping data.
  • CDC transport - Debezium streams database changes into Kafka topics for downstream consumers.
  • Stream processing source - Flink, Kafka Streams, and Spark Structured Streaming read topics to compute aggregations, joins, and enrichments in motion.

Common pitfalls

Under-partitioning caps your future parallelism - you can add partitions but it scrambles key ordering, so size with headroom. Committing offsets before processing finishes turns at-least-once into "lose data on crash." Leaving retention too short can expire data before a lagging consumer reads it. And Kafka is not a database: it is a log you scan, not a store you query by key, so don't reach for it when you need random lookups or long-term queryable history - land the data in a table for that.

The KRaft shift

Kafka historically depended on ZooKeeper for cluster metadata. Recent versions replace it with KRaft, a built-in Raft quorum that removes the separate ZooKeeper deployment, simplifies operations, and raises the ceiling on partition counts. New clusters should start on KRaft.