Data streaming and messaging systems move events between the services that produce them and the ones that consume them, in near real time. They decouple producers from consumers so each side can run, scale, and fail independently - the backbone of event-driven architectures, real-time analytics, and the ingestion side of most modern data platforms.

Two broad designs sit in this category. Log-based streaming systems (Kafka and its managed kin) keep an ordered, durable, replayable record of events that many consumers can read at their own pace; they suit high-throughput pipelines, event sourcing, and stream processing. Message brokers (RabbitMQ, and protocol brokers like NATS) route individual messages to consumers and excel at task queues, request/reply, and flexible routing, often with the message removed once acknowledged.

The practical choice turns on whether you need replay and multiple independent readers of the same stream (favoring a log) or flexible per-message routing and work distribution (favoring a broker), plus how much you want to operate yourself versus consume as a managed service. The pages below cover the main options - from self-managed brokers to the cloud-managed Kafka services on AWS, Azure, and Red Hat.