Amazon MSK (Managed Streaming for Apache Kafka) is AWS's managed Kafka service. It runs real, open-source Apache Kafka - same API, same protocol, same clients - while AWS handles broker provisioning, patching, replication setup, and monitoring. You get Kafka without operating ZooKeeper/KRaft, broker failover, and version upgrades yourself.

What "managed" covers

MSK provisions brokers across availability zones, applies patches, replaces failed nodes, and integrates with AWS networking and IAM. Because it is genuine Kafka, existing producers, consumers, Kafka Connect, and Streams apps work unchanged - you point them at the MSK bootstrap brokers. That protocol compatibility is the main reason to choose it over a proprietary queue: no lock-in at the API level.

Provisioned vs Serverless

  • Provisioned - you choose broker instance types and count and manage storage/throughput; cheapest at steady high volume, and you tune partitions and retention directly.
  • Serverless - AWS scales capacity automatically and bills on usage; good for spiky or unpredictable workloads and for teams that don't want to size a cluster.

AWS integration

MSK uses IAM for authentication and authorization (alongside TLS and SASL/SCRAM), so access control lives in the same place as the rest of your AWS permissions. It ships metrics to CloudWatch, supports MSK Connect (managed Kafka Connect) for source/sink connectors, and pairs with Kinesis Data Analytics / Managed Flink for stream processing. Data lands naturally into S3, Redshift, or OpenSearch through connectors.

When to choose it

  • You're on AWS and want Kafka's open API without running the cluster.
  • You need IAM-based security and tight integration with S3, Flink, and CloudWatch.
  • You want to avoid the proprietary lock-in of a non-Kafka queue while offloading operations.

Tradeoffs

MSK manages infrastructure but not your Kafka design - you still choose partition counts, replication factor, retention, and consumer-group strategy, and you still monitor consumer lag. It costs more than self-hosting at scale but far less in engineering time. Compared with Confluent Cloud, MSK gives you closer-to-vanilla Kafka and deep AWS integration, while Confluent adds a richer feature layer (Schema Registry, ksqlDB, more connectors) across clouds. The choice usually comes down to how much of the surrounding ecosystem you want managed for you versus staying close to open Kafka on AWS.