Apache Hadoop is the framework that launched the big-data era: a way to store and process datasets too large for any single machine using clusters of commodity hardware. Its two foundational pieces are HDFS, a distributed file system that splits files into blocks and replicates them across nodes, and MapReduce, a programming model for processing those files in parallel. Later, YARN was added as a cluster resource manager so engines other than MapReduce could share the same hardware.

HDFS: storage that tolerates failure

HDFS stores each file as a sequence of large blocks (128 MB by default) spread across DataNodes, with a NameNode tracking where every block lives. Each block is replicated - three copies by default - so a disk or node failure loses no data and the job keeps running. The design assumes hardware fails routinely and builds reliability in software, which is what made clusters of cheap servers viable for petabyte-scale storage.

MapReduce: move compute to the data

MapReduce processes data in two phases. A map function runs on each block, ideally on the node that already holds it - moving computation to the data instead of shipping terabytes across the network. A shuffle groups intermediate records by key, and a reduce function aggregates each group. It is powerful but writes intermediate results to disk between stages, which makes it slow for iterative or interactive work.

Where Hadoop stands now

MapReduce as a programming model has largely been displaced by faster engines - Spark keeps intermediate data in memory and runs the same jobs far quicker - but the Hadoop ecosystem persists. HDFS and YARN still underpin many on-prem clusters, and tools like Apache Hive and Apache Pig were built on top of it. In the cloud, object storage like S3 has replaced HDFS for many new deployments, but the architectural ideas - partition, replicate, process locally - are everywhere.

Frequently asked questions

What are Hadoop's three core components? HDFS (distributed storage), MapReduce (parallel processing), and YARN (resource management) that lets multiple engines share a cluster.

Why is MapReduce considered slow? It writes intermediate results to disk between the map, shuffle, and reduce stages, so iterative and interactive jobs pay heavy I/O costs that in-memory engines like Spark avoid.

Is Hadoop obsolete? MapReduce is rarely chosen for new work, but HDFS and YARN remain in production, and cloud object storage plus Spark cover the same needs for newer platforms.