Apache Pig is a platform for writing data-flow programs that run on Hadoop. Its language, Pig Latin, sits between raw MapReduce and SQL: you describe a sequence of transformations - load, filter, group, join, store - as a script, and Pig compiles it into MapReduce (or later, Tez) jobs. It was built at Yahoo for engineers who found MapReduce too low-level but wanted more procedural control than HiveQL's declarative SQL gave them.

Pig Latin: a data pipeline as steps

A Pig script reads as an explicit pipeline, with each statement producing a named relation you can reference in the next. This procedural style makes complex multi-stage flows - several joins, intermediate filtering, custom logic - easier to express and reason about than a single deeply nested SQL statement.

events = LOAD 'events' AS (user:long, action:chararray);
clicks = FILTER events BY action == 'click';
grouped = GROUP clicks BY user;
counts = FOREACH grouped GENERATE group, COUNT(clicks);
STORE counts INTO 'click_counts';

UDFs and nested data

Pig handles nested and semi-structured data well through its bag, tuple, and map types, and it is extensible with user-defined functions in Java, Python, or other languages for logic that doesn't fit built-in operators. This made it popular for ETL over messy log data where records didn't fit a clean tabular schema.

Where Pig stands now

Pig solved a real problem in the early Hadoop years, but the field moved on. Spark's DataFrame API and SQL cover the same procedural-pipeline ground with better performance and a larger ecosystem, and most new work that would once have used Pig now uses Spark or Hive. Pig scripts still run in legacy Hadoop pipelines, but it is rarely chosen for greenfield projects.

Frequently asked questions

How is Pig different from Hive? Pig Latin is procedural - you write an explicit sequence of transformations - while HiveQL is declarative SQL. Pig suits complex multi-step ETL; Hive suits set-based queries.

What does Pig compile to? Originally MapReduce jobs, and later Apache Tez, which it runs on a Hadoop cluster.

Should I use Pig for new pipelines? Generally no - Spark covers the same procedural data-flow needs with better performance and active development; Pig mostly lives on in existing Hadoop systems.