Apache Oozie is a workflow scheduler built for the Hadoop ecosystem. It runs DAGs of Hadoop jobs - MapReduce, Hive, Pig, Sqoop, and Spark - defined in XML, and triggers them on time schedules or when input data arrives. Oozie was the standard orchestrator for on-prem Hadoop clusters; in modern stacks it has largely been displaced by Airflow and cloud-native schedulers, but it remains in service across long-lived Hadoop estates.

The three building blocks

  • Workflow - a DAG of action nodes (a Hive query, a Spark job) with control nodes (start, decision, fork/join, end), defined in an XML file (workflow.xml).
  • Coordinator - runs a workflow on a schedule and/or when data dependencies are satisfied (e.g., "run when yesterday's partition exists").
  • Bundle - groups multiple coordinators to manage them together.
<workflow-app name="daily-etl" xmlns="uri:oozie:workflow:0.5">
  <start to="hive-load"/>
  <action name="hive-load">
    <hive xmlns="uri:oozie:hive-action:0.5">
      <script>load_daily.hql</script>
    </hive>
    <ok to="end"/> <error to="fail"/>
  </action>
  <kill name="fail"><message>ETL failed</message></kill>
  <end name="end"/>
</workflow-app>

How it differs from Airflow

Oozie workflows are declared in XML and tightly bound to Hadoop actions; Airflow defines pipelines in Python with operators for almost any system. Oozie's data-trigger coordinators were a strong feature for partition-driven Hadoop pipelines, but the XML is verbose, testing is awkward, and it assumes a Hadoop cluster. Airflow's Python authoring, broad integrations, and richer UI are why most teams migrated to it, with cloud schedulers covering the rest.

Where it fits

  • Existing on-prem Hadoop clusters where Oozie already orchestrates jobs.
  • Partition/data-availability-driven scheduling of Hadoop workloads via coordinators.

For new pipelines, Airflow, Dagster, or a cloud-native scheduler are the usual choices; Oozie is mostly encountered when maintaining or migrating legacy Hadoop workflows. (Note: Oozie is a workflow scheduler rather than a containerization tool - it appears in this category to match the site's existing taxonomy.)