Azkaban is a batch workflow scheduler built at LinkedIn to run Hadoop jobs in dependency order. You define a workflow as a set of jobs and the dependencies between them, package the job files into a ZIP, and upload it through a web UI that then schedules, runs, and monitors the flow. It predates Airflow and shares its purpose - ordered execution of dependent batch jobs - but expresses workflows through property files and a project ZIP rather than Python code. It remains in use in Hadoop-centric shops, though most new projects choose a code-defined orchestrator.
Jobs, dependencies, and the project ZIP
Each job is a .job properties file naming a job type and its command, with a dependencies key listing the jobs that must finish first. Azkaban reads those keys to build the DAG.
# extract.job
type=command
command=python3 extract.py
# transform.job
type=command
command=python3 transform.py
dependencies=extract
# load.job
type=command
command=python3 load.py
dependencies=transformYou ZIP the .job files plus any scripts, upload the ZIP as a project version, and Azkaban renders the flow graph. Running load pulls in extract and transform first, in order.
Architecture and execution
Azkaban runs as two services: a web server that hosts the UI, projects, scheduling, and authentication, and one or more executor servers that actually run the jobs. A MySQL database holds projects, schedules, and execution history. Splitting web from executors lets the scheduling tier stay responsive while heavy Hadoop jobs run on the executor pool. Built-in job types cover plain commands, Java, Hadoop, Pig, and Hive, with plugin types for more.
| Component | Role |
|---|---|
| Web server | UI, projects, scheduling, auth, history |
| Executor server | Runs the jobs in a flow |
| MySQL | Projects, schedules, execution records |
| .job files | Job type, command, and dependencies |
Where it fits today
Azkaban's strengths are a simple dependency model, a clear UI for Hadoop batch flows, and per-job permissions suited to a shared cluster. Its limits show against modern tools: workflows are static property files rather than dynamic code, there's no rich templating or dynamic task generation, and the connector ecosystem is narrow. Teams running an existing Azkaban installation often keep it; greenfield data platforms generally pick Airflow or Dagster for code-defined workflows and broader integrations.
Frequently asked questions
How does Azkaban compare to Airflow? Both schedule dependent batch jobs, but Azkaban uses property-file projects uploaded as a ZIP while Airflow defines DAGs in Python. Airflow's dynamic generation, templating, and integrations are why it became the default for new work.
Is Azkaban still maintained? The open-source project is stable but low-activity. Evaluate the release cadence and your need for new integrations before standardizing on it for new pipelines.
What job types does it support? Command, Java, Hadoop (MapReduce), Pig, and Hive out of the box, plus a plugin mechanism for additional types.
Can it retry failed jobs? Yes - jobs can be configured with retry counts and intervals, and a failed flow can be resumed or restarted from the failed node through the UI.