Apache Airflow is a platform for authoring, scheduling, and monitoring data pipelines as code. Workflows are written in Python as directed acyclic graphs (DAGs) of tasks; Airflow works out the execution order from the dependencies, runs the tasks on a schedule or trigger, retries failures, and exposes the whole thing through a web UI. Created at Airbnb, it became the default open-source orchestrator for batch data engineering.

The model: pipelines as DAGs of tasks

A DAG defines what to run and in what order; each node is a task, and edges are dependencies. "Acyclic" matters - a task cannot depend on itself directly or in a loop, which guarantees a finite, well-defined run order. Airflow's job is scheduling and dependency resolution, not heavy computation: tasks should hand work off to the systems built for it (a warehouse, Spark, dbt, a container) rather than crunching large data inside the Airflow worker.

from airflow import DAG
from airflow.operators.python import PythonOperator
import pendulum

with DAG(
    dag_id="daily_sales",
    schedule="0 6 * * *",                 # 06:00 every day, cron syntax
    start_date=pendulum.datetime(2026, 1, 1, tz="UTC"),
    catchup=False,                        # don't backfill all missed runs on deploy
    default_args={"retries": 2},
) as dag:

    extract = PythonOperator(task_id="extract", python_callable=pull_orders)
    load    = PythonOperator(task_id="load",    python_callable=load_to_warehouse)

    extract >> load                       # 'extract' must succeed before 'load'

Scheduling, intervals, and the catchup trap

Airflow schedules around data intervals: a run labeled for a given period kicks off after that period ends. The most common newcomer surprise is catchup=True (the historical default): deploy a DAG with a start_date months in the past and Airflow immediately queues a run for every interval since, flooding the cluster. Set catchup=False unless you genuinely want a backfill, and trigger intentional backfills explicitly.

Operators, sensors, and the task ecosystem

• Operators are task templates - PythonOperator, BashOperator, plus provider operators for Snowflake, BigQuery, dbt, Spark, Kubernetes, and hundreds more.
• Sensors wait for a condition (a file arrives, a partition lands, an external job finishes). Use mode="reschedule" or deferrable sensors so a waiting task frees its worker slot instead of blocking it.
• The TaskFlow API lets you write tasks as decorated Python functions and pass return values between them directly, which reads more naturally than wiring operators by hand.

from airflow.decorators import dag, task

@dag(schedule="@daily", start_date=pendulum.datetime(2026,1,1,tz="UTC"), catchup=False)
def sales_etl():
    @task
    def extract():
        return pull_orders()

    @task
    def transform(rows):
        return clean(rows)

    transform(extract())

sales_etl()

What Airflow is good at - and what it isn't

It excels at scheduled, dependency-heavy batch orchestration: nightly ELT, model training pipelines, report generation, anything that is "run these steps in this order, on this cadence, and tell me when something breaks." It is a weak fit for low-latency streaming (use Flink or Kafka Streams), for sub-minute schedules, or for passing large datasets between tasks - XComs are for small metadata, not dataframes. Treat the worker as a conductor, not a compute engine.

Architecture you operate

A production deployment has several moving parts: the scheduler (decides what runs when), the metadata database (Postgres holds all run state), workers (execute tasks, typically via the Celery or Kubernetes executor), and the webserver (UI and API). The KubernetesExecutor runs each task in its own pod, isolating dependencies and scaling to zero between runs; the CeleryExecutor keeps a warm worker pool for lower per-task latency. The metadata database is the system of record - back it up, and don't let it fill with years of task history (prune with the built-in cleanup).

Idempotency and retries

Because Airflow retries tasks and reruns intervals on backfill, every task should be idempotent: running it twice for the same data interval must produce the same result. The standard pattern is delete-then-insert (or MERGE) for the task's partition, keyed on the run's data interval, rather than blind appends. Without this, a single retry silently double-loads data.

Where it sits among alternatives

Airflow's strengths are maturity, the breadth of provider integrations, and a large operator ecosystem. Newer orchestrators target its rough edges: Dagster centers on data assets and typed inputs/outputs with strong local testing, and Prefect emphasizes dynamic, Pythonic flows with less boilerplate. Airflow remains the safe default when you need broad integrations and a known operational model; the others are worth evaluating when asset-awareness or developer ergonomics dominate your requirements.