Dagster is a Python orchestrator built around data assets rather than tasks. Where most schedulers ask "what operations run, in what order," Dagster asks "what tables, files, and models should exist, and how is each produced." You declare an asset with the @asset decorator; its function parameters name the upstream assets it depends on, and Dagster builds the dependency graph from those signatures. That asset-centric model makes lineage, freshness, and partition-by-partition backfills first-class instead of bolted on.

Software-defined assets

An asset is a function that computes and persists one object - a database table, a Parquet file, an ML model. The return value is handed to an I/O manager that decides where it lands, so business logic stays separate from storage. Dependencies are expressed by parameter name, not by wiring task order.

from dagster import asset

@asset
def raw_orders() -> pd.DataFrame:
    return pd.read_sql("SELECT * FROM orders", conn)

@asset
def daily_revenue(raw_orders: pd.DataFrame) -> pd.DataFrame:
    return (raw_orders
            .groupby(raw_orders.created_at.dt.date)["amount"]
            .sum().reset_index())

Dagster reads daily_revenue's dependency on raw_orders from the argument, materializes upstream first, and records the lineage. The asset graph in the UI then shows what's stale and lets you rematerialize only the downstream assets affected by a change.

Partitions, backfills, and the I/O manager

Assets can be partitioned - by date, by region, by any keyspace - and Dagster tracks which partitions exist. A backfill becomes "materialize partitions 2026-01-01 through 2026-03-31," launched and monitored as one run with per-partition status. I/O managers handle the read/write side: one manager can persist every asset to S3 as Parquet, another to a warehouse table, without touching the asset code.

ConceptRole
AssetA persisted data object, defined by the function that builds it
I/O managerPluggable read/write layer (S3, warehouse, local) per asset
PartitionA slice of an asset (e.g. one day) tracked independently
Sensor / scheduleTriggers materialization on time or on external events

Scheduling, sensors, and deployment

Schedules launch runs on a cron expression; sensors poll for external state - a new S3 file, an upstream asset freshness violation - and launch runs in response. Dagster runs as a long-lived daemon (for schedules and sensors) plus a web server (Dagit) for the asset graph and run logs. Code lives in code locations loaded over gRPC, so you redeploy a project's logic without restarting the whole instance. For production, the daemon and web server are typically separate containers backed by a Postgres run storage.

Frequently asked questions

How is Dagster different from Airflow? Airflow orchestrates tasks; Dagster orchestrates assets and derives the task graph from declared data dependencies. Dagster's asset lineage, typed inputs/outputs, and local testability are the usual reasons teams pick it for new data platforms.

Does it replace dbt? No - it orchestrates dbt. Dagster loads a dbt project so each dbt model appears as an asset in the same graph as your Python assets, giving end-to-end lineage across both.

Can I test assets locally? Yes. An asset is a plain Python function, so you call it directly in a unit test, and I/O managers can be swapped for in-memory versions, which makes Dagster pipelines unusually testable.

What backs a production deployment? A Postgres run/event-log storage, the Dagster daemon for schedules and sensors, a web server, and per-project code locations. Dagster+ (the hosted product) and Helm charts cover managed and Kubernetes setups.