Prefect is a Python orchestrator that turns ordinary functions into observable, retryable workflows with decorators. You write normal Python; adding @flow and @task gives you scheduling, retries, caching, logging, and a UI without restructuring the code into a static DAG file. Prefect's design philosophy is dynamic: the workflow graph is discovered at runtime as the code executes, so loops, conditionals, and data-dependent fan-out are native rather than awkward.

Flows and tasks

A flow is the orchestration unit Prefect tracks as a run; tasks are the retryable, cacheable units inside it. Because the graph is built by running the function, you can call tasks in a loop or branch on their return values directly.

from prefect import flow, task

@task(retries=3, retry_delay_seconds=10)
def fetch(region: str) -> list:
    return api_pull(region)

@flow
def ingest(regions: list[str]):
    for region in regions:          # dynamic fan-out, no static DAG
        rows = fetch(region)
        load(rows)

ingest(["us", "eu", "apac"])

The fetch task retries three times on failure independently per region; the flow run records each task's state. Run that same file anywhere - locally, in CI, in a container - and it reports to the same Prefect API.

Deployments, work pools, and workers

To schedule a flow, you create a deployment: metadata that pairs the flow with a schedule, parameters, and an infrastructure target. Deployments submit runs to a work pool; workers poll the pool and execute each run on the configured infrastructure - a subprocess, a Docker container, a Kubernetes job, or a serverless push pool. This decoupling means the orchestration API never executes your code; it only tracks state and hands work to your own compute.

Concept	Role
Flow	The tracked workflow run, a decorated Python function
Task	Retryable, cacheable unit inside a flow
Deployment	Flow + schedule + parameters + infrastructure target
Work pool / worker	Queue plus the process that runs submitted flow runs

State, caching, and the backend

Every task and flow run moves through explicit states (Pending, Running, Completed, Failed, Retrying), and Prefect persists them so the UI can show exactly where a run is and why it failed. Task caching keys a result on inputs and a cache policy, so a re-run skips work whose inputs haven't changed. The backend is either Prefect Cloud (hosted) or self-hosted Prefect Server, both backed by a Postgres database; the open-source server runs the API, scheduler, and UI in one process for small setups and splits out for scale.

Frequently asked questions

How does Prefect differ from Airflow? Airflow defines a static DAG up front; Prefect builds the graph dynamically as the code runs, so loops and conditionals are native. Prefect flows are also plain Python you can run and test without a scheduler.

Does the Prefect API run my code? No. The API tracks state and schedules; workers in your own infrastructure pull and execute runs. Your data and code never live on Prefect Cloud's servers.

Prefect 1 versus 2 versus 3? Prefect 2 dropped the rigid DAG model for dynamic flows; Prefect 3 refined work pools, events, and performance. New projects should start on Prefect 3 - the 1.x "Orion" concepts no longer apply.

How is it different from Dagster? Prefect orchestrates tasks and is lightweight to adopt incrementally; Dagster centers on data assets and lineage. Choose Prefect when you want to wrap existing Python with minimal restructuring.