Luigi, Prefect, and Keboola all promise to keep your data pipelines running on schedule, in the right order, with sensible behavior when something fails. But they sit at different points on a spectrum: Luigi and Prefect are Python orchestration libraries you assemble and host yourself, while Keboola is a managed platform that bundles connectors, storage, and transformation around the orchestration. The choice is less about features and more about how much of the stack you want to own.

Luigi: the original Python task runner

Built at Spotify, Luigi models a pipeline as a graph of Task classes. Each task declares what it requires(), what it produces as output(), and what to do in run(); Luigi resolves the dependency graph and refuses to run a task until its inputs exist. The mental model is deliberately blunt - a target is a file (or a row, or an object in S3) that either exists or doesn't, and existence is how Luigi decides what to skip on a re-run.

class AggregateCustomerMetrics(luigi.Task):
    date = luigi.DateParameter(default=date.today())

    def requires(self):
        return CleanCustomerData(date=self.date)

    def output(self):
        return luigi.LocalTarget(f"data/metrics/{self.date}.csv")

    def run(self):
        df = pd.read_csv(self.input().path)
        df.groupby("region").agg({"revenue": "sum"}) \
          .to_csv(self.output().path)

That target-existence model is Luigi's strength and its ceiling. It handles clear, file-based batch dependencies cleanly, and it costs nothing to run. But it has no native scheduler beyond its own central daemon (most teams trigger it with cron or Airflow), dynamic task generation is awkward, and the visualization UI is bare. Luigi fits Python-comfortable teams running batch workloads on their own infrastructure who want something small and predictable.

Prefect: orchestration that keeps Python's execution model

Prefect was designed to fix the friction in older orchestrators. Instead of forcing you into a new paradigm, it decorates ordinary Python functions as @task and composes them into a flow. Tasks can be generated at runtime, which makes fan-out and conditional branches natural; state handling is first-class, so retries, failure callbacks, and skips are configuration rather than custom code. Deployment scales from a local process to a self-hosted server to Prefect Cloud without rewriting the flow.

The trade you make is observability and dynamic control in exchange for running the orchestration layer yourself (or paying for the managed tier). Prefect suits teams building complex, branching workflows - ML retraining loops, pipelines whose shape depends on what data arrived - where Luigi's file-existence model would strain and you want detailed run history without building it.

Keboola: the whole data-ops platform, orchestration included

Keboola is a different category. Rather than a library you wire into your own stack, it's a managed SaaS covering the full lifecycle: hundreds of pre-built source and destination connectors, managed storage and compute, transformations in SQL, Python, or R, plus lineage, versioning, and access control. Orchestration - scheduling, dependencies, notifications - is one feature inside that platform, configured through a UI rather than authored in code.

That breadth is the point for teams without a large engineering bench: you connect Google Analytics, Facebook Ads, and a CRM, write the joins in SQL, and ship a dashboard without provisioning a single server. The cost is the usual managed-platform trade - you accept the platform's boundaries and pricing in exchange for not maintaining infrastructure, and non-engineers can work alongside engineers in the same workspace.

How to decide

• You have Python engineers and want to own the stack. Luigi for simple batch DAGs with file dependencies; Prefect when workflows are dynamic, branchy, or need real observability.
• You want flexibility between local, self-hosted, and cloud. Prefect's deployment range is the widest of the three.
• You're light on engineering or need connectors and transformation, not just scheduling. Keboola does the most out of the box and lets non-developers participate.

These aren't mutually exclusive. A common split is Keboola for broad data integration and a code-first orchestrator like Prefect for specialized ML or analytics workflows that need finer control. Match the tool to how much of the pipeline you actually want to build and maintain, and the decision mostly makes itself.