Metaflow is a Python framework for building and running data science and ML workflows, originally developed at Netflix. It targets the data scientist's path from a laptop notebook to a production run on the cloud, keeping the same code throughout. You structure work as a flow of steps; Metaflow handles versioning, artifact storage, and scaling individual steps onto cloud compute, so the modeling code stays clean while the infrastructure concerns move into decorators and flags.

Flows and steps

A flow is a class subclassing FlowSpec with methods marked @step. Each step assigns instance attributes that become versioned artifacts, automatically passed to the next step and persisted for later inspection. The self.next() call defines the linear or branching control flow.

from metaflow import FlowSpec, step, resources

class TrainFlow(FlowSpec):
    @step
    def start(self):
        self.data = load_features()
        self.next(self.train)

    @resources(memory=16000, cpu=4)
    @step
    def train(self):
        self.model = fit(self.data)   # artifact, versioned + stored
        self.next(self.end)

    @step
    def end(self):
        print("done")

if __name__ == "__main__":
    TrainFlow()

The @resources decorator tells Metaflow how much compute the step needs; with a cloud backend, that step runs on a right-sized machine while the rest run locally.

Artifacts, versioning, and the Client API

Every artifact (anything assigned to self) is content-addressed and stored, so each run is fully reproducible and inspectable. The Client API lets you reach into any past run from a notebook - Run('TrainFlow/42').data.model pulls the exact model object from run 42 without re-executing. This run-as-data model is Metaflow's defining feature for experiment tracking: there's no separate logging step, the artifacts are the record.

Concept	Role
FlowSpec / step	The workflow class and its discrete stages
Artifact	Versioned self. attribute, persisted and inspectable
@resources / @batch	Run a step on cloud compute with set CPU/memory/GPU
Client API	Read any past run's artifacts from code

Scaling and scheduling

Locally, every step runs as a subprocess. Point Metaflow at a cloud backend and steps decorated with @batch or @kubernetes run on AWS Batch or Kubernetes, with artifacts stored in S3 and metadata in a tracked service. The foreach construct fans a step out over a list - one task per shard - for embarrassingly parallel work like hyperparameter sweeps. For recurring production runs, Metaflow exports a flow to a scheduler: AWS Step Functions or Argo Workflows execute it on a cron, so the same code a scientist ran by hand becomes a scheduled job without rewriting.

Frequently asked questions

Is Metaflow an orchestrator or an ML framework? Both, narrowly. It orchestrates Python steps and scales them to the cloud, but it is opinionated toward data science and ML rather than general data pipelines. For broad data orchestration, see Dagster or Prefect.

Do I need AWS to use it? No for local development, but the scaling story (@batch, S3 artifacts, Step Functions) is most mature on AWS. Kubernetes and Argo backends broaden it beyond AWS.

How does it track experiments? Implicitly. Artifacts assigned to self are versioned per run, so the Client API can retrieve any run's exact inputs, model, and metrics later - no separate tracking call.

Can two steps run in parallel? Yes, via branching (split into multiple self.next targets, then join) or foreach for data-parallel fan-out across a list.