Snakemake is a workflow manager from the bioinformatics world, modeled on GNU Make but written in Python. You describe a pipeline as a set of rules, each declaring its input files, output files, and the shell command or Python code that turns one into the other. Snakemake reads the rules, works backward from the files you request, and builds the dependency graph by matching outputs to inputs through filename patterns. It then runs only the rules whose outputs are missing or older than their inputs - the same incremental logic as Make, applied to data analysis.
Rules and file-based dependencies
A rule names its inputs and outputs as file paths, often with wildcards that let one rule apply to many samples. Snakemake resolves the chain by pattern-matching: it sees you want results/sample_A.sorted.bam, finds the rule that produces {sample}.sorted.bam, and recursively resolves that rule's inputs.
rule sort:
input: "mapped/{sample}.bam"
output: "results/{sample}.sorted.bam"
threads: 4
shell: "samtools sort -@ {threads} {input} -o {output}"
rule all:
input: expand("results/{sample}.sorted.bam",
sample=["A", "B", "C"])Requesting rule all expands to three target files; Snakemake runs sort three times, in parallel up to the core limit, skipping any sample whose sorted output is already up to date.
Reproducibility: conda, containers, and dry runs
Each rule can declare its own conda environment or container, so Snakemake provisions the exact software a step needs and the pipeline is reproducible across machines. The --dry-run flag prints the jobs that would run without executing them - essential for checking a large run before committing compute. Because dependencies are file timestamps, re-running after editing one input recomputes only the affected downstream files.
| Feature | What it gives you |
|---|---|
| Wildcards | One rule applied across many samples via pattern matching |
| Incremental builds | Re-run only outputs older than or missing relative to inputs |
| conda / container per rule | Reproducible, isolated software environments |
| Cluster / cloud executors | Submit jobs to SLURM, Kubernetes, or cloud batch |
Scaling beyond one machine
The same workflow runs on a laptop, an HPC cluster, or the cloud by swapping the executor. Snakemake generates one job per rule instance and submits them to SLURM, SGE, Kubernetes, or cloud batch services, respecting the declared threads and resources. This portability and the file-timestamp model make it a fit for scientific pipelines with thousands of per-sample steps; it is less suited to event-driven or streaming orchestration, where a service-style scheduler fits better.
Frequently asked questions
How is Snakemake different from Airflow? Snakemake is file- and rule-driven, like Make: dependencies are inferred from filename patterns and re-runs are incremental. Airflow is task- and schedule-driven for recurring service pipelines. Snakemake dominates in bioinformatics and reproducible research.
Does it only work for bioinformatics? No - that's its origin, but any file-transformation pipeline (data prep, simulations, ETL over files) fits the model. The strength is incremental, reproducible batch processing.
How does it decide what to re-run? By comparing output and input file modification times, plus optional tracking of code and parameter changes, so editing one input recomputes only its downstream outputs.
Can it run on a cluster? Yes. A profile maps rules to SLURM, SGE, Kubernetes, or cloud batch, submitting one job per rule instance with its declared CPU, memory, and time resources.