Marquez is an open-source metadata service for collecting, storing, and visualizing data lineage. It is the reference implementation of the OpenLineage standard: pipelines emit OpenLineage events, Marquez records them, and its UI renders the resulting graph of how datasets and jobs depend on each other. Originally built at WeWork, it's now an LF AI & Data project.
What it stores
Marquez keeps a model of datasets, jobs, and runs. Every pipeline execution is a run with recorded inputs and outputs, schemas, and timing. Because it captures this per run, you get not just a static lineage diagram but a history: which run produced a given table version, when a job's inputs changed, and where a failure propagated.
Lineage and operational metadata
- Lineage graph - dataset-to-job-to-dataset dependencies, navigable in the UI.
- Run history - start/end times, states, and the exact inputs/outputs of each execution.
- Dataset versions and schemas tracked over time.
- An API to query lineage programmatically for impact analysis and tooling.
Where it fits
Marquez is the natural backend when you've instrumented pipelines with OpenLineage (via the Airflow, Spark, or dbt integrations) and want a place to store and explore the lineage without adopting a full catalog. It answers "where did this data come from, and what breaks if I change it?" with run-level accuracy.
How it compares
Marquez is focused - a lineage and run-metadata store with a viewer, not a full discovery/governance catalog. Broader platforms like DataHub and OpenMetadata also consume OpenLineage and add search, documentation, quality, and governance on top. Choose Marquez when lineage collection and visualization is the specific need (often paired with OpenLineage); choose a full catalog when you also want discovery and governance in one place.