Data lineage is the record of where data comes from, how it's transformed, and where it goes - the family tree of every table, column, and report in a data platform. When an executive asks "where does this number come from?" or a compliance officer asks "which reports use customer email?", lineage is what turns a week of manual investigation into a query. It's the connective tissue that makes a warehouse auditable and debuggable rather than a black box.
Table-level and column-level lineage
Lineage exists at two granularities. Table-level lineage maps which tables feed which - source → staging → mart → dashboard - and is enough to answer "what's upstream of this report?" Column-level lineage goes finer, tracing how a specific column was derived (this revenue field is amount divided by 100 from the payments source), which is what you need for true impact analysis and for tracking sensitive fields like PII through every transformation. Column-level is harder to capture and far more valuable when it matters.
How it's captured
Modern tools generate lineage automatically rather than asking people to document it. dbt produces a complete model-to-model graph from its ref() calls. Query engines expose the parse trees of executed SQL, from which table and column dependencies can be inferred. Orchestration tools like Airflow define pipeline-to-pipeline relationships. Standards like OpenLineage emit lineage events from jobs into a metadata store (such as Marquez) so the graph spans tools. The challenge is unifying these sources - dbt, Spark, the warehouse, the BI tool - into one coherent graph rather than five partial ones.
Why it earns its keep
Lineage pays off in three recurring situations. Impact analysis before a change: "if I rename this upstream column, what breaks?" - answered by walking the graph downstream instead of by surprise in production. Debugging a wrong number: trace the dashboard back through its transformations to find where the value went wrong. And compliance: prove where sensitive data flows, which is exactly what regulations like GDPR demand. Partial lineage is the dangerous kind - it looks like coverage and isn't - so the goal is end-to-end, and the gaps tend to reveal themselves during an incident, the worst possible time to discover them.
Frequently asked questions
What's the difference between table-level and column-level lineage? Table-level maps which tables feed which; column-level traces how an individual column was derived, which is what enables true impact analysis and PII tracking.
How is lineage captured automatically? From dbt's model graph, parsed query plans, orchestration dependencies, and standards like OpenLineage that emit events into a metadata store.
Why does lineage matter for compliance? It proves where sensitive data flows across the platform - required to answer "where does customer email appear?" under regulations like GDPR.
Why is partial lineage risky? It looks like full coverage but leaves blind spots, and those gaps usually surface during an incident when there's no time to investigate manually.