Metadata management is the practice of recording and serving data about your data - what a table contains, where it came from, who owns it, when it last loaded - so people and tools can find, trust, and reason about datasets without reading the pipeline code. In a warehouse with thousands of tables, the metadata is what turns an opaque pile of schemas into something searchable and governable.
Three kinds of metadata
It helps to separate metadata by what it describes and who consumes it.
| Type | Describes | Examples |
|---|---|---|
| Technical | Structure | Schema, column types, partition keys, primary keys |
| Operational | Runtime behavior | Last load time, row counts, job duration, freshness |
| Business | Meaning | Definitions, owner, sensitivity/PII tags, glossary terms |
Technical metadata is usually harvested automatically from the warehouse. Operational metadata is emitted by the pipeline as it runs - the same freshness signals that feed the timeliness checks in data quality management. Business metadata is the part humans have to supply, and the part that decays fastest if no one owns it.
Catalogs, lineage, and schema registry
A data catalog - DataHub, Amundsen, OpenMetadata - is the searchable front end: type "revenue" and find the certified table, its owner, and its column descriptions. Lineage records how datasets derive from each other - column-level lineage lets you trace a dashboard metric back through dbt models to the raw source. OpenLineage is the open standard for emitting these events from Airflow, dbt, and Spark so the catalog stitches a graph automatically rather than from hand-drawn diagrams.
# OpenLineage event (abridged)
{
"job": { "namespace": "airflow", "name": "build_orders" },
"inputs": [ { "name": "raw.orders" } ],
"outputs": [ { "name": "mart.fct_orders" } ]
}A schema registry (Confluent Schema Registry, AWS Glue) governs the structure of streaming events - Avro/Protobuf schemas with compatibility rules - so a producer can't silently break every consumer of a Kafka topic, including a CDC stream.
Why metadata powers governance
Three jobs depend entirely on metadata. Discovery: people find the right table instead of rebuilding it. Governance: PII tags and ownership drive access policy and retention. Impact analysis: before dropping a column, lineage shows every downstream model and dashboard that breaks if you do - the question that's otherwise unanswerable until something breaks in production.
Frequently asked questions
What's the difference between technical and business metadata? Technical metadata describes structure (schemas, types, keys) and is mostly auto-harvested; business metadata describes meaning (definitions, ownership, sensitivity) and is supplied by humans.
What does lineage let me do? Trace a dataset's origins and dependencies - answer "what feeds this metric?" and "what breaks if I drop this column?" before making the change.
What is OpenLineage? An open standard for emitting lineage events from tools like Airflow, dbt, and Spark, so a catalog (DataHub, Amundsen) can build the dependency graph automatically.
Why a separate schema registry for streaming? Streaming events have no central schema by default; the registry stores event schemas with compatibility rules so producers can't push a change that breaks existing consumers.