Data Quality Management

Data quality management is the set of checks, contracts, and processes that keep bad data from reaching the people who trust it. A pipeline that runs green every night but loads duplicated orders or null customer IDs is worse than one that fails loudly - the silent corruption gets baked into reports before anyone notices. Quality work makes the failure visible at the point it enters the warehouse.

The six dimensions

Quality is usually broken into measurable dimensions so a vague "the data looks wrong" becomes a specific, testable assertion.

Dimension	Question it answers	Example check
Completeness	Are required values present?	`customer_id IS NOT NULL`
Accuracy	Does it match reality?	order total = sum of line items
Consistency	Do related values agree?	ship_date ≥ order_date
Timeliness	Is it fresh enough?	max(loaded_at) within 2 hours
Uniqueness	Any unintended duplicates?	one row per order_id
Validity	Does it fit the allowed format/range?	status in known enum

Testing in the pipeline

The checks belong inside the pipeline, not in a separate audit after the fact. Three common tools express the same dimensions in different ways:

dbt tests - declarative assertions in YAML next to the model, run as part of the transform step.
Great Expectations - Python "expectation suites" that validate a dataframe or table and emit data docs.
Soda - checks written in SodaCL/YAML, often pointed at the warehouse on a schedule.

# dbt schema.yml
models:
  - name: stg_orders
    columns:
      - name: order_id
        tests: [unique, not_null]
      - name: status
        tests:
          - accepted_values:
              values: ['pending','shipped','cancelled']

Quarantine vs. fail-fast, and SLAs

When a check fails you have two responses. Fail-fast stops the run so nothing bad propagates - right for the kind of breakage where partial data is dangerous (a broken revenue table). Quarantine routes the offending rows to a side table and lets the clean rows through, so one malformed record doesn't block the other 99.99%. Quarantined rows are then triaged like the dead-letter records in error handling. The choice should be explicit per check, not accidental.

Quality targets get written down as SLAs and SLOs: an SLO might be "99.5% of orders complete and valid"; the SLA is the commitment to consumers - e.g. "daily sales table fresh by 07:00, or the data team is paged." These pair with the timeliness dimension and with checks applied to CDC streams, where out-of-order events can violate consistency.

Frequently asked questions

Quarantine or fail-fast - which is the default? Fail-fast for breakage where partial loads corrupt downstream numbers; quarantine for row-level defects where the clean majority should still flow. Decide per check.

dbt tests vs. Great Expectations vs. Soda? dbt tests live with the model and run in the dbt graph; Great Expectations gives rich Python suites and data docs; Soda targets scheduled checks against the warehouse. They overlap - pick by where your transforms already live.

What's the difference between an SLA and an SLO? The SLO is the internal target (e.g. 99.5% valid rows); the SLA is the external commitment to consumers, with consequences when missed.

Why test inside the pipeline instead of auditing later? In-pipeline checks catch defects before they reach reports, and can fail or quarantine the load; a later audit only tells you the bad data already shipped.