Great Expectations is an open-source framework for data quality. You declare expectations - assertions about what valid data looks like - and it validates batches of data against them, failing loudly when reality diverges. It catches bad data at the pipeline boundary, before it reaches a warehouse table or a dashboard and quietly corrupts decisions.
Expectations as declarative tests
An expectation is a readable assertion: a column has no nulls, values fall in a range, a column is unique, a date is not in the future. You group them into a suite for a dataset and run it against incoming data. The assertions are declarative and self-documenting - the suite doubles as a living specification of what the data should be.
validator.expect_column_values_to_not_be_null("order_id")
validator.expect_column_values_to_be_between("amount", min_value=0, max_value=100000)
validator.expect_column_values_to_be_in_set("status", ["paid", "pending", "refunded"])
validator.expect_column_values_to_be_unique("order_id")Data Docs and profiling
Two features make it more than an assertion library. Data Docs auto-generates human-readable HTML reports of expectations and validation results - a shareable, always-current data quality report. A profiler can inspect a dataset and suggest an initial suite, so you don't write every expectation by hand. Validation results can gate a pipeline: stop the run, or quarantine the batch, when data fails.
Where it fits in a pipeline
- Validate raw data at ingestion before it's trusted downstream.
- Check transformed outputs (often run as an Airflow/dbt step) before publishing.
- Produce shareable quality documentation for stakeholders.
How it compares
Great Expectations is the most established open-source data quality framework, strong on expressiveness and documentation, but it carries setup and configuration overhead. dbt tests cover quality inside the dbt transformation layer more simply (and suit teams already on dbt); Soda offers a lighter YAML-based checks approach; Apache Griffin targets big-data/streaming quality. Choose Great Expectations when you want rich, documented validations across diverse sources; choose dbt tests or Soda when you want lighter checks close to your transformations.