Apache Griffin is an open-source data quality framework built for big data. It lets teams define quality requirements, measure data against them at scale on Spark, and track the results over time - across both batch and streaming sources. It originated at eBay to put a consistent quality model over a large, heterogeneous data platform.

A model-based approach

Griffin organizes quality around dimensions - accuracy, completeness, validity, timeliness, uniqueness - and lets you define measures for each. A common pattern is accuracy by comparing a target dataset against a source of truth (for example, validating that a processed table matches the upstream system within tolerance). Measurements run as Spark jobs, so they scale to large datasets, and results are stored and trackable as metrics rather than one-off checks.

Batch and streaming

Because it runs on Spark, Griffin can measure quality on batch datasets and on streaming data (via Spark Streaming), which suits big-data platforms that need the same quality model across both. It exposes results as metrics that can feed dashboards and alerts.

Where it fits

• Hadoop/Spark big-data platforms needing scalable, model-based quality measurement.
• Source-to-target accuracy validation across large datasets.
• Teams wanting quality tracked as ongoing metrics rather than pass/fail tests alone.

How it compares

Griffin targets big-data scale and a formal quality-dimensions model. Great Expectations is more popular for general pipelines with rich, documented expectations; dbt tests and Soda suit warehouse-centric checks with less setup. Griffin's niche is Spark/Hadoop environments where quality must be measured at volume across batch and streaming. As an Apache project its activity is more modest than the mainstream options, so evaluate community momentum for your needs.