Consider a mid-size company - 300 engineers, 14 domain teams, a drowning central data team of 8, and a backlog of 200+ data requests - that pitched a full data mesh in early 2024: stop bottlenecking everything through the center, give domains ownership, build a self-serve platform. Two years on, the lived experience looks nothing like the conference talks. This is the honest retrospective - where it succeeded, where it failed, and what to do instead.
What data mesh promises
Zhamak Dehghani's framework rests on four principles: domain ownership (the teams that produce data own and serve it), data as a product (treat data outputs with product rigor), self-serve platform (infrastructure that makes domains autonomous), and federated computational governance (decentralize governance while keeping global standards). On paper it's compelling - the central team is a bottleneck in every growing org, domain experts understand their data best, decentralization scales. It all makes sense until you implement it with real humans in a real organization.
The starting point
Before mesh, the architecture was the classic central bottleneck: Airbyte → S3 raw, dbt and 300+ Airflow DAGs all owned by the data team, a single Snowflake account, Looker on top. The data team of 8 carried a 200+ request backlog and a 3-6 week average time to deliver. Domain teams submitted requests, waited weeks, got something that didn't match their mental model, sent it back, repeated - while the central team burned out maintaining DAGs and debugging quality issues in domains it barely understood.
What was actually built
The rollout took about nine months. The platform came first (months 1-4) - you can't ask domain teams to own data without tools. Domain teams defined a YAML manifest for each data product, and the platform generated the Airflow DAGs, dbt models, Snowflake schemas, Great Expectations checks, and catalog entries automatically:
class DataProductManifest:
name: str # "payments.transactions"
owner_team: str # "payments"
sla: SLAConfig # freshness, availability targets
schema: SchemaDefinition # versioned Avro/JSON schema
quality_checks: list[QualityRule]
access_policy: AccessPolicy # who can read, PII handling
output_ports: list[OutputPort] # Snowflake table, Kafka topic, API
Two platform engineers, four months, worth every hour. Ownership transfer (months 4-7) is where it got painful. Of three pilot domains, Payments had an enthusiastic senior engineer and improved data quality within weeks (success); Catalog assigned a part-time junior and treated it as a chore (partial failure); User Identity flatly refused - "we're measured on identity-service uptime and auth latency, not on whether analytics can run reports," and they weren't wrong. Governance and discovery (months 7-9) added a catalog, naming and schema-evolution standards, and a biweekly "data council" whose attendance dropped to three people within two months.
What worked
Data-as-a-product thinking was the single most valuable concept - and it requires no decentralization. Giving every output an SLA, documentation, quality checks, and an owner transformed a Snowflake graveyard of undocumented tables into governed products with freshness targets, anomaly checks, and a breaking-changes policy. That alone cut data-quality incidents by about 60%, with nothing to do with who owns the pipeline. The self-serve platform let engaged teams stand up a data product in hours instead of weeks - and the key lesson was to make it opinionated: early attempts at maximum flexibility failed, but narrowing to exactly three output patterns (Snowflake table, Kafka topic, REST API) drove adoption, because engineers want a paved road, not infinite choice. Explicit data contracts between domains caught breaking schema changes in CI/CD before they silently broke downstream pipelines:
def validate_schema_change(old_schema, new_schema) -> ContractResult:
breaking = BreakingChangeDetector().find_breaking_changes(old_schema, new_schema)
if breaking:
for consumer in get_downstream_consumers(old_schema["product_name"]):
notify_slack(consumer.slack_channel,
f"Breaking change for {old_schema['product_name']}: {breaking}. "
f"Migration window: 14 days.")
return ContractResult(approved=False, reason=f"Breaking changes: {breaking}")
return ContractResult(approved=True)
What failed
Domain teams mostly don't want to own data. Of 14 teams, exactly 3 genuinely adopted ownership; the rest were passive resisters (agreed, then assigned their least experienced engineer and let quality degrade) or active resisters (pushed back through managers, correctly noting their OKRs were about product features, not data products). The root problem: data mesh requires an incentive structure most companies don't have - unless domains are measured and rewarded for data-product quality, they prioritize their primary product, and changing incentives across 300 engineers is a multi-year organizational transformation, not a data-architecture project.
Federated governance became no governance. Without a central authority that could enforce, standards became suggestions - three different definitions of "active user," inconsistent date formats. Federation works only when all parties have equal motivation to maintain standards; in reality governance is only as strong as its least engaged member. Decentralization's hidden costs were large - the Snowflake bill rose 40% in the first six months from domain teams running inefficient queries on separate warehouses with no shared optimization:
| Cost factor | Centralized | Data mesh |
|---|---|---|
| Snowflake compute | One optimized warehouse | 14 domain warehouses, many idle |
| Duplicated transformations | Single owner | 3 teams built their own "active user" metric |
| Cross-domain queries | Simple JOINs | Cross-product contracts, latency |
| Incident response | One team, clear ownership | Finger-pointing between domains |
And cross-domain analytics got harder, not easier - the dirty secret of data mesh. Most valuable analytics join data across domains; the "customer 360" every executive wants combines payments, identity, catalog, support, and marketing. In a centralized model that's a SQL JOIN; in a mesh it's a cross-domain product needing contracts with five teams, schema-alignment meetings, and freshness coordination.
Mesh vs. fabric vs. hybrid
| Dimension | Centralized | Data mesh | Hybrid |
|---|---|---|---|
| Best for | <100 engineers, single product | 1000+ engineers, strong data culture | 100-1000 engineers, mixed maturity |
| Ownership | Central team owns all | Domains own all | Domains own sources, central owns cross-domain |
| Governance | Central authority | Federated | Central standards, domain execution |
| Cross-domain analytics | Easy | Hard | Moderate |
| Org change required | None | Massive | Moderate |
Data fabric - metadata-driven automation and a unified access layer regardless of where data lives - is often framed as the alternative to mesh, but it isn't either/or. Fabric ideas (automated discovery, intelligent integration, unified governance) are complementary and can layer on top of any ownership model. The "mesh vs. fabric" framing is largely a vendor-driven false dichotomy.
Where it landed: a pragmatic hybrid
After two years, the honest description is hybrid, not pure mesh. A platform team of 6 (down from 8) owns the self-serve platform, shared infrastructure, the cross-domain data products (customer 360, revenue metrics), the catalog, governance tooling, and cost management. About 5 domains that genuinely adopted - Payments, Catalog, Search - own their own products, some with a dedicated data engineer. The other ~9 stay platform-managed: Identity (focused on auth, no data interest - platform runs the pipelines, domain reviews PRs), Marketing (no engineering capacity - platform runs it, marketing defines requirements), Support (limited needs - a platform-managed Airbyte sync). The realization: not every domain needs to or should own its data products, and forcing mesh principles on unwilling or unable teams produces worse outcomes than a well-run central model.
What to do differently
- Start with data-product thinking, not org change. You get ~80% of the value - SLAs, documentation, quality checks, versioned schemas - without changing who owns the pipelines. Do all of that before discussing domain ownership.
- Build the platform first, transfer ownership second. Asking domains to own data without a mature self-serve platform is asking them to do data devops from scratch. The platform must be good enough that ownership feels incremental, not like a second job.
- Let domains opt in. The voluntary adopters succeeded; the pressured ones failed. Make the platform attractive and let adoption happen organically.
- Keep cross-domain data centralized. Any product joining multiple domains should be owned by the platform team - the coordination cost is too high for one domain to bear.
- Invest in governance tooling, not meetings. Replace the biweekly council with automated gates in CI/CD - naming linters, PII detection, schema-evolution validation, documentation completeness - so governance is a byproduct of using the platform:
def governance_gate(manifest) -> GateResult:
checks = [
NamingConventionChecker(rules={"table_names": "snake_case", "id_columns": "ends_with _id"}),
PIIDetector(action="require_masking_policy",
patterns=["email", "phone", "ssn", "credit_card"]),
SchemaEvolutionValidator(allow_remove_column=False, require_deprecation_period="14d"),
DocumentationCompleteness(required_fields=["description", "owner", "sla", "quality_checks"]),
]
failed = [r for c in checks for r in [c.validate(manifest)] if not r.passed]
return GateResult(passed=not failed,
blocking=any(r.severity == "critical" for r in failed),
messages=[r.message for r in failed])
The numbers after two years
| Metric | Before (2023) | After (2025) |
|---|---|---|
| Time to deliver a new data product | 3-6 weeks | 1-5 days |
| Data-quality incidents / month | ~12 | ~5 |
| Central data team size | 8 | 6 (platform) |
| Snowflake annual cost | $180k | $220k (+22%) |
| Data products with SLAs | 0 | 34 |
| Domains owning their data | 0 of 14 | 5 of 14 |
| Data request backlog | 200+ | ~30 |
The backlog collapse from 200+ to ~30 is the headline, and it transformed the relationship between the data team and the rest of engineering - but most of that came from the self-serve platform and data-product thinking, not from domain ownership specifically.
The uncomfortable conclusion
Data mesh as its advocates describe it - full domain ownership with federated governance - didn't work here, and probably doesn't work for most companies under 1000 engineers. The organizational change is massive, the hidden costs are real, and the governance model assumes a data maturity most teams don't have. But the ideas inside it are genuinely valuable: data-as-a-product is transformational, self-serve platforms are essential at scale, and data contracts prevent whole categories of incidents - and domain ownership works where the team is willing and capable. The mistake is treating mesh as an all-or-nothing architecture choice. It's a set of principles; adopt the ones that fit and leave the rest. Five of fourteen domains owning their data with a central team handling the rest isn't failure - it's pragmatism. Invest in the platform, adopt data-product thinking everywhere, let ownership emerge organically, and don't feel guilty about keeping a central team for the domains that aren't ready. The conference talks will say that isn't "true" data mesh; the engineers and the Snowflake bill will thank you for ignoring them.




Leave a Comment