You ship your first lakehouse pipelines, and then reality arrives: nightly jobs crawl, tables bloat with tiny files, schemas shift underneath you, and MERGE starts throwing errors. These are symptoms of healthy growth, not failure - and Delta Lake on Databricks has specific, production-safe fixes for each. Here are the three problems you'll meet first and how to handle them.

Problem	Symptom	Root cause	First aid
Small files	Many 10-200 KB Parquet files, slow scans	Streaming/batch writes with tiny partitions	OPTIMIZE bin-packing, optimized writes, auto-compaction
Schema drift	New/renamed columns; writes fail or silently drop	Upstream changes, CSV/JSON variability	Per-operation schema evolution, rescue column, column mapping
MERGE conflicts	Duplicate/ambiguous matches; concurrent write failures	Bad join keys, races across writers	Pre-dedupe source, narrow ON, idempotent patterns

Small files

Lots of tiny files mean excess metadata and I/O. The fixes, in order of priority:

1. Optimized writes and auto-compaction as table properties keep small files from accumulating as data lands, not just after the fact: ALTER TABLE sales SET TBLPROPERTIES (delta.autoOptimize.optimizeWrite = true, delta.autoOptimize.autoCompact = true);
2. Run OPTIMIZE regularly for bin-packing, targeting recent windows first: OPTIMIZE sales WHERE sale_date >= current_date() - INTERVAL 7 days;
3. Adopt liquid clustering instead of managing partitions or Z-ORDER by hand - it's GA on recent DBR and replaces the old partitioning/Z-ORDER guidance for new tables: ALTER TABLE sales CLUSTER BY AUTO; then OPTIMIZE sales; to recluster.
4. Right-size target file sizes at write time if you control the writer.
5. VACUUM safely to remove old unreferenced files after compaction - keep the default 7-day retention.

Two gotchas: don't Z-ORDER and liquid-cluster the same table (they're mutually exclusive), and don't VACUUM under 7 days - you can break long-running readers or corrupt active transactions.

Schema drift

Evolve schemas per write or MERGE, not via a long-lived session default - explicit and auditable. An append with evolution uses mergeSchema = true as a writer option; a MERGE enables evolution only for that operation:

MERGE INTO silver_events AS t
USING staged_changes AS s
ON t.id = s.id
WHEN MATCHED THEN UPDATE SET *
WHEN NOT MATCHED THEN INSERT *;   -- evolution enabled on this operation only

Databricks recommends enabling evolution on the operation rather than setting a broad Spark conf, though spark.databricks.delta.schema.autoMerge.enabled exists if you genuinely need it. Two more tools help: a rescue column (_rescued_data) captures unexpected fields from semi-structured sources without failing the job (pair it with a weekly review to promote columns), and column mapping lets you rename columns on existing tables without Parquet field-name collisions. The pitfall to avoid is turning on autoMerge everywhere - that produces uncontrolled schema creep, so keep explicit evolution to ingestion jobs only.

MERGE conflicts

Delta uses optimistic concurrency control: if another writer changes files you read, your commit may be rejected. Separately, duplicate matches in the source cause semantic errors. A robust playbook:

1. Pre-dedupe the source to one row per business key before the MERGE - this removes most multi-match ambiguity.
2. Narrow the ON condition to the minimal stable key set; avoid wide, mutable predicates.
3. Make merges idempotent so a rerun on the same input produces the same table state (dedupe by watermark, keys plus sequence numbers, or CDC last-writer-wins).
4. Handle deletes explicitly with WHEN NOT MATCHED BY SOURCE when applying CDC.
5. If conflicts persist, consider deletion vectors to avoid full file rewrites for row-level deletes/updates under high churn, then REORG … APPLY (PURGE) when needed.

A deduped CDC-style MERGE pulls it together:

WITH latest AS (
  SELECT s.*, ROW_NUMBER() OVER (PARTITION BY id ORDER BY ts DESC) rn
  FROM cdc_stage s
)
MERGE INTO silver.events t
USING (SELECT * FROM latest WHERE rn = 1) s
ON t.id = s.id
WHEN MATCHED AND s.op = 'U' THEN UPDATE SET *
WHEN NOT MATCHED AND s.op = 'I' THEN INSERT *
WHEN NOT MATCHED BY SOURCE AND s.op = 'D' THEN DELETE;

Daily operations checklist

• Layout: ALTER TABLE … CLUSTER BY AUTO to let Databricks choose keys; schedule OPTIMIZE on hot ranges and verify the file-size distribution.
• Ingest: enable schema evolution per write; review _rescued_data promotions weekly.
• CDC and upserts: pre-dedupe and watermark; add WHEN NOT MATCHED BY SOURCE for deletes.
• Maintenance: keep VACUUM at 7 days unless you fully model reader behavior; use predictive optimization where available.
• Governance: track the table feature protocol and drop features only when necessary, since it can truncate history.

Common failures and fast fixes

• NON_LAST_NOT_MATCHED_CLAUSE_OMIT_CONDITION: a non-final WHEN NOT MATCHED clause lacks a condition - add it or reorder the clauses.
• "Concurrent modification" errors: retry with backoff, reduce the data touched (smaller batches), split hot keys, and ensure idempotency.
• Time-travel queries failing after cleanup: retention was too aggressive - align delta.deletedFileRetentionDuration and logRetentionDuration with your SLAs.

Reference snippets

Create a new table with automatic clustering and optimized writes:

CREATE OR REPLACE TABLE bronze.clicks (id BIGINT, ts TIMESTAMP, user_id STRING, country STRING)
CLUSTER BY AUTO;
ALTER TABLE bronze.clicks SET TBLPROPERTIES (
  delta.autoOptimize.optimizeWrite = true,
  delta.autoOptimize.autoCompact  = true);

Weekly compaction of recent data, and a safe VACUUM at the 7-day default:

OPTIMIZE bronze.clicks WHERE ts >= current_date() - INTERVAL 14 days;
VACUUM  bronze.clicks RETAIN 168 HOURS;   -- 7 days; keep unless you fully understand the risks

The recap

Small files, schema drift, and MERGE conflicts respond to three habits: preventive layout (liquid clustering plus optimized writes), explicit per-operation schema evolution rather than global autoMerge, and deterministic, pre-deduped, idempotent merges. Start by enabling optimized writes, scheduling OPTIMIZE on hot data, and converting merges to the deduped pattern above - then review retention and table features so maintenance doesn't fight your SLAs.