Error handling in data pipelines is the difference between a job that recovers on its own at 3 a.m. and one that pages an engineer. Failures are not exceptional - networks blip, source APIs throttle, a single row arrives malformed. A pipeline that treats every failure as fatal is brittle; one that distinguishes the kinds of failure and responds appropriately keeps running. The first decision is always: is this error transient or permanent?

Transient vs. permanent errors

A transient error fixes itself on retry - a timeout, a 503, a deadlock, a rate limit. A permanent error will fail identically no matter how many times you retry - a malformed JSON payload, a schema violation, a row that breaks a not-null constraint. Retrying a permanent error wastes time and can mask the real problem. The handling rule follows directly: retry transient errors, route permanent ones aside for inspection.

Idempotency, retries, and dead-letter queues

Idempotency is the foundation that makes everything else safe. An operation is idempotent if running it twice produces the same result as running it once - so a retry, or a replayed CDC event, can't double-count. The usual implementations are a MERGE keyed on a natural key, or deleting a partition before reloading it.

MERGE INTO orders t
USING staging_orders s ON t.order_id = s.order_id
WHEN MATCHED THEN UPDATE SET ...
WHEN NOT MATCHED THEN INSERT ...;

Retries with backoff handle transient errors without hammering a struggling dependency - wait 1s, 2s, 4s, 8s, with jitter, up to a cap. After the cap, a poison message (a record that fails every time) must not block the queue forever; route it to a dead-letter queue (DLQ) - a side topic or table holding failed records with the error and a timestamp - so the rest of the stream proceeds. DLQ records are triaged the same way quarantined rows are in data quality management.

Checkpointing and partial-failure recovery

Checkpointing records how far a job got so a restart resumes from there instead of from the beginning - Spark Structured Streaming writes offsets to a checkpoint directory; a batch job records the last processed partition. Combined with idempotency, this gives clean partial-failure recovery: if a run dies after loading 7 of 10 partitions, the rerun skips or safely re-loads those 7 and finishes the last 3, with no duplicates. Without idempotency, resuming risks double-loading; without checkpointing, the only safe option is reprocessing everything.

Frequently asked questions

Why is idempotency the first thing to get right? Because retries, replays, and resumes all re-run work. If the operation isn't idempotent, every recovery mechanism risks duplicating data.

What goes in a dead-letter queue? Records that keep failing - poison messages and permanent errors - together with the error and timestamp, so the main stream isn't blocked and the failures can be inspected and reprocessed later.

How should retry backoff be tuned? Exponential delays (1s, 2s, 4s...) with jitter to avoid thundering herds, a cap on attempts, and a DLQ for what exceeds the cap. Retry only transient errors.

What does checkpointing recover? The job's progress position, so a restart resumes from the last committed point rather than reprocessing from the start - only safe to skip re-loads when the writes are idempotent.