Apache Spark – Data/ML Engineer Blog

Apache Spark for Data Engineers: From “It Works Locally” to Production-Grade at Scale

Why this matters (intro)

You’ve got terabytes landing hourly, dashboards choking, and a data science team asking for features “yesterday.” Apache Spark is the engine that turns that mess into minutes. But most teams run Spark like a laptop script on a 100-node cluster—and then blame “infra.” Let’s fix that.

What Spark is (and isn’t)

Spark is a distributed compute engine for batch and streaming that runs on top of cluster managers (YARN, Kubernetes, Standalone) and storage layers (S3, ADLS, HDFS). It’s not a database. It doesn’t magically optimize your schema. It executes your transformations in parallel, and the moment you push bad joins, skewed keys, or chatty UDFs—you pay with shuffles and retries.

Core pieces in one breath

Catalyst: query optimizer that rewrites your logical plan into an efficient physical plan.
Tungsten: memory and codegen layer that keeps execution tight.
RDD → DataFrame/Dataset: use the DataFrame API unless you truly need low-level control.
Structured Streaming: micro-batch or continuous processing with exactly-once sinks (when designed correctly).
AQE (Adaptive Query Execution): adjusts join/shuffle strategies at runtime based on real stats.

Architecture at a glance

Driver: builds the DAG, optimizes it, and coordinates tasks. Guard it—OOM here kills the job.
Executors: run tasks, hold caches, spill to disk under pressure.
Shuffle: the tax collector of distributed computing—paid whenever you repartition, groupBy, or wide-join.
Storage: object stores (S3/ADLS), HDFS, JDBC sources, and NoSQL systems (MongoDB, Cassandra, Elasticsearch, Redis) via connectors.
Metastores & Lakehouse: Hive Metastore or Unity/Glue Catalog; Delta Lake/Iceberg/Hudi bring ACID and time travel.

When to choose Spark (quick table)

Scenario	Good Fit for Spark?	Why
Large-scale ETL/ELT over object storage	✅	Column pruning, predicate pushdown, parallel IO
ML feature engineering at scale	✅	Wide ecosystem, pandas API on Spark, UDF/UDAF options
Incremental upserts to a data lake	✅	Delta/Iceberg/Hudi with MERGE/OPTIMIZE/VACUUM
Low-latency sub-second stream processing	⚠️	Possible, but consider Flink for hard real-time
Ad-hoc BI/SQL with lots of small queries	⚠️	Presto/Trino can be a better fit
OLTP/serving	❌	Use databases/NoSQL stores instead

Real example: Batch + Streaming on a Lakehouse (PySpark)

1) Batch: build a dimensional table with sane partitions

from pyspark.sql import functions as F

raw = (spark.read
       .format("delta")
       .load("s3://lake/raw/orders/")
       .withColumn("order_date", F.to_date("created_at"))
       .withColumn("order_ym", F.date_format("order_date", "yyyy-MM")))

# Handle duplicates and late arrivals
dedup = (raw
         .withColumn("rn", F.row_number().over(
             Window.partitionBy("order_id").orderBy(F.col("ingest_ts").desc())))
         .where("rn = 1")
         .drop("rn"))

(dedup
 .repartition("order_ym")                 # keep shuffle bounded
 .write
 .format("delta")
 .mode("overwrite")
 .partitionBy("order_ym")                  # manageable number of partitions
 .option("overwriteSchema", "true")
 .save("s3://lake/curated/fct_orders/"))

2) Streaming: exactly-once to Delta with watermarks

events = (spark.readStream
          .format("kafka")
          .option("kafka.bootstrap.servers", "broker:9092")
          .option("subscribe", "payments")
          .load())

parsed = (events
          .selectExpr("CAST(value AS STRING) as json")
          .select(F.from_json("json", payment_schema).alias("p"))
          .select("p.*")
          .withWatermark("event_time", "20 minutes"))

agg = (parsed
       .groupBy(F.window("event_time", "5 minutes"), "merchant_id")
       .agg(F.sum("amount").alias("gross_amount")))

(agg.writeStream
    .format("delta")
    .outputMode("update")
    .option("checkpointLocation", "s3://lake/_chk/payments/")
    .start("s3://lake/curated/payments_rollups/"))

3) Fast joins without pain: broadcast wisely

dim = spark.read.table("curated.dim_merchants")
big = spark.read.table("curated.fct_orders")

# If dim merch is small (< ~10–100 MB), broadcast avoids shuffle
joined = big.join(F.broadcast(dim), "merchant_id", "left")

Integrating NoSQL (Spark as the heavy-lift engine)

MongoDB: use the Spark Connector. Push filters, avoid exploding documents, consider $project to trim.
Cassandra: model queries first; narrow partitions; use spark.cassandra.input.split.size_in_mb judiciously.
Elasticsearch/OpenSearch: selective fields, es.scroll.size, and align partitioning to index shards.
Redis: not a primary sink; great as a cache/lookup during jobs via Redis connector or custom UDFs.

Performance playbook (brutally honest)

1) Kill small files
Object stores + partitions easily create millions of <128MB files. Use compaction (e.g., Delta OPTIMIZE) and write with target file sizes (128–512MB). Small files destroy listing time and query planning.

2) Control partitions

repartition(n) for global reshuffle; coalesce(n) to shrink without full shuffle.
Partition on columns with bounded cardinality (e.g., yyyy-MM, not user_id).
Avoid >10k partitions per table; you will hate your life at maintenance time.

3) Join strategy

Prefer broadcast-hash join for small dims (and set spark.sql.autoBroadcastJoinThreshold).
Skew? Use salting or skew join hints; AQE can auto-switch to shuffle hash or sort-merge.

4) Caching
Cache only if reused multiple times in the same job and the dataset fits memory. Otherwise you create spills and make it slower.

5) UDFs last
Native functions first. If you must use UDFs, consider pandas UDFs (vectorized) and keep them pure and fast.

6) Shuffle is the bill
Every wide dependency (groupBy, join on non-partition keys) writes to disk and across network. Design pipelines around minimizing shuffles.

7) Cluster sizing

Scale executors first, then cores; leave headroom for GC.
Use dynamic allocation and autoscaling; lock minimums for steady streams.
Watch task time vs. GC time. If GC >10–15%, you’re memory-bound or leaking.

8) Schema discipline
Define schemas explicitly. inferSchema=True in production is amateur hour. Enforce with expectations/tests.

Common pitfalls (seen in the wild)

Writing partitioned tables by high-card columns (e.g., customer_id).
Millions of tiny files due to per-batch streaming writes without compaction.
Overusing caches, then wondering why the next stage spills.
Driver OOM from collecting large DataFrames (.collect() is not a debugging tool).
Pushing ML training into a single jumbo executor instead of distributed strategies.
Treating Delta/Iceberg like a SQL warehouse and skipping OPTIMIZE/VACUUM hygiene.

Settings that actually move the needle (starters)

spark.conf.set("spark.sql.adaptive.enabled", "true")                 # AQE on
spark.conf.set("spark.sql.shuffle.partitions", "auto")               # or tuned number
spark.conf.set("spark.sql.files.maxPartitionBytes", 134217728)       # ~128MB
spark.conf.set("spark.sql.broadcastTimeout", 600)                    # slow networks
spark.conf.set("spark.databricks.io.cache.enabled", "true")          # if on Databricks

(Adjust per platform; value sanity > cargo-culting.)

Testing & reliability

Contract tests for schemas and invariants (row count ranges, nullability).
Deterministic checkpoints and idempotent sinks for streaming.
Data Quality: Deequ/Great Expectations to fail fast.
Observability: capture job lineage, shuffle reads/writes, and stage retries; alert on regressions.
Backfills: design partition-aware replays; do not reprocess the universe for one bug.

Cheatsheet: Spark vs Flink vs Trino (one-liners)

Engine	Superpower	Where Spark loses
Spark	Unified batch + streaming + ML	Lowest-latency streaming
Flink	True streaming, event-time wizardry	Ad-hoc SQL over lakes
Trino	Blazing interactive SQL federation	Complex stateful transforms

Internal link ideas

“Choosing Partition Keys for Data Lakes”
“Delta Lake Maintenance: Optimize, Z-Order, Vacuum—When and Why”
“Skewed Joins: Detection and Salting Patterns”
“Great Expectations in CI for Data Pipelines”
“Streaming Checkpointing and Exactly-Once Semantics Explained”

Conclusion & takeaways

Spark is a power tool, not magic. Treat shuffles like a tax to be minimized, design partitions with discipline, and let AQE help—but don’t abdicate thinking. If you enforce schemas, compact files, pick sane join strategies, and test your invariants, Spark becomes boringly fast. That’s the goal.

Do next:

Turn on AQE and inspect the physical plan.
Compact your worst small-file table.
Replace one UDF with native functions.
Add a contract test to fail a bad schema before it ships.

Image prompt

“A clean, modern architecture diagram of an Apache Spark lakehouse pipeline: sources (Kafka, S3), Spark driver/executors, shuffle stages, Delta Lake tables (bronze/silver/gold), with compaction and AQE annotations — minimalistic, high-contrast, 3D isometric style.”

Data/ML Engineer Blog