What is Apache Spark?

What is Apache Spark?

Apache Spark was created at UC Berkeley's AMPLab in 2009 and open-sourced in 2010. It became an Apache top-level project in 2014. Today it's the most widely-used engine for distributed data processing, deployed at companies like Uber, Netflix, Airbnb, and virtually every large data engineering team.

Spark's key innovation was moving computation into memory. Hadoop MapReduce writes intermediate results to disk after every step — Spark keeps data in RAM across a full pipeline. For multi-step transformations and iterative ML algorithms, this change is the difference between hours and minutes.

The primary interface for modern data engineering is the DataFrame / Spark SQL API, optimized by Spark's Catalyst engine and Tungsten execution layer. Around it sits a library ecosystem — Structured Streaming for stateful stream processing, MLlib for distributed machine learning, GraphX for graph analytics — all sharing the same cluster and unified API.

Why does Spark matter?

• 10-100x faster than MapReduce via in-memory processing and DAG-aware execution
• Scales linearly from a laptop to thousands of nodes processing petabytes
• One API for batch, streaming, SQL, and ML — no separate stacks to maintain
• Catalyst optimizer rewrites your query plan, predicate-pushes, and prunes partitions automatically
• Adaptive Query Execution (AQE) re-plans joins and partition sizes at runtime
• Native readers for Delta Lake, Iceberg, Parquet, ORC, Avro — the lakehouse default

How does Spark work?

Spark uses a Driver–Executor model. The Driver process plans the job, builds a logical query plan, and hands physical tasks to executors. A Cluster Manager (YARN, Kubernetes, or Spark Standalone) allocates resources across nodes. Executors run tasks in parallel on partitioned data. Storage lives outside the cluster — S3, HDFS, Delta Lake, or Iceberg tables.

When you call an action (.write, .show, .count), Spark builds a DAG of stages, splits each stage into tasks (one per partition), and schedules tasks across executors. Failed tasks are retried automatically; lost partitions are recomputed from lineage.

A typical Spark batch job in Python (PySpark):

from pyspark.sql import SparkSession
from pyspark.sql.functions import col, sum

spark = (
    SparkSession.builder
    .appName('daily_etl')
    .config('spark.sql.shuffle.partitions', 200)
    .getOrCreate()
)

# Read raw events from S3
df = spark.read.parquet('s3://bucket/events/date=2026-01-01/')

# Transform: aggregate by user
result = (
    df
    .filter(col('event_type') == 'purchase')
    .groupBy('user_id')
    .agg(sum('amount').alias('total_spend'))
)

# Write to Delta Lake
(
    result.write
    .format('delta')
    .mode('overwrite')
    .save('s3://bucket/gold/user_spend/')
)

Spark vs pandas vs MapReduce

Use pandas when the dataset fits in one machine's RAM — it's faster to write, faster to debug. Reach for Spark when data exceeds single-node memory, when you need cluster parallelism, or when you want one engine for batch, SQL, streaming, and ML. MapReduce is largely legacy — most teams migrated to Spark years ago.

What can you build with Spark?

• Batch ETL at scale — process terabytes of raw logs, events, or files into clean, partitioned tables
• Lakehouse transforms — read from Delta Lake, Iceberg, or Parquet; apply medallion architecture (bronze/silver/gold) at TB+ scale
• Near-real-time streaming — Structured Streaming processes Kafka topics with exactly-once semantics and watermarking for late data
• ML feature engineering — join 10+ tables, compute aggregates, and write feature vectors to a feature store in one job
• Interactive SQL — query petabyte-scale tables with Spark SQL or notebooks (Databricks, Jupyter) at sub-second latency on hot data
• Performance tuning playground — broadcast joins, partition pruning, AQE, bucketing — Spark exposes the full toolbox for 10x speedups

Common mistakes (and what to do instead)

• Calling .collect() on large DataFrames — pulls all rows into the Driver's memory and OOMs it. Use .show(), .limit(), or write to storage instead.
• Python UDFs instead of built-in functions — Python UDFs break Catalyst optimization and run 10-100x slower than equivalent SQL. Use col(), when(), and pyspark.sql.functions wherever possible.
• Wrong shuffle partition count — the default spark.sql.shuffle.partitions=200 is too high for small datasets, too low for large ones. Target ~128MB per partition. Enable AQE to let Spark adjust dynamically.
• Iterating row-by-row with .collect() + a for loop — Spark is not pandas. Use DataFrame transformations, map(), or window functions to keep work on the cluster.
• Not caching DataFrames that are reused — if you reference the same DataFrame twice, Spark recomputes it. Call .cache() or .persist() before the second use.
• Skewed joins on one hot key — one task can stall a whole job. Salt the key, broadcast the smaller side, or enable AQE skew handling.

Who is Spark for?

Spark is built for data engineers and ML engineers working with data that doesn't fit in one machine's RAM. If your daily workload includes lakehouse transforms, terabyte-scale aggregations, or streaming pipelines, Spark is the default engine.

Teams that benefit most:

• Platform teams running medallion lakehouses on Delta Lake or Iceberg at the petabyte scale
• Analytics engineering teams replacing slow warehouse jobs with Spark on cheaper object storage
• ML engineers running feature pipelines that join hundreds of millions of rows before training
• Streaming teams who need exactly-once semantics on Kafka topics with Structured Streaming