10 Data Engineering Spark Interview Questions and Answers

Short answer: The driver builds the plan and coordinates execution, the cluster manager (YARN, Kubernetes, Standalone) allocates resources, and executors on the workers run tasks and hold data in memory. Every action launches a job, which the scheduler cuts into stages at shuffle boundaries, and each stage into tasks by partition count.

In depth:

1. Driver — the process running your code and the SparkSession. It builds the logical and physical plan, the DAG of stages, schedules tasks, and collects results. If the driver dies, the whole application dies.
2. Cluster manager — negotiates resources: how many executors, how many cores and how much memory each. It does no computation itself.
3. Executors — JVM processes on the workers. They run tasks, cache partitions, and serve data during shuffles. They live for the application's lifetime.
4. Work hierarchy — Job (one per action) → Stage (bounded by shuffles) → Task (one per partition, the smallest unit of parallelism).

            ┌──────────────┐
            │    Driver    │  plan + DAG + scheduler
            │ SparkSession │
            └──────┬───────┘
                   │ request resources
            ┌──────▼───────┐
            │Cluster Manager│  YARN / K8s / Standalone
            └──────┬───────┘
        ┌──────────┼──────────┐
   ┌────▼────┐ ┌───▼────┐ ┌───▼────┐
   │ Executor│ │Executor│ │Executor│  tasks + cache
   └─────────┘ └────────┘ └────────┘

⚠️ Common mistake: conflating executor cores with tasks — parallelism is capped at num executors × cores, and if there are fewer partitions than slots, some cores sit idle.

Short answer: A DataFrame is a structured API on top of RDDs, backed by the Catalyst optimizer and the Tungsten engine: they rewrite the plan and operate on a columnar binary representation, so DataFrames are almost always faster and lighter on memory. You drop to RDDs when you need full control over low-level logic or the data is fundamentally unstructured.

In depth:

1. Catalyst — the query optimizer: predicate and projection pushdown, join reordering, constant folding. RDDs get none of it — Spark runs your code as-is.
2. Tungsten — off-heap binary format and code generation; far less GC pressure than the JVM objects an RDD holds.
3. When RDDs are justified — custom partitioning logic, truly raw data, fine-grained low-level control, or legacy code.

⚠️ Common mistake: writing a pipeline on RDDs "for flexibility" when you don't need to — you switch off Catalyst and Tungsten and almost always lose speed for nothing.

Short answer: Transformations (map, filter, join) are lazy — they only append steps to the DAG and compute nothing until an action (count, collect, write) launches a job. Narrow transformations need no data movement between partitions; wide ones require a shuffle and therefore form a stage boundary.

In depth:

1. Laziness — Spark accumulates the plan and optimizes it as a whole through Catalyst before running; redundant steps are pruned and predicates are pushed to the source.
2. Transformations vs actions — a transformation returns a new DataFrame and triggers nothing; an action returns a result to the driver or writes it out and triggers the computation.
3. Narrow — each output partition depends on one input partition: map, filter, union. Pipelined within a single stage, no network.
4. Wide — an output partition depends on many input partitions: groupBy, join, distinct, repartition. They require a shuffle → a new stage.

⚠️ Common mistake: assuming df.filter(...) has already filtered the data — nothing runs before an action, and repeated actions recompute the whole chain from scratch unless you cache it.

Short answer: A shuffle redistributes data across partitions and over the network so that records with the same key land on the same executor. It's expensive because of serialization, map-side disk writes, and network transfer, and it's usually the job's bottleneck. Wide transformations trigger it: groupBy, join, distinct, repartition.

In depth:

1. Why it's expensive — map tasks write key-sorted blocks to local disk (shuffle write) and reduce tasks pull them over the network (shuffle read), plus serialization and GC pressure.
2. What triggers it — any wide transformation: aggregations, joins, distinct, orderBy, an explicit repartition.
3. How to minimize — broadcast-join a small table instead of sort-merge; filter before the join, not after; avoid needless repartition; enable AQE so Spark coalesces small post-shuffle partitions itself.

 Map stage               shuffle            Reduce stage
 [P0: a,b,a] ─┐   (write to disk,     ┌─► [key a: a,a,a]
 [P1: b,c,a] ─┼─► transfer over net) ─┼─► [key b: b,b]
 [P2: c,a,b] ─┘   re-hash by key      └─► [key c: c,c]

⚠️ Common mistake: dropping a repartition before every operation "to even things out" — you add an extra full shuffle; often AQE or a coalesce (no shuffle) is enough.

Short answer: Skew is an uneven distribution of data by key, where one or two partitions get disproportionately many records. It shows up as a single "stuck" task: 199 of a stage's tasks finish while the 200th runs many times longer or dies on memory. Fix it with key salting, a broadcast join, AQE skew join, or deliberate repartitioning.

In depth:

1. How to spot it — in the Spark UI the per-task duration and shuffle-read distribution is heavily uneven; the max task is an order of magnitude slower than the median.
2. Salting — append a random suffix to the hot key to split it across N partitions; on the other join side, explode the key across the same N values.
3. AQE skew join — from Spark 3+, enabling spark.sql.adaptive.skewJoin.enabled splits skewed partitions at runtime automatically.
4. Broadcast — if one side is small, a broadcast join removes the shuffle and the skew entirely.

from pyspark.sql import functions as F

# salt the hot key into N buckets to break the skew
N = 16
facts = df.withColumn("salt", (F.rand() * N).cast("int"))
dims = dim.withColumn("salt", F.explode(F.array(
    *[F.lit(i) for i in range(N)])))

result = facts.join(dims, ["key", "salt"], "inner")

⚠️ Common mistake: growing executor memory to "survive" the skew — that treats the symptom at the cost of resources; salting or AQE fixes the cause, not gigabytes of heap.

Short answer: A partition is the smallest unit of parallelism: one task processes one partition. repartition(n) does a full shuffle and can increase or decrease the partition count with reshuffling; coalesce(n) only decreases it by merging adjacent partitions without a shuffle, so it's cheaper but can produce skew.

In depth:

1. Partition count — a guideline: 128–256 MB per partition and at least as many partitions as total slots (executors × cores), ideally a multiple, so there are no trailing stragglers.
2. repartition — full shuffle, even distribution; use it when you have too few partitions or need to redistribute by key (repartition(col)).
3. coalesce — no shuffle, only merges up; ideal before a write to avoid producing thousands of tiny files.
4. Too many partitions — scheduling overhead and small files; too few — underused parallelism and spill.

⚠️ Common mistake: running repartition(1) before a write to get a single file — you funnel all data onto one executor; use coalesce to merge up, and only build a single file on genuinely small volumes.

Short answer: A sort-merge join shuffles both tables by key, sorts, and merges — the general option for two large datasets. A broadcast (map-side) join ships the small table in full to every executor, and the join runs locally with no shuffle of the large side. Broadcast wins when one table fits in memory (by default up to ~10 MB, the threshold is configurable).

In depth:

1. Sort-merge — the default strategy for large-large: two shuffles, a sort, a merge. Expensive but it scales.
2. Broadcast hash join — the small table is collected on the driver and broadcast; a local hash is built against each partition of the large table — no shuffle of the large side.
3. When broadcast wins — one side is small (a lookup, a dimension); with AQE, Spark often switches to broadcast itself after measuring the size at runtime.

from pyspark.sql import functions as F

# explicitly request a broadcast of the small dimension table
result = big_facts.join(
    F.broadcast(small_dim),
    on="dim_id",
    how="inner",
)
# the large table is not shuffled — the join runs map-side

⚠️ Common mistake: broadcasting a table that doesn't fit in driver/executor memory — it triggers an OOM; broadcast only a genuinely small side, and the threshold is spark.sql.autoBroadcastJoinThreshold.

Short answer: Caching pays off when the same DataFrame is used several times — otherwise, because of laziness, Spark recomputes the whole chain on every action. cache() is persist() with the MEMORY_AND_DISK level; persist(level) lets you pick the level. The cache occupies executor memory, so caching "just in case" is harmful — you evict useful data and provoke spill.

In depth:

1. When it helps — iterative algorithms, reuse of an intermediate result, a plan that branches from one point.
2. Storage levels — a trade-off between memory, CPU, and disk; MEMORY_ONLY is faster, but on shortage the partition is dropped and recomputed.
3. Cost of the mistake — a needless cache eats heap, raising GC and spill; release it with unpersist() once it's no longer needed.

⚠️ Common mistake: caching a DataFrame that's read exactly once — you spend memory and time materializing it and gain nothing; a cache only makes sense with reuse.

Short answer: Start not from guesses but from the Spark UI: find the stage with the longest task, disk spill, and skew in shuffle read. Then size partitions toward 128–256 MB, enable AQE so Spark picks the post-shuffle partition count and switches join strategy itself, and balance executor memory against spill.

In depth:

1. Read the Spark UI — Stages tab: max/median task duration (skew), Spill (Memory/Disk) as a sign of memory shortage, Shuffle Read/Write as network volume.
2. Partition sizing — too large → spill and OOM; too small → overhead; target 128–256 MB.
3. AQE — spark.sql.adaptive.enabled=true: coalescing small partitions, handling skew joins, auto-broadcast by actual size.
4. Memory and spill — if there's heavy Disk Spill, raise memory or shrink partitions; watch the share going to shuffle.

spark.conf.set("spark.sql.adaptive.enabled", "true")
spark.conf.set("spark.sql.adaptive.coalescePartitions.enabled", "true")
spark.conf.set("spark.sql.adaptive.skewJoin.enabled", "true")
# baseline parallelism matched to your slots
spark.conf.set("spark.sql.shuffle.partitions", "400")

⚠️ Common mistake: leaving the default spark.sql.shuffle.partitions=200 on large data — partitions become huge and spill to disk; tune the count to the volume and slots (or let AQE do it).

Short answer: MapReduce processes data in three phases: map turns records into key-value pairs, shuffle groups them by key, and reduce aggregates. Classic Hadoop MapReduce materializes each phase's result to disk (HDFS), whereas Spark keeps intermediate data in memory and builds a DAG of many stages, so it's several times faster on iterative jobs.

In depth:

1. Map — in parallel across partitions, turns input into key-value pairs (e.g. (word, 1)).
2. Shuffle — redistributes pairs so one key lands on one reducer; this is the network and disk phase.
3. Reduce — folds the values of one key (sum, count, concatenation).
4. Why Spark is faster — it doesn't write intermediates to HDFS between steps, it pipelines narrow transformations, caches reused data, and optimizes the whole DAG through Catalyst.

  Input       Map            Shuffle           Reduce
 "a b a" ─► (a,1)(b,1)(a,1) ─► a:[1,1] b:[1] ─► a:2  b:1
 "b c"   ─► (b,1)(c,1)      ─► c:[1]         ─► c:1

⚠️ Common mistake: believing Spark is "always in memory and never touches disk" — it writes to local disk during shuffles and spill too; the win is that there's no mandatory HDFS materialization between stages, not that it avoids disk entirely.