What Actually Happens When You Run a Spark Job? – Unite.AI

Introduction

As datasets grow, processing them efficiently on a single machine can become challenging due to limitations in CPU capacity, memory, and storage throughput.

Apache Spark addresses these challenges by providing a distributed data processing engine that can divide workloads into partitions and execute tasks in parallel across CPU cores and multiple machines.

However, Spark is not simply about adding more machines.

Understanding how its components work together—Drivers, Executors, RDDs, DataFrames, Partitions, Jobs, Stages, and Tasks—is essential to designing efficient and reliable data pipelines.

In this article, I’ll walk through Spark’s architecture, execution mechanics, and key performance considerations using a practical e-commerce sales aggregation example.

1. Why Apache Spark – Parallel Data Processing

Apache Spark is a distributed computing engine designed for large-scale data processing and data engineering.

Spark processes big data leveraging parallel data processing:

Compared with a single node processing (right, Figure A), Spark divides work into tasks using multiple nodes, executing them across resources, when available, across machines.

Here, distributed does not mean multiple machines are mandatory. Spark can run in local mode on a single machine. Even there, data can be partitioned and processed in parallel.

2. Core Apache Spark Architecture

Spark consists of several key components:

• Spark driver (driver): The central coordination process of Spark application.
• Cluster manager: An external service responsible for allocating hardware resources like CPU cores and memory.
• Worker nodes (worker): A physical / virtual machine or compute node in the cluster.
• Executors: Distributed worker processes launched on worker nodes.
• Java Virtual Machine (JVM): The runtime environment required to execute Spark’s bytecode (since Spark is written in Scala/Java).

First, the driver coordinates a Spark application and transform it into a directed acyclic graph (DAG) of physical execution tasks, and schedules them across the executors.

Then, cluster manager allocates resources under the deployment’s resource-management system.

Worker nodes supply compute resources to executors, which execute tasks and hold data during the task execution.

Some managed environments like Databricks use one executor per worker, but this is not a universal Spark rule. A worker can host multiple executors depending on cluster configuration.

2-1. E-commerce Use Case – Input Data Structure

Let us imagine a boutique e-commerce company.

It has 10,000 sales records, each with a stock keeping unit (SKU) and quantity sold. Assume three possible SKUs: A, B, and C.

Table 1. E-Commerce sales record breakdown & expected aggregations example.

Our goal is to calculate total valid units sold per SKU:

1. Filter out records whose qty , then
2. Aggregate samples by SKU.

In the illustrative full dataset, suppose 9,800 of the 10,000 records remain after filtering. The final totals expected from Spark are:

[("A", 8000), ("B", 10500), ("C", 9200)]

2-2. Spark Cluster

I’ll define a simplified cluster for our use case.

Worker Nodes (Workers)

I’ll set 3 worker nodes, each of which has an instance type of 8 vCPU with 32 GiB RAM:

Figure B. Resource allocation across Worker Nodes (Created by Kuriko IWAI)

In total, 24 vCPU are available in the system.

Spark Executors and CPU/Memory Allocation

Let’s say, I’ll secure:

• 7 executors.
• 2 vCPU cores for each executor.
• 3GiB JVM heap memory for each executor.

Spark application requests the cluster manager to allocate resources to these 7 executors.

Then, the cluster manager decides CPUs and memory allocations based on the resource usage of each worker.

For instance, it decides to create 14 tasks slots – 2 vCPU core per executor x 7 executors:

Figure C-1. vCPU cores, memory, and Executor allocation across Workers with resource imbalance in Worker A

Note that the 3 GiB memory allocated to Executor is not its total memory because it has to secure memory overhead for Workers and other executions.

Assuming that a single task requires 1 vCPU (spark.task.cpus = 1), each worker allocates its vCPU:

Figure C-2. Parallel vCPU allocation across Workers

This shows two key points:

1. Each worker has its unique CPU usage rate (In Figure C-2, Worker A’s usage is 75%, while Workers B and C remain 50%).
2. Even though the system has 24 vCPU cores (8 vCPU/worker x 3 Workers), it can only use 14 vCPU cores; the rest 10 remain unused for these tasks.

2-3. Java Virtual Machine (JVM) Behind Spark

Apache Spark is built on the Java Virtual Machine (JVM).

JVM is a runtime environment required to execute Spark’s bytecode.

Since Spark is written in Scala/Java:

• When using Scala or Java in application: The code runs natively inside the JVM.
• When using PySpark (Python): A Python process runs the code, but it communicates with a JVM-based driver via a bridge such as Py4J. The Spark executors on the worker nodes still run inside a JVM, which then manages Python workers to execute Python code and Python User-Defined Functions (Python UDFs).

In either case, the JVM manages execution threads, memory allocation, and garbage collection for both the driver and executors.

Driver & Executor JVMs in PySpark Application

In a PySpark application, the Python driver communicates with the Spark driver running in the JVM;

• Python driver: Where application-level PySpark code is written and executed.
• Driver JVM: Coordinates Spark execution and scheduling.
• Executor JVM: Runs Spark tasks and manages execution resources. Python worker processes are involved when tasks require Python execution.
• Client JVM (client process or submission machine): JVM running on the local machine or an edge/gateway node.

As shown in the figure, the client JVM first connects to the driver JVM, which subsequently coordinates with the executor JVMs:

Figure D. Overall relation of client, driver, and executor JVMs

From here, the driver JVM hosts the Spark Driver to manage the application, while the executor JVMs host the individual executors responsible for executing those tasks.

Each JVM driver has JVM heap to handle data processing, caching, and user objects efficiently.

2-4. Resilient Distributed Datasets (RDDs) Deep Dive

A Resilient Distributed Dataset (RDD) is an immutable, partitioned collection of elements that Spark can process in parallel.

RDDs track dependencies between transformations, allowing Spark to recompute lost partitions in many failure scenarios.

The name captures three properties:

• Resilient: Spark records lineage—which RDD depends on which previous RDD—and can often reconstruct missing partitions instead of maintaining replicas of every intermediate result.
• Distributed: The dataset is represented as partitions that Spark may process on different executors. Multiple physical machines are not required; local mode also supports RDDs.
• Dataset: An RDD can contain key-value pairs, integers, strings, tuples, and other supported elements—not only relational rows. RDD APIs include operations such as map, filter, reduceByKey, join, and aggregateByKey.

An RDD is a logical collection and computation abstraction, not a declaration that particular partitions are permanently assigned to particular workers.

Partitions

A partition is a logical chunk of data and normally corresponds to one task within a stage.

Let’s say we create 100 partitions (These 100 initial partitions are an explicit teaching choice).

With 10,000 source elements, it gives approximately 100 elements per partition if the input is distributed roughly evenly:

Figure E. Parallel partition allocation across 100 initial data splits

3. RDD API vs DataFrame API: Internal Catalyst Optimization

The original RDD example helps understand lineage, partitions, and shuffle.

In typical structured ETL, however, the DataFrame API is the preferred starting point because it provides columns, schema information, SQL-like expressions, and optimizer-visible operations.

Below diagram illustrates the relationship:

Figure F. How Spark’s Graph concepts relate; RDD linage, DataFrame, and Spark Execution DAG

Just like RDD has RDD lineage to define the transformation, DataFrame query plan represents its structured query.

Spark constructs execution work from dependencies and physical operators.

DataFrame execution may use RDD-based mechanisms internally, but that does not mean users need to manually build RDD lineage when writing DataFrame code.

3-1. DataFrame Query Plan

A DataFrame query plan is the sequence of logical and physical steps an execution engine generates (purple box in Figure L).

A DataFrame query moves through several representations:

1. Logical plan: A description of the requested operations (scan, filter, group, aggregate).
2. Analyzed and optimized logical plan: Spark resolves references and applies applicable logical optimizations, such as column pruning or predicate pushdown.
3. Physical plan: Spark chooses executable operators, e.g. scans, partial/final aggregates, exchanges, joins, and sorts.
4. Execution: The physical plan is run using Spark’s execution mechanisms, ultimately involving tasks on executors.

In our use case, the physical plan can include the following steps:

• Partial hash aggregation,
• Exchange by SKU, and
• Final hash aggregation.

3-2. Comparing Actions: RDD API vs DataFrame API

Let us compare RDD API and DataFrame API, taking the SKU aggregation for an example.

RDD API

The elements are (sku, qty) pairs. Our Python aggregation logic is explicit, and Spark follows RDD dependencies:

result_rdd = (
    raw_rdd
    .filter(lambda row: row[1] > 0)
    .reduceByKey(lambda a, b: a + b, numPartitions=4)
)

rdd_results = result_rdd.collect()

# Example:
# [('A', 8000), ('B', 10500), ('C', 9200)]

DataFrame API

Assume sales_df has columns sku (string) and qty (numeric):

from pyspark.sql import functions as F

result_df = (
    sales_df
    .filter(F.col("qty") > 0)
    .groupBy("sku")
    .agg(F.sum("qty").alias("total_qty"))
)

df_results = result_df.collect()

# Example:
# [Row(sku='A', total_qty=8000),
#  Row(sku='B', total_qty=10500),
#  Row(sku='C', total_qty=9200)]

Here, both examples solve the same business problem:

Filtering invalid sales records and aggregating valid quantities by SKU.

And in both cases, collect() triggers Spark execution and returns all resulting records to the Driver.

The key difference here is in the result representation:

• RDD API: Returns Python elements, such as (sku, total_qty) tuples.
• DataFrame API: Returns structured Row objects with named fields, such as sku and total_qty.

Here’s an overall comparison of RDD and DataFrame APIs:

Table 2. RDD API vs. DataFrame API – feature & abstraction comparison:(Created by Kuriko IWAI)

4. Wrapping Up

The goal of Spark tuning is not to maximize partition count or CPU usage independently.

It is to reduce end-to-end execution cost and latency without introducing instability.

In this article, we learn the core relationships in its system:

• Resources: Workers host executors; executors run tasks; configured CPU and memory limit concurrency.
• RDD programming: Transformations produce immutable RDD definitions and lineage; actions trigger computation.
• DataFrame programming: Transformations produce logical query operations; optimization chooses executable physical operators; actions trigger computation.
• Execution: Spark organizes work into jobs, stages, and tasks, and tasks process partitions on executors. Shuffles and other dependencies create coordination boundaries among stages.

Understanding how Spark executes distributed workloads is only the foundation.

You can learn a seven-step computation flow using RDD API.