Introduction To Spark

 spark is fast general-purpose distributed data processing engine compatible with Hadoop data. On top of the Spark core data processing engine, there are libraries for SQL, machine learning, graph computation, and stream processing, which can be used together in an application.

It utilizes in-memory caching, and optimized query execution for fast analytic queries against data of any size.

that why it is widely used for big data workloads.

Spark is written in Scala but provides rich APIs for  Scala, Java,  Python, and R. Apache Spark is a tool for Running Spark Applications. Spark is 100 times faster than Bigdata Hadoop and 10 times faster than accessing data from disk. It can be integrated with Hadoop and can process existing Hadoop HDFS data.

 Resilient Distributed Dataset – RDD: 

 RDD is an acronym for Resilient Distributed Dataset. It is the fundamental unit of data in Spark. Basically, it is a distributed collection of elements across cluster nodes. Also performs parallel operations. Moreover, Spark RDDs are immutable in nature. Although, it can generate new RDD by transforming existing Spark RDD

 Lazy Evaluation: 

Spark Lazy Evaluation means the data inside RDDs are not evaluated on the go. Basically, only after an action triggers all the  computation is performed. Therefore, it limits how much work it has to do.

Apache Spark is an open-source, distributed data processing framework designed for processing large-scale, fault-tolerant, and high-performance data workloads.

It is widely used in modern data platforms for:

• Batch Processing – large volumes of historical data
  → Using Spark Core and Spark SQL
• Stream Processing – real-time data streams
  → Using Structured Streaming (built on Spark SQL)
• ETL (Extract, Transform, Load) – data preparation and transformation
  → Using Spark SQL, DataFrame API, and Datasets
• Machine Learning – scalable ML pipelines
  → Using MLlib
• Analytics – interactive SQL and reporting
  → Using Spark SQL
• Graph Processing – relationship and network analysis
  → Using GraphX

What are the main components of Spark?

Answer:
The core components of Apache Spark are:

1.      Spark Core – basic functionality and task scheduling

2.      Spark SQL – structured data processing

3.      Spark Streaming – real-time data processing

4.      MLlib – machine learning library

5.      GraphX – graph processing

 

1. Batch Processing

Batch processing collects data over a period of time and processes it all at once as a single "batch." This usually happens on a schedule (e.g., every night at 2 AM).

·         How it works: Data is collected, stored, and then processed in bulk.

·         Latency: High (minutes to days).

·         Best for: Massive volumes of data where immediate results aren't needed.

·         Example: Calculating monthly payroll, generating daily sales reports, or clearing bank transactions at the end of the day.

·         Tools: Apache Spark (RDD/DataFrames), AWS Glue, Snowflake, dbt, Airflow.

2. Streaming (Near Real-Time)

Streaming processes data records one by one as they arrive. It’s designed for continuous flow.

·         How it works: Instead of waiting for a "bucket" to fill up, the system processes each "drop" of data as it falls.

·         Latency: Low (seconds to milliseconds).

·         Best for: Monitoring systems where a delay of a few seconds is acceptable.

·         Example: Tracking GPS locations of delivery drivers, updating a live "trending topics" list on social media, or monitoring website traffic logs.

·         Tools: Apache Kafka, Spark Streaming, Amazon Kinesis, Google Pub/Sub.

 

Data skew:

Data skew occurs when one partitions has significantly more data ,leading to performance bottlenecks during shuffle operations like joins and aggregations

Where Data Skew Commonly Happens

  groupByKey ()
  reduceByKey()
  join()
  repartition()
 Aggregations
 Window functions

Basically → Any shuffle operation.

Types of Data Skew

1  Key Skew

One key dominates dataset.

2 Partition Skew

Uneven partition size.

3 Join Skew

One key in join causes explosion.

 

How to Fix Data Skew:

   AQE → Enable spark.sql.adaptive.skewJoin.enabled (auto splits skewed partitions in Spark 3+).

  Broadcast Join → broadcast(small_df) to avoid shuffle.

  Salting → Add random suffix to heavy join key to spread data.

  Isolate Skew → Process heavy/NULL keys separately, then union.

  Repartition → df.repartition(n, "col") to balance partitions.

 

 

Typical Architecture:

1.     Data Ingestion

o    Kafka / API / Files

2.     Processing

o    Spark SQL (batch)

o    Structured Streaming (real-time)

3.     Storage

o    Data Lake (Parquet, Delta)

o    Data Warehouse

4.     Orchestration

o    Airflow

 

Why Spark is Important for Data Engineers

✅ Distributed & Scalable
✅ In-memory processing (faster than Hadoop MapReduce)
✅ Unified engine (Batch + Streaming + ML)
✅ Strong ecosystem support
✅ Cloud-native compatibility

Partitions

A Partition is a logical chunk of your data.

DE Perspective: This is the most critical component for a DE. You use repartition() or coalesce() to control how data is spread across the cluster. Correct partitioning prevents Data Skew.

 

Speculative Execution (The Backup Plan)

In a large cluster, sometimes a single machine (Worker Node) becomes slow because of hardware issues, network congestion, or a noisy neighbor. This is called a Straggler.

How it works: If Spark detects that a Task is taking much longer than the average time of other tasks in the same stage, it launches a "speculative copy" of that same task on a different executor.

The Race: Spark now has two versions of the same task running. Whichever one finishes first, Spark accepts that result and kills the other one.

Settings: You turn this on using spark.speculation = true.

What  is check pointing

Check  pointing stores intermediate results to reliable storage like:

.HDFS

.Amazon S3

Used in

Streaming

Fault tolerance 

 

Spark Core (Foundation / Execution Engine)

Spark Core is the base engine responsible for:

·         Distributed task scheduling

·         Memory management

·         Fault tolerance

·         I/O operations

·         Interaction with storage systems (HDFS, S3, ADLS)

Key Internal Components:

Driver

·         Entry point of the application

·         Builds the execution plan (DAG)

·         Sends tasks to executors

Executors

·         Run tasks in parallel

·         Store cached/shuffled data

·         Return results to driver

Cluster Manager

·         Allocates CPU & memory resources

·         Examples: YARN, Kubernetes, Standalone

Data Engineering Importance:

·         Enables parallel ETL processing

·         Handles large-scale batch workloads

·         Ensures fault tolerance using lineage

---

 

Spark SQL (Structured Data Processing Layer)

Spark SQL processes structured and semi-structured data using:

·         SQL queries

·         DataFrame API

·         Dataset API

Key Features:

·         Catalyst Optimizer (query optimization)

·         Tungsten Engine (memory & CPU optimization)

·         Hive integration

Data Engineering Use Cases:

·         Data cleaning and transformation

·         Schema enforcement

·         Working with JSON, Parquet, ORC, Avro

·         Building curated datasets for analytics

Example:
Raw logs → Transform → Store as Parquet in Data Lake

---

3  Structured Streaming (Real-Time Processing)

Spark’s real-time processing engine built on Spark SQL.

Features:

·         Micro-batch processing

·         Event-time processing

·         Watermarking (handles late data)

·         Exactly-once guarantees

 

Data Engineering Use Cases:

·         Kafka stream processing

·         Real-time dashboards

·         Fraud detection

·         IoT event processing

Example pipeline:
Kafka → Spark Structured Streaming → Delta Lake

---

4  MLlib (Machine Learning Library)

Distributed ML library inside Spark.

From Data Engineering View:

·         Build feature engineering pipelines

·         Prepare large-scale training datasets

·         Support ML teams with scalable preprocessing

Typically, data engineers focus more on data preparation than model building.

---

5  GraphX (Graph Processing Engine)

Used for graph-based computations.

Use Cases:

·         Social network analysis

·         Recommendation engines

·         Fraud network detection

Not common in standard ETL but useful in advanced analytics.

---

6  Deployment & Cluster Management

Spark can run on:

·         Hadoop YARN

·         Kubernetes

·         Standalone clusters

·         Cloud platforms (EMR, Databricks)

Why Important for Data Engineers:

·         Resource tuning (memory, partitions, cores)

·         Cost optimization

·         Scalability planning

·         Production deployment management

Q: What is the difference between a Transformation and an Action?

·         Transformations: Functions that create a new RDD/DataFrame from an existing one (e.g., map, filter, join). They are Lazy—Spark only records them in a DAG.

·         Actions: Operations that trigger computation and return results to the driver or write to storage (e.g., count, collect, write).

Q: What is a DAG (Directed Acyclic Graph)?

It is a logical representation of the sequence of transformations applied to the data. It allows Spark to optimize the execution plan (e.g., pipelining transformations) before actually running the tasks.

 

2. Data Abstractions: RDD vs. DataFrame vs. Dataset

Q: Why choose DataFrames over RDDs?

• Optimization: DataFrames use the Catalyst Optimizer and Tungsten execution engine, which optimize query plans and memory usage. RDDs have no built-in optimization.
• Structure: DataFrames provide a schema-based view (columns/rows), making them easier to use for SQL-like operations.

Q: When would you still use an RDD?

• When you need low-level control over data distribution or physical placement.
• When dealing with unstructured data that doesn't fit a schema.
• When performing complex functional programming that isn't easily expressed in SQL/DSL.

---

3. Performance Tuning & Optimization (Critical for Data Engineers)

Q: How do you handle Data Skew? Data skew occurs when one partition has significantly more data than others, causing one task to run for hours while others finish in seconds.

• Salting: Add a random "salt" key to the join key to redistribute the data.
• Broadcast Join: If one table is small, broadcast it to all nodes to avoid shuffles.
• Iterative Filtering: Filter out the skewed keys and process them separately.

 

Q: What is the difference between repartition() and coalesce()?

• repartition(): Can increase or decrease partitions. It triggers a full shuffle, ensuring data is distributed uniformly.
• coalesce(): Only used to decrease partitions. It avoids a full shuffle by merging existing partitions, making it much more efficient for reducing file counts after a filter.

Q: Explain the Catalyst Optimizer. It is the engine behind Spark SQL/DataFrames. It works in four phases:

1.     Analysis: Resolving table/column names.

2.     Logical Optimization: Applying rules like Predicate Pushdown (filtering data at the source) and Projection Pruning (selecting only needed columns).

3.     Physical Planning: Generating multiple physical plans and picking the one with the lowest cost.

4.     Code Generation: Generating Java bytecode for execution.

---

4. Storage & Memory Management

Q: Cache vs. Persist?

• cache(): Shortcut for persist(StorageLevel.MEMORY_ONLY).
• persist(): Allows you to specify the storage level (e.g., MEMORY_AND_DISK, DISK_ONLY, or MEMORY_ONLY_SER to save space via serialization).

 

Q5 : What are Broadcast Variables and Accumulators?

• Broadcast Variables: Read-only variables cached on every machine rather than being sent with every task. Use this for small lookup tables in joins.
• Accumulators: Write-only variables used for counters or sums across the cluster (e.g., counting corrupted records). Only the Driver can read the final value.

---

 Structured Streaming

Q: What is Watermarking? Watermarking is a threshold used in streaming to handle late data. It tells Spark how long to wait for late-arriving events before dropping them from the state. For example, a 10-minute watermark means data arriving more than 10 minutes late will be ignored.

Q: Explain Checkpointing. Checkpointing saves the current state (offsets, metadata) to a fault-tolerant storage like S3/HDFS. If the stream fails, it can restart from the exact point it left off, ensuring exactly-once processing.

---

6. Real-world Scenario / Coding

Q: "My Spark job is slow. How do you debug it?"

1.     Open Spark UI: Look for the longest-running Stage.

2.     Check for Skew: If one task is taking much longer than others, it's a skew issue.

3.     Check for Spill: If you see "Disk Spill" in the UI, it means your executors don't have enough RAM for the operation, and data is being moved to disk (very slow).

4.     Check Shuffle Read/Write: High shuffle indicates a need for Broadcast Joins or better partitioning.

7. My job failed with an OutOfMemory (OOM) error. How do you fix it?

Case A: Driver OOM

·         Cause: You probably ran .collect() on a massive dataset. This sends all the data from the workers to the single Driver machine.

·         The Fix: Don't use .collect(). Write the data to a file (S3/HDFS) instead, or increase spark.driver.memory.

Case B: Executor OOM

·         Cause: Your partitions are too big (Data Skew), or you have too many concurrent tasks running on one executor.

·         The Fix:

1.     Increase Partitions: Use .repartition() to make the data chunks smaller.

2.     Increase Memory: Adjust --executor-memory.

3.     Reduce Cores: If you have 5 cores per executor, 5 tasks are fighting for the same RAM. Reducing cores to 2 or 3 gives each task more "breathing room."