Thank you for sending your enquiry! One of our team members will contact you shortly.
Thank you for sending your booking! One of our team members will contact you shortly.
Duration 21 hours
Course Outline
Module 1: Big Data & Spark Foundations
- Survey of the Big Data ecosystem and Spark’s pivotal role in contemporary data platforms
- Decoding Spark architecture: drivers, executors, cluster managers, lazy evaluation, DAGs, and execution planning
- Distinguishing between RDD and DataFrame APIs and determining the appropriate use case for each
- Setting up SparkSessions and mastering the fundamentals of application configuration
Module 2: PySpark DataFrames
- Ingesting and exporting data from enterprise sources and formats: CSV, JSON, Parquet, and Delta
- Mastering PySpark DataFrames: transformations, actions, column expressions, filtering, joins, and aggregations
- Implementing advanced operations like window functions, timestamp management, and handling nested data structures
- Integrating data quality checks and writing reusable, maintainable PySpark code
Module 3: Kubernetes & Docker Fundamentals for Spark Users
- Conceptualizing the relationship between Docker images and Kubernetes pods in the context of Spark drivers and executors—focused on cluster consumers rather than administrators
- Tracing the lifecycle of a Spark job on Kubernetes: pod scheduling, resource requests/limits, and mapping session settings to cluster resources
- Defining the boundary between user-controlled Spark session parameters and platform-managed cluster provisioning, and why this distinction is critical for tuning
- Hands-on exercise: Examine a live Spark-on-Kubernetes job in a sandbox cluster to identify driver and executor pods
Module 4: Spark Session Configuration on Kubernetes — Deep Dive
- Analyzing the trade-offs between executor count and size, specifically regarding memory versus core allocation
- Managing driver and executor memory, overhead, and the translation of these settings into pod resources
- Evaluating dynamic allocation on Kubernetes: identifying scenarios where it conserves resources versus when it does not
- Hands-on exercise: Execute identical jobs under varying executor and core configurations to benchmark runtime and resource consumption
Module 5: Scaling Behaviour & Cost Optimisation
- Assessing how cluster scaling (adding/removing nodes) impacts job behavior, completion time, and resource spend
- Weighing the performance and cost implications of many small executors versus fewer large ones
- Optimizing shuffle behavior and partition tuning, including estimating the financial impact of configuration choices
- Hands-on exercise: Scale a sandbox cluster from five to ten nodes and monitor the effects on job completion time and resource usage
Module 6: Efficient Data Ingestion & Partitioning
- Addressing the small-files problem: why numerous 1–5 MB Parquet files degrade performance and disrupt partitioning
- Implementing repartitioning and coalescing strategies to optimize data flow
- Controlling partition sizes during both read and write operations
- Writing Parquet data efficiently to prevent downstream small-file issues
- Hands-on exercise: Process a dataset consisting of many small Parquet partitions, apply various repartitioning strategies, and compare pre- and post-optimization performance
Module 7: Pandas Memory Management & Failure Diagnosis
- Identifying root causes of out-of-memory errors in Pandas and recognizing early warning signs
- Implementing memory-efficient conversion patterns between Spark and Pandas
- Preventing memory spikes when writing large datasets to CSV
- Utilizing chunked processing and dtype optimization for resource-constrained environments
- Hands-on exercise: Simulate a typical Pandas out-of-memory scenario and resolve it using chunking and dtype optimization techniques
Module 8: Polars as a Complementary Tool
- Positioning Polars relative to Pandas: analyzing performance traits, lazy evaluation, and memory behavior
- Integrating Polars into modern cloud data stacks (including AWS) alongside PySpark and Pandas, and navigating migration roadmaps
- Hands-on exercise: Refactor a Pandas-heavy transformation in Polars to benchmark memory usage and processing speed
Module 9: Applying Optimisation to ETL & ML Workloads
- Applying configuration, partitioning, and memory management principles to a realistic ETL pipeline
- Navigating optimization nuances for machine learning workloads sharing the same cluster resources
- Establishing a systematic tuning workflow to diagnose cost and performance issues
- Hands-on exercise: Complete an end-to-end mini-project involving data loading, transformation, and basic model training, while independently tuning Spark configurations
Requirements
Participants are expected to possess:
- Practical proficiency in Python programming, covering functions, modules, and fundamental object-oriented principles.
- Basic to intermediate experience with Pandas and tabular data processing workflows.
- Foundational knowledge of PySpark and Spark DataFrames, including data ingestion, transformations, actions, joins, and aggregations.
- A general understanding of SQL and data processing concepts, such as filtering, grouping, and joining datasets.
- Basic familiarity with Docker and Kubernetes concepts, including containers, images, and pods. No prior Kubernetes administration experience is necessary.
- Basic understanding of common data formats, including CSV, JSON, and Parquet.
Participants are not required to be Kubernetes administrators or infrastructure specialists. The course centers on enabling data engineers, developers, and data scientists to comprehend and optimize their Spark workloads on Kubernetes from an application and configuration standpoint.
Target Audience
This program is tailored for professionals who develop, maintain, or optimize data processing and machine learning workloads using Python and Spark in cloud or containerized settings.
It is particularly beneficial for:
- Data Engineers handling PySpark, distributed processing, and ETL pipelines.
- Data Scientists working with large datasets or executing machine learning tasks using Spark, Pandas, or Polars.
- Python Developers building data-intensive applications who aim to enhance memory efficiency and processing speed.
- Machine Learning Engineers managing data preparation and model training on shared Kubernetes or cloud infrastructure.
- Analytics Engineers processing large datasets who seek to boost data processing efficiency.
- DevOps, Platform, and Cloud Engineers supporting Spark workloads on Kubernetes who need to understand how application configuration influences resource usage and performance.
- Technical Leads and Solution Architects involved in designing or refining modern data platforms.
Testimonials (1)
I liked that it was practical. Loved to apply the theoretical knowledge with practical examples.
