Class Central is learner-supported. When you buy through links on our site, we may earn an affiliate commission.

LinkedIn Learning

Stream Processing Design Patterns with Spark

via LinkedIn Learning

Overview

Learn how to build popular streaming design patterns efficiently with Apache Spark.

Stream processing is becoming more popular as more and more data is generated by websites, devices, and communications. Apache Spark is a leading platform that provides scalable and fast stream processing, but still requires smart design to achieve maximum efficiency. This course helps developers use best practices and validated design patterns to implement stream processing in Apache Spark. Instructor Kumaran Ponnambalam shows how to set up your environment and then walks through four design patterns and real-world use cases: streaming analytics, alerts and thresholds, leaderboards, and real-time predictions. In chapter six, he introduces a start-to-finish project that shows how to go from design to executed job using Spark, Apache Kafka, MariaDB, and Redis. By the end of the course, you'll understand all the capabilities of this powerful platform and be able to incorporate it in your own data engineering solutions.

Syllabus

Introduction
  • Streaming with Spark
  • Prerequisites
1. Stream Processing with Spark
  • What is stream processing?
  • Streaming opportunities and challenges
  • Streaming with Apache Spark
  • Spark Structured Streaming APIs and SQL
  • Setting up the exercise files
  • Setting up Kafka
  • Setting up MariaDB and Redis
2. Streaming Analytics
  • Streaming analytics: Pattern
  • Streaming analytics: Use case design
  • Streaming analytics: Helper classes
  • Streaming analytics: Pipeline implementation
  • Streaming analytics: Results review
3. Alerts and Thresholds
  • Alerts and thresholds: Pattern
  • Alerts and thresholds: Use case design
  • Alerts and thresholds: Helper classes
  • Alerts and thresholds: Pipeline implementation
  • Alerts and thresholds: Review
4. Leaderboards
  • Leaderboards: Pattern
  • Leaderboards: Use case design
  • Leaderboards: Helper classes
  • Leaderboards: Pipeline implementation
  • Leaderboards: Review
5. Real-Time Predictions
  • Real-time predictions: Pattern
  • Real-time predictions: Use case design
  • Real-time predictions: Helper classes
  • Real-time predictions: Pipeline implementation
  • Real-time predictions: Review
6. Use Cases
  • Use case definition
  • Design of the project
  • Code walk-through
  • Execute and analyze
Conclusion
  • Next steps

Taught by

Kumaran Ponnambalam

Related Courses

Reviews

Start your review of Stream Processing Design Patterns with Spark

Never Stop Learning!

Get personalized course recommendations, track subjects and courses with reminders, and more.

Sign up for free