Take your skills to the next level with our Professional Data Engineer track. This advanced track is designed to build on the Associate Data Engineer in SQL and Data Engineer in Python tracks. It equips you with the cutting-edge knowledge and tools demanded by modern data engineering roles.
Throughout this journey, you'll master modern data architectures, enhance your Python skills with a deep dive into object-oriented programming, explore NoSQL databases, and harness the power of dbt for seamless data transformation.
Unlock the secrets of DevOps with essential practices, advanced testing techniques, and tools like Docker to streamline your development and deployment processes. Immerse yourself in big data technologies with PySpark and achieve mastery in data processing and automation using shell scripting.
Engage in hands-on projects and tackle real-world datasets to apply your knowledge, debug complex workflows, and optimize data processes. By completing this track, you'll not only gain the advanced skills needed to conquer complex data engineering challenges but also the confidence to apply them in the dynamic world of data engineering.
Overview
Syllabus
- Understanding Modern Data Architecture
- Discover modern data architecture's key components, from ingestion and serving to governance and orchestration.
- Introduction to Shell
- The Unix command line helps users combine existing programs in new ways, automate repetitive tasks, and run programs on clusters and clouds.
- Containerization and Virtualization Concepts
- Learn the essentials of VMs, containers, Docker, and Kubernetes. Understand the differences to get started!
- Introduction to dbt
- This course introduces dbt for data modeling, transformations, testing, and building documentation.
- Introduction to Object-Oriented Programming in Python
- Discover the fundamental concepts of object-oriented programming (OOP), building custom classes and objects!
- Introduction to NoSQL
- Conquer NoSQL and supercharge data workflows. Learn Snowflake to work with big data, Postgres JSON for handling document data, and Redis for key-value data.
- DevOps Concepts
- In this Introduction to DevOps, you’ll master the DevOps basics and learn the key concepts, tools, and techniques to improve productivity.
- Introduction to Testing in Python
- Master Python testing: Learn methods, create checks, and ensure error-free code with pytest and unittest.
- Debugging Code
- Introduction to Docker
- Gain an introduction to Docker and discover its importance in the data professional’s toolkit. Learn about Docker containers, images, and more.
- Introduction to PySpark
- Master PySpark to handle big data with ease—learn to process, query, and optimize massive datasets for powerful analytics!
- Introduction to Big Data analysis with Spark
- Programming in PySpark RDD’s
- PySpark SQL & DataFrames
- Cleaning an Orders Dataset with PySpark
- Downloading Data on the Command Line
- Data Pipeline on the Command Line
- Streaming Concepts
- Learn about the difference between batching and streaming, scaling streaming systems, and real-world applications.
- Introduction to Apache Kafka
- Master Apache Kafka! From core concepts to advanced architecture, learn to create, manage, and troubleshoot Kafka for real-world data streaming challenges!
- Introduction to Kubernetes
- In this course, you will learn the fundamentals of Kubernetes and deploy and orchestrate containers using Manifests and kubectl instructions.
- Impactful Data Engineering—with Datadog's Wouter de Bie
Taught by
Filip Schouwenaars, Mike Metzger, Tim Sangster, Cem Sakarya, Alexander Levin, Miller Trujillo, Jake Roach, Frank Heilmann, George Boorman, and Julia Ostheimer
