The course teaches how to develop elegant workflows in Python code using Apache Airflow. The learning outcomes include understanding basic Airflow concepts, defining workflows in Python code, extending Airflow with custom task operators, sensors, and plugins. The course covers topics such as orchestrating data processing pipelines, directed acyclic graphs (DAGs), task retries, and resilience in workflow design. The teaching method involves examples and demonstrations. The intended audience for this course is Python developers interested in orchestrating and automating data processing pipelines.
Overview
Syllabus
Intro
ABOUT ME
WHAT IS A WORKFLOW?
A TYPICAL WORKFLOW
EXAMPLES EVERYWHERE
WORKFLOW MANAGERS
APACHE AIRFLOW
WHAT FLOWS IN A WORKFLOW?
SOURCE AND TRIBUTARIES
DISTRIBUTARIES AND DELTAS
BRANCHES
AIRFLOW CONCEPTS: DAGS
AIRFLOW CONCEPTS: OPERATOR
AIRFLOW CONCEPTS: SENSORS
AIRFLOW CONCEPTS: XCOM
SCAN FOR INFORMATION UPSTREAM
REUSABLE OPERATORS
CONDITIONAL EXECUTION
BASH COMMANDS AND TEMPLATES
AIRFLOW PLUGINS
Taught by
EuroPython Conference
