Monitoring and maintaining Apache Airflow workflows can be complex and critical for ensuring reliable data pipelines. In this course, Monitor and Maintain Apache Airflow, you’ll gain the ability to effectively oversee and manage your Airflow workflows. First, you’ll explore how to set up a DAG to read files and store filtered and processed data in a SQLite database, and you'll monitor the run of the workflow using the Airflow UI. Next, you’ll discover how to configure the email provider settings in Airflow configuration to be able to send notifications for your DAG runs. Finally, you’ll learn how to configure task-level and DAG-level SLAs, receive email notifications for SLA breaches, observe SLA misses in the Airflow UI, and configure DAG failed/retry actions. When you’re finished with this course, you’ll have the skills and knowledge of Apache Airflow needed to monitor, maintain, and ensure the reliability of your data workflows.
Overview
Monitoring and maintaining Apache Airflow workflows can be complex and critical for ensuring reliable data pipelines. In this course, Monitor and Maintain Apache Airflow, you’ll gain the ability to effectively oversee and manage your Airflow workflows. First, you’ll explore how to set up a DAG to read files and store filtered and processed data in a SQLite database, and you'll monitor the run of the workflow using the Airflow UI. Next, you’ll discover how to configure the email provider settings in Airflow configuration to be able to send notifications for your DAG runs. Finally, you’ll learn how to configure task-level and DAG-level SLAs, receive email notifications for SLA breaches, observe SLA misses in the Airflow UI, and configure DAG failed/retry actions. When you’re finished with this course, you’ll have the skills and knowledge of Apache Airflow needed to monitor, maintain, and ensure the reliability of your data workflows.
Syllabus
- Course Overview 1min
- Configure Email Notifications and Track SLA Misses 20mins
Taught by
Janani Ravi
