AI-Powered ETL Pipeline Orchestration - Multi-Agent Systems in the Era of Generative AI

Explore a 27-minute talk that delves into the revolutionary impact of AI on ETL pipeline orchestration through the lens of multi-agent systems. Journey through the evolution of data workflows, from basic cron jobs to sophisticated cloud-native architectures and AI-driven orchestration systems. Follow along with a practical case study of a hypothetical e-commerce platform, GlobalShop, to understand how AI optimizes resource allocation, implements predictive scheduling, and manages complex dependencies in modern data ecosystems. Master essential concepts in data engineering, machine learning, and AI applications for scalable ETL solutions, while gaining hands-on knowledge of key tools including Apache Airflow, Kubernetes, and TensorFlow. Discover emerging trends shaping data integration's future and acquire valuable insights for implementing AI-powered orchestration in data engineering workflows.