Overview
This conference talk from Conf42 DevOps 2025 explores how to effectively manage Kubernetes without overlooking critical aspects. Discover the evolution of data engineering, from traditional approaches to modern methodologies, and understand the common challenges faced in conventional data pipelines. Learn how AI is revolutionizing data pipeline management with real-time adaptability, illustrated through practical case studies. Explore the key technologies driving AI-powered pipelines and get practical advice on implementing AI in your data engineering workflows. The presentation also covers emerging trends in data engineering and provides actionable next steps for implementation. The 30-minute session is structured in a comprehensive format, covering everything from introductory concepts to future outlook.
Syllabus
00:00 Introduction and Speaker Background
00:21 Evolution of Data Engineering
02:22 Challenges in Traditional Data Pipelines
03:28 AI's Role in Transforming Data Pipelines
05:28 Real-Time Adaptability and Case Studies
10:44 Key Technologies Driving AI-Powered Pipelines
12:36 Getting Started with AI in Data Engineering
13:53 Future Trends in Data Engineering
15:03 Conclusion and Next Steps
Taught by
Conf42