Mimecast's Journey from Batch to Real-Time Processing

This conference talk explores how Mimecast transformed their data architecture from traditional batch-processing to a scalable, real-time data platform. Follow along as Craig Metcalfe (Director of Data Platforms at Mimecast) and Ismail Makhlouf (Sr. Analytics SA at AWS) detail the architectural evolution, showcasing how they integrated AWS services like MSK and Glue alongside open-source technologies such as Trino to build a robust data ingestion pipeline capable of handling diverse workloads. Gain insights into their strategic decisions around domain-based cluster separation, data validation approaches, and intelligent query routing while learning practical lessons from their experience managing seasonal traffic patterns and ensuring system reliability. This 36-minute presentation from the AWS AI and Data Conference provides valuable knowledge for organizations looking to modernize their data processing capabilities.