Evaluating Retrieval-Augmented Generation Systems: Challenges and Practices

This 25-minute conference talk by Yue Liu, Machine Learning Engineer at Modulai, explores the challenges and practices in evaluating Retrieval-Augmented Generation (RAG) systems. Dive into the complexities of assessing RAG performance, which requires evaluating both the quality of generated text and the relevance and correctness of retrieved information. Learn about emerging frameworks and metrics in this evolving field where standardized evaluation methods are still developing. Discover current approaches to RAG evaluation and engage with key research questions that remain open in this rapidly advancing area. The speaker, Yue Liu, brings expertise from her work at Modulai, a Swedish ML consultancy, where she has implemented AI solutions across healthcare, legal, and finance sectors. Her background includes PhD research at KTH focused on AI applications for breast cancer assessment and a Master's in Computer Science from KTH, Sweden and TU Delft, Netherlands. This talk was recorded at the 2025 GAIA Conference on April 11 at Svenska Mässan in Gothenburg, Sweden.