What Goes into the Making of LLMs

In this 10-minute conference talk from Conf42 LLMs 2025, Antara Raman Sahay explores the intricate process behind creating large language models. Discover the critical role of code data in pre-training, understand the various phases of training language models, and learn about experimental frameworks used in model development. Examine how code data impacts model performance, with insights on optimal proportions and quality requirements. The presentation covers key findings, practical recommendations for LLM development, and concludes with future research directions and essential takeaways for anyone interested in language model creation.