Learn to build a Llama 4 large language model from scratch in this comprehensive 3.5-hour course taught by @vukrosic. Dive deep into the architecture and inner workings of modern LLMs, starting with fundamentals like text generation, token prediction, and sampling techniques. Progress through implementing your own tokenizer with Byte Pair Encoding, building self-attention mechanisms, understanding Query-Key-Value operations, implementing Rotary Positional Embeddings (RoPE), and creating Feed-Forward Networks. The course provides both theoretical explanations and hands-on coding, making complex concepts accessible even to beginners. All code and presentations are available on GitHub, allowing you to follow along and build your own implementation of this cutting-edge language model architecture.
Overview
Syllabus
- 0:00:00 Introduction to the course
- 0:00:15 Llama 4 Overview and Ranking
- 0:00:26 Course Prerequisites
- 0:00:43 Course Approach for Beginners
- 0:01:27 Why Code Llama from Scratch?
- 0:02:20 Understanding LLMs and Text Generation
- 0:03:11 How LLMs Predict the Next Word
- 0:04:13 Probability Distribution of Next Words
- 0:05:11 The Role of Data in Prediction
- 0:05:51 Probability Distribution and Word Prediction
- 0:08:01 Sampling Techniques
- 0:08:22 Greedy Sampling
- 0:09:09 Random Sampling
- 0:09:52 Top K Sampling
- 0:11:02 Temperature Sampling for Controlling Randomness
- 0:12:56 What are Tokens?
- 0:13:52 Tokenization Example: "Hello world"
- 0:14:30 How LLMs Learn Semantic Meaning
- 0:15:23 Token Relationships and Context
- 0:17:17 The Concept of Embeddings
- 0:21:37 Tokenization Challenges
- 0:22:15 Large Vocabulary Size
- 0:23:28 Handling Misspellings and New Words
- 0:28:42 Introducing Subword Tokens
- 0:30:16 Byte Pair Encoding BPE Overview
- 0:34:11 Understanding Vector Embeddings
- 0:36:59 Visualizing Embeddings
- 0:40:50 The Embedding Layer
- 0:45:31 Token Indexing and Swapping Embeddings
- 0:48:10 Coding Your Own Tokenizer
- 0:49:41 Implementing Byte Pair Encoding
- 0:52:13 Initializing Vocabulary and Pre-tokenization
- 0:55:12 Splitting Text into Words
- 1:01:57 Calculating Pair Frequencies
- 1:06:35 Merging Frequent Pairs
- 1:10:04 Updating Vocabulary and Tokenization Rules
- 1:13:30 Implementing the Merges
- 1:19:52 Encoding Text with the Tokenizer
- 1:26:07 Decoding Tokens Back to Text
- 1:33:05 Self-Attention Mechanism
- 1:37:07 Query, Key, and Value Vectors
- 1:40:13 Calculating Attention Scores
- 1:41:50 Applying Softmax
- 1:43:09 Weighted Sum of Values
- 1:45:18 Self-Attention Matrix Operations
- 1:53:11 Multi-Head Attention
- 1:57:55 Implementing Self-Attention
- 2:10:40 Masked Self-Attention
- 2:37:09 Rotary Positional Embeddings RoPE
- 2:38:08 Understanding Positional Information
- 2:40:58 How RoPE Works
- 2:49:03 Implementing RoPE
- 2:56:47 Feed-Forward Networks FFN
- 2:58:50 Linear Layers and Activations
- 3:02:19 Implementing FFN
Taught by
freeCodeCamp.org
