Dive into this one-hour talk from Oxen that explores when small language models (SLMs) are appropriate and when they should be avoided, featuring a comparative evaluation of Mistral 3.1 and Gemma-3 models. Learn about the benefits of SLMs, data flywheels, and why smaller models are gaining importance in the AI landscape. The presentation includes practical demonstrations evaluating various models on SimpleQA and Rust programming tasks, along with insights on multimodal capabilities. Access comprehensive resources including datasets, slides, and community links to further explore the concepts discussed. The talk covers everything from model evaluation methodologies to training considerations, with a special focus on how to match the right model size to specific use cases.
Small Language Models - When and When NOT to Use Them + Mistral 3.1 & Gemma-3 Comparison
Overview
Syllabus
0:00 Welcome to Arxiv Dive
1:12 $ whois
1:59 $ whoami
3:18 What is Oxen.ai
4:24 Lets Talk Smol Lms4:35 Benefits of Smol Lms
6:33 When Not to Use Smol LMs
7:47 What is a Data Flywheel
9:01 Why Smol LMs Are Important Now
13:42 Did I Use a Framework for SFT or RL
14:09 Only Your Data and Criteria Matters
16:18 Gemma-3 vs. Mistral-3.1 Evals
16:41 How to Evaluate a Model
26:49 o3-mini, Mistral Small-3.1, and Gemma-3 on SimpleQA
28:17 Training a Model to Program in Rust
34:45 o3-mini, Mistral Small-3.1, and Gemma-3’s Eval on Rust
38:17 Questions
43:36 What About Smol Multimodal Models?
48:56 Test a Homemade Phi-4 Multimodal Chatbot
58:45 QR Code for Free Compute Credits
Taught by
Oxen
