This course explores transfer learning in Natural Language Processing (NLP) focusing on open questions, current trends, limits, and future directions. Participants will walk through interesting papers and research directions on model size, computational efficiency, out-of-domain generalization, model evaluation, fine-tuning, sample efficiency, common sense, and inductive biases. The teaching method includes a presentation of slides and a discussion of key concepts. This course is intended for individuals interested in the latest advancements and future directions in NLP.
Overview
Syllabus
Intro
Open questions, current trends, limits
Model size and Computational efficiency
Using more and more data
Pretraining on more data
Fine-tuning on more data
More data or better models
In-domain vs. out-of-domain generalization
The limits of NLU and the rise of NLG
Solutions to the lack of robustness
Reporting and evaluation issues
The inductive bias question
The common sense question
Taught by
Hugging Face
