This course covers the concepts of self-attention, cross-attention, hard and soft attention, and the Transformer architecture. Students will learn about set encoding, key-value stores, queries, keys, and values in self-attention and cross-attention. The course includes implementation details and a practical PyTorch implementation of a Transformer encoder using a Jupyter Notebook. The intended audience for this course is individuals interested in deep learning, natural language processing, and neural networks.
Self - Cross, Hard - Soft Attention and the Transformer
Overview
Syllabus
– Welcome to class
– Listening to YouTube from the terminal
– Summarising papers with @Notion
– Reading papers collaboratively
– Attention! Self / cross, hard / soft
– Use cases: set encoding!
– Self-attention
– Key-value store
– Queries, keys, and values → self-attention
– Queries, keys, and values → cross-attention
– Implementation details
– The Transformer: an encoder-predictor-decoder architecture
– The Transformer encoder
– The Transformer “decoder” which is an encoder-predictor-decoder module
– Jupyter Notebook and PyTorch implementation of a Transformer encoder
– Goodbye :
Taught by
Alfredo Canziani
Tags
Related Courses
-
Create a Large Language Model from Scratch with Python – Tutorial
-
Pytorch Transformers from Scratch - Attention Is All You Need
-
The Complete Guide to Transformer Neural Networks
-
Prompt Engineering - Understanding Large Language Models with ChatGPT
-
Let's Build GPT - From Scratch, in Code, Spelled Out
-
NYU Deep Learning
