Course 02
Transformer-based models and tricks
- Wednesday, 28 October 2026
- 12:15 – 13:45
- Online
Attention on its own carries no information about the order of tokens. We look at how models represent position and how the approaches evolved up to RoPE.
The second half covers encoder models, BERT in particular, and how they are used for classification and search.
On the programme
- 01Attention approximation
- 02MHA, MQA, GQA
- 03Position embeddings (regular, learned)
- 04RoPE and applications
- 05Transformer-based architectures
- 06BERT and its derivatives