Positional Encodings and Group Theory in Transformers
Educator; Researcher
Grant Sanderson and Alok Puranik explore the theoretical foundations of positional encodings in transformer networks, applying group theory to analyze methods like RoPE and ALiBi.
Grant Sanderson of 3Blue1Brown and Alok Puranik, a researcher at Jane Street, work through Alok’s latest blog post on positional encodings in transformers. Starting from a few natural assumptions about attention, they use group and representation theory to explore what forms a positional encoding can take. Along the way, they connect the theory to familiar methods like RoPE (Rotary Positional Embedding) and ALiBi (Attention with Linear Biases), and investigate whether it reveals anything new.
Read Alok's blog post here: https://blog.janestreet.com/using-group-theory-to-explore-positional-encodings-attention/