Reinventing Shannon Information Theory: Entropy & Optimal Prefix Codes
Derives Claude Shannon's 1948 entropy formula H(X) = -Σ p(x) log₂ p(x) from Kraft's inequality on binary prefix trees and connects lossless compression directly to language modeling.
How to study the compressibility of language.
Check out our virtual career fair: https://3b1b.co/talent
See new projects before they go live: https://3b1b.co/support
Comments
Add a comment...Sign in