Sign inSign in
ConceptAI & ML

RMS norm

Root Mean Square normalization scales inputs by their root mean square values without subtracting the mean, mapping data points onto a hypersphere in high dimensions.
as Phillip Isola defines itMITThe wording closest to all 3 definitions
Watch it explained

Know first

Named as a prerequisite by the lectures that define this. The number is how many say so. A grey one has no page yet.

Euclidean norm1High dimensional space1Layer normalization1Normalization layers1Root mean square1Vector dimension1

Opens up

Concepts that name this one as a prerequisite.

Lecturers

3

Each in their own terms, from the easiest level.

  • Chris Ré

    Stanford University · Graduate

    A simplified version of layer normalization that scales inputs by their root mean square without centering them around a mean.
    • 1:14:39Neural Networks I: Feedforward Architectures & Backpropagation
  • Jeremy Bernstein

    MIT · Graduate

    A norm for a vector in ℝd\mathbb{R}^d defined as the Euclidean norm scaled by one over the square root of the dimension.
    • 1:00:19Scaling Rules for Optimization

Connections

7

The lectures that define it or assume it, and the concepts next to it.

3 lecturers. Definitions are written by AI from each lecture’s transcript. A time with a play mark starts the lecture at the moment the lecturer says it.