ConceptAI & ML
Residual connection
A connection that adds the input of a layer to its output, allowing the network to learn residual mappings.
Know first
Named as a prerequisite by the lectures that define this. The number is how many say so. A grey one has no page yet.
Opens up
Concepts that name this one as a prerequisite.
Lecturers
3Each in their own terms, from the easiest level.
Andrej Karpathy
Residual connections add the input of a block directly to its output via a skip connection, creating a gradient superhighway that facilitates training deep neural networks.Sara Beery
A connection defining layer output as the sum of the layer's processed input and the original input itself.
Lectures that assume it
3Connections
14The lectures that define it or assume it, and the concepts next to it.
Concepts8
- Backpropagation
- Deep neural networks
- Gradient flow
- Skip connections
- Pre-layer norm
- Residual layer weight initialization