Sign inSign in
ConceptAI & ML

Speculative decoding

An approach for faster decoding where a lighter model generates outputs and a heavier model scores and reranks them.
as Ashish Vaswani defines itStanford UniversityThe wording closest to all 3 definitions
Watch it explained

Know first

Named as a prerequisite by the lectures that define this. The number is how many say so. A grey one has no page yet.

Autoregressive generation2Auto-tuning compiler1Decoder models1Dynamic shapes1Tensor programs1Transformer1

Lecturers

3

Each in their own terms, from the easiest level.

  • Yue Guan

    GPU MODE · Graduate

    A dynamic shape problem at the operator level that involves complex runtime scheduling at the computation graph level.
    • 51:11Mercury Multi-GPU Operator Optimization
  • John Langford

    UC Berkeley · Research

    Using a fast approximate model rollout to guess ahead in parallel and verify with a precise model.
    • 35:20An Occam's Razor Principle for Transformers?

Connections

5

The lectures that define it or assume it, and the concepts next to it.

3 lecturers. Definitions are written by AI from each lecture’s transcript. A time with a play mark starts the lecture at the moment the lecturer says it.