Opening Retrieval Augmented Language Models
Retrieval Augmented Language Models
- Length
- 1 h 19 min
- Level
- Graduate
- Recorded
- Stanford University2024
Chapter 1 of 11: Before the lecture
Kiela contrasts frozen retrieval pipelines with RAG, Retro and Atlas, which train the retriever end to end.
Retrieval augmented language models pair a generator with an external document store to reduce hallucination and get past the limits of parametric memory. Kiela, speaking in Stanford's CS25 Transformers United series, starts from sparse retrieval with BM25 and moves to dense passage retrieval and approximate nearest neighbour search with FAISS. He contrasts frozen pipelines that rely on in-context learning with systems trained end to end, including RePlug, RAG, Fusion-in-Decoder, Retro, Realm and Atlas. Later sections weigh long-context models against retrieval and consider active retrieval and tool use. The seminar assumes familiarity with neural language models, embeddings and backpropagation, and closes with questions on domain adaptation, hardware and hallucination.
Retrieval augmented language models pair a generator with an external document store to reduce hallucination and get past the limits of parametric memory. Kiela, speaking in Stanford's CS25 Transformers United series, starts from sparse retrieval with BM25 and moves to dense passage retrieval and approximate nearest neighbour search with FAISS. He contrasts frozen pipelines that rely on in-context learning with systems trained end to end, including RePlug, RAG, Fusion-in-Decoder, Retro, Realm and Atlas. Later sections weigh long-context models against retrieval and consider active retrieval and tool use. The seminar assumes familiarity with neural language models, embeddings and backpropagation, and closes with questions on domain adaptation, hardware and hallucination.
Natural Language Processing With Modular PDP Networks and Distributed Lexicon
Neural probabilistic language models
This lecture
Talk 11 of 21 of Transformers United.
Choose a node to see how it connects to this lecture. What comes before is on the leftat the top. What comes after is on the rightat the bottom. The ring is what it teaches. People and places are above. Other lectures and books are below.
Press a chosen node again to open it.
Your comment is kept while you sign up.Sign in
Parametric and non-parametric memory in language models
DrQA
OrQA
Dense Passage Retriever
FAISS
ColBERT
Formal et al 2021
Lin et al 2023
Retrieval augmentation reduces hallucination paper
RePlug (Shi et al 2023)
Retrieval-Augmented Generation for Knowledge-Intensive NLP Tasks
Leveraging Passage Retrieval with Generative Models for Open Domain Question Answering
Generalization through Memorization: Nearest Neighbor Language Models
Improving language models by retrieving from trillions of tokens
Retro++
REALM: Retrieval-Augmented Language Model Pre-Training
REALM
Atlas
Mixtral 7B
RAG
Retro architecture
FLARE
TRIME
SILO
Lost in the Middle
Lost in the Middle: How Language Models Use Long Contexts
WebGPT: Improving the factual accuracy of language models through web browsing
Toolformer: Language Models Can Teach Themselves to Use Tools
Self-Reflective Retrieval-Augmented Generation (Self-RAG)
InstructRetro
RA-DIT
Hypothetical Document Embeddings (HyDE)
Multimodal RAG
Light Convolutions