Opening From Large Language Models to Large Multimodal Models
From Large Language Models to Large Multimodal Models
- Length
- 1 h 20 min
- Level
- Undergraduate
- Recorded
- Stanford University2024
Chapter 1 of 11: Before the lecture
Ding shows how CogVLM adds vision experts to a frozen LLM and argues diffusion beats autoregressive image generation.
Large multimodal models join visual perception to large language models, and Ming Ding of Zhipu AI builds the case from the language side first. He reviews BERT, his own GLM, scaling laws and chat alignment, arguing that downstream performance tracks pretraining loss and that data cleaning matters more than architecture. Training infrastructure follows: DeepSpeed, Megatron, activation checkpointing and context parallelism for long windows. Ding then compares BLIP-2, LLaVA, CogVLM and CogAgent, showing how CogVLM adds vision-expert parameters while keeping the original language weights intact. A section on image generation sets autoregressive tokenization against diffusion models such as DiT and Relay Diffusion. The session, part of Stanford CS25, assumes familiarity with transformers and ends with video understanding, embodied AI and audience questions.
Large multimodal models join visual perception to large language models, and Ming Ding of Zhipu AI builds the case from the language side first. He reviews BERT, his own GLM, scaling laws and chat alignment, arguing that downstream performance tracks pretraining loss and that data cleaning matters more than architecture. Training infrastructure follows: DeepSpeed, Megatron, activation checkpointing and context parallelism for long windows. Ding then compares BLIP-2, LLaVA, CogVLM and CogAgent, showing how CogVLM adds vision-expert parameters while keeping the original language weights intact. A section on image generation sets autoregressive tokenization against diffusion models such as DiT and Relay Diffusion. The session, part of Stanford CS25, assumes familiarity with transformers and ends with video understanding, embodied AI and audience questions.
All NLP tasks are generation tasks: A general pretraining framework (GLM)
InstructGPT
Transformer paper

This lecture
Talk 17 of 21 of Transformers United.
Choose a node to see how it connects to this lecture. What comes before is on the leftat the top. What comes after is on the rightat the bottom. The ring is what it teaches. People and places are above. Other lectures and books are below.
Press a chosen node again to open it.
Your comment is kept while you sign up.Sign in
DeepSpeed
CogLTx: Applying BERT to Long Texts
DPO paper
CogView
iGPT
DALL-E
Parti
CogView2
DiT
CogVideo