Mixed-modal Language Modeling: Chameleon, Transfusion, and Mixture of Transformers

Explore a lecture by Luke Zettlemoyer from the University of Washington, presented at the Simons Institute as part of "The Future of Language Models and Transformers" series. Delve into recent advancements in mixed-modal language modeling, examining how researchers are moving beyond specialized architectures designed for specific modalities toward universal models capable of processing arbitrary sequences of images and text. Compare two contrasting architectural approaches—Chameleon and Transfusion—which represent different paradigms for handling mixed-modal data. Learn about the shift from "tokenize-everything" methods to hybrid models combining autoregressive transformers with diffusion techniques. Discover strategies for training such models at scale while avoiding excessive modality competition through the Mixture of Transformers technique. This hour-long presentation outlines a potential foundation for truly universal models that can understand and generate data across any modality, while also identifying crucial next steps needed to achieve this ambitious goal.