Causal Representation Learning: A Natural Fit for Mechanistic Interpretability

This talk by Dhanya Sridhar from IVADO, Université de Montréal, and Mila explores sparse shift autoencoders (SSAEs) as a solution for steering large language models without extensive supervision. Learn how SSAEs map differences between embeddings to sparse representations that capture concept shifts, offering an identifiable approach that enables accurate steering of single concepts without requiring labeled data. The presentation demonstrates empirical results using Llama-3.1 embeddings across both semi-synthetic and real-world language datasets, positioning causal representation learning as a natural framework for mechanistic interpretability in the context of safety-guaranteed LLMs. The 59-minute talk was presented as part of the Simons Institute's series on Safety-Guaranteed LLMs.