Class Central is learner-supported. When you buy through links on our site, we may earn an affiliate commission.


Information Theory

Indian Institute of Technology Bangalore and NPTEL via Swayam


This is a graduate level introductory course in Information Theory where we will introduce the mathematical notion of information and justify it by various operational meanings. This basic theory builds on probability theory and allows us to quantitatively measure the uncertainty and randomnessin a random variable as well as information revealed on observing its value. We will encounter quantities such as entropy, mutual information, total variation distance, and KL divergence and explain how they play a role in important problems in communication, statistics, and computer science.Information theory was originally invented as a mathematical theory of communication, but has since found applications in many areas ranging from physics to biology. In fact, any field where people want to evaluate how much information about an unknown is revealed by a particular experiment, information theory can help. In this course, we will lay down the foundations of this fundamental field.
Senior undergraduate and graduate students interested in probability, statistics, communication, theoretical computer science, machine learning, quantum information and statistical physics
PREREQUISITES :Undergraduate level probability (sets and events, probability distributions, probability density functions, probability mass functions, random variables, expected value, variance, popular probability laws, Markov inequality, Chebyshev in equality, central limit theorem, law of large numbers)



Week 1:Introduction to entropy as a measure of uncertainty and randomness
Week 2:Binary hypothesis testing: bayes optimal binary hypothesis testing and total variation distance, Neyman-Pearson formulation, Stein's lemma,and KL divergence Week 3:Measures of information and their properties: Chain rule and additivity,concavity, and variational formulaeWeek 4:Data processing inequality, Pinsker's inequality, and Fano's inequality
Week 5:Data compression: _xed and variable length source coding theoremsand entropyWeek 6:Hu_man code, Shannon-Fano-Elias code, arithmetic code, hash tablesWeek 7:Universal compressionWeek 8:Channel coding: Channel capacity theorem, sphere packing bound,maximal code construction
Week 9:Random coding and ML decodingWeek 10:PLDPC and Polar codesWeek 11:QuantizationWeek 12:Minmax lower bounds in statistics

Taught by

Prof. Himanshu Tyagi

Related Courses


Start your review of Information Theory

Never Stop Learning!

Get personalized course recommendations, track subjects and courses with reminders, and more.

Sign up for free