Harvard University homepage
HARVARD.EDU

Madhu Sudan: Transmission of Information

Course number: 6.441
Prereq: 6.041
Time: TuTh 11:00-12:30pm
Location: 32-124
3-0-9 H-Level Grad Credit
Lecturer: Madhu Sudan
TA: Chung Chan
Official Homepage: http://stellar.mit.edu/S/course/6/sp06/6.441

Bulletin Board

This course is now history!
Thanks to all who participated!

Lecture notes.
The official website.
Course announcement

Course announcement

Prereq: 6.041
Time: TuTh 11:00-12:30pm
Location: 32-124
3-0-9 H-Level Grad Credit
Homepage: http://theory.csail.mit.edu/~madhu/ST06/

Transmission of Information

This course teaches the theory behind the transmission of information. Rough collection of topics includes:

  •  Entropy and Information: Definitions, variations of the notions, their properties.
  • Source Coding: Estimating the amount of information produced by a source and compressing this information. Introduces the Asymptotic Equipartion Property (AEP, an extension of the law of large numbers).
  • Channel Coding: Models of noisy channels. Coding theorem and its converse. Joint source-channel coding.
  • Network Information Theory.
  • Applications of Information Theory to other domains:  Stock Markets, Gambling etc.

See course homepage http://theory.csail.mit.edu/~madhu/ST06/ for more details. Most of the material for the course is covered in Elements of Information Theory by Thomas M. Cover and Joy A. Thomas. Also see http://web.mit.edu/6.441/www/ for information on previous offerings of this course.

Instructor: Madhu Sudan

TA: Chung Chan

Topics covered:  (Compilation of all scribed lecture notes)

  • Lecture 01 (02/07): Introduction. The Problem of Information Transmission. The Shannon architecture. My notes. Scribe notes-1. Scribe notes-2.
  • Lecture 02 (02/09): Entropy and Information: Basic Properties. My notes. Scribe notes.
  • Lecture 03 (02/14): Entropy and Information: Relative entropy, Data Processing Theorem, Fano’s Inequality. My notes. Scribe notes.
  • Lecture 04 (02/16): Fano’s inequality (contd.).  Asymptotic Equipartition Property (AEP). My notes. Scribe notes.
  • Tuesday (02/21): No lecture: MIT Monday.
  • Lecture 05 (02/23): Markov chains and Entropy rates. My notes. Scribe notes.
  • Lecture 06 (02/28): Source coding aka Data Compression. My notes. Scribe notes.
  • Lecture 07 (03/02): Source Coding: Shannon codes, Huffman codes. My notes. Scribe notes.
  • Lecture 08 (03/07): Source Coding: Universal Coding. My notes. Scribe notes.
  • Lecture 09 (03/09): Source Coding: Ziv-Lempel Coding. My notes. Scribe notes.
  • Lecture 10 (03/14): Channel Coding: Discrete memoryless channels, BSC,  Erasure channel. My notes. Scribe notes.
  • Lecture 11 (03/16): Channel Coding: Coding theorem for symmetric channels. My notes. Scribe notes.
  • Lecture 12 (03/21): Channel Coding: Coding Theorem for all DMC; and a converse. Scribe notes.
  • No Lecture (03/23): Midterm Exam.
  • (03/27 – 03/31): Spring break
  • Lecture 13 (04/04): Error Exponents. My notes. Scribe notes.
  • Lecture 14 (04/06): Feedback channel. Joint Source-Channel Coding. Continuous Random Variables. My notes. Scribe notes.
  • Lecture 15 (04/11): Capacity of a continuous channel. My notes. Scribe notes.
  • Lecture 16 (04/13): AWGN channel and its capacity. My notes. Scribe notes.
  • Tuesday (04/18): Patriots Day Holiday.
  • Lecture 17 (04/20): AWGN channel capacity (contd.); Colored noise. My notes. Scribe notes.
  • Lecture 18 (04/25): Network Information Theory: Multiple access channels. My notes. Scribe notes.
  • Lecture 19 (04/27): Network Information Theory: Multiple access channels; Coding for correlated sources. My notes. Scribe notes.
  • Lecture 20 (05/02): Network Information Theory:  Correlated source coding. Slepian-Wolf theorem. My notes. Scribe notes.
  • Lecture 21 (05/04): Network Information Theory: Broadcast channel. My notes. Scribe notes.
  • Lecture 22 (05/09): Network Information Theory: Broadcast channel (contd.). My notes. Scribe notes.
  • No Lecture (05/11): No lecture due to project presentations.
  • Lecture 23 (05/16): Computational aspects of Information Theory: Efficient Channel Coding. My notes. Scribe notes.
  • Lecture 24 (05/18): Computational aspects of Information Theory: Kolmogorov complexity. My notes. Scribe notes.

References

The principal reference for this course

    Thomas M. Cover and Joy A. Thomas. Elements of Information Theory. Wiley Series in Telecommunications, 1991.