Announcements

• [June 13] Answers to Exercises 3 posted.
• [June 10] Slides of Thursday, Week 10 posted. The slides of Tuesday have been updated and now contain the material about rate distortion theory for Gaussian channels. The slides of Thursday contain all the last minute information about the review session, the paper, and the material and rules for the Final.
• [June 7] Exercises 3 posted.
• [June 6] Slides of Tuesday, Week 10 posted.
• [June 3] Slides of Week 9 posted.
• [May 30] "Not the usual channels", a popular article from The Economist about modern codes.
• [May 30] Slides of Week 8 posted.
• [May 19] Slides and handout of Week 7 posted.
• [May 14] Slides of Week 6 posted.
• [May 6] An email was sent about the projects. For those who are missing on the ulist for this course, it said: "Now that the Midterm is out of the way, it is time to get serious about the project for the course "Information Theory" (CS225/ECE205a). At the end of quarter you should have written a short paper (~ 5 pages) on an information theoretic topic (see the slide 4 of Week 3 for some suggestions). To get things started, I would like all of you to email me before the end of the weekend an abstract outlining the paper that you want to work on. -WvD"
• [May 6] Answers to the Midterm posted.
• [May 2] Answers to Exercises 2 posted.
• [May 1] Correction: Question 1a in Exercises 2 should say H(X|Z) instead of H(X)Z.
• [April 29] Thursday May 4 we have the Midterm on the material of the first four weeks. See the slides of Week 4 for the details. Bring your questions to next Tuesday's class.
• [April 28] Exercises 2 are posted, try to solve them before next Tuesday's class.
• [April 27] Slides of Week 4, Thursday posted.
• [April 26] Slides of Week 4, Tuesday posted.
• [April 25] Answers to Exercises 1 posted.
• [April 20] Slides of Week 3 posted.
• [April 14] Exercises 1 are posted; try to solve them before next Tuesday's class.
• [April 13] Slides of Week 2, Thursday posted.
• [April 12] Slides of Week 2, Tuesday posted.
• [April 10] Slides of Week 1 posted.
• [March 23] The required text book for this course is Elements of Information Theory by Cover and Thomas.
• [February 6, 2006] Installment of this web site for the graduate course Information Theory (CS225/ECE205a).

Course Info

Description:

• List of topics: Entropy, mutual information, and Shannon's coding theorems, lossless source coding, Huffman, Shannon-Fano-Elias, and arithmetic codes, channel capacity; aspects of rate-distortion theory, lossy source coding, source-channel coding, Kolmogorov/algorithmic complexity and information; applications of information theory in various fields
• This 4 unit, graduate course is listed both for the CS department and the ECE department, and will be taught alternately by Wim van Dam (CS) and Kenneth Rose (ECE). For CS students this course will satisfy the 'theory track'.

Prerequisites:

• ECE 140 or PSTAT 120A-B

Required Texts:

• Elements of Information Theory, Thomas M. Cover and Joy A.Thomas, John Wiley, 1991
• "A Mathematical Theory of Communication" by Claude E. Shannon

Grading/Exams:

• Midterm+Final+Project : 1+2+3 = 6

Weekly Schedule:

• Tuesday 15:00-16:30: Class in Phelps 1401
• Thursday 15:00-16:30: Class in Phelps 1401

Contacting/Questions:

• I prefer that you use my office hours for your questions, rather than doing a lengthy Q&A exchange via email. Ideally, you could email me about the issue in advance so that I can look into it before my office hour session. Thanks.

Schedule and Slides

• Week 1 [April 2 - April 8]
  Topics: formalities, introduction, Claude Shannon, noisy channels, repetition codes, channel rate, discrete random variables, probability theory, entropy, joint entropy, conditional entropy
  Slides Week 1 (ppt|pdf)
  
  
• Week 2 [April 9 - April 15]
  Topics: entropy, relative entropy, mutual information, entropic inequalities; data compression, prefix codes, Kraft's inequality
  Slides Week 2, Tuesday (ppt|pdf)
  Slides Week 2, Thursday (ppt|pdf)
  Exercises 1 / Answers 1
  
  
• Week 3 [April 16 - April 22]
  Topics: data compression (lossless source coding), Kraft inequality, entropic inequalities, Jensen's inequality
  Slides Week 3 (ppt|pdf)
  
  
• Week 4 [April 23 - April 29]
  Topics: Shannon's noiseless source coding theorems, Huffman codes, Shannon-Fano-Elias codes, arithmetic Codes, Lempel-Ziv coding
  Slides Week 4, Tuesday (ppt|pdf)
  Slides Week 4, Thursday (ppt|pdf)
  Exercises 2 / Answers 2
  
  
• Week 5 [April 30 - May 6]
  Topics: miscellanea; Midterm
  Tuesday: Discussion of the answers to Exercises 2.
  Thursday: Midterm on Entropies and Data Compression (Week 1-4; see Slides of Week 4 for details)
  Answers to Midterm
  
  
• Week 6 [May 7 - May 13]
  Topics: channel capacity, AEP, channel coding theorem
  Slides Week 6 (ppt|pdf)
  
  
• Week 7 [May 14 - May 20]
  Topics: proof of the channel coding theorem, error correction in practice, minimum distance decoding
  Handout on noisy channel coding theorem
  Slides Week 7, Thursday (ppt|pdf)
  
  
• Week 8 [May 21 - May 27]
  Topics: Hamming codes, linear codes, parity check matrices, perfect codes, real world codes
  Slides Week 8 (ppt|pdf)
  
  
• Week 9 [May 28 - June 3]
  Topics: Kolmogorov complexity, universal probability
  Slides Week 9 (ppt|pdf)
  
  
• Week 10 [June 4 - June 10]
  Topics: rate distortion theory for binary channels and Guassian channels; last minute information about the last two weeks of the course, overview of the material for the Final.
  Slides Week 10, Tuesday (ppt|pdf)
  Slides Week 10, Thursday (ppt|pdf)
  Exercises 3 / Answers 3
  
  
• Finals Week [June 11 - June 17]
  Sunday: Paper is due
  Tuesday: Review session in Phelps 1401, 15:00-16:30
  Thursday: Final Examination on all material in Phelps 1401, 16:00-19:00
  

	This work is licensed under a Creative Commons Attribution-NonCommercial-ShareAlike 2.5 License.
Created February 6, 2005 / last updated June 13, 2006 by vandam@cs