EE 299 – Information Theory

Spring 2003

W F 02:20 PM - 03:35 PM

Instructor:

Lawrence Carin

176 Hudson Hall

PH: 660-5270

Webpage: www.ee.duke.edu/~lcarin

Email: lcarin@ee.duke.edu

Office Hours:

No set times, set up a meeting as desired

Text:

Thomas M. Cover and Joy A. Thomas, Elements of Information Theory, Wiley, 1991

Available to Duke students online at:

http://www3.interscience.wiley.com/cgi-bin/booktoc?Type=DD&ID=86512748

Grading policy:

1. HW collected and graded (15%)
2. Exam 1 (25%)
3. Exam 2 (25%)
4. Final Exam (35%)

Rough Course Outline

1. Entropy, relative entropy and mutual information (Ch. 2)
2. Asymptotic equipartition property (Ch. 3)
3. Entropy rates of stochastic processes (Ch. 4)
4. Data compression (Ch. 5)
5. Channel capacity (Ch. 8)
6. Differential entropy (Ch. 9)
7. Maximum entropy and spectral estimation (Ch. 11)
8. Information theory and statistics (Ch. 12)
9. Rate-distortion theory (Ch. 13, and also the book by Toby Berger)
10. Optimal experiments and information theory in sensing (notes & papers from literature)

********************************************

No Class April 2

Last four classes of semester: April 9, 11, 16 and 18

********************************************

Homework Assignments:

HW #1: Ch. 2/ problems 2, 3, 5, 6, 8, 9, 14, 15, 16, 19 (due 1/17)

HW #2: Ch. 2/ problems 20, 21, 24, 25, 27, 29, 32, 34 (due 1/31)

HW #3: Ch. 3/ problems 1,3,5,7 (due 2/7)

HW #4: Ch. 5/ problems 3, 4, 5, 8, 9, 14, 22, 25 (due 2/17)

HW #5: Ch. 8/ problems 3, 5, 6, 7, 9 (due 3/5)

HW #6: Ch. 9/ problems 1, 3, 4, 5, 6 (due 3/17)

HW #7: Ch. 10/ problems 1, 2, 3 (due 3/24)

HW #8: Ch. 13/ problems 1, 3, 5, 6, 8 (due 4/18)

HW #9: Ch. 4/ problems 1, 2, 5, 7, 9, 12 (due 4/25) EXTRA CREDIT

Useful Additional Notes:

On AEP: http://www.inference.phy.cam.ac.uk/mackay/info-theory/course.html (Lecture 2)

Joint Classification and Reconstruction