INF 723: Proseminar in Information Science (Info. Theory)

Spring 2008 Syllabus

Instructor Information

Sanjay Goel picture Sanjay Goel
Office: BA 310b (Uptown Campus)
Hours: M 11:30AM-1PM & by appt.
PH: (518) 442-4925
FX (518) 442-2568
Email: [email protected]


Time:M 9:00-11:00am
Room:Draper 313
Dates:April 7 - May 12
Call #:9325

Text & Reference Books

See Resources section for required reading.


In the first two lectures of this part of the class, we will learn how to measure information content; how to compress data and; how to communicate reliably across noisy channels. This includes the source coding and noisy channel coding theorems proposed by Claude Shannon.

In the next three lectures, we will learn the basics of decision analysis and techniques to make decisions under uncertainty. Decision analysis typically involves use of decision trees where the nodes of the tree represent events, arcs connecting the nodes represent dependencies between events, and uncertainties are represented via probabilities of events. In this class, students will solve problems using decision trees. Bayesian Probability Theory will also be discussed and multistage decision problems using Bayesian analysis will be covered. Utility functions that model the decision maker's attitudes and multi-attribute utility functions to model trade-offs among different decisions will be covered if time permits.

Supplemental readings for the class are primarily provided to not only ease the understanding of the material, but also help you reflect on the writers' thought process. Students are expected to be able to discuss the content of the readings during the class specified.

Learning Objectives

Students will be able to:
  1. Understand the differences between information theory and coding theory
  2. Learn the algorithms for measuring information content in data
  3. Able to use theories proposed by Claude Shannon for error correction in communication channels


The assessment and grading is based on the papers that were assigned in the overall syllabus.

Course Schedule

4/7Coding Theory IStephen F. Bush (Guest Speaker)
4/14Coding Theory IISanjay Goel
4/28Bayesian Decision Theory ISanjay Goel
5/5Bayesian Decision Theory IISanjay Goel
5/12Bayesian Decision Theory IIISanjay Goel


April 7 & 14, 2008
Title: Coding Theory I & II
Details: In this lecture, we address the problem of reliable communication through a noisy channel including error-correcting strategies of repetition and hamming code as well as Shannon's Noisy Channel Coding theorem. We also discuss definitions of entropy, conditional entropy, and mutual information. Zero Error Information Theory and Graph Entropy will also be defined. A brief introduction to quantum information theory and classical information theory and their differences will also be presented. In addition, the lecture will cover Network Coding, Huffman Coding, and Arithmetic Coding. Finally, application to video compression will be presented.
The second lecture will be a continuation of the previous lecture where we look at compression algorithms including block codes, uniquely decodable codes, optimum code lengths and Huffman codes. Kolmogorov Complexity as well as estimation techniques for complexity will also be covered.

  1. The original classic paper by Shannon: The Bell System Technical Journal, Vol. 27, pp. 379-423, 623-656, July, October 1948,
  2. Lecture 3 slides from Dr. Stephen F. Bush from GE Global Research
  3. Dr. Mackay's class notes from MIT

April 28, 2008
Title: Bayesian Decision Theory I
Details: In this lecture, we will discuss the basics of probability theory and solve decision analysis problems using decision tables. Students learn both deterministic and probabilistic approaches for decision analysis as well as value of perfect information.
Readings: Tversky, A and Kahneman, D., The Framing of Decisions and the Psychology of Choice, Science, New Series, Vol. 211, No. 4481. (Jan. 1981), pp. 453-458.

May 5, 2008
Title: Bayesian Decision Theory II
Topics: In this class, we review Bayesian Probability Theory and problems in multistage decision analysis. Students will also learn the use of a decision tree tool for solving complex problems. Simple decision trees will also be covered in the class.
Readings: Howard, R. A., Heathens, Heretics, and Cults: The religious Spectrum of Decision Making, Interfaces 22:6, Nov.-Dec., 1992, pp. 15-27.

May 12, 2008
Title: Bayesian Decision Theory III
Topics: In this lecture, we look at the basics of utility theory and its application to decision analysis. Students will also understand the issues related to information security risk analysis.
Readings: Pat´┐Ż-Cornell, M. E., Dillon, R. L., The Respective Roles of Risk and Decision Analyses in Decision Support, Decision Analysis, Vol. 3, No. 4, December 2006, pp. 220-232

Download syllabus: inf723syllabus.pdf