Information Theory, STA865, call# 7358
Instructor: Andrew A. Reilly, Ph. D.
Meeting dates: Monday 4:15 and Tuesday, 5:35 PM.
Room C6.
The increasing use of computers has been accompanied with an explosion of the use of the term information: Infomercials, Management Information Systems (MIS), Cheif Information Officer (CIO), System Information Analysis (SIA), Geographic Information Systems (GIS), Information Science, Information Technology, Informatics (Medical and non), Library Information Science, Information Infastructure, etc. Some of these uses refer to only the existance of and ability to access data. Others clearly refer to either the ability to extract some knowledge from data or measuring the amount of information it contains. This course will focus on statistical information theory which is clearly centered on the latter objectives. Applications of the theory to recent developments in the biological sciences will be examined.
| 1 | Basic algebraic relationships of entropy | Feb 1 | 1-28 |
| 2 | Second Law of Thermodynamics, Sufficient Statisitics | Feb 2 | 29-49 |
| 3 | Equipartition property, Entropy rates of stochastic processes | Feb 8 | 50-77 |
| 4 | Data compression, Huffman and Shannon codes | Feb 9 | 78-100 |
| 5 | Optimal coding | Feb 16 | 101-124 |
| 6 | Data compression examples | Feb 22 | 125-143 |
| 7 | Kolmogorov complexity | Feb 23 | 144-182 |
| 8 | Channel capacity, Hamming codes | Mar 8 | 183-211 |
| 9 | Joint source coding, Differential Entropy | Mar 9 | 212-238 |
| 10 | Gaussian channels | Mar 15 | 239-264 |
| 11 | Max Ent, Spectral Estimation, information theory and statistics | Mar 16 | 266-290 |
| !! | Midterm Exam, lectures 1-10 | Mar 22 | !!!! |
| 12 | Relationship between information theory, statistics, and likelihood | Mar 23 | 291-317 |
| 13 | Lempel-Ziv coding, Cramer-Rao inequality | Mar 30 | 318-335 |
| 14 | Rate distortion definition and rate calculation | Apr 5 | 336-361 |
| 15 | Rate function characterization and channel capacity | Apr 6 | 362-372 |
| 16 | Network information theory | Apr 12 | 374-406 |
| 17 | Encoding correlated sources | Apr 13 | 407-427 |
| 18 | Source coding with side information | Apr 19 | 428-457 |
| 19 | Information Theory and the Stock Market | Apr 20 | 459-481 |
| 20 | Inequalities in information theory. | Apr 26 | 482-509 |
| !! | Second Exam, lectures 11-20 | Apr 27 | !!!! |
| 21 | Molecular Information Theory: | May 3 | Ref 1 |
| 24 | Information content of genetic sequences | May 4 | Ref 2 |
| 25 | Information analysis of binding sites | May 10 | Ref 3 |
| 26 | Channel Capacity of Molecular Machines | May 11 | Ref 4 |
| !! | Final Exam/Project | May 17 | !!!! |
| ! | Commencement | May 23 |
There will be 2 midterms and a final exam. Participants will make one presentation in an information area of their own choosing.
Textbook: Cover, Thomas and Thomas, Joy. 1991. Elements of information Theory. Wiley.