We're going to original sources for InformationTheory. This serves the joint purpose of a) providing some background for UniformInformationDensity, and b) giving us all a good starting point for using InformationTheory in our StatsMiniCourse.
To read
The foundational paper in InformationTheory is [attachment:shannon-48.pdf Shannon '48].
Other sources
An elaboration of Shannon's ideas with a more computational linguistics bent is [http://lccn.loc.gov/63016192 Abramson '63].
- An application of the ideas in the '48 paper is in [attachment:shannon-51.pdf Shannon '51], "Prediction and Entropy of Printed English".
- Shannon's most recent paper on the topic that I could find is [attachment:shannon-84.pdf Shannon '84]. It discusses transmission rates under different noise conditions.