|
Measuring Information
Quantity of information Quantifying information (details)
Information and Uncertainty
Quantifying information (continued)
Entropy
Value of information
Amount versus value
Gould's measure
Other resources
Contact Information
Department of Zoology |
Information and Communication The justification that we have just presented for Log(p) as the appropriate measure for information has been rather ad hoc. Here we present an alternative derivation which may be useful in understanding why information is measured in this way. Roman (1992) presents a more formal derivation along these lines. Suppose we want to send as a message a three letter word: CAT, DOG, BOY, NQG, TXR, etc. For simplicity, we assume that all three letter words - legitimate English words or not - are equally probable as the chosen message. We could send the message in a number of different ways. For example, we could send it one letter at a time. In this case, the total information contained in the message would be Information(Letter 1) + Information(Letter 2) + Information(Letter 3)Since each of the 26 letters is assumed to be equally likely, this sum is simply Information(26 equally likely)+Information(26 equally likey)+Information(26 equally likely). Alternatively, we could have a great big table of individual symbols, each of which would correspond an entire three letter word. To do this, we would need to have 26*26*26 symbols, each of which would be sent with equal probability under the assumptions that we have made. Thus the message sent this way would have Information(Word-symbol) =Information(26*26*26 equally likely)Now here is the crux. It seems that any useful measure of information would have property that a given message contains the same amount of information, regardless of how it was transmitted or coded. That is, any useful measure of information would have the property that Information(26*26*26 equally likely alternatives)=3 * Information (26 equally likely alternatives).Similar equalities would be required for analogous message systems or coding systems. One can show that the Log function has this property, and moreover that it is the only "well-behaved" function that does have this property. We are free to choose the base of the logarithm with which we wish to work.
[ Previous Page ] [ Next Page ]
Last modified April 2, 2006 |