|
Measuring Information
Quantity of information Quantifying information (details)
Information and Uncertainty
Quantifying information (continued)
Entropy
Value of information
Amount versus value
Gould's measure
Other resources
Contact Information
Department of Zoology |
Information and Communication How can we measure information quantitatively? Let us consider an example. Suppose we receive a message m that could take any of four possible forms, A, B, C, or D, each with equal probabiliy. How much information is associated with the message m? Well, the message allows us to distinguish among four different alternatives (A, B, C, or D), so we might be tempted to say that m conveys 4 units of information. But suppose that we receive two such messages, m1 and m2, one after the other. Intuitively, it would nice to say that this pair of messages gives us twice as much information as did the single message m. But notice that this pair of messages actually allows us to distinguish among not eight but rather sixteen equally likely possibilities. By doubling the number of messages, we have quadrupled the number of alternatives among which we can distinguish:
What to do? Do we have to say that doubling the number of messages more than doubles the amount of information conveyed? In a series of early (1917-1928) papers, Harry Nyquist and R. V. L. Hartley pointed out that if we measure information by the logarithm of the number of alternatives that can be distinugished, the problem is resolved. The message m gives us log(4) units of information. The pair of messages m1 and m2 together give us log(16)=2 log (4) units of information - exactly twice that which we obtained from the single message alone. This justification for Log(n) as the appropriate measure for information has been rather ad hoc; some readers may wish to read a more detailed explanation of why information is measured in this Log(n) form.[ Previous Page ] [ Next Page ]
Last modified April 2, 2006 |