Information and Communication
Measuring information when the probabilities of messages differ.
Now what happens if the different messages have different probabilities of occuring? Suppose that message A is sent with probability 7/10, while messages B, C, and D occur with probability 1/10 each. In this situation, it seems that if message B comes through we've learned more than if messages A comes through. Each message - A through D - allows us to distinguish among four alternatives - but somehow we seem to have learned more when we receive message B than when we receive message A. After all, in the absense of prior knowledge we would have been "expecting" the signal A anyway, so when A does arrive this doesn't come as a particular surprise. Can we capture this somehow in our definition of information?Consider another example. Suppose there are 10 possibile states of the world: A1, A2, A3, A4, A5, A6, A7, B, C, and D. Then if we receive signal B, this allows us to distingush among 10 states of the world. Signals C and D are the same; each of these provides us with log (10) units of information.
If A1 occurs, this also has information log (10) - but if we simply receive the signal A in response to this event, we actually we don't find out whether A1, A2, A3, A4, A5, A6, or A7 actually occurred. Thus we have lost the ability ot distinguish among 7 alternatives; the net amount of inforamtion that we get in then
Log (10) - Log (7) = Log (10/7)
This suggests a measure of the information provided by a signal S that occurs with probability p:
Information(S) = - Log p
Applying this to our example above:
Last modified April 2, 2006