TheWay writes:
Are you not re-defining information from the abstract? If I laid out a 1 1 1 1 (four ones) sequence we could say that it is information because we can add these up to get something new like the number four. So abstractly a sequence can manifest a certain idea. Also, we can attach meaning to each (1) number and have it be represented accordingly.
Information theory, as first conceived by Claude Shannon of Bell Labs, addresses the problem of how to communicate information from sender to receiver in the presence of noise, for example, interference on a telephone landline. There is one particular facet of the definition of "information" in information theory that is of extreme importance, and I'll quote Shannon from his landmark paper,
A Mathematical Theory of Communication:
Shannon writes:
The fundamental problem of communication is that of reproducing at one point either exactly or approximately a message selected at another point. Frequently the messages have meaning; that is they refer to or are correlated according to some system with certain physical or conceptual entities. These semantic aspects of communication are irrelevant to the engineering problem.
In other words, information theory is not concerned with meaning, and in this context it is important not to confuse information with meaning.
Notice that Shannon mentions communicating a message. In order for a sender to communicate with a receiver, both must know the message set. For example, let's say the message set consists of the 26 letters plus some punctuation. When the sender sends a message to the to the receiver, he is sending a letter, say "A" or "G". In order to send that message over a wire the letter must be encoded, and so "A" might be encoded by the bits "01000001" and "G" might be encoded by the bits "01000111". The sender can send the string "AG" by first sending the message "01000001" followed by the message "01000111".
But notice that these numbers "01000001" and "01000111" have no inherent
meaning. They have meaning to the sender and receiver in that one means "A" and the other means "B", but the numbers themselves have no meaning. When the sender sends those numbers down the wire, the wire has no knowledge of the meanings of those numbers, and it doesn't matter to the wire or to the larger problem of communication what they actually mean. It could be '"01000001" if by land and "01000111" if by sea' for all the wire cares.
The amount of information being sent down the wire is the log
2 of the number of messages in the message set. A message set consisting only of "A" and "G" is of size 2, and so log
22 = 1 bit means that it really takes only one bit to encode our message. The encoding would be obvious in this case, you'd define A=0 and G=1, or A=1 and G=0. In other words, if our message set isn't the 26 letters plus punctuation but only two letters then we don't need the longer encodings I mentioned above, which happen to be the ASCII codes for A and G.
You asked how much information is in "1111". Assuming that "1111" is the sequence of bits you're going to send down the wire, and assuming that your message set size is 16 (four bits can take on 16 different values, from "0000" through "1111"), the amount of information in your message is log
216, which is 4 bits, which we knew already since we were sending 4 bits.
It would be much more informative to ask something like, "How much information would I have to send in order to communicate the 20 amino acids." This is straightforward. The log
220 = 4.32 bits. In other words, 4.32 bits is the minimum number of bits necessary to communicate a single message from a message set of size 20. Unfortunately there are no simple equations to tell you the most efficient encoding, but we can easily invent our own that takes sometimes 4 bits and sometimes 5 bits, because we'll assume that a leading 0 means the message consists of 4 bits, while a leading 1 means it consists of 5 bits, e.g.:
0000 Alanine
0001 Arginine
0010 Asparagine
0011 Aspartic acid
0100 Cysteine
0101 Glutamic acid
0110 Glutamine
0111 Glycine
10000 Histidine
10001 Isoleucine
10010 Leucine
10011 Lysine
10100 Methionine
10101 Phenylalanine
10110 Proline
10111 Serine
11000 Threonine
11001 Tryptophan
11010 Tyrosine
11011 Valine
We can tell we haven't achieved the most efficient encoding, because assuming each amino acid is equally likely to be sent, then the average number of bits in each message is 4.6, which is above the 4.32 bits that the Shannon equation tells us is the minimum. Finding the most efficient coding is often very difficult.
The first couple pages of the Shannon paper are worth a read, they are very comprehensible. After that point it gets rather complicated mathematically.
--Percy
™ Version 4.2
