Introduction to Information

....."Hey, guess what?"What, you don’t know? Of course you don’t (at least not if this is the first time you’ve read this). You are completely uncertain about what I am going to tell you because I have given you no information at all. In other words, your uncertainty is maximum and the information you have is minimum......"Hey, guess what my dog just did?"Ah, now your uncertainty is not as great. Now you know that I wasn’t going to say something about our solar system, or about earthquakes, or about cars, or about a chair, or about a slew of other things. Your uncertainty has decreased because I have given you more information this time: it’s about my dog. Still, you have little idea exactly what I will say......"Hey, my dog just shook hands for the first time!"Now you know exactly what I was going to say. Your uncertainty is now minimum and the information you have is maximum.If you noticed, uncertainty decreases as information increases. The opposite holds true: if your uncertainty increases, the amount of information you have must decrease. Therefore, information can be defined as a reduction in uncertainty.Let’s see another example of uncertainty decreasing as information increases, this time keeping track of the amount of uncertainty (and therefore, information) we have at each step. Suppose I shuffle a normal deck of cards and you pick a card at random. How many yes/no questions must I ask to find out what card you have picked? 52? 26? You might be surprised. Let’s suppose you picked the ace of spades (it really doesn’t matter what card you picked: all cards require at most the same number of questions).Before I ask the first question I have complete uncertainty and no information: your card could be any of 52.Hearts: 2, 3, 4, 5, 6, 7, 8, 9, 10, J, Q, K, A
Diamonds: 2, 3, 4, 5, 6, 7, 8, 9, 10, J, Q, K, A
Clubs: 2, 3, 4, 5, 6, 7, 8, 9, 10, J, Q, K, A
Spades: 2, 3, 4, 5, 6, 7, 8, 9, 10, J, Q, K, ASo the process of eliminating possibilities begins.Question 1: Is your card a red card?Because you have the ace of spades, your answer is, No, which eliminates all red cards from consideration.Clubs: 2, 3, 4, 5, 6, 7, 8, 9, 10, J, Q, K, A
Spades: 2, 3, 4, 5, 6, 7, 8, 9, 10, J, Q, K, AInstead of your card being any one of 52, it is now any one of 26: a reduction in uncertainty due to an increase in information.Question 2: Is the suite of your card clubs?Your answer is, No, which eliminates all clubs from consideration.Spades: 2, 3, 4, 5, 6, 7, 8, 9, 10, J, Q, K, ANow, I know the suit of your card: it must be spades. And, instead of your card being any one of 26, it is now any one of just 13.Question 3: Is your card greater than 7?Your answer is, Yes, which eliminates all spades from 2 to 7 from consideration.Spades: 8, 9, 10, J, Q, K, ANow, instead of your card being any one of 13, there are only 7 possibilities.Question 4: Is your card greater than 10?Your answer is, Yes, which eliminates the 8, 9, and 10 of spades.Spades: J, Q, K, ANow, instead of your card being any one of 8, it is any one of just 4.Question 5: Is your card greater than a Queen?Your answer is, Yes, which eliminates the Jack and Queen of spades.Spades: K, AThis leaves only two possibilities: your card is either the King of spades or the Ace of spades.Question 6: Is your card the King of spades?Your answer is, No, which eliminates the King.Spades: AI now know for sure that your card is the Ace of spades. At this point, after just 6 questions, I have no uncertainty whatsoever and complete information.That’s it: any card can be determined with a maximum of just 6 yes/no questions, if those questions are chosen correctly. The idea here —
and in other information gathering processes — is to eliminate half of the remaining possibilities with each question. That gives you the maximum amount of information, or equivalently, reduces your uncertainty as much as possible, for each question and therefore allows you to zero in on the solution as quickly as possible. For example, randomly guessing cards, Is it the 2 of clubs?, wouldn’t guarantee success until the 52nd question (and that is only if you kept track of all your guesses to avoid repeating any).Measuring Information: BitsIt was stated abpve that the most information is obtained per question by devising questions that cut the number of remaining possibilities in half. If we use some math, we can see why it takes at most six questions to figure out which card is selected.If there are two possibilities — such as red or black — then a single question can determine which one of those two possibilities is the case.1 out of 2 possibilities = 1 questionIf there are four possibilities — such as hearts, diamonds, clubs, or spades — then two questions can determine which one is the case.1 out of 4 possibilities = 2 questionsThis is because the first question cuts it down from 1 out of 4 to 1 out of 2, and then the second question narrows it down from there to just one.The same logic applies to larger numbers. For 32 possibilities:1 out of 32 possibilities = 5 questionsThe fist question cuts it down from 1 out of 32 to 1 out of 16; the second question cuts it down from 1 out of 16 to 1 out of 8; the third question cuts it down from 1 out of 8 to 1 out of 4; and as we saw above, two more questions will narrow it down to just 1.There is a mathematical relationship between the number of halving questions needed and the number of beginning possibilities. In simple form, all we do is multiply 2 times itself as many times as needed to end up with a number that is at least as big as the number of possibilities: each 2 used in the multiplication represents one question.2 = 2
2 x 2 = 4
2 x 2 x 2 = 8
2 x 2 x 2 x 2 = 16
2 x 2 x 2 x 2 x 2 = 32
2 x 2 x 2 x 2 x 2 x 2 = 64So, as the last equation above indicates, if one out of 64 possibilities is selected, you need a maximum of 6 questions to find which one was chosen. And that is why it takes a maximum of 6 questions to figure out what card was selected: 52 is more than 32 (so 5 questions won’t do), but is less than 64 (so 6 questions will do). If you could somehow ask partial questions, then it wouldn’t take a full 6 questions to figure out a card; but you can’t, so it does.When gaining information, each division of possibilities in half corresponds to one bit of information (the term bit here being a technical term referring to base two or log2 numbers, not merely a small piece). In other words, each halving question answered reduces your uncertainty by half and consequently gives you an additional one bit of information.Note that a more mathematical way of expressing the above method of calculating how many bits of information (halving questions) is required to figure something out is as follows:I = -log2(1/N)where I represents information (in bits) and N represents the number of equally likely possibilities. If partial questions (that is, fractional bits) are not allowed under the given circumstances, you need to round the calculated value for I up to the nearest whole number.

I hoped you were going to define the kind of information that doesn't increase due to mutations and other genetic changes.

Could you, please, calculate the "information" content of each sentence and show us that the first has the least information and the last has the most?

I gave you the tools to do that yourself.You just need to estimate how many equally probable possibilities there are in the Universal set and how many of those remain after each statement is given.Of course, these will be ballpark figures only.***********************************************Shoot, I'll give a rough idea of how it could be done......"Hey, guess what?"It could be any of a trillion different things. Let's assume that's a good number, thus setting the cardinality of the Universal set to 10^12. At this point, after so far not narrowing it down at all, if I were to give you the one thing I was talking about I would be handing you I = -log2(1/10^12) bits of information. That comes to 40 bits of info. But this one statement gives you 0 bits as it eliminates no uncertainty......"Hey, guess what my dog just did?"From the Universal set, we could assume that maybe 1,000 things could relate to my dog (these are ballpark figures). Thus, after this more descriptive - more informative - statement, if I were to now give you the one thing I was talking about I would be handing you just I = -log2(1/10^3) bits of information; just 10 bits of info.Therefore, the statement "Hey, guess whay my dog just did?" would contain the amount of information associated with the difference, that is, associated with the reduction in uncertainty. Using the previous ballpark figures:I = 40 bits - 10 bits = 30 bitsThe final statement, which actually does give the one thing I was going to say, eliminating all possibilities but that one, gives an additional 10 bits of information.So under the numerical assumptions used to get these ballpark figures, the statements would give 0, 30, and 10 bits of information respectively.[This message has been edited by DNAunion, 12-13-2003]

quote:

---

I gave you the tools to do that yourself.

---

You may have given people tools, but nothing to work with. Where is the substance?You will notice when forced to use those tools yourself, you had to start by ASSUMING. What's worse is you then ASSUMED in a way that supports the very conclusion you were driving for.I can agree that there are more possibilities in the first statement than in the next, but there was no accurate quantification of the difference. You even say "ball park" figures. What good are estimated ballpark figures when one has no real concept of the ballpark (which is what we are facing with cosmological or evolutionary or biochemical systems)?We know you and I, and we know dog, and we know what dogs can do. Can you tell me anything we do know on that order about complex systems we are trying to model.I can almost buy into your system of evaluating (quantifying) information content in relative numerics based on halves. The problem is you first need an objective way to MEASURE initial numeric values of info content of some kind. We must move beyond assumption, or the whole argument you make is circular.This also does nothing to suggest that information is more than manmade, there is no intrinsic information content gained or lost in a molecule that must bind in a certain way to another under STP conditions.------------------
holmes

quote:

---

You will notice when forced to use those tools yourself, you had to start by ASSUMING. What's worse is you then ASSUMED in a way that supports the very conclusion you were driving for.

---

That's a flawed "counter".The person asked me a question for which accurate numbers cannot be assigned. Then, when I assumed certain values to demonstrate the method you "gripe" that I had to make assumptions, implying that therefore the methodology is flawed. Warped logic. (What's the probability that the Earth will be hit by a mile-wide asteroid within the next 5 years? Oh, you can't give me an accurate number so probability theory is invalid???? See how silly that kind of reasoning is).I CAN give accurate information calculations for things to which accurate numbers can be applied. For example, the main demonstrative example I gave involving cards.

quote:

---

You even say "ball park" figures.

---

Because of the specific question posed to me, which cannot have accurate numbers assigned. It's kind of the old GIGO - if the input into a calculation cannot have accurate numbers assigned, then one shouldn't be at all surprised to learn that the output is not an accurate number. That doesn't mean the processing is invalid.

quote:

---

What good are estimated ballpark figures when one has no real concept of the ballpark (which is what we are facing with cosmological or evolutionary or biochemical systems)?

---

Which is not what my post was about in the first place. It was about information: an introduction to what it is and how it can be calculated.

quote:

---

I can almost buy into your system of evaluating (quantifying) information content in relative numerics based on halves. The problem is you first need an objective way to MEASURE initial numeric values of info content of some kind. We must move beyond assumption, or the whole argument you make is circular.

---

Wrong.1) "My" method of calculating information is correct (a bit simplified, but correct)2) The specific question someone then asked me about cannot have accurate numbers assigned so assumptions were used. That does not make the method flawed.3) I made no argument that could be considered circular. You've read more into my statements than were there, then attacked your own interpretation.

quote:

---

This also does nothing to suggest that information is more than manmade, there is no intrinsic information content gained or lost in a molecule that must bind in a certain way to another under STP conditions.

---

Sure there is, but until I can get you to see that the method I used to calculate information is valid, there's no point going off on other tangents that rely upon this foundation.[This message has been edited by DNAunion, 12-14-2003]

Here's a calculation of information that uses accurate numbers, based on my primary example of calcuating information that I gave originally.

quote:

---

DNAunion: Let’s see another example of uncertainty decreasing as information increases, this time keeping track of the amount of uncertainty (and therefore, information) we have at each step. Suppose I shuffle a normal deck of cards and you pick a card at random. How many yes/no questions must I ask to find out what card you have picked? 52? 26? You might be surprised. Let’s suppose you picked the ace of spades (it really doesn’t matter what card you picked: all cards require at most the same number of questions).

​

Before I ask the first question I have complete uncertainty and no information: your card could be any of 52.

​

Hearts: 2, 3, 4, 5, 6, 7, 8, 9, 10, J, Q, K, A
Diamonds: 2, 3, 4, 5, 6, 7, 8, 9, 10, J, Q, K, A
Clubs: 2, 3, 4, 5, 6, 7, 8, 9, 10, J, Q, K, A
Spades: 2, 3, 4, 5, 6, 7, 8, 9, 10, J, Q, K, A

​

So the process of eliminating possibilities begins.

​

Question 1: Is your card a red card?

​

Because you have the ace of spades, your answer is, No, which eliminates all red cards from consideration.

​

Clubs: 2, 3, 4, 5, 6, 7, 8, 9, 10, J, Q, K, A
Spades: 2, 3, 4, 5, 6, 7, 8, 9, 10, J, Q, K, A

​

Instead of your card being any one of 52, it is now any one of 26: a reduction in uncertainty due to an increase in information.

---

To start with, before any cards were eliminated, the one card chosen could have been any one of 52 equally likely possibilities. Thus, if you had told me the card you selected at this point you would have handed me 5.700439718141 bits of information.I = -log2(1/52) = 5.700439718141 bits of informationThe answser to question 1 eliminated half of the possibilities, giving me 1 bit of information according to what I said about halving uncertainties. That checks out. I’ll show you.After question 1 had been answered, the one card you selected could be any one of just 26; so if you were then to tell me what card you chose you would be handing me 4.700439718141 bits of information.I = -log2(1/26) = 4.700439718141 bits of informationSo that halving question dropped the number of bits of information needed to determine the one card selected from its starting value of 5.700439718141 to 4.700439718141. That is, the answer to that question gave me exactly 1 bit of information, which matches what I said above.[This message has been edited by DNAunion, 12-14-2003]

For those who want a link to a primer on information...http://www.lecb.ncifcrf.gov/...paper/primer/latex/index.html[This message has been edited by DNAunion, 12-14-2003]

quote:

---

The person asked me a question for which accurate numbers cannot be assigned. Then, when I assumed certain values to demonstrate the method you "gripe" that I had to make assumptions, implying that therefore the methodology is flawed.

---

Where to begin...1) The "question" asked was the first example YOU supplied for your methodology. Why did you bring it up if it was a bad example for using your methodology?2) When replying to that "question" you never said it was a bad example because it forced you to assume numbers, and your methodology should only be applied when accurate numbers can be applied. Frankly, if you had said this rather than attempting to show how right your methodology was I wouldn't have said anything.3) Probability theory (especially with regard to asteroids) has little to do with information theory. We understand that probabilities are made based on estimates from human experience (which may not be accurate). Information theory (or its firm adherents) tend to inflate "information" into a real entity... like energy or something.

quote:

---

I made no argument that could be considered circular. You've read more into my statements than were there, then attacked your own interpretation.

---

If you were not attempting to prove the validity of your information theory in addressing the "question" (which once again I must point out was YOUR opening example of increasing information, not his) then I apologize.It really read as if you were suggesting your opening intro was a valid example of increasing info in a sentence, and that you were showing how it could be measured, even in a ballpark sort of way.If you were trying to do that then your argument was circular in its defense in that you front loaded estimates of information content, to show how your methods would show its information increasing (by possibilities decreasing).I don't believe your ballpark estimates were correct either, but that is a whole other subject.

quote:

---

Sure there is, but until I can get you to see that the method I used to calculate information is valid, there's no point going off on other tangents that rely upon this foundation.

---

I can see that if information is broken into "bits", which grows as whatever that content is is doubled (ex... a two layer cake has 2 bits of cake info compared to a 1 layer cake) then your methodology is sound.The point I am making is that methodology and accuracy with regard to info is only for human consumption. It is modelling and to view "information" as some inherent property of an entity being modelled, rather than a clever device for human understanding, is incorrect.I noticed within the link you gave in a later post that this very thing is mentioned...http://www.lecb.ncifcrf.gov/...alls.html#consensus_sequencesHeck, I came to really like that author's work as he skewered Dembski at just about every turn.Information theory gives humans wonderful models to help understand and predict. It does nothing to measure an actual property which exerts change inside or on another entity.------------------
holmes

What information would that be?

quote:

---

Information can be defined as a reduction in uncertainty.

---

That seems reasonable enough, but still doesn't give
a direct link to the data.In your question list, your analysis of which has the most
information relies on (amongst others):- Knowing english.
- Knowing what a dog is.
- Knowing what 'shaking hands' means.
- Knowing what 'shaking hands' means wrt dogs.
- Not having been told already.There is still an interpretive act involved in the 'reduction
of uncertainty', and it is only in very well-structured examples
(like a deck of cards) where the data-information relationship
is very simple, that this uncertainty is fully measurable (even
then one has to know about cards).As soon as our knowledge of the 'system' in question is
incomplete our ability to measure the reduction in uncertainty
evaporates ... leaving it as vague a definition of information
as any other.

That is, indeed, the question. Many Creationists (e.g. Lee Spetner) claim that information cannot be increased by evolutionary mechanisms, and invoke their own version of information theory to "back up" the claim ... but there are severe flaws in all their claims.The kind of information DNAUnion is considering is increased by gene duplication followed by a mutation in one of the genes, so it can't be the kind that never increases by evolutionary mechanisms.

Thanks peter for restating the point I was trying to make in much much clearer wording.------------------
holmes

quote:

---

That is, indeed, the question. Many Creationists (e.g. Lee Spetner) claim that information cannot be increased by evolutionary mechanisms, and invoke their own version of information theory to "back up" the claim ... but there are severe flaws in all their claims.

---

I agree completely.Probably the biggest problem with trying to apply any sort of "information theory" to DNA is one simple thing: Information requires at least one participant for which there is a context that gives the information meaning (otherwise, it's not information). For example, if I were to encode in a DNA strand the Gettysburg Address, it would only be information *in the context* of my encoding scheme; it would be random gibberish in the context of, say, cellular replication. Thus, for DNA to hold information, there must be a context in which it has some sort of meaning. If there is a God who designed life, it has meaning to that God (such as if that God declared life to have meaning, the DNA would carry information about life). However, if there was no God who created life and no inherent meaning to it, then there is no inherent "purpose" to the DNA, and thus, no more information than a snowflake contains; it is just a cycle that continues itself.The only thing "interpreting" our DNA is the universe itself, and its laws of chemistry. Just like the snowflake, however intricate, contain no information (because it is not designed as such), the same holds true with DNA. There is no "meaning" behind it, just selective factors.Now, while we as humans could ascribe meaning to a carrot plant (and declare it "food"), or a rock (and declare it "weapon"), that doesn't mean that these things were designed for that purpose. We are selecting specific objects out of a random dataset, and by the very process of our selection, we encode information to it. We do the same to DNA. Given a strand of DNA, we "encode" meaning into the strand in our own context. We may declare a gene to have the purpose of encouraging mitosis, or for apoptosis. Just like the carrot *was* good for food and the rock *was* good for a weapon, these genes *are* good for what we declare them for, *despite* not being created for these specific purposes. To declare that something was created *for* a purpose, one must already have the worldview that there is some sort of inherent meaning or purpose in life itself; you'll have a hard time getting a naturalist to believe that.------------------
"Illuminant light,
illuminate me."

quote:

---

The person asked me a question for which accurate numbers cannot be assigned. Then, when I assumed certain values to demonstrate the method you "gripe" that I had to make assumptions, implying that therefore the methodology is flawed.

---

quote:

---

Where to begin...

​

1) The "question" asked was the first example YOU supplied for your methodology. Why did you bring it up if it was a bad example for using your methodology?

---

It’s NOT a bad example of my methodology. It shows quite clearly the link between a reduction in uncertainty and an increase in information; it does exactly what it was intended to do — introduce the main ideas in a very simple way. Then, as the MAIN example, I dealt with cards and probability, which involved more math and thus was less suited for use as an opening example.

quote:

---

2) When replying to that "question" you never said it was a bad example because it forced you to assume numbers,

---

It’s not a bad examplethe person who asked the question chose the wrong example of two to ask about in terms of calculating information (at least if one wants numbers other than ballpark figures based on assumptions).The one with the dog shaking hands merely introduces, in a simple manner that even a child could grasp, the link between reduction in uncertainty and gain of informationthat’s it. When I discussed calculating information in my original post, I used the other example that deals with cards and probability — the MAIN example I discussed. If one wanted to see accurate numbers calculated the OBVIOUS choice would have been to ask about the cards example.

quote:

---

and your methodology should only be applied when accurate numbers can be applied.

---

No. First, it’s not my methodology. Second, it works even when accurate numbers cannot be assigned: it just doesn’t work quantitatively.

quote:

---

Frankly, if you had said this rather than attempting to show how right your methodology was I wouldn't have said anything.

---

It’s not my methodology.And I assumed the person who selected the wrong example of two to ask about did so for a reason. I assumed the idea was to see if the method could be applied to a wide range of situations, instead of to just those that involve obvious numerical values. It can be, but the resulting numeric values can’t be expected to be accurate since the input numbers cannot be accurately assigned.

quote:

---

3) Probability theory (especially with regard to asteroids) has little to do with information theory.

---

Wrong, information theory relies upon probability theory.

quote:

---

I made no argument that could be considered circular. You've read more into my statements than were there, then attacked your own interpretation.

---

quote:

---

If you were not attempting to prove the validity of your information theory in addressing the "question" (which once again I must point out was YOUR opening example of increasing information, not his) then I apologize.

---

To reiterate...it’s not my information theory, and it is valid.

quote:

---

It really read as if you were suggesting your opening intro was a valid example of increasing info in a sentence, and that you were showing how it could be measured, even in a ballpark sort of way.

---

It IS a valid example of increasing information linked to a reduction in uncertainty. But I never claimed or implied that it was a good example of how information can be measured quantitatively, and in fact I repeatedly qualified my statements with words such as ballpark and assume, and, also used a different example when discussing calculating information.

quote:

---

If you were trying to do that then your argument was circular in its defense in that you front loaded estimates of information content, to show how your methods would show its information increasing (by possibilities decreasing).

---

It’s not circular at all. It doesn’t matter what reasonable numbers you use for that example. When you know nothing, your uncertainty is maximum and your information is minimum. When I reduce the number of possibilities by giving you information — telling you that it relates specifically to my dog — I have reduced your uncertainty. ANY legitimate numbers you assign to this example will show that uncertainty drops and information rises.Again, this is NOT my methodology: this is straight-forward (though simplified) information theory.