Introduction to Information

....."Hey, guess what?"What, you don’t know? Of course you don’t (at least not if this is the first time you’ve read this). You are completely uncertain about what I am going to tell you because I have given you no information at all. In other words, your uncertainty is maximum and the information you have is minimum......"Hey, guess what my dog just did?"Ah, now your uncertainty is not as great. Now you know that I wasn’t going to say something about our solar system, or about earthquakes, or about cars, or about a chair, or about a slew of other things. Your uncertainty has decreased because I have given you more information this time: it’s about my dog. Still, you have little idea exactly what I will say......"Hey, my dog just shook hands for the first time!"Now you know exactly what I was going to say. Your uncertainty is now minimum and the information you have is maximum.If you noticed, uncertainty decreases as information increases. The opposite holds true: if your uncertainty increases, the amount of information you have must decrease. Therefore, information can be defined as a reduction in uncertainty.Let’s see another example of uncertainty decreasing as information increases, this time keeping track of the amount of uncertainty (and therefore, information) we have at each step. Suppose I shuffle a normal deck of cards and you pick a card at random. How many yes/no questions must I ask to find out what card you have picked? 52? 26? You might be surprised. Let’s suppose you picked the ace of spades (it really doesn’t matter what card you picked: all cards require at most the same number of questions).Before I ask the first question I have complete uncertainty and no information: your card could be any of 52.Hearts: 2, 3, 4, 5, 6, 7, 8, 9, 10, J, Q, K, A
Diamonds: 2, 3, 4, 5, 6, 7, 8, 9, 10, J, Q, K, A
Clubs: 2, 3, 4, 5, 6, 7, 8, 9, 10, J, Q, K, A
Spades: 2, 3, 4, 5, 6, 7, 8, 9, 10, J, Q, K, ASo the process of eliminating possibilities begins.Question 1: Is your card a red card?Because you have the ace of spades, your answer is, No, which eliminates all red cards from consideration.Clubs: 2, 3, 4, 5, 6, 7, 8, 9, 10, J, Q, K, A
Spades: 2, 3, 4, 5, 6, 7, 8, 9, 10, J, Q, K, AInstead of your card being any one of 52, it is now any one of 26: a reduction in uncertainty due to an increase in information.Question 2: Is the suite of your card clubs?Your answer is, No, which eliminates all clubs from consideration.Spades: 2, 3, 4, 5, 6, 7, 8, 9, 10, J, Q, K, ANow, I know the suit of your card: it must be spades. And, instead of your card being any one of 26, it is now any one of just 13.Question 3: Is your card greater than 7?Your answer is, Yes, which eliminates all spades from 2 to 7 from consideration.Spades: 8, 9, 10, J, Q, K, ANow, instead of your card being any one of 13, there are only 7 possibilities.Question 4: Is your card greater than 10?Your answer is, Yes, which eliminates the 8, 9, and 10 of spades.Spades: J, Q, K, ANow, instead of your card being any one of 8, it is any one of just 4.Question 5: Is your card greater than a Queen?Your answer is, Yes, which eliminates the Jack and Queen of spades.Spades: K, AThis leaves only two possibilities: your card is either the King of spades or the Ace of spades.Question 6: Is your card the King of spades?Your answer is, No, which eliminates the King.Spades: AI now know for sure that your card is the Ace of spades. At this point, after just 6 questions, I have no uncertainty whatsoever and complete information.That’s it: any card can be determined with a maximum of just 6 yes/no questions, if those questions are chosen correctly. The idea here —
and in other information gathering processes — is to eliminate half of the remaining possibilities with each question. That gives you the maximum amount of information, or equivalently, reduces your uncertainty as much as possible, for each question and therefore allows you to zero in on the solution as quickly as possible. For example, randomly guessing cards, Is it the 2 of clubs?, wouldn’t guarantee success until the 52nd question (and that is only if you kept track of all your guesses to avoid repeating any).Measuring Information: BitsIt was stated abpve that the most information is obtained per question by devising questions that cut the number of remaining possibilities in half. If we use some math, we can see why it takes at most six questions to figure out which card is selected.If there are two possibilities — such as red or black — then a single question can determine which one of those two possibilities is the case.1 out of 2 possibilities = 1 questionIf there are four possibilities — such as hearts, diamonds, clubs, or spades — then two questions can determine which one is the case.1 out of 4 possibilities = 2 questionsThis is because the first question cuts it down from 1 out of 4 to 1 out of 2, and then the second question narrows it down from there to just one.The same logic applies to larger numbers. For 32 possibilities:1 out of 32 possibilities = 5 questionsThe fist question cuts it down from 1 out of 32 to 1 out of 16; the second question cuts it down from 1 out of 16 to 1 out of 8; the third question cuts it down from 1 out of 8 to 1 out of 4; and as we saw above, two more questions will narrow it down to just 1.There is a mathematical relationship between the number of halving questions needed and the number of beginning possibilities. In simple form, all we do is multiply 2 times itself as many times as needed to end up with a number that is at least as big as the number of possibilities: each 2 used in the multiplication represents one question.2 = 2
2 x 2 = 4
2 x 2 x 2 = 8
2 x 2 x 2 x 2 = 16
2 x 2 x 2 x 2 x 2 = 32
2 x 2 x 2 x 2 x 2 x 2 = 64So, as the last equation above indicates, if one out of 64 possibilities is selected, you need a maximum of 6 questions to find which one was chosen. And that is why it takes a maximum of 6 questions to figure out what card was selected: 52 is more than 32 (so 5 questions won’t do), but is less than 64 (so 6 questions will do). If you could somehow ask partial questions, then it wouldn’t take a full 6 questions to figure out a card; but you can’t, so it does.When gaining information, each division of possibilities in half corresponds to one bit of information (the term bit here being a technical term referring to base two or log2 numbers, not merely a small piece). In other words, each halving question answered reduces your uncertainty by half and consequently gives you an additional one bit of information.Note that a more mathematical way of expressing the above method of calculating how many bits of information (halving questions) is required to figure something out is as follows:I = -log2(1/N)where I represents information (in bits) and N represents the number of equally likely possibilities. If partial questions (that is, fractional bits) are not allowed under the given circumstances, you need to round the calculated value for I up to the nearest whole number.

I gave you the tools to do that yourself.You just need to estimate how many equally probable possibilities there are in the Universal set and how many of those remain after each statement is given.Of course, these will be ballpark figures only.***********************************************Shoot, I'll give a rough idea of how it could be done......"Hey, guess what?"It could be any of a trillion different things. Let's assume that's a good number, thus setting the cardinality of the Universal set to 10^12. At this point, after so far not narrowing it down at all, if I were to give you the one thing I was talking about I would be handing you I = -log2(1/10^12) bits of information. That comes to 40 bits of info. But this one statement gives you 0 bits as it eliminates no uncertainty......"Hey, guess what my dog just did?"From the Universal set, we could assume that maybe 1,000 things could relate to my dog (these are ballpark figures). Thus, after this more descriptive - more informative - statement, if I were to now give you the one thing I was talking about I would be handing you just I = -log2(1/10^3) bits of information; just 10 bits of info.Therefore, the statement "Hey, guess whay my dog just did?" would contain the amount of information associated with the difference, that is, associated with the reduction in uncertainty. Using the previous ballpark figures:I = 40 bits - 10 bits = 30 bitsThe final statement, which actually does give the one thing I was going to say, eliminating all possibilities but that one, gives an additional 10 bits of information.So under the numerical assumptions used to get these ballpark figures, the statements would give 0, 30, and 10 bits of information respectively.[This message has been edited by DNAunion, 12-13-2003]

quote:

---

You will notice when forced to use those tools yourself, you had to start by ASSUMING. What's worse is you then ASSUMED in a way that supports the very conclusion you were driving for.

---

That's a flawed "counter".The person asked me a question for which accurate numbers cannot be assigned. Then, when I assumed certain values to demonstrate the method you "gripe" that I had to make assumptions, implying that therefore the methodology is flawed. Warped logic. (What's the probability that the Earth will be hit by a mile-wide asteroid within the next 5 years? Oh, you can't give me an accurate number so probability theory is invalid???? See how silly that kind of reasoning is).I CAN give accurate information calculations for things to which accurate numbers can be applied. For example, the main demonstrative example I gave involving cards.

quote:

---

You even say "ball park" figures.

---

Because of the specific question posed to me, which cannot have accurate numbers assigned. It's kind of the old GIGO - if the input into a calculation cannot have accurate numbers assigned, then one shouldn't be at all surprised to learn that the output is not an accurate number. That doesn't mean the processing is invalid.

quote:

---

What good are estimated ballpark figures when one has no real concept of the ballpark (which is what we are facing with cosmological or evolutionary or biochemical systems)?

---

Which is not what my post was about in the first place. It was about information: an introduction to what it is and how it can be calculated.

quote:

---

I can almost buy into your system of evaluating (quantifying) information content in relative numerics based on halves. The problem is you first need an objective way to MEASURE initial numeric values of info content of some kind. We must move beyond assumption, or the whole argument you make is circular.

---

Wrong.1) "My" method of calculating information is correct (a bit simplified, but correct)2) The specific question someone then asked me about cannot have accurate numbers assigned so assumptions were used. That does not make the method flawed.3) I made no argument that could be considered circular. You've read more into my statements than were there, then attacked your own interpretation.

quote:

---

This also does nothing to suggest that information is more than manmade, there is no intrinsic information content gained or lost in a molecule that must bind in a certain way to another under STP conditions.

---

Sure there is, but until I can get you to see that the method I used to calculate information is valid, there's no point going off on other tangents that rely upon this foundation.[This message has been edited by DNAunion, 12-14-2003]

Here's a calculation of information that uses accurate numbers, based on my primary example of calcuating information that I gave originally.

quote:

---

DNAunion: Let’s see another example of uncertainty decreasing as information increases, this time keeping track of the amount of uncertainty (and therefore, information) we have at each step. Suppose I shuffle a normal deck of cards and you pick a card at random. How many yes/no questions must I ask to find out what card you have picked? 52? 26? You might be surprised. Let’s suppose you picked the ace of spades (it really doesn’t matter what card you picked: all cards require at most the same number of questions).

​

Before I ask the first question I have complete uncertainty and no information: your card could be any of 52.

​

Hearts: 2, 3, 4, 5, 6, 7, 8, 9, 10, J, Q, K, A
Diamonds: 2, 3, 4, 5, 6, 7, 8, 9, 10, J, Q, K, A
Clubs: 2, 3, 4, 5, 6, 7, 8, 9, 10, J, Q, K, A
Spades: 2, 3, 4, 5, 6, 7, 8, 9, 10, J, Q, K, A

​

So the process of eliminating possibilities begins.

​

Question 1: Is your card a red card?

​

Because you have the ace of spades, your answer is, No, which eliminates all red cards from consideration.

​

Clubs: 2, 3, 4, 5, 6, 7, 8, 9, 10, J, Q, K, A
Spades: 2, 3, 4, 5, 6, 7, 8, 9, 10, J, Q, K, A

​

Instead of your card being any one of 52, it is now any one of 26: a reduction in uncertainty due to an increase in information.

---

To start with, before any cards were eliminated, the one card chosen could have been any one of 52 equally likely possibilities. Thus, if you had told me the card you selected at this point you would have handed me 5.700439718141 bits of information.I = -log2(1/52) = 5.700439718141 bits of informationThe answser to question 1 eliminated half of the possibilities, giving me 1 bit of information according to what I said about halving uncertainties. That checks out. I’ll show you.After question 1 had been answered, the one card you selected could be any one of just 26; so if you were then to tell me what card you chose you would be handing me 4.700439718141 bits of information.I = -log2(1/26) = 4.700439718141 bits of informationSo that halving question dropped the number of bits of information needed to determine the one card selected from its starting value of 5.700439718141 to 4.700439718141. That is, the answer to that question gave me exactly 1 bit of information, which matches what I said above.[This message has been edited by DNAunion, 12-14-2003]

For those who want a link to a primer on information...http://www.lecb.ncifcrf.gov/...paper/primer/latex/index.html[This message has been edited by DNAunion, 12-14-2003]

quote:

---

The person asked me a question for which accurate numbers cannot be assigned. Then, when I assumed certain values to demonstrate the method you "gripe" that I had to make assumptions, implying that therefore the methodology is flawed.

---

quote:

---

Where to begin...

​

1) The "question" asked was the first example YOU supplied for your methodology. Why did you bring it up if it was a bad example for using your methodology?

---

It’s NOT a bad example of my methodology. It shows quite clearly the link between a reduction in uncertainty and an increase in information; it does exactly what it was intended to do — introduce the main ideas in a very simple way. Then, as the MAIN example, I dealt with cards and probability, which involved more math and thus was less suited for use as an opening example.

quote:

---

2) When replying to that "question" you never said it was a bad example because it forced you to assume numbers,

---

It’s not a bad examplethe person who asked the question chose the wrong example of two to ask about in terms of calculating information (at least if one wants numbers other than ballpark figures based on assumptions).The one with the dog shaking hands merely introduces, in a simple manner that even a child could grasp, the link between reduction in uncertainty and gain of informationthat’s it. When I discussed calculating information in my original post, I used the other example that deals with cards and probability — the MAIN example I discussed. If one wanted to see accurate numbers calculated the OBVIOUS choice would have been to ask about the cards example.

quote:

---

and your methodology should only be applied when accurate numbers can be applied.

---

No. First, it’s not my methodology. Second, it works even when accurate numbers cannot be assigned: it just doesn’t work quantitatively.

quote:

---

Frankly, if you had said this rather than attempting to show how right your methodology was I wouldn't have said anything.

---

It’s not my methodology.And I assumed the person who selected the wrong example of two to ask about did so for a reason. I assumed the idea was to see if the method could be applied to a wide range of situations, instead of to just those that involve obvious numerical values. It can be, but the resulting numeric values can’t be expected to be accurate since the input numbers cannot be accurately assigned.

quote:

---

3) Probability theory (especially with regard to asteroids) has little to do with information theory.

---

Wrong, information theory relies upon probability theory.

quote:

---

I made no argument that could be considered circular. You've read more into my statements than were there, then attacked your own interpretation.

---

quote:

---

If you were not attempting to prove the validity of your information theory in addressing the "question" (which once again I must point out was YOUR opening example of increasing information, not his) then I apologize.

---

To reiterate...it’s not my information theory, and it is valid.

quote:

---

It really read as if you were suggesting your opening intro was a valid example of increasing info in a sentence, and that you were showing how it could be measured, even in a ballpark sort of way.

---

It IS a valid example of increasing information linked to a reduction in uncertainty. But I never claimed or implied that it was a good example of how information can be measured quantitatively, and in fact I repeatedly qualified my statements with words such as ballpark and assume, and, also used a different example when discussing calculating information.

quote:

---

If you were trying to do that then your argument was circular in its defense in that you front loaded estimates of information content, to show how your methods would show its information increasing (by possibilities decreasing).

---

It’s not circular at all. It doesn’t matter what reasonable numbers you use for that example. When you know nothing, your uncertainty is maximum and your information is minimum. When I reduce the number of possibilities by giving you information — telling you that it relates specifically to my dog — I have reduced your uncertainty. ANY legitimate numbers you assign to this example will show that uncertainty drops and information rises.Again, this is NOT my methodology: this is straight-forward (though simplified) information theory.

quote:

---

The point I am making is that methodology and accuracy with regard to info is only for human consumption. It is modelling and to view "information" as some inherent property of an entity being modelled, rather than a clever device for human understanding, is incorrect.

​

I noticed within the link you gave in a later post that this very thing is mentioned...

​

http://www.lecb.ncifcrf.gov/...alls.html#consensus_sequences

---

I think you need to explain exactly how that link shows this very thing, as opposed to it being a case of your interpreting the statements the way you want.Getting off track a bit, but...an organism’s DNA does contain information, stored in the sequence of bases (analogous to the way that these sentences store information in the sequence of characters). This cellular information is encoded instructions used by the cells to direct the synthesis of their constituents and to maintain themselves. That information, stored in the DNA of countless organisms. is there regardless whether or not humans exist or read it. If it weren’t there, living cells and organisms could not exist.

quote:

---

I think you need to explain exactly how that link shows this very thing, as opposed to it being a case of your interpreting the statements the way you want.

---

quote:

---

The page that opens is a link of improper uses or mistakes made by those who like to conflate information theory. In specific the link brings you to the issue of...

​

"Confusing a model with reality"

​

It uses a sequence as an example, but the point is the same one I was talking about.

---

You still haven't explained, and in fact, you seem confused.

quote:

---

I think it is up to you to explain how that link does not show what I was saying...

---

No, silly, it is up to YOU to support YOUR statements.

quote:

---

and if it isn't exactly what is the author saying?

---

I'm getting the distinct impression YOU don't know what the author is saying.[This message has been edited by DNAunion, 12-15-2003]

quote:

---

I will note that if you continue reading the page it also addresses some other mistakes you have been making.

---

Another assertion you must support.

quote:

---

an organism’s DNA does contain information, stored in the sequence of bases (analogous to the way that these sentences store information in the sequence of characters). This cellular information is encoded instructions used by the cells to direct the synthesis of their constituents and to maintain themselves. That information, stored in the DNA of countless organisms. is there regardless whether or not humans exist or read it. If it weren’t there, living cells and organisms could not exist.

---

quote:

---

Exactly who is reading these instructions?

---

Not who, what.I understand your desire to put a conscious spin on everything — WHO instead of WHAT - trying to twist my position into something it isn’t. It’s typical — substituting tricks for logic and evidence.Information doesn’t require consciousness to exist, as I have made clear in numerous of my preceding posts.

quote:

---

Cells? They read some "code" and then know what to do?

---

KNOW what to do? Again with the conscious-distorting of my statements. Information doesn’t require consciousness to exist, as I have made clear in numerous of my preceding posts.

quote:

---

These are simply chemicals. Molecules. Cells are collections of chemicals that interact in a specific way with each other.

---

And why are those specific biologically important molecules made instead of others? For example, why do our cells make phophoglucoisomerase, triose phosphate isomerase, pyruvate decarboxyolase, etc., instead of only random, nonfunctional polymers? Are those enzymes molecules that are found in nature outside of cells? Nope. Yet your cells, and my cells, and CrashFrog’s cells, and Rei’s cells, and NosyNed’s cells — in fact, all humans’ cells — make these specific molecule. How? FROM THE INFORMATION STORED IN CELL’S DNA USING BASE SEQUENCES. We all have genes that encode the instructions needed for making those specific molecules: the genes specify the exact order that amino acids need to be hooked up in order to end up with phosophoglucoisomerase, triose phosphate isomerase, pyruvate decarboxyolase, etc.

quote:

---

Do molecular enzymes pass instructions for the other chemicals to speed up?

---

Do enzymes select out of a myriad of molecules they encounter just the one(s) they interact with? Yes. And how do they do that? By their specific three-dimensional conformation. And what determines their conformation? Ultimately, the DNA (genes) that encodes them: they supply the information needed for the cell to synthesize those specific enzymes.

quote:

---

Do not confuse models with reality.

---

I’m not. DNA really contains information.You are the one who seems to have problems with models. You seem to be confusing the use of a model with a belief that only the model is real: that that which is being modeled does not exist. Do you believe that atoms exist? Maybe you don’t: after all, scientists use models to explain them.

quote:

---

Second, it works even when accurate numbers cannot be assigned: it just doesn’t work quantitatively.

---

quote:

---

By which you are suggesting it works qualitatively?

---

By which I mean exactly what I said. It works, just not quantitatively.

quote:

---

How exactly does that work if one cannot assign accurate numbers?

---

By showing the link between a gain of information and a reduction in uncertainty.

quote:

---

Wrong, information theory relies upon probability theory.

---

quote:

---

You'll have to explain how determining the probability of floods or asteroids based on experienced frequencies has anything to do with "information".

---

No, I don’t have to show that at all since that is not what I said.

quote:

---

You may be right that information theory relies upon probability

---

I was correct.

quote:

---

but the reverse is not true

---

Which isn’t what I said.

quote:

---

Information theory methodology is helpless in measuring anything with unknowns.

---

Maybe, but it can still tell us that something contains information, even if we can’t accurately measure it.And with DNA, we can actually MEASURE the information contained in some regions.[This message has been edited by DNAunion, 12-15-2003]

quote:

---

You still haven't explained, and in fact, you seem confused.

---

quote:

---

You have got to be kidding me.

---

Not at all. I’ll explain to you below.

quote:

---

I am unsure what is unclear, especially starting with the title of that section which is confusing models with reality. One is not supposed to confuse a model with an existing reality in nature.

---

Which is not what you originally said. Look again.

quote:

---

The point I am making is that methodology and accuracy with regard to info is only for human consumption. It is modelling and to view "information" as some inherent property of an entity being modelled, rather than a clever device for human understanding, is incorrect.

​

I noticed within the link you gave in a later post that this very thing is mentioned...

​

http://www.lecb.ncifcrf.gov/...alls.html#consensus_sequences

---

Where does the section you linked to state that information is not an inherent property of DNA, but rather is a clever device for human understanding?Where does it say that accuracy with regard to information is only for human consumption?[This message has been edited by DNAunion, 12-15-2003]

quote:

---

If an atom holds information, how special is DNA in this respect. How do you separate the information found in one atom and the information found in a polymer?

---

One reason is probability.Under undirected, nonbiological conditions, the probability that carbon will bond with 4 other atoms or groups of atoms is very high, but the probability that hemoglobin will form is virtually zero. For hemoglobin to exist - not just one copy, but copy after copy, and in cell after cell, and in human after human, and in generation after generation - there has to be some information that stores the instructions needed to produce it. Can you guess where that information is stored?

quote:

---

And, you still haven't shown how the informatin stored in DNA is no different that the information in general chemical reactions.

---

I explained why the information stored in DNA is huge compared to that of carbon bonding with 4 other atoms. Think about the probability of hemoglobin forming again, and that it doesn't occur just one time but rather time after time in cell after cell in human after human in generation after generation, and you should be able to grasp this.Here, let's do a quick BALLPARK calculation (I gotta get back to work so I am throwing this together quickly).What is the probability that a carbon atom will bond to 4 other atoms or groups of atoms? Well, it basically does this all the time except when double or triple bonds are formed. In a hurry, so let's ASSUME that carbon forms 4 bonds 80% of the time. Using the information calculation I = -log2P(e) we get about 0.322 bits of information.What about hemoglobin? Well, its over 500 amino acids long and each of 20 amino acids could be found in any position of a polypeptide thrown together randomly. Let's assume - to be very safe - that any 10 of 20 amino acids could suffice at every position (a HUGE concession) and that hemoglobin is just 500 amino acids long. The probabilty of spontaneous formation, considering only getting an functional sequence, would be about 1 in 2^500, or about 1 in 10^150. I = -log2(1/2^500) = 500 bits of information. That's an astronomically huge difference in information content. And that would be for just ONE gene of thousands in a genome.

quote:

---

He is not specifically stating that information is not an inherent property of anything. What he is saying is that models which are based in information theory should not be confused with representations of what is actually existent in nature.

---

No, that is not waht he is saying.What he appears to be saying in the section you linked to is that people shouldn't confuse a CONSENSUS SEQUENCE - which is an abstraction (model) - with actual BINDING SITES - which are actual sequences.In simple terms, if 100 organisms' sequences for a given binding site are aligned and compared, the consensus sequence is the "average" sequence. That is, each site in the consensus sequence has the nucleotide that occurs most frequently at that site in the 100 samples. But it may be that not one of the 100 organisms actually has that sequence for that binding site. So a consensus sequence is an abstraction - a model.Note that the author does NOT use consensus sequences when calculating the information found in DNA binding sites - he uses actual sequences.I don't see any statement - explicit or implicit - by the author in that section you linked to that indicates information in DNA is only for us humans and doesn't actually exist.[This message has been edited by DNAunion, 12-16-2003]