Social Statistics (How many samples are enough/too much to speak about a population?)

Melatonin began a thread on a study suggesting atheists are the most distrusted minority in America. Amongst other potential issues with the study, I made a negative comment regarding the fact that it involved only 2000 subjects to extrapolate the views of America. My concern for this was blown a bit out of proportion, but its apparently an important issue to some, and its intriguing enough for me to follow as an issue. There may be more than one person interested in how statisticians explain that small numbers of people can be used to represent/model the feeilngs of large populations. This is a thread to explain/debate how this is done.I think my opponents and I agree that a small sample can represent a large group of the smaller group is sufficiently randomized such that it reaches a true cross-section of the larger group. Randomization and "true cross-section" depend on understanding demographics so that sampling "points" can be picked correctly.My argument is that size of population and geographic region covered (as well as how populations group within that region) complicate sampling such that one cannot simply assert that 2000 people are enough. Although I admit I have no definitive evidence saying it can't, I suggest that there are enough examples of failures in polling and no studies to support my opponent's argument, so that their position is not so easy to assume.I will start by answering the last points made to me by opponents in the other thread... I want some support for this claim. Other than that it takes more resources, why do more samples of a population make it harder to sample "reasonably well"?Does this mean that for a population of 3 million in one city and 30 billion across three planets, all we need to accurately understand the entire populations, and 4000 would make the study less accurate?To a similar question regarding whether 2000 was enough to model the humans on the planet, the same poster answered... But that is not accurate and what I am trying to argue. It is not an estimate of the mean of all humans, but rather the mean of the samples. If the samples do not create an appropriate cross-section then the mean is not related to "humans" as a whole. This only amplifies my question. How can 3 and 3 million have no significant difference at all in AIDING accuracy of a study? That seems a bit of an exaggeration and I'd like a better explanation. What is an "accurate list"? And then you add that you have to consider demographics to remedy known biases... what does that mean other than that demographics CAN effect a study and so simple blind sampling is not necessarily accurate? I am answering this to straighten out a question, but I don't want to get into Kinsey specifics. The point was raised that 2000 is enough and more would bring in complications. I raised Kinsey as an example that more than 2000 does not start acting as a detriment to study.Along similar lines I asked if a scientist would look down on a study involving 10K as opposed to 2K samples, the answer was... I wasn't talking about "many cases" and "most purposes". I made it very clear that I was talking about getting an accurate picture of social prefs in very large populations (100+mil) over very large areas such that there may be many subcultures. And my question was not what a researcher would choose to do because of the realities of living on a budget constraint.If a scientist sees a study with a 10K sample, versus a 2K sample, is that scientist correct in assuming the 10K is less representative, and its conclusions inherently problematic (or more problematic)? Or would it be looked at as a boon for providing more data, and suggest a greater level of validity?I don't see how any scientist would judge it less accurate, drawing a conclusion that because a researcher managed to get more samples that they don't know statistics and the study must be flawed. Indeed that would seem to be a pretty big logical fallacy.To wrap up I want to pose a theoretical question. Is it correct to believe that whether we are studying a city of 4 million, a continent of 400 million, a world of 4 billion, or multiple worlds of 400 billion, size of sample required to get an "accurate" random sample is never an issue? Thus we can choose 3, 2K, 20K, or 20million and we'd get some accurate representation of the population under study?If this is true why do scientists ever use more than three? Why do they engage in multiple studies, as well as meta-data studies to create larger bases of data?

Hi, holmes.Let me try to present some information from statistical theory that I haven't seen anyone mention yet. Maybe it will shed some light on things, and maybe it will not. But maybe it will suggest some further questions to ask. By the way, I am not a statistician, and I do not use statistics professionally. I am a mathematics instructor, and all I really know about statistics is what I have learn from the intro to statistics course that I took as an undergraduate and have taught as a math teacher.Let us suppose that a population has, say, 200,000,000 people. We want to estimate the average of some quantity, say height. Now these people each has a height that can be measured, and these 200,000,000 numbers can be averaged together to form the mean, call it mu. Now we don't know what value mu actually has; but short of actually measuring the height of each person we can only estimate it. Also, let us assume that the distribution of heights makes a classical bell curve; there is going to be a number called the standard deviation, sigma, which tells us how close to this mean most people's height is. The smaller sigma is, the more people there are who have a height close to this average and the fewer people there are who are very much taller than average and who are very much shorter than average.Suppose that we take a random sample of 2000 people. We actually do measure the heights of each of these people. We can calculate the average height, x, and the standard deviation, s. Now we use this value x, which really only pertains to this sample, as an estimate of mu, which is the number that we really want.Of course, the problem is that, even if the sample is totally random, without any biases, just happened to have chosen 2000 people who are very tall. How do we determine how likely it is that our sample of 2000 is unrepresentative of the whole population?Let us consider every possible way of choosing 2000 people from the 200,000,000. There are a lot of ways of choosing 2000 people from 200,000,000 (in fact, my calculator cannot even calculate this). So we have a lot of possible values for x (the average for a given sample of 2000). However, if we assume a classical bell curve, it is possible to calculate the average of all the x's. This average value for x turns out to be equal to mu. In other words, the "average" 2000 person sample will have an average heigth that is equal to the average height of the population. How spread out are these averages? Are most 2000 person samples near this value mu, or are there many 2000 person samples that are very, very unrepresentative? This, too, can be calculated exactly (assuming a standard bell curve). The standard deviation (the spread) of the x's from the average mu is sigma divided by the square root of 2000. In other words, the more people in your sample, the less likely the average will be very far from the true mean. And this does not depend at all on the size of the actual population. One can also calculate the average value and standard deviation of all the possible values of s.In theory, one has the true mean, mu, and wants to know mu plus or minus what will contain 95% of all possible x's. This what will depend on the value of sigma divided by the square root of 2000. Depending what sigma is, this what may be, say, 3%. Thus, 95% of all possible samples of 2000 people will give an average, x, that is within mu plus or minus 3%.Of course one does not know the value of sigma, either. Instead we have a value x and a value s for a particular sample of 2000 people. We estimate the average height for the 200,000,000 people is x. How close are we? We estimate sigma to be close to s, and so we use s in place of sigma.In effect, when the results of a survey are report plus or minus 3%, what is being said is that we are estimating that only 5% of all possible 2000 person samples will give an average more than 3% from the true average. So, roughly, we expect that the true mean is within 3% of the average value, x, that we obtain from the sample because we assume that only 5% of possible 2000 person samples will have an average further from the true average.Hopefully, this was more or less clear. If it is, maybe it will suggest a fruitful direction of questions and comments.Edited the penultimate paragraph for clarity.Added by edit:
I'm currently looking at a statistics text book. There is a cartoon involving two anthropomorphized bears. One bear says, "Statisticians are really nice people!" The other replies, "I know! Even the mean statistician is nice!"Edited again to correct a serious typo.This message has been edited by Chiroptera, 02-Apr-2006 12:01 AM

---

"Religion is the best business to be in. It's the only one where the customers blame themselves for product failure."
-- Ellis Weiner (quoted on the NAiG message board)

Very well explained.I'll just add a comment or two.In the case of a typical survey, we are trying to find the proportion p of people who have a particular opinion (say, that atheists cannot be trusted). Thus we are taking 2000 random samples. The randomness (actually the independence of the sampling) assures that there is a probability p for that opinion for each sample. The overall distribution for 2000 such choices is a binomial distribution. The binomial distribution for large n is well approximated by the normal distribution. If we are counting numbers, then the theoretical mean is np and the theoretical standard deviation is SQRT(np(1-p)). Divide these by n if you are looking at proportions.For n=2000, and with the worst case assumption p=0.5, the standard deviation for the proportion having the particular opinion is smaller than .012. Checking a normal distribution table shows that 95% of the time, the result will be within 1.96 standard deviations, so your answer will be within around 2.3 percentage points of the correct proportion.

Thanks chiro, I think your post was very helpful in that it gave us a common set of terms and explanations about stats to work with. I think it can guide discussion in a positive direction.That said, I just spent well over an hour writing a reply using what you set out and lost it all right before posting. I actually feel sick now (because of that) and do not want to rewrite it.I doubt I will be in the mood for the rest of the day. I really really hate when that happens. Especially on detailed discussions which involve technical and boring, even if important, subjects. Grrrrrrrr.I will reply to this Tuesday at the latest (Monday I suspect I might be too busy). Right now I am going outside to avoid punching in my screen.

---

holmes
"Some day the piecing together of dissociated knowledge will open up such terrifying vistas of reality, and of our frightful position therein, that we shall either go mad from the revelation or flee from the light into the peace and safety of a new dark age." (Lovecraft)

Ugh. Sorry to hear that, holmes. I've lost enough long posts that I understand how you feel.Anyway, I just want to add a little bit more to what I've already said to show how statistics are reported. nwr points out that there are some differences in analyzing results that take only two values (like heads/tails in flipping a coin, and yes/no in an opinion survey) but the underlying concepts are the same, so I will stick to finding average height because I think that it is conceptually a little bit easier.Suppose that we have a population, and we are intersted in knowing what the average height is. I won't specify how large the population is; it turns out to be irrelevant to the analysis that I am about to do. I will assume that the distribution of heights fits a bell curve; it turns out that this isn't even necessary, either, but that can be discussed as well.Suppose that, even though we don't know it, the mean (average) height is 66 inches (sorry non-Americans), and the standard deviation (a measure of how wide or narrow the bell curve is) is 10 inches. This is now all the information that I need. I can now make the following statements:Suppose that we randomly choose a sample of 500 individuals. Random means that Ahmud, a Muslim nurse living in a suburb of Chicago, has as much chance of being chosen for this survey as Nancy, a Baptist housewife living in rural New Mexico. I can calculate the mean height and standard deviation of this sample, which will, in general, be different from the mean and standard deviation of the entire population. (I won't exlain how these calculations are made, but someone can check them if they want. I have already found a mistake in my work!)90% of all the possible 500 person samples will have an average height between 65.26 inches and 66.74 inches.
95% of all possible 500 person samples will have an average height between 65.12 inches and 66.88 inches.
99% of all possible 500 person samples will have an average height between 64.85 inches and 67.15 inches.Now suppose someone else chooses a random sample of 2000 persons.90% of all possible 2000 person samples will have an average height between 65.63 inches and 66.37 inches.
95% of all possible 2000 person samples will have an average height between 65.56 inches and 66.44 inches.
99% of all possible 2000 person samples will have an average height between 65.42 inches and 66.58 inches.Note two things. The larger the sample size produces a narrower range in which a "typical" sample will fall. This is why larger sample sizes are nice: they allow us to make more precise estimates of the true population average.The other thing is the "confidence", the numbers 90%, 95%, and 99%. These numbers tell us what we mean by a "typical" sample; a sample is "typical" if it is one of the 90% that are near true population mean, or 95%, or 99%. The higher the confidence, the less likely our sample is atypical, but at the cost of less precision, that is the range of possible values is wider.So the researcher will first of all choose a confidence level. I believe that 95% is typical, although 90% and 99% (and other values) are used as well.Suppose that the researcher chooses a random sample of 500 individuals, and she measures that this sample has an average height of 66.25 inches and a standard deviation of 9.5 inches. Since she doesn't know the population standard deviation she will use her sample standard deviation as an estimate and calculate that 95% of the samples will fall between plus or minus 0.83 inches of the true mean. She will then claim that her sample must be within 0.83 inches of the true mean. She will report:Her results show that the average height of the population is between 65.43 inches and 67.08 inches with 95% confidence.Notice that she reports the confidence level. This informs us that, even if she correctly choose a completely random sample, there is still a 1 in 20 chance that this sample actually lies further than 0.83 inches from the true mean.I hope that this example was also informative, and will give people a common understanding how statistics work in real life. And, as nwr points out, things are a little bit different with different problems, but the essential ideas are the same. (I also hope that this is actually correct.)Edited to correct a minor typo.This message has been edited by Chiroptera, 02-Apr-2006 05:06 PM

---

"Religion is the best business to be in. It's the only one where the customers blame themselves for product failure."
-- Ellis Weiner (quoted on the NAiG message board)

Okay, I walked it off in the windy streets. But this may be a little briefer than I had originally intended. Let me start by summarizing what you said...Our intent is to measure a quality of a population. The "true average of the population" we can call mu. Sigma describes how the population varies from that average.Since it is difficult or impractical to measure an entire population in total, we sample a portion of the population. The "true average of the sample" we can call X, and the variation of the sampled individuals from X is described by s.(First assumption issues... not all sociological issues, like preference or comparative preference, can be measured in the same way as some objective quality like "height". The statistics above are based on an assumption of bell shape curves, which may not be what is found or would be applicable. In fact any curve, or mean, may be an artifact of forcing people to choose in a wholly artificial manner.Here is a very quick example: If asked if one would prefer to work with an christian, a jew, a muslim, an atheist, a gay, a lesbian, or an immigrant, the resulting answer is not necessarily cleanly able to be put in a graph as one would with "height". It does not measure an objective quality and does not actually reflect an actual "range". I'm not going to address complications added if the question is asked in the form of different rankings. The point I am getting at is that the assumption social questions can have significant results in terms of averages and standard deviations is not exactly right.)Okay so what statistics does is makes a series of assumptions such that we can speak about mu based on x. They are held to have a relationship to each other, and the stronger the relationship is (or can be assumed) the more confident we are that what we found in the sample can be said to be true for the total population. Likewise sigma and s have a relationship. Right, as N (the number of samples) increases the closer x will be to mu, regardless of population. Thus more samples does not inherently hurt a study.
And as you relate (or seem to suggest), since we don't often have high sample N there could be a number of possible x's, and we want to know the "confidence" we have in any particular x that it does not deviate dramatically from mu. This is related through sigma. Now is where we really start hitting some problems, especially when the sample size is low compared to the population size and we must take into account the effects of population densities over large areas. It is not easy to assert sigma is close to s, when we don't have a collection of all in one hat and are simply picking samples.
Let's use the example of 2K sample for a nation of 200Mil. That is one person per 100K people. Realistically one can have many separate views within that number of people. Lets say we take 3 people for 300K (using three to at least create a baseline x and s). That is making a rather large assumption to suggest the random 3 in that body, which may actually involve 3-10 different communities results in an s which matches the true sigma of the entire population.
Given sociological realities demographics are important to creating "accurate" random sampling, and thus may result in more than 2000 samples to get a good x with an s which does reflect sigma.
Let me grow the example, if one has a nation of 20K cities each with 10K population and are known to have unique outlooks (let's say each is based around a unique religion though there might be some variation within any city's population) and relationships with each other, a 2K sample will not hit each of the unique environments and thus come close to getting at any average for the nation. You will only have hit (at best) 1/10th of the cities. So how can you possibly know (or be confident from that one study) if city A is least preferred among the people of all 20 cities?
It should be remembered that I was not saying that there was no value at all to such a study. My suggestion was that as populations vastly outnumber the sample, and demographics play a part in effecting the proposed measured quality, less can be taken as seriously from any one study.
My description was of having a rough sketch done. With more studies one may get more sketches and as they support each other a clearer image emerges. But one isolated initial study, is just a very rough sketch and may not be adequately representative of an entire nation. In this specific case, not enough to announce that atheists are the most distrusted minority of America.

---

holmes
"Some day the piecing together of dissociated knowledge will open up such terrifying vistas of reality, and of our frightful position therein, that we shall either go mad from the revelation or flee from the light into the peace and safety of a new dark age." (Lovecraft)

Argh, and now a cross posting! Okay, check to see if my post#6 makes sense.

---

holmes
"Some day the piecing together of dissociated knowledge will open up such terrifying vistas of reality, and of our frightful position therein, that we shall either go mad from the revelation or flee from the light into the peace and safety of a new dark age." (Lovecraft)

Heh. I'll wait and see if the discussion of confidence levels takes care of one of your objections. Last night I thought of another example that might be relevant to some of your other objections; if it does then I will agree that your objection has some validity. But I'll wait until you have a chance to read my second post.

---

"Religion is the best business to be in. It's the only one where the customers blame themselves for product failure."
-- Ellis Weiner (quoted on the NAiG message board)

Okay, I read it. Again very interesting and useful description of how stats are done.Unfortunately the problems remain. Things like height are naturally bounded and have an actual "range" with some sort of meaning. The same is not true of social phenomena.You mentioned "yes/no" answers on surveys, but it can be much more dynamic than that (artificial ranges which obviously have a minimum built in sdev which does not mean the same thing as that around a height).And this is worse when one is trying to get to comparisons between multiple items. That when asked I will put that I prefer gays to atheists, does not suggest that in fact there is some spectrum I am on with someone who likes gays equal to atheists but will on forced selection choose gays last and someone that hates both but will choose one last and another that chooses the other last. There will be a quantification of some kind, but the results do not have the same concept of a s dev, because it isn't a natural range.In this case the subject was getting at which was the most distrusted minority. Imagine the vast number of minorities and numbers of minority opinions about other minorities comparative to all other minorities within the enormous US population. The concept of a bell curve or binomial distribution starts to not be adequate as an assumption.And if the minorities tend to cluster together, then being able to hit as many clusters as possible becomes a very real issue.Let me know if this is starting to make sense.This message has been edited by holmes, 04-02-2006 07:05 PM

---

holmes
"Some day the piecing together of dissociated knowledge will open up such terrifying vistas of reality, and of our frightful position therein, that we shall either go mad from the revelation or flee from the light into the peace and safety of a new dark age." (Lovecraft)

It is important and there is a discussion on these issues (certainly in mathematical psychology). Attitude scales are not true interval/ratio data such as height, distance etc, rather ordinal data. Some believe such data is not suitable for a parametric analysis such as ANOVA, others see no issue. So, we just use it and let the statisticians bash it out.This message has been edited by melatonin, 04-02-2006 01:06 PM

The answer given is completely objective. It consists of marks in some of 7 boxes.What the answer means is subjective, because the terms are vague and different people mean different things, and perhaps mean something different on weekdays from what they mean on weekends. However, the actual answer give is objective.The statistical sampling claims only to measure the objective answers given. It does not pretend to get at the subjective meaning behind those answers. The press, when reporting such surveys, might claim that they address the subjective issues, but that's a problem with the media and not with the sampling methodology. Sorry, but that's where you are confused. We can find crude upper bounds on sigma, even before we try to estimate s. And those crude upper bounds are sufficient to determine confidence intervals. You ask 7 questions, with a yes/no answer for each. There are only 128 possible combinations of answer. So where does this 20K unique outlooks come from?The statistical sampling is estimating a national average. If you want to find an average for one of those 20K cities, then you can only use the sampling done within that city. You won't get good results on a city by city breakdown with only a 2000 nation wide sample. But you will still do pretty well on estimating the national average.

Hi, holmes.I think I do understand what you are saying here, and I agree with them. My post was mainly to answer the question whether the choice of sample size should depend on the size of the entire population; it turns out that it doesn't (at least, not usually -- one might be able to think of "pathological" situations where it does, I don't know).Now we can discuss how meaningful the results are. You have express some concern about the possibilities of different communities which are very distinct from one another but internally homogenous. Let me explain the example I mentioned, and tell me if this is the sort of thing you are discussing.Suppose, in this totally artificial example, that 60% of the people live in cities and 40% live in rural areas. Suppose all city dwellers are exactly 60 inches tall, and all rural dwellers are exactly 72 inches tall. It's easy to calculate the statistics in this case: the mean height is 64.8 inches, and the standard deviation is 5.9 inches.Now if a researcher chooses a random sample of 1000 individuals, then roughtly 600 should be city dwellers and roughly 400 should be rural dwellers. Of course, by unlucky chance she may have chosen 700 rural dwellers, but the confidence level tells us how likely this sort of thing is. So if she reports that the mean height is between 64.5 inches and 64.9 inches at the 95% confidence level, then I will trust her results. Of course, she might report the mean as between 65.1 inches and 65.5 inches, which is wrong, but she has already told me that there is a 5% chance that her results are bogus. But, as in all scientific studies, repitition and reproducibility are important.The actual problem is that, in this case, the true mean, 64.8 inches, is pretty meaningless. No one is 64.8 inches tall. In height, 5 inches is usually considered quite a large amount, and so no one is even close to 64.8 inches.Similarly, if all city dwellers are raving Democrats and rate GWB a 0 on a scale of 1 to 10, while all rural dwellers are rabid Rebublicans who rate GWB a 10, then reporting that GWB's approval rating is 4.0 may not be all that informative. And of course, the other types of things you mentioned are also important.There is also a more subtle, but fun, paradox that can come about. I forget the name of it, but it is discussed in intro to stats texts. Suppose that we are considering two airlines, A and B. Because the weather in Seattle is often bad, flights run by A are typically delayed by 1 hour while flights run by B are typically delayed by 1.5 hours. In Phoenix, where the weather is better, A's flights are only delayed by 5 minutes on average, while B's flights are delayed by 15 minutes on average. So clearly, if you had the choice, you should choose to fly with airline A, at least if potential delays are important to you.But suppose that Seattle is A's hub, so that A has 100 flights out of Seattle and only 10 out of Phoenix. Meanwhile, B has 15 flights out of Seattle, but 200 out of Phoenix. So the overall average delay for airline A is 55 minutes, while the overall average delay for B is 7 minutes. So if you are not careful, you would conclude that airline B is more timely.The moral of the story is that you can perform statistics completely correctly, and get very precise results that turn out to be meaningless. An intro to stats course, if done properly, doesn't just teach how to do some of the calculations, but also teaches to beware of sampling bias, loaded questions, and results that may not mean a whole lot.This is why I usually reserve a certain amount of skepticism of any statistical study or poll that I read about until I have more information on the methods and questions that were used. And this is why, when I taught the intro to stats class, I tried to emphasize that one should not simply accept statistics at face value. Even when done correctly, it is possible that the results do not justify the conclusions that are claimed.In scientific studies, like medical research, I tend to place more faith in the results since I assume that the researchers are experienced in the use of statistics and understand the potential problems. However, opinion polls, as you point out, are sensitive to what is being asked and how it is being asked. In all cases, it is always preferrable to have several different studies done on the same question (and using different methods, if possible) so that the results can be compared.

---

"Religion is the best business to be in. It's the only one where the customers blame themselves for product failure."
-- Ellis Weiner (quoted on the NAiG message board)

it's not about the number of samples... it's about the randomness of the sampling. and there is a formula to determine generalizability. it's not a question. take research methods and stop taking up board space. f'real.

I keep having doubts... were you serious? At first I assumed so, but now I'm not so sure.This message has been edited by holmes, 04-03-2006 12:53 PM

---

holmes
"Some day the piecing together of dissociated knowledge will open up such terrifying vistas of reality, and of our frightful position therein, that we shall either go mad from the revelation or flee from the light into the peace and safety of a new dark age." (Lovecraft)

You misunderstood my statement. I said that the measured quality is not an objective quality and does not reflect an actual "range".Height is an objective quality, while preference may be objectively real for a person it is not some objective property that fits in an order of measurements with others.2 meters v 1.5 meters is not the same as I like Xians v I hate Jews, or I rank atheists 4 and gays 2 on people I'd like to have my kids marry. There are statistical assumptions made regarding the relation between x and mu which do not make as much sense when the property being measured is not a true (objectively consistent) range. Don't just say it, explain how this is done. I think I wasn't clear. The second example I used was not the same as the first. I was suggesting that there were 20K cities, each with 10K residents. Each city has a unique relationship with each other city. The question was to determine what the most distrusted city was among the population of that nation.Now for purposes of this hypothetical we will assume a person can get through a survey ranking their prefs for 20K cities. There is no chance there is only 128 possible combinations.But in any case we can even reduce the number of items they are trying to evaluate. By the nature of the example, with highly solid opinions within any one city, differing between city, that means one misses a large section of population pref by missing any one city.1/10th of all cities will not be enough to get an "accurate" random sample. Especially as you are stuck with only 1 person (at best) from any city and you could (though low probability) hit an outlier for that city. To be accurate to any city then would really require 3, but then even smaller numbers of cities will be tested.I guess you can look at this as my getting at densities within densities. You already admit that population density can make a difference in choosing how to sample, but that may be different that preferential density. You'd need to know both in advance in order to properly randomly sample.This doesn't play as big a factor on smaller scales, but it does start playing as a factor when one must make a very large commitment for each sample taken, particularly in the case of social stats.I will add that in both the first and the second example you assumed yes/no, and as I have pointed out they can also included rankings which drastically increases possible combinations of answers.

---

holmes
"Some day the piecing together of dissociated knowledge will open up such terrifying vistas of reality, and of our frightful position therein, that we shall either go mad from the revelation or flee from the light into the peace and safety of a new dark age." (Lovecraft)