As I noted earlier in this thread, though it may have been before you began posting and so you may have missed it, my time here (at EvC) is running short and will soon be practically nil. I apologize in advance for this. I am a bit edgy at ending my time here leaving something hanging. But I'll see what I can do...
You appear to be holding 2 opposing positions on statistical analysis simultaneously. Initially you seem to agree that a sample size of 2,000 is sufficient to estimate the value of a quality with a 95% level of confidence. However you then pose the question about sampling across 4 worlds and ask if 3,000 is an adequate sample?
Well I didn't just ask if 3000 was enough for 4 worlds, that was to start a ball rolling. And I think you went on to deal with that ball (quite well too) from a statistical viewpoint.
The appearance you mention is (I think) caused by the fact that I keep getting forced to deal with two different subjects: statistical analysis and the practical reality of sample collection to achieve useful data that fulfills requirements for statistical analysis.
An unbiased sampling method will ensure that the distribution of the sample will reflect the distribution of the total populaton. The probability that this is not the case is given by the level of confidence. So there would be a 5% probability that the sample of 2,000 does not reflect the total population's mean given all possible permutations and combinations of sampling 2,000 out of the total population.
You were correct in your analysis involving "Europa" population and impact. However, you have gone on to reassert two points of assumption. Unbiased sampling method and confidence level.
Sample size can effect sampling bias through inability to get enough unique environments (since populations or "prefs" within populations do sort to some degree). They sort of go hand in hand due to the possibility of regional (or some form of) demographic variability. Thus while what you say may be true about Europa, if many many other environments are missed as well, then they all add up. Thus retesting is a part of a larger unbiased sampling method when small sample pops are used.
For a very weak defense of my point, I am going to point you to
this article on quantitative marketing research. Toward the bottom of the page is a list of errors. Within the section on "random sampling errors" the first two mentioned are:
sample too small
sample not representative
Yes I realize this is weak and somewhat semantic, but I think that is where most of the difference is coming in here.
Lets move on to a more important and more easily discussed topic, which is confidence.
You'll have to start getting more specific about the assumptions which you consider questionable.
In specific most stats, and certainly
confidence levels, are based on an assumption of a
normal distribution. From these wiki articles...
The normal distribution is a convenient model of quantitative phenomena in the natural and behavioral sciences. A variety of psychological test scores and physical phenomena like photon counts have been found to approximately follow a normal distribution. While the underlying causes of these phenomena are often unknown, the use of the normal distribution can be theoretically justified in situations where many small effects are added together into a score or variable that can be observed. The normal distribution also arises in many areas of statistics: for example, the sampling distribution of the mean is approximately normal, even if the distribution of the population the sample is taken from is not normal.
While it is certainly theoretically justifiable, and most often useful, it is NOT always appropriate, even for things that seem to be straightforwardly "gaussian". My point would be that in social pref research, and especially new prefs (initial study) we have not looked at before, it is not an assumption that should be taken for gospel.
Here's some interesting, and pointed exceptions we have found (color to emphasize issues with specific bearing on preferences or highly sectionalized societies)...
The overwhelming biological evidence is that bulk growth processes of living tissue proceed by multiplicative, not additive, increments, and that
therefore measures of body size should at most follow a lognormal rather than normal distribution. Despite common claims of normality, the sizes of plants and animals is approximately lognormal. The evidence and an explanation based on models of growth was first published in the 1932 book Problems of Relative Growth by Julian Huxley.
Differences in size due to sexual dimorphism, or other polymorphisms like the worker/soldier/queen division in social insects, further make the joint distribution of sizes deviate from lognormality...
Because of the exponential nature of interest and inflation, financial indicators such as interest rates, stock values, or commodity prices make good examples of multiplicative behavior. As such, they should not be expected to be normal, but lognormal...
Other examples of variables that are not normally distributed include the lifetimes of humans or mechanical devices. Examples of distributions used in this connection are the exponential distribution (memoryless) and the Weibull distribution.
In general, there is no reason that waiting times should be normal, since they are not directly related to any kind of additive influence.
and from this on the "
fat tail" phenomenon, which exists within normal distributions, because of the difference between theory and reality...
The fat tail is a phenomenon of approximately normal distribution probability distributions that emerge in practice; that is, in the real world. According to the theoretical distribution, events that deviate from the mean by five or more standard deviations ("5-sigma event") are extremely rare, with 10- or more sigma being practically impossible. However, under many applications, such events are more common than expected; 15- or more sigma events have happened in finance, for example. Because the real-world commonality of high-sigma events is much greater than in theory, the distribution is "fatter" at the extremes ("tails") than a truly normal one.
I realize those are not all directly applicable, but they are suggestive. The following is much more important...
A great deal of confusion exists over whether or not IQ test scores and intelligence are normally distributed... As a deliberate result of test construction, IQ scores are always and obviously normally distributed for the majority of the population. Whether intelligence is normally distributed is less clear. The difficulty and number of questions on an IQ test is decided based on which combinations will yield a normal distribution. This does not mean, however, that the information is in any way being misrepresented, or that there is any kind of "true" distribution that is being artificially forced into the shape of a normal curve. Intelligence tests can be constructed to yield any kind of score distribution desired. All true IQ tests have a normal distribution of scores as a result of test design; otherwise IQ scores would be meaningless without knowing what test produced them. Intelligence tests in general, however, can produce any kind of distribution.
For an example of how arbitrary the distribution of intelligence test scores really is, imagine a 20-item multiple-choice test entirely composed of problems that consist mostly of finding the areas of circles. Such a test, if given to a population of high-school students, would likely yield a U-shaped distribution, with the bulk of the scores being very high or very low, instead of a normal curve. If a student understands how to find the area of a circle, he can likely do so repeatedly and with few errors, and thus would get a perfect or high score on the test, whereas a student who has never had geometry lessons would likely get every question wrong, possibly with a few right due to guessing luck. If a test is composed mostly of easy questions, then most of the test-takers will have high scores and very few will have low scores. If a test is composed entirely of questions so easy or so hard that every person gets either a perfect score or a zero, it fails to make any kind of statistical discrimination at all and yields a rectangular distribution. These are just a few examples of the many varieties of distributions that could theoretically be produced by carefully designing intelligence tests.
I realize that that might suggest survey methodology is what is at stake, but again I would view that as a semantics issue. The fact is different tests are valid and useful DESPITE not generating normal distributions.
Even more than strict cognitive ability measurement, preferences may be nonnormal in population, or in many diverse subpopulations. Thus making direct extrapolation quite difficult.
So while people keep mentioning confidence, as I mentioned it is also based on an assumption, which is the normal distribution. In this
standard deviation diagram you can see where confidence levels are derived. When this does not exist in a population, or in a sample population, extrapolations become more difficult. Or maybe I should say the confidence of such extrapolations becomes more difficult.
Is it some sort of political correctness sneaking into social sciences? Why would it be necessary to ensure that a sample includes some/many Hispanics in the sample as long as the methodology does not systematically exclude Hispanics? Why would some quota of christians be necessarily included? The methodology of sample selection should not be biased in including or excluding such subgroups but that does not mean that they MUST be sampled.
I have no idea why you suggested that political correctness has anything to do with it. It is about sociological reality. Societies are made up of cultures and subcultures which may cluster according to different demographics. These cultures and subcultures will effect (or rather reflect and so are based upon) common or generally unified preferences.
If one is going to accurately describe a society's preferences then one will have to get samples from an adequate number of cultures/subcultures. And specifically the kinds that are likely to effect such preferences. One can make a mistake in getting too specific (overestimating the number of useful subpops), and so get repetitive data, but as long as one does not break one's budget that is a lesser error than being too general and missing important subcultures.
Let's run with your example. That a study method will not exclude hispanics, is generally not enough if the question under study may have special relevance to hispanics. Lets say it is an evaluation of trust of hispanics in the US. If by random chance you have pulled up no hispanics, or a lesser number than is in the population, then that would likely result in a skewed extrapolation from sample to population, especially if such attitudes are NOT best described by a bell shape for the population.
Thus it is useful IN SOME CASES to ensure that the subcultures sampled to some degree reflect the population, with randomness involving the individual taken from the subpopulation to form the sample.
If your argument is that a truly random process SHOULD result in ~5 of 100 chosen samples being X, if X is 5% of the total population, my practical question is what real world truly random process are you talking about? That's the "handwaving" I can't seem to get an answer to. Planned sampling methods are about the only way to achieve that goal.
As the number of potentially unique environments grow, so the minimum number of samples grow that one needs to make sure to hit in order to have a truly random test sample of the entire population.
What is important is that the range of possible values is small compared to the sample size and that all possible values could be held by a sample point.
This is also true, and works its way into things like comparative preference surveys (like determining least trusted minority from all minorities).
Okay, well I had to rush through this so I apologize for any shortcomings. And I apologize if I cannot reply to any responses you have, or at least not in any timely way. I suppose in effect I will have to concede to whatever you reply with.
That said, I wholly agree you have demonstrated the theoretical answer to the issues I posed. I'm trying to draw the line between theory and what we must do in practice... or perhaps how people should view conclusions of singular small sample studies in their daily lives. If I have not made my case to your satisfaction, I hope it was at least useful or interesting to go over.
This message has been edited by holmes, 04-10-2006 04:10 PM
holmes
"Some day the piecing together of dissociated knowledge will open up such terrifying vistas of reality, and of our frightful position therein, that we shall either go mad from the revelation or flee from the light into the peace and safety of a new dark age." (Lovecraft)
™ Version 4.2