Is experimental psychology science?

Many times on this board, people claim that human behavior is the domain of science. Yet this is done without qualification. Human behavior is certainly much less predictable than the motion of a ball or a well-controlled chemical reaction.The question is, is experimental psychology science? If it is, how do the available methodoliges compare to those in other "hard" sciences? What are the benefits? What are the pitfalls? And what does it all mean in developing a science of how humans work?This topic was approached in another thread. To summarize on MY terms, Robinrohan calls psychology "pseudo-science." Nwr believes psychology is "soft science", where some genuine science is done, but somehow different than "hard science." This has something to do with the predictability of human behavior, he suggests. Schraf takes umbrage with robinrohan's claim, and proposes a specific article to discuss. Schraf makes no explict claim as to the "type" of science that experimental psychology is, only challenging robinrohan to show that it's pseudoscience.As for myself... it's a lot of material to cover at once. I'd ask the admins to promote this to "Is it Science" and allow me to address the points individually.Ben

Certain human behaviors are more surprising than we'd expect... but:

Still not as predictable as what we can measure in physics.

​

Galvanic skin response; to what degree of accuracy can you measure it for a single person across multiple trials taken over a series of days? Even if you control for well-known factors, such as time of day? Surely the presence or absence is pretty reliable, but I'd equate that to predicting whether a ball is going to drop or not in physics. It's not a very fine-grained statistic.

​

And take something only slightly more complex; a reaction time test. With what accuracy can we predict somebody's reaction time (say a button press to an unpredictable stimulus) across trials across different days? We'd be lucky to be able to predict within 1% error (we'd be lucky to get within 5 or 10% of error, I think), no matter how many factors you try and control.

The overwhelming majority of behaviors are so variable!

​

Take another simple one, one related to your husband's work: working memory span. Take digit span. How consistent are people across trials of digit span tasks? 10% variation? 20% variation?

​

What about playing chess for amateurs? Or decision-making? Go more complex; social intereaction, etc.

​

How about hitting a baseball off of a tee? Or putting a golf ball across a flat green?

I agree there are predictable behaviors, as in "will it happen or not" some of the time; to actually question whether as a sum total, "human behavior" is as predictable as the motion of a ball seems preposterous to me.Ben

It's interesting; can you tell us more about meteorology maybe?I'm curious to know why this distinction seems to be made. Is it actually due to some difference in the systems (total number of factors that control behavior, object-based systems vs. neural-networks, ability to break meterology down to any level of analysis and test it in a lab wheras there are certain types of experiments we cannot run on humans), or simply our perception of the situation (one is physical and caused, the other seems mental and "free").Ben

Hi nwr,I did my best to summarize when pulling things over here; thanks for clarifying.I agree that one of the big factors driving difficulties in cognition is the type of experimentation we use is so limited due to ethical considerations. We have beautiful maps of connectivity principles of macaque visual cortices, but the same work can only be attempted using much weaker experimental setups for humans. We know much more about the rat hippocampus than we know about the human one.It's a real concern; what can actually be accomplished with techniques that:

examine a subject for 1-2 hours, have little control over the internal state of a subject leading up to an experiment, and due to these factors and analysis techniques must use group data? This includes fMRI/PET and ERP/EEG/MEG.

​

And I haven't even gotten into crazy analysis techniques that further decrease the power of the data.

can only be run on diseased patients on cortex you're going to remove (single unit recording in epileptic patients)

where actual incidence / development of the "manipulation" are completely uncontrolled and possibly unknown (case-studies of "natural experiments")

Where does it leave us? Is experimental psychology "hard science"? What is our criterion? For example, what behaviors are predictable to even a 5% accuracy (our arbitrary test for significance) for a single trial on a single subject? Is that a fair criterion?Ben

Schraf,I think the original link you gave is to a website that requires a subscription; I found a reprint copy online that is free:Dissociable neural mechanisms underlying response-based and familiarity-based conflict in working memory Thought that might make things easier to pick up and discuss.Ben

So, now that you've shown that research papers are not accessible to the layman, let's address the real question.What's the operational definition of "emotion" here, and how does that relate to robinrohan's usage of emotion?robinrohan's claim might not be based on logic, but that doesn't mean his intuition is incorrect. Making RR retract is just debate and sophistry. Let's actually address the interesting question.To put my own thoughts out there, so as to be transparent, I've never seen any research paper address in any rigorous way "feelings". Maybe I just haven't read enough yet. I'm pretty sure that the paper below does not address "emotions" in a layman's sense, but a very strict operational definition where emotion=behavior. I'm thinking Damasio's separation between "emotion" and "feeling" here.

It is interesting theoretically to know not just how the average human behaves in a situation, but the actual mechanisms behind it. It is interesting theoretically to know the sources of variation in an individual. So, we want models that "in principle" allow us to predict an individual's behavior. The actual predictions part... not so interesting.Fitt's law isn't very interesting to me because it doesn't explain behavior; it just describes it. It's interesting to people who are trying to describe group behaviors, but not nearly as interesting to someone like me, who is interested in describing the mechanisms that make an individual work. I'm not making my point explicitly enough. The point is about a lab environment. Ballistic motion is not very complex, there are few confounding factors. And you can take it into a laboratory environment, remove the confounds, and given the underlying theory, predict with high precision what the motion will be.The problem with people is twofold; first has to do with confounds. Even figuring out what confounding factors might be present is super hard; we don't have an "external" or "objective" perspective, we tend to get fooled by our folk-psychological intutions. Second, actually controling for those confounds is a real problem. Third, when you take people into a lab, you change the behaviors. People's behavior is dependent on all sorts of contextual cues. And you change that when bringing them into the lab. Even the paper you posted isn't "out in the field." Experiments are usually done in lab environments, where confounding factors can be controlled. I thought I could assume this experimental point. Now I'll make it really explicit.Taking a ball into the lab doesn't fundamentally change the laws that govern the ball. Taking a person into the lab and running "controlled" tasks fundamentally changes the way their mind processes information. A real problem in cognitive science is the validity of extending lab studies to actual human behavior that happens "in the field." There is not nearly the same problem in something like ballistic motion. Tell me how Baddelley's Working memory model accounts for variability. Do you actually find it to be "scientific", or a complete, utter gloss? I find it to lack rigor. Any model that predicts "approximately 7" without specifying the set of conditions that change the result is not a rigrous model of congition.I don't find the answer "he didn't have his cup of coffee" to be the level of rigor I'm looking for. Neither is "he was tired" or any other folk psychology-based "explanation."I would claim that anywhere you see folk psychology, whatever's being investigated is not scientific. Folk psychology pervades cognitive science in motivating the hypotheses, in identifying confounds, in being parts of completely underspecified models, and in being the preferred method of "glossing" results. Granted, it's not this way in every study, every model, or for every researcher. But we're talking about a discipline as a whole.That's why I prefer computational models. They're predictive. And that's why I prefer behavioral measurements, I'm OK with some imaging measurements... sometimes. But they're prone to misuse and misinterpretation. Just another way that the field fails to be really rigorous. But that's best saved for another post, focused on imaging techniques, data processing procedures, and the types of sketchy stuff that can, and does, go on there.Ben

You missed my point. Newton's laws aren't interesting to quantum physicists, except that you want quantum physics, in the limit of large objects, to reduce to newton's laws.This wasn't an argument why it's not science; it's an argument that it doesn't address the level of cognition that I'm personally interested in. I'm interested in describing the behavior of individuals.

It's an indepenedent level of analysis. You don't need ANY understanding of how groups of particle work in order to understand how an individual particle works (i.e. quantum mechanics).Without any isolation of any cognitive system, the result is meaningless. It can only be used at the end, when you have a model of the cognitive system, to see if the behavior of the overall system is consistent with that measure. It doesn't help at all in understanding the underlying system.It's completely a side point. The only point I wanted to make is that Fitt's law doesn't address things at the level that I'm questioning of whether it's science. You can do studies like that all day, come up with mathematical fits, but in the end it doesn't address the mechanisms underlying any behavior. It's too complex.

Hi Chiroptera,I'll try to respond to your post here, like you suggested I think this proposal is really limited in it's ability to be scientific. You can collect data, but how do you relate different sets of data? You have to have an underlying abstraction or model that tells you whether one data set can be related to another; what is that model here?I would suggest that the above method uses folk psychological concepts of things like perception, emotion, feeling. In other words, all your data and the relations between them will sit on top of folk psychology.That's not scientific. Folk psychology is notoriously ... wrong, folky, common-sensy, can't be the foundation of any rigorous method of study.So what I'm saying is, you can collect surveys, and that's great, but to actually relate the data from two different surveys, you have to justify why that's valid to do. And your justification is going to be folk-psychological. You have to have a model for designing your experiments and identifying confounds--your model is going to be folk psychology. Just like when you use logic on top of false premises, you don't get a good argument--if you use scientific methodology on unscientific premises (folk psychology), what you get out isn't science....Maybe once we actually either accept or reject most of folk psychology, we'll have science. Until then, we'll be stuck with protoscience or pseudoscience. It's really not unusual for people in the field not to identify their folk-psychological assumptions. That is a type of pseudoscience.Cognitive studies are "too close to our belt." It's so hard to be objective. You need to be objective to do science. Anything else is protoscience or pseudoscience; you'll make unwarranted assumptions, you'll give underspecified theories, and you'll make leaps in logic that you shouldn't.I gotta start fishing for some examples. All this general talk is not gonna get us far.Ben

I don't know what I said that gave you that impression. I was just wondering if there's a fundamental difference between meteorology and psychology, or if differences are merely in the way we perceive them.One reason I've been trying to describe behind why I think experimental psychology sometimes has elements of pseudoscience is this element of perception; even cognitive scientists often speak in terms of free will and folk psychology. Part of it is simply a matter of language; part of it is actually imprecise thinking, I believe.

Hey Schraf,I know I haven't got back to Zhimbo yet, but there are things going on that don't have to do with "larger error bars." For example,Take the fMRI technology, the data processing techniques, and the derivations of results from it. There are two big problems that I have with many studies:

1. Arbitrary choosing of p-values for "significance" of activation - when it comes to processing fMRI data, suddenly P < 0.05 no longer fits the bill; experimenters choose their own P-values, usually <0.01 or <0.001. How do they choose it? Honestly, it depends on the experimenter. I worked in a lab where P was chosen based on HOW THE RESULT MATCHED WHAT YOU'RE LOOKING FOR. And I know this isn't uncommon; P-values are usually chosen so that activation patterns are localized. This is not science.
   ​
   I couldn't tell if this was happening in the paper that you cited; my sophistication level of data analysis is still not high. ROI analysis was done at P < 0.01.
​
2. The failure to reject the null hypothesis is taken as evidence FOR the null hypothesis - this is one of the fundamental no-no's of null hypothesis testing. But it often happens in fMRI studies--a failure for a region to show significant activation as compared to another condition is taken to mean that the areas were not differentially important in processing.
   ​
   For example, in the study you cited, the lack of significant differential activation of the anterior cingulate cortex was NOT taken to mean that no differential activation was found on this trial, but rather that anterior cingulate cortex was not differentially involved. It seems clear to me that this is using a failure to reject the null hypothesis (a failure to reject that there are no processing differences) as evidence FOR the null hypothesis.
   ​
   That's not scientific where I come from.

Sorry, it's been bothering me for a while; I had to get that off my chest.Ben

Whether we laben psychology "science" or "pseudoscience", it doesn't really matter to me. The REAL purpose is to point out how psychology is different than other sciences, and to show real problems in the field. Let's be intellectually honest and transparent about how things work in the field.That way people don't have a false impression about what's going on in the study of mind....With that said, these are definitely methodological concerns for the "harder" scientific parts of psychology. There are other problems with other imaging methods. Real problems.Then there's theoretical concerns that I was bringing up before, that I haven't fleshed out well with Zhimbo. I'll try to work on that. There are some really important problems in there too. Let's bring them out!Ben

Well, when your methods are shaky... and your theoretical distinctions are based in non-science... and large contributions in your field come from philosophers and linguists...Here's a simple question: why do you think we don't have satisfactory, agreed upon answers for the following BASIC questions: do apes have language? are other animals conscious? do other animals have conceptual knowledge?It's because NONE of those questions use concepts with rigorous definitions. 90% of what we do in psychology is discover how the concepts that we've built are bunk. Conceptual knowledge? Bunk. Language? Bunk. "Human reasoning"? Bunk. "Visual percpetion?" Bunk. None of these things work in the ways our "intuition" tell us they might. None of these are wholistic things; they are commonsense words that obscure multipart, distributed, even unrelated systems.When your fundamental distinctions are made up and do more to obscure the facts than to discover them, it's good to really question what's going on. Like I said, it's not the ultimate goal for me. The important point is the path that we take in getting the answer. Labels don't mean much; it's what constitutes the label that holds the meaning.This message has been edited by Ben, Tuesday, 2005/10/18 07:04 AM

Exactly my point. We can't honestly investigate these things or make sense out of the data because we don't rigorously define what it is that we're investigating! It's frustrating!I just read 10 papers about "human reasoning" where reasoning was never defined, where it was clear that some papers operationalized reasoning in a way that a single-celled bacterium following a chemical gradient could be called "rational" wheras others focused on explicit, rule-based reasoning...and not only that, but the ultimate purpose of most of the papers was to argue the completely unscientific point of "are humans rational?" Since when was this part of judgment science? It all depends on what definitions you put out there, and if you're going to answer such silly questions, you're going to be stuck using the commonsense "definitions"; if you use other definitions and then try to answer that question, you're going to make invalid generalizations.Now we're studying "visual imagery", and the question of whether it's different than visual perception. What is visual imagery? Who knows. Seems to encompass a whole lot of disparate things (remembering, mental manipulations, dreaming, hallucinations, etc). Why are they bound together into a single "scientific" concept? Because the QUALATATIVE experience of these things seem similar. Where is the scientific "objectivity"? If we defined these things behaviorally, we wouldn't be constantly making the "homunculus error".But that's like saying "if we weren't human, we'd...". We are human. That's why I think psychology is more prone to error than other sciences. Because we "work from the inside." We can't get on the outside to look at things objectively. Absolutely. It's not at the same level as other sciences. The point of this thread for me is to point out exactly why that is.The one described here is that we can't get our basic concepts down, and we have trouble even recognizing it because we're "working from the inside."Ben