The Research Methodology Thread

Arouet

Member
This is a continuation of the thread I started in the old forum: http://forum.mind-energy.net/skepti...logy-high-quality-vs-low-quality-studies.html

The idea is that it is a thread to discussion research methodology issues in general. Here's my OP from that thread:

Arouet;167112 said:
This is a thread to discuss research methodology in general.

What is meant by a high quality study vs. a low quality study?

What is not meant by a high quality study vs. a low quality study?

Why does it matter?

How do we figure it out?

Are there accepted definitions and standards?

What is meant by methodological bias and imprecision?

What are the different biases and should they be dealt with?

And I'm sure many more.

Though I'm the OP and as some know I've been pushing for this discussion for quite a while, I'm not claiming to be an expert or even all that well-informed on these questions. This thread is as much for my benefit as I hope it will be for the other forum members. I'm learning this stuff as I go.

I'll start off by posting what seems to be a good general overview of these issues:

GrADE: an emerging consensus on rating quality of evidence and strength of recommendations
My hope is that we can come to some better understanding of these issues and perhaps some informal consensus about what is important about these issues and then in a separate thread analyse some parapsychological studies according to this kind of criteria.

I'd like to keep the focus on research methodology in general and not bring in specific debates about whether this or that parapsychological study met a particular criteria or not.

Let's also try and keep the culture war out of this thread if possible.
 
I'd say, in a nutshell, that high-quality studies are designed to investigate as many confounding factors and alternative interpretations of the new and previous findings as "reasonably" possible. ("Reasonable" probably being in the eye of the beholder :) )

Low-quality studies tend to ignore both confounding factors and alternative interpretations of the data.

At the end of the day, a lot of this boils down to bias and ignorance; nobody knows everything, and everyone is biased. We all evaluate data based on a series of "operating assumptions". Someone who, for whatever reason, "believes in" orgone and luminiferous aether will, rightly by their "operating assumptions", see most modern "orthodox" science as flawed and low-quality, because the heterodox interpretations (and potential confounding factors the heterodox assumptions would imply exist) are never even considered.
 
Thanks for bumping this thread. To be clear, we're not talking in this thread about personal biases (ie: prejudice, and prexisting beliefs) - but rather methodological ones.

Here's the definition of bias that Cochrane uses: http://handbook.cochrane.org/

8.2.1 ‘Bias’ and ‘risk of bias’

A bias is a systematic error, or deviation from the truth, in results or inferences. Biases can operate in either direction: different biases can lead to underestimation or overestimation of the true intervention effect. Biases can vary in magnitude: some are small (and trivial compared with the observed effect) and some are substantial (so that an apparent finding may be entirely due to bias). Even a particular source of bias may vary in direction: bias due to a particular design flaw (e.g. lack of allocation concealment) may lead to underestimation of an effect in one study but overestimation in another study. It is usually impossible to know to what extent biases have affected the results of a particular study, although there is good empirical evidence that particular flaws in the design, conduct and analysis of randomized clinical trials lead to bias (see Section 8.2.3). Because the results of a study may in fact be unbiased despite a methodological flaw, it is more appropriate to consider risk of bias.

As I wrote in the other forum:

The important concepts I get from this is that:

a) we are talking about systematic error - not the motives or prejudices or pre-conceived notions of experimenter

b) because it is impossible to know how much actual bias got into the results, what we are concerned with is really "risk of bias." That is: its possible, even with methodological flaws, that no bias actually got into the study. It is also possible, even with few methodological flaws, that plenty of error got in. The problem is we're never going to know. However, while certainty is never assured, our confidence in the results rises with fewer risks of bias being present, and lowers with greater risks of bias being present.
 
How do you address the realities of error, systematic or otherwise, without addressing bias? BTW, I don't mean to demonize bias. I don't consider bias either avoidable or even necessarily always "bad". Bias is a thing which won't alter the raw numbers themselves. Not upon super-careful examination of the research, at least, where you trace citations back to all their origins.

I understand that you're talking about methodological bias, but those stem from more "personal" biases, don't they? For example, let's consider the skewing of numbers that happens with active vs passive surveillance of reported outcomes. If a study is based on passive surveillance, and the authors don't address the weakness of the surveillance system, that's personal bias (based on their operating assumptions) manifesting as a bias in the method. (I'm assuming the effect is not an intentional error better described as "fraud.")
 
How do you address the realities of error, systematic or otherwise, without addressing bias?

I agree with you that everyone has personal biases. One of the goals this thread is to discuss how different research methodology can raise or lower the risk of those personal biases impacting on the results of the study.

BTW, I don't mean to demonize bias. I don't consider bias either avoidable or even necessarily always "bad". Bias is a thing which won't alter the raw numbers themselves. Not upon super-careful examination of the research, at least, where you trace citations back to all their origins.

Well, I'm not sure that's quite true. For example there are researcher degrees of freedom that can lead researchers to - quite innocently - manipulate the numbers in a certain direction. Different research methodologies can increase or decrease that freedom.

I understand that you're talking about methodological bias, but those stem from more "personal" biases, don't they? For example, let's consider the skewing of numbers that happens with active vs passive surveillance of reported outcomes. If a study is based on passive surveillance, and the authors don't address the weakness of the surveillance system, that's personal bias (based on their operating assumptions) manifesting as a bias in the method. (I'm assuming the effect is not an intentional error better described as "fraud.")

Discussing the pros and cons of active vs. passive surveillance might be a good topic for this thread. What you're arguing is that one method has a lower risk of bias than the other - which is exactly the topic of this thread.

Basically if its related to the "Methodology" portion of a paper, its relevant for this thread.
 
Kay, that's an interesting way to put it - personal prejudice/bias leads you to pick weak methodologies or to ignore their limitations?

Arouet, I'm thinking you have to give up on using the term "bias". No matter how many times you say otherwise, nobody is going to regard it as anything but "personal prejudice". How about using PIDIA instead? It means what "bias" means (Problems In Design, Implementation or Analysis which lead to false results).

Linda
 
Kay, that's an interesting way to put it - personal prejudice/bias leads you to pick weak methodologies or to ignore their limitations?

Yes, or lack of knowing about potential limitations means that a researcher is literally unable to even attempt to account for it, or discuss it, or anything. This is probably the biggest factor, and while the words "ignorance" and "prejudice" apply, they have a negative, judgmental tone that's not "fair" in my opinion.

The research that might inform a scientist's understanding of their study's limitations is often fairly obscure (and the reasons why incredibly interesting, "solid", relevant research remains obscure is an interesting, incredibly complicated question.) Active vs passive surveillance obviously comes to mind. The old stories from back when HeLa cells contaminated ALL the cell substrates also does (that is one crazy, true story!) Modern problems in culturing some types of bacteria to get an epidemiological picture of the incidence of clinical vs subclinical infections is another.

I like the idea of abandoning "bias" in favor of PIDIA (I'll think of it as "rPITA"..."really, PITA factors." :) )
 
Well, I'm not sure that's quite true. For example there are researcher degrees of freedom that can lead researchers to - quite innocently - manipulate the numbers in a certain direction. Different research methodologies can increase or decrease that freedom.

Right. But unless fraud or extreme gross incompetence is present, the researchers will overtly state where and how they got their numbers, so as long as the raw data is present or can be scavenged from elsewhere, it doesn't really matter. It only matters in the sense that skewed results can be commonly believed to be representative of objective reality, because people who cite the study (or post it on facebook, or whatever) never see the problems with the study design and interpretation.
 
When you say active vs passive surveillance, would I get correct in assuming that it could apply to how you gather reports of, say NDEs? I have no expertise in anything so humor me as you would a child.

I feel like on the one hand, actively engaging people in telling their experiences is arguably "leading" and could conceivably produce "false positives", while on the other hand, simply waiting for people to come forward could lead to the opposite.

This is one thing that I can think of right now that very much depends on our understanding of human psychology, which is itself an incomplete area of research and one which perhaps couldn't or shouldn't become as objective as say physics.

I hear "numbers don't lie", which is why I support the banning of human researchers and all other phenomenological researchers.

We should just let numbers run all experiments and theoretical formulations from now on.
 
When you say active vs passive surveillance, would I get correct in assuming that it could apply to how you gather reports of, say NDEs?
Yep. :)

I feel like on the one hand, actively engaging people in telling their experiences is arguably "leading" and could conceivably produce "false positives", while on the other hand, simply waiting for people to come forward could lead to the opposite.
There's that, too. But there are ways of aggressively pursuing data that minimize the "leading" effect, or control for it.

This is one thing that I can think of right now that very much depends on our understanding of human psychology, which is itself an incomplete area of research and one which perhaps couldn't or shouldn't become as objective as say physics.
Agreed.

I hear "numbers don't lie", which is why I support the banning of human researchers and all other phenomenological researchers.

We should just let numbers run all experiments and theoretical formulations from now on.

:) Until then, I think the ideal best method would be gathering data in a variety of ways, comparing and contrasting the results, and going from there.
 
Right. But unless fraud or extreme gross incompetence is present, the researchers will overtly state where and how they got their numbers, so as long as the raw data is present or can be scavenged from elsewhere, it doesn't really matter.

From what I understand about degrees of freedom issues there doesn't necessarily need to be fraud or gross incompetence involved. Here's one description I found: http://andrewgelman.com/2012/11/01/researcher-degrees-of-freedom/

It is unacceptably easy to publish “statistically significant” evidence consistent with any hypothesis.

The culprit is a construct we refer to as researcher degrees of freedom. In the course of collecting and analyzing data, researchers have many decisions to make: Should more data be collected? Should some observations be excluded? Which conditions should be combined and which ones compared? Which control variables should be considered? Should specific measures be combined or transformed or both?

It is rare, and sometimes impractical, for researchers to make all these decisions beforehand. Rather, it is common (and accepted practice) for researchers to explore various analytic alternatives, to search for a combination that yields “statistical significance,” and to then report only what “worked.” The problem, of course, is that the likelihood of at least one (of many) analyses producing a falsely positive finding at the 5% level is necessarily greater than 5%.

It only matters in the sense that skewed results can be commonly believed to be representative of objective reality, because people who cite the study (or post it on facebook, or whatever) never see the problems with the study design and interpretation.

Frankly I don't think a lot of people spend all that much time reading much more than the conclusion section of most studies. However I think the methodology section should often get the most attention. That's one of the reasons I started this series! To draw attention to research methodology issues.
 
  • Like
Reactions: Kay
Hmm...

That is a really, really interesting full text referenced there. :)
http://pss.sagepub.com/content/22/11/1359.full#aff-2

The core problem they seem to be referring to is not so much "bad data" as it is presenting "fishing expeditions" as hypothesis-testing. They raise some other issues, too, but those come closer to what I think of as "fraud" (even their clever "Music can make you literally older" study example, while a prank, would be fraudulent if presented as fact in objective reality. Yes, I know that's their whole point, but still...)

The authors don't address the potential usefulness of what I think of as "fishing-like" data collection methods, also. For example, there could be a hypothetical study that looked at the relationship between NDEs and the incidence of altruistic behaviors before and after the experience. Looking at the total population of NDE experiencers might not reach statistical significance (or much above "technical" significance) on any particular measure studied, but if (again, this is hypothetical!) there were something like a tiny sub-group (say, 3 people) who, upon careful examination of their experience report, apparently experienced something like "hell", and all three of them subsequently engaged in a >100-fold increase in altruistic behaviors, that would be...really fascinating. Small numbers and post-hoc analysis issues be damned.

Anyway, on this (from the above link):

Posting materials and data
We are strongly supportive of all journals requiring authors to make their original materials and data publicly available. However, this is not likely to address the problem of interest, as this policy would impose too high a cost on readers and reviewers to examine, in real time, the credibility of a particular claim.

My alternative suggestion (this might be kind of "out there" :) ) would be for the peer review process to include a "designated debunker" (or several) to add a "Possible PIDIAs" list to the abstracts (it could go under "conclusions" in the abstract) and discussion (although high-quality research already has those listed in the discussion section, what I consider "high quality research" is relatively rare.)
 
Yes, or lack of knowing about potential limitations means that a researcher is literally unable to even attempt to account for it, or discuss it, or anything. This is probably the biggest factor, and while the words "ignorance" and "prejudice" apply, they have a negative, judgmental tone that's not "fair" in my opinion.

The research that might inform a scientist's understanding of their study's limitations is often fairly obscure (and the reasons why incredibly interesting, "solid", relevant research remains obscure is an interesting, incredibly complicated question.) Active vs passive surveillance obviously comes to mind. The old stories from back when HeLa cells contaminated ALL the cell substrates also does (that is one crazy, true story!) Modern problems in culturing some types of bacteria to get an epidemiological picture of the incidence of clinical vs subclinical infections is another.

I like the idea of abandoning "bias" in favor of PIDIA (I'll think of it as "rPITA"..."really, PITA factors." :) )
The knowledge level among medicine tends to be pretty decent, as the evidence-based movement systematically educates students on some of these issues. Anyone with training in epidemiology even more so. My perspective will be skewed by having practised in an academic environment, though.

I've often wondered where other fields stand. I get the sense that some of these fields, like psychology, are asking the same questions, making the same kinds of complaints, and offering the same kinds of suggestions which medicine went through about 20 years ago.

Linda
 
Right. But unless fraud or extreme gross incompetence is present, the researchers will overtly state where and how they got their numbers, so as long as the raw data is present or can be scavenged from elsewhere, it doesn't really matter. It only matters in the sense that skewed results can be commonly believed to be representative of objective reality, because people who cite the study (or post it on facebook, or whatever) never see the problems with the study design and interpretation.
Full transparency may save us from placing much (if any) weight on research which doesn't warrant confidence in the results. But the research is still unreliable and invalid (but now we know about it). We should be encouraging a culture where valid and reliable research is performed, not one where it is okay to perform poor quality research as long as it is possible for sleuths to catch you out. Because ultimately we want this to be about progress in discovery, not about packing citations onto your CV. Make it so that self-interest coincides with progress.

Homeopathy is a good example where a culture change would be needed, except that the field would disappear, which is a stronger disincentive than anything I could come up with. Their claim for positive studies depends strongly on PIDIA. In particular, a researcher seems to be able to measure dozens of outcomes and then claim the idea proven based on finding a "statistically significant" difference on any one of them. And they seem to do this with impunity amongst their peers - they are lauded for finding proof rather than criticized for their overt use of researcher degrees of freedom.

Linda
 
When it comes to the fishing expedition research, I think there's a culture problem in how the researchers communicate (or don't) with the press release writers, too. I've seen examples before of media reports that wildly distort the actual study results, and a lot of times the "seed" of the skew can be traced to the press release. Of course, sometimes researchers aren't quite as cautious in terms of "precise language" in media interviews as they are when phrasing things to pass peer review, it seems. (Seems like the NDE scientists have that problem, as well, from what I've gathered in threads here.)
 
Sometimes the press releases are not written by the researchers, but by someone from the university with a journalism/communications background, rather than a science background. I agree that much of the problem is introduced at the point where the research findings are presented for public consumption. But your point is taken - researchers may also play up the results to a degree that they can't get away with from mainstream peer-review.

OTOH, I had a conversation with a journalist once, who insisted that the job of a journalist was to 'interpret' and simplify what the researcher said in order to make it accessible for the public. When I pointed out that this process led to reporting which was dead wrong, since the researcher already likely dumbed-down the results considerably, they were unrepentant. I can't remember who said it, but someone (maybe Feynman) made the point that when it comes to conveying these complex ideas, it is necessary to make it a bit wrong to make it understandable.

Linda
 
Last edited:
I didn't know researchers commonly wrote the press releases (outside of the really obscure stuff where the researchers are overtly in the self-promotion game. )

Learn something new every day. :)
 
I can't remember who said it, but someone (maybe Feynman) made the point that when it comes to conveying these complex ideas, it is necessary to make it a bit wrong to make it understandable.

Linda

Do you personally agree with that point?
 
I didn't know researchers commonly wrote the press releases (outside of the really obscure stuff where the researchers are overtly in the self-promotion game. )

Learn something new every day. :)
I don't know if it's "common" or if we are thinking of the same thing. But I think we agree that media reporting is untrustworthy. The problem is avoided by going to the primary source (the published research report) instead. But then the information becomes less accessible (in terms of who gets access and in terms of whether someone has the capacity to understand it).

Linda
 
Back
Top