On the other hand, Bem's second way of modelling the content bias and testing for its presence found that it rendered the previously significant difference only marginally significant (two-tailed). And then Bierman's third way of modelling the content bias found that it rendered the significant difference in scoring on dynamic vs. static targets non-significant.
Bem's second way of modeling the content bias was a two sample comparison test, which has less power than a one sample test. They answer different questions about the data. Note that the first method Bem used produced a p-value that was virtually indistinguishable from the original one, thus why it was rounded to the same decimal place. Bierman's finding could very easily be explained by coincidence. Randomness is randomness.
Johann's response is making me a bit uneasy. My criticism, as well as Kennedy's, has been that parapsychologists have focused on debunking debunkers instead of focussing on performing well-designed, well-powered experiments. Too much emphasis has been put on Bem coming up with tests (of unknown validity and reliability to detect the effects of the bias in the first place) which attempt to rule out an effect from one kind of bias as a way to claim that it doesn't matter that the ganzfeld tests aren't well-performed. Kennedy's and my point has always been that the research shows pretty clearly that this doesn't work, that the better strategy is to move forward with performing well-designed and well-powered experiments instead. Especially since there seems to be general agreement that regardless of whether or not "it accounts for all of the effects in the ganzfeld studies", these are just some of the ways in which the hit rates can be biased (another example would be the autoganzfeld session 302). Why Johann's response makes me uneasy is because Johann previously told me that he and Maaneli had co-authored a paper in which they recommended that the way to move forward was to perform experiments in ways which reduce bias. So this now seems like a step backwards for him to criticize Kennedy and me for making that same suggestion by bringing up Bem's attempt to rationalize away attempts to address bias.
Linda, you state repeatedly that the ganzfeld studies aren't well designed. You are entitled to that opinion. However, part of the basis for this conclusion has been Kennedy's report, and specifically his suggestion that the power of ganzfeld studies is low. You specifically asked me in another thread why I thought parapsychologists had not already taken care to use their best subjects to rectify this. Well, my response is that many of them have.
As I have implied before, we are under-informing ourselves when we focus on only the summary measure of a meta-analysis. The power calculations done by Max have taught me something that in retrospect should have been obvious; that is, that the statement "parapsychologists do (or have been doing) x" is usually a misleading one. IMO, the attempt to generalize over the whole field is a weakness that infects arguments from both advocates and counter-advocates of parapsychology, on a consistent basis. If we shift our focus away from the false homogeneity that has often been implied for the field, to its true diversity, we can abjure some our misconceptions and obtain a more realistic picture of the situation. Nothing aids improvement like an accurate understanding of where we are now.
For example, if we look at the success of experiments
specifically designed to test artists and musicians because of their previously reported higher hit rates, we find a string of breathtaking successes with very high power (please note: these studies were aggregated in Max's power paper; I owe him a great debt of gratitude for pointing them out to me):
Bem & Honorton 104/105b (1988): N = 20; HR = 50%; z = 2.20; p = .013 - Original Study
Morris et al. (1993): N = 32; HR = 41%; z = 1.78; p = .037
McDonough et al (1994): N = 20; HR = 30%; z = 1.02; p = .382
Morris et al (1995): N = 97; HR = 33%; z = 1.67; p = .047
Dalton (1997): N =128; HR = 47%; z = 5.20; p = 7.072*10^-08
Parker & Westerlund study 4 (1998): N = 30; HR = 47%; z = 2.40; p = .008
Morris, Summers & Yim (2003): N =40; HR = 38%; z = 1.64; p = .054
Let's do the meta-analysis.
For all studies: N = 367; X = 152; HR = 41.4%; z = 7.26; p = .00000000000436
For all studies minus the first study: N = 347; X = 142; HR = 40.92%; z = 6.85; p = 0.0000000000591
Note the remarkable characteristic of these studies: to my knowledge, there is
nothing post hoc about them! Every single one mentions its intention to use the results of previous studies with high-scoring creatives to enhance their own results (by using creatives); a meta-analysis on this subset is therefore not only wholly justified, but probably more informative as to the reality of psi than a meta-analysis which takes an all-inclusive approach. There is also the benefit that little to no selection bias is likely to exist for them, since studies specifically using creatives are well-known, and since there is very little ambiguity in the single criterion "selection for creativity". Furthermore, all confirmatory studies reached independently significant results except for one, for a 5/6 or 83.33% proportion of positive results. Putting aside the fact that a binomial test with a 5% alpha on the study count here is wildly conservative, the probability of 5/6 significant studies or more is p = 0.0000018.
In spite of my argument against the file-drawer, I think it may still be worthwhile to apply an internal consistency check; that is, to ascertain whether selection bias is a viable hypothesis. We can do this with the Ioannidis & Trikalinos (2007) excessive significance test (applied by Francis, 2012, on Bem's studies), which uses pooled ES to predict how many studies should reach significance. Although the test is impaired in the presence of significant heterogeneity, these studies are not significantly heterogenous. So, we find that 4.46 studies should have reached significance, and 5 did. Looks pretty good to me, but not, as Francis would say, "too good to be true". Considering that the test is overly generous towards the file-drawer when the true power is greater than .5 (which it inevitably is for these studies) these results would thus be very difficult to explain with selective reporting, whether of studies or of individual trials.
For a collection of just six replication studies to demonstrate a cumulative deviation of greater than six sigma (which I'll remind people again is the threshold for the discovery of the Higgs) is nothing short of flabbergasting. Yet how well-known is this fact? I've never seen it mentioned in the parapsychology literature, or the skeptical literature, with the exception on Max's still unpublished power paper, and even then without this meta-analysis. My intention is to publish a small analysis of these studies eventually, in which I do a systematic statistical and methodological survey of each study, and identify success characteristics.
Note: The removal of Dalton (1997) leaves a binomial p = 0.0000309 and a z = 4.24
So let's recapitulate. These six studies cover a period from 1993 to 2003. According to Storm et al, they would constitute about 10% of the 60 study post-PRL database. That's 10% of studies that have gone specifically towards the confirmation of a hypothesis, with astounding success. But it's only a fraction of the picture, because when we also think about selected subjects in general, where the finding is slightly weaker (owing to the marginally higher ambiguity in the "selected" criterion), we still find 14 studies in the database of Storm et al (2010)**, which is about 47% (14/30) of the studies conducted from 1997 to 2008. These 47% studies have much greater power than the rest of the Storm et al database, even when we remove the studies with creatives.
But let's do that anyway. We remove Dalton (1997), Parker & Westerlund (1998), and Morris, Summers & Yim (2003) from Storm et al (2010). That still leaves a hit rate of 40.18%, with an average sample size of 50 and a power of 77%.
In sum:
Kennedy says in his paper the following:
By the usual methodological standards recommended for experimental research, there have been no well-designed ganzfeld experiments. Based on available data, Rosenthal (1986), Utts (1991), and Dalton (1997b) described 33% as the expected hit rate for a typical ganzfeld experiment where 25% is expected by chance. With this hit rate, a sample size of 201 is needed to have a .8 probability of obtaining a .05 result one-tailed. No existing ganzfeld experiments were preplanned with that sample size. The median sample size in recent studies was 40 trials, which has a power of .22.
Kennedy also says that "cases with a smaller number of studies and/or possible methodological problems*** sometimes have replication rates outside of this range", which is perhaps an indication that he at least remembered some of what Max mentioned in their email exchange. But the image conferred on ganzfeld research by Kennedy's paragraph is, IMO, pretty convincingly false. Why? Because 37% of ganzfeld experiments in the most recent meta-analysis by Storm et al (2010) have a power (based on their mean
and median sample sizes, which both happen to be 50) in the vicinity of 77%. An unknown proportion of these have been designed to be confirmatory (In other words, I don't know, but the data is out there for anyone to find out). A further 10% have a power of around 69%, possessing both a larger effect size and the virtue of all being confirmatory (but with a smaller median sample size of n = 36).
