To be fair, that is quite a big number. I reckon the probability of getting the desired 5 or more out of 7 would be about 0.44%, while that of getting 4 out of 7 would be about 1.83%, which obviously would normally be considered statistically significant.
Considering that - as Hyman points out - the protocol stated that the idea of the test was to "help to decide whether further studies of Natasha’s claimed abilities are warranted," it's difficult to see how an objective observer could agree that a result with 1.83% probability indicated that further studies
weren't warranted, whatever flaws there were in the design and/or execution of the experiment:
http://www.csicop.org/si/show/testing_natasha
Although the test was methodologically flawed I still think it is interesting to get a precise handle on the probabilities involved. It comes down to the fairly simple task of counting permutations that fixes a certain number of elements. So let D(n,k) denote the number of permutations of n elements that fixes k elements. If we let Binom(n,k) denote the binomial coefficient and we have the values of D(k,0) from
here, we use D(n,k) = Binom(n,k)*D(n-k,0) to obtain:
D(7,0) = 1854
D(7,1) = 1855
D(7,2) = 924
D(7,3) = 315
D(7,4) = 70
D(7,5) = 21
D(7,6) = 0
D(7,7) = 1
And so if P_k denotes the probability of exactly k hits out of 7, we have P_k = D(7,k)/7!:
P_0 = 1854/7! ≈ 36,79%
P_1 = 1855/7! ≈ 36,79%
P_2 = 924/7! ≈ 18,33%
P_3 = 315/7! ≈ 6,25%
P_4 = 70/7! ≈ 1,39%
P_5 = 21/7! ≈ 0,42%
P_6 = 0/7! ≈ 0,00%
P_7 = 1/7! ≈ 0,02%
The probability of getting 4 or more hits is the probability of getting either 4,5,6 or 7 hits. And since these events are disjoint we can just add their probabilities to get our answer:
P(at least 4 hits) = P_4 + P_5 + P_6 + P_7 = 92/7! ≈ 1,83%.
This is approximately 1 in 55.
It is also interesting to calculate the expected number of hits. So if we weigh the number of hits with their corresponding probability, we get:
Expected #hits = 0*P_0 + 1*P_1 + ... + 7*P_7 = (0*1854 + 1*1855 + 2*924 + 3*315 + 4*70 + 5*21 + 6*0 + 7*1)/7! = 1.
CSICOP required at least 5 hits for a success. The probability of that happening by chance is:
P(at least 5 hits) = P_5 + P_6 + P_7 = 22/7! ≈ 0,44%.
This is approximately 1 in 229.
So Demkina got 4 hits where you would expect 1 hit on average, and that has a 1 in 55 probability of happening by chance. CSICOP considered this a failure since they required a result that has a 1 in 229 probability of happening by chance.