Belief in the law of small numbers

I like this sort of thing

Its a really famous paper, and still THE go-to source to cite when you're concerned about studies making claims based on small samples.

Though this is not the biggest problem afflicting psyc research into ME. The biggest problem is systematic bias. I reckon you could run the PACE trial a hundred times and although it wouldn't reach p < .05 every time, overall there'd be a significant outcome. Its just that outcome is not meaningful because of the bias inherent in the design.

 

Many if not most people seem to think statistically significant means correct. This is easily exploitable as you say, do enough trials you can cherry pick the one you like and voila you have "proved" lies.
There was a Dilbert episode where he is forced to prove Chronic Cubicle Syndrome exists and goes to his company's statistical distortion department, whose employee says no need to prove it, everyone knows it exists since its talked about on tv and in national discourse.
Perhaps the biggest thing modern science has to weed out is the people who are not interested in the truth but want to use science as a screen for harming others or legitimizing lies.

 

I hope they were not making fun of Chronic Fatigue Syndrome.

 

They were not, Dogbert invented a new condition to show how gullible the public is to media driven lies (eerily accurate for being written almost 20 years ago)

 

The most notable part of this is this bias is still common amongst scientists today.