ME/CFS Skeptic - How many scientific papers are fake?

[ME/CFS Skeptic]

New blog post about fraud and research misconduct in scientific research.
https://mecfsskeptic.com/how-many-scientific-papers-are-fake/

Twitter summary:
https://twitter.com/user/status/1901672256738095441


1) How many scientific papers are fake?

A new review argues that fabrication and falsification of scientific results may be more common than previously thought. It estimates that approximately one in seven papers are fake.

2) In the past few people were actively looking for fraud but in the past 15 years, a group of data sleuths have developed various techniques to screen for research misconduct.

3) This has resulted in a flood of retractions: from highly cited papers on Alzheimer’s, controversial COVID-19 treatments to publications by the Dana-Farber Cancer Institute. Even the president of Stanford University had to resign.

4) In our blog post we look at some high profile examples of research misconduct, the smart tools data sleuths use to detect problematic papers, and how this is slowly changing the scientific literature.

5) Although still at a small scale and driven mostly by volunteers, the valuable work of research integrity advocates signals a hopeful change in the psychological and biomedical sciences.

 

[EndME]

I'm not sure whether "scientific papers" is the right term here, even though it is the term used by Heathers. Perhaps "psychological and biomedical papers" is more appropriate? I suspect that the differences between subjects will be massive and there are many disciplines where this argument doesn't apply, since there is essentially no data that can be fabricated (certain topics in philosophy, pure mathematics, theoretical physics etc), but where the other substantial problems in academic publishing still exist.

I think it might be meaningful to point out that for now, apart from a few positive signs, there are many signs of worsening, especially with how people are using ChatGPT (which is fraud by definiton of almost every journal, but still gets published almost everywhere) and the like to write all sorts of nonsensical papers in their academic h-index competitions.

 

[Wonko]

I'm 'surprised' at the '1 in 7 are fake', i'd be less surprised at '1 in 7 aren't fake', because in many areas that i have seen virtually all papers are clearly fake, whole swathes of the internet returns thousands of fakes, or at least highly dubious papers.

But this is just my impression, maye there are loads of papers, accurate papers, on subjects I have no interest in, like tire rubber eating bacteria studies etc.

 

[Utsikt]

This might be of interest:
https://www.nature.com/articles/d41586-025-00455-y

Exclusive: These universities have the most retracted scientific articles
A first-of-its-kind analysis by Nature reveals which institutions are retraction hotspots.

 

[EndME]

I'm 'surprised' at the '1 in 7 are fake', i'd be less surprised at '1 in 7 aren't fake', because in many areas that i have seen virtually all papers are clearly fake, whole swathes of the internet returns thousands of fakes, or at least highly dubious papers.

But this is just my impression, maye there are loads of papers, accurate papers, on subjects I have no interest in, like tire rubber eating bacteria studies etc.

The one in 7 is a reference to fabrication and falsification of data, not papers that are dubious, full of errors, nonsensical or simply wrong. That would be a very high number given that all the other problematic things don't fall into this category. I think the Alzheimers example is a pretty daunting example of how bad the situation is in many fields, but fabrication and falsification of data is only the end of the tip of the iceberg.

 

[Kitty]

The one in 7 is a reference to fabrication and falsification of data

Yep. That word "data" does a lot of work too.

The bloke at the former pizza place insisted he got a 10% response to leafletting littering our road. This was because somebody wrote something about 10% response rates, in a context loosely connected with marketing, at some point in the 1980s. Data!

Sadly doesn't help if you've no idea how to make edible pizzas.

 

[Hutan]

This might be of interest:
https://www.nature.com/articles/d41586-025-00455-y

Exclusive: These universities have the most retracted scientific articles
A first-of-its-kind analysis by Nature reveals which institutions are retraction hotspots.

Two days before the end of 2021, administrators at Jining First People’s Hospital in Shandong, China, issued a highly unusual report. The hospital announced that it had disciplined some 35 researchers who had been linked to fraud in publications, such as fabricating data. These sanctions were part of a countrywide crackdown motivated by concerns about a flood of sham medical papers emanating from hospitals.

The problem was that some young physicians at hospitals had purchased fake manuscripts from paper mills: companies that churn out fraudulent scientific reports to order. These doctors were under pressure because they were required to publish papers to get jobs or earn promotions, says integrity sleuth Elisabeth Bik in California. Sleuths such as Bik soon began spotting signs of this problem, identifying duplicated images in large numbers of papers. They publicized the issue and a wave of retractions followed.

Many other Chinese hospitals are retraction hotspots. But universities and institutes in China, Saudi Arabia, India, Pakistan and Ethiopia feature in the data as well. Retractions can be for honest mistakes and administrative errors, but evidence suggests the majority of cases in these data are related to misconduct.

8 out of the top 10 hotspots identified are in China. I suspect we have seen CFS papers from the listed Chinese institutions.

I think the identified institutions have to be be given some credit for actually retracting papers. I'm sure many institutions with stronger reputations just quietly sweep "issues" under the carpet.

 

[Utsikt]

I think the identified institutions have to be be given some credit for actually retracting papers. I'm sure many institutions with stronger reputations just quietly sweep "issues" under the carpet.

The US retracts papers at a lower rate than most. I doubt it’s because they are much more honest in their research.

 

[Utsikt]

The main point of posting this article here was the difference between the retraction rates and the suspected rate of falsified papers. 0.2 % suggests that less than 1 in 10 of falsified papers gets retracted.

 

[Hutan]

Great blog as usual @ME/CFS Skeptic.

Another prominent case is that of Dutch cardiologist Don Poldermans. His research suggested that administering beta blockers before surgery could reduce complications such as heart attacks and strokes. Based on his findings, some medical guidelines recommendedthis approach. However, in 2012 an inquiry by his former employer, Erasmus Medical School, concluded that Poldermans had used fictitious data. A 2014 review that excluded his studies found that beta blockers led to a 27% increase in mortality and urged a rapid update to clinical guidelines. It is unclear how much damage the false data caused, but some estimates go as far as hundreds of thousands of deaths.

It's clear that this sort of fraud can cause great harm. It's interesting to think about whether authors of fraudulent papers can be guilty of manslaughter.

Let’s look at a third, more recent, example. A 2018 Cochrane review concluded that giving steroids before a cesarean section would improve breathing in premature babies. These findings found their way into several guidelines. The data on late pregnancy, however, were based on only 1 British and 3 Egyptian studies. The latter were fraught with statistical issues and unrealistic data. The largest trial on more than 1000 pregnant women reported a proportion of female babies of nearly 60%, a figure the authors could not explain (it should be close to 50%). The journal retracted the paper with the following warning:

“The editorial board has raised concerns regarding the integrity of this paper and remaining in the public domain. It is by far the largest trial of corticosteroids for this indication, overwhelming other similar trials in Cochrane and other systematic reviews and is likely leading to widespread prescription of this drug, which may have serious side effects on fetal brain development. If the data is unreliable, women and babies are being harmed.”

In 2021, Cochrane introduced a new policy that recommended identifying ‘problematic’ trials and excluding them from reviews. An update of the 2018 review without the Egyptian trials now concluded there was insufficient evidence to reach a conclusion.

We've talked about this study, but I was not aware of Cochrane's policy. That's potentially relevant to the Larun et al review. A new review undertaken in compliance with that policy might have resulted in the PACE trial being excluded.

 

[Yann04]

I enjoyed the blog post. I continually wondered about survivorship bias when reading your blog. What papers aren’t getting flagged. I mean, you have to be reasonably knowledgeable to get to a place where you can publish academic papers. I’ll assume many of those falsifying their data are “competent” at falsification.

 

[ME/CFS Skeptic]

I'm not sure whether "scientific papers" is the right term here, even though it is the term used by Heathers. Perhaps "psychological and biomedical papers" is more appropriate?

True, this is specified later in the blog and Twitter thread but was difficult to include in the title.

I think it might be meaningful to point out that for now, apart from a few positive signs, there are many signs of worsening, especially with how people are using ChatGPT

Yes, good point. I remember people posting studies where the text included sentences of the prompting or an introduction such as: 'Certainly, here is a possible introduction for your topic'. EDIT: I now see that it's Cabanac who posted this, the researchers who highlighted the problem of tortured phrases.

The one in 7 is a reference to fabrication and falsification of data, not papers that are dubious, full of errors, nonsensical or simply wrong.

To me the 1/7 estimate seems quite high. I mostly read ME/CFS literature and there there are almost always big methodological flaws that produce 'significant' results so that probably means less need to fabricate data. Often researchers present their findings as quite impressive when the actual data is very weak or unconvincing. That seems like a sign that the data is likely not totally fabricated, otherwise they would have probably come up with something better!

It's interesting to think about whether authors of fraudulent papers can be guilty of manslaughter.

Yes or in the case of Alzheimer's, fraudulent studies might have wasted millions of dollars and hold back research to an effective drug, so the detrimental effects for society can be enormous. Begs the question if it should be dealt with like a crime. Now it seems that the worst thing that happens is that researchers get fired and can no longer work in academia.

 

[Utsikt]

Begs the question if it should be dealt with like a crime. Now it seems that the worst thing that happens is that researchers get fired and can no longer work in academia.

The burden of proof to convict would probably have to be quite high, so you’d end up with many ‘false, but legal’ papers. I’m not sure if that’s a problem.

 

[MrMagoo]

Off topic but ME/CFS Data Sleuth would be a great name!

 

[EndME]

To me the 1/7 estimate seems quite high. I mostly read ME/CFS literature and there there are almost always big methodological flaws that produce 'significant' results so that probably means less need to fabricate data. Often researchers present their findings as quite impressive when the actual data is very weak or unconvincing. That seems like a sign that the data is likely not totally fabricated, otherwise they would have probably come up with something better!

Yes, it seems too high for me as well. It's hard to believe that in 1/7 situations people would feel the need to fake data when in fact they can just get away with "faking a story" without that having any repercussions or even being frowned upon.

 

[Trish]

Fascinating article, thank you.

 

[Utsikt]

Off topic but ME/CFS Data Sleuth would be a great name!

For some reason I’ve always connected the word ‘sleuth’ with ‘sloth’. Not being a native speaker probably contributes to the confusion!

 

[rvallee]

The US retracts papers at a lower rate than most. I doubt it’s because they are much more honest in their research.

Such low retraction rates pretty much make a mockery of the idea of science being self-correcting. Corrections and retractions should be common, because the process is so abysmal that it lets nonsense pass through.

It really looks like one of the key features of the system has not been functioning in a long time, and no one noticed. Like an alarm that hasn't gone off, because it was taped over to prevent it from going off.

What's most annoying is how this plays perfectly into the anti-science movement, which is looking at trashing the entire concept of science. Even though, somehow, the discipline with by far the highest rate of bad research has been entirely ignored. Because it pleases a lot of people.

Fake news. Fake research. Fake people. Damn the future is bleak.

 

[ME/CFS Skeptic]

Side-thought: it must be really frustrating to be a true scientist amongst this mess of fraud and incompetence. Imagine funding going to 'exciting' but totally untrustworthy or incredible studies over and over again, while valuable work gets overlooked. Promotions going to people with 50 publications per year, etc.

I assume true scientists try to overlook most studies and only follow researchers in their field that they know are reliable. But I suspect it must be very frustrating.

 

[Yann04]

For some reason I’ve always connected the word ‘sleuth’ with ‘sloth’. Not being a native speaker probably contributes to the confusion!

same!