The Rise and Fall of Peer Review, 2022, Mastroianni

[Lilas]

Absolutely, @Jonathan Edwards, I should have specified "among other things". I was only talking about the statistical aspect but in my mind I did not exclude other weaknesses... including the one you mentioned in the thread of this study. Moreover, I am in no way in a position to contradict the expert that you are in this field.

 

[Ravn]

I think the big problem is that peer review is done by peers!

People in any specific area of research all have a collaborative approach to taking things forward along the lines they think correct.

These are the peers who get asked to review, because the are “knowledgeable” in the area.

So if “your” research produces the outcomes that support the current collective view of that group, what incentive is there for “me” to criticise your work? I wouldn’t want to start undermining the whole foundations of the work we do, nor would I want “you” to later examine my projects with the same scrutiny.

So methods that might be dubious pass peer reviews, and the more often they do, the more they become accepted as standard, and the harder it becomes to critically assess them.

Group-think thus gets given authority, which becomes more and more difficult to question as time goes on, and as the original authors become increasingly senior and established.

Meh

This. Risks leading to group think as well as to a culture of you rub my back I rub yours. Both seem highly prevalent in ME/FND/MUS BPS circles, circle being the operative word here.

I think there is serious value in a culture of semi-formal preprint review. Probably one of the more important developments in science in recent years. Done right it could save us all a lot of time and energy, and heartache. Could also be a good, and mutually beneficial, training ground for emerging scientists, citizen and professional.

I'd go a step further. Instead of [ETA: or as well as] peer review or even preprint review I'd like to see public preview of protocols and methods. Preferably pre any allocation of funding.

Obviously this wouldn't be workable for purely commercial research funded entirely by industry. But it should be a requirement for anything that uses any sort of public funding, however indirect. Research funded by patient donations would also benefit from undergoing a similar, more rigorous and public preview of protocols and methods.

 

[FMMM1]

The chronic illness disaster is the best example of how this system fails miserably.

I know a researcher, who works on Alzheimer's disease, and for much of his career (he's close to 60 now) he struggled to get funding.

Jonathan highlighted that his colleagues couldn't make progress in ME/CFS - they tried a few things and, when those didn't work, then they gave up. I think there are better (and improving) tools like GWAS - so there are now opportunities to understand diseases where there hasn't been any progress - like ME/CFS. We need Government/public funding for good research.

it's like a brute force approach to finding the right key among millions that will unlock that one lock, except they keep trying the same 10 or so keys

Reminds me of @Snow Leopard point ---they keep looking in the places they looked before and found nothing! They even seem to apply the same techniques to new diseases like long covid --- Jonathan's comment comes to mind "The vultures are circling - and babbling as they fly." [https://www.s4me.info/threads/news-...e-cfs-2020-to-2021.14074/page-151#post-314441]

 

[FMMM1]

It is about checking a good result didn't appear as a matter of coincidence to replicate, it isn't about repeating the same issues and method precisely in order to repeat the same patterns of bias, but about testing from different angles to remove them. Putting in a blinded control wouldn't have stopped it from being a test of the treatment approach

Yea for some researchers it's about including objective outcome indicators like Actimetry (FitBit) since you cannot blind adequately. Failure to objectively measure (post intervention) outcomes seems to be de rigueur for some scientists and medical professionals --- makes sure they can prove their pet intervention/income stream works I guess!

 

[CRG]

 

[rvallee]

You successfully self-published an article. If everyone who wrote a paper did that, there would be 100s of manuscripts uploaded weekly with zero quality control and zero discoverability unless like a self-publishing fiction author you work your ass off at social media to get noticed.

Heh. As opposed to...?

In our own corner of the universe, things are even worse since the same thing is happening except it may as well literally be the same 5 papers published over and over again, with zero quality control, and clearly little discoverability as evidenced by recent "systematic reviews" that didn't find over 90% of the papers published on the same topic.

The criticism usually points to the best research out there as evidence that the system still works, but the reality is so much worse. In psychology, the poverty of peer review is essentially abused all the time, especially the massive HYPING of every small result from tiny studies that never replicate anything.

Right now we pretty much have the flaws of a giant self-serving bureaucratic mess, but without the benefits it can bring: no coordination or leadership, excessive redundancy and nothing is leveraged, no economies of scale, no sharing or pooling of resources while nothing builds on itself, everyone doing their small thing in their small corner, unable to bring the scale of work that other scientific disciplines like physics have succeeded at. It's eternal starting from scratch about a tradition that is nearly a religion, always promising, never delivering yet somehow seems to have always been there.

The author got the point right: this is about monoculture, about the lack of differing viewpoints. I can't say for other disciplines, but medical research is crippled by this, by groupthink and cultural enforcement of what's acceptable and what's forbidden. It's even worse than that when you consider the example of BPS ideologues who claim, and likely believe, that their excessively generic and coercive groupthink is the real courageous thinking, they literally boast about being fringe thinkers as they enforce their monoculture in secret behind closed doors.

The monoculture of medicine is probably its biggest flaw, it even prevents medicine from seeing its flaws, forcing the monoculture to remain the only acceptable way to think. Frankly it's borderline church-like behavior in terms of how conformist every idea is and how enforcement is cultural and political, never on merit, never on the actual evidence since medicine doesn't have math to settle debates, only a popularity contest where only the monoculture is allowed. Shockingly, one of the least curious group of people I've ever seen.

 

[SNT Gatchaman]

See also The Preprint Club - A cross-institutional, community-based approach to peer reviewing (2023)

 

[Kitty]

From @MSEsperanza:

(Not a reply on Mastroianni but thought it could serve as a comment. Sorry it's pessimistic.)

James Heathers: The Right to Be Wrong Isn’t The Freedom From Consequences

And From Where Consequences Might Arise

https://jamesclaims.substack.com/p/the-right-to-be-wrong-isnt-the-freedom

this is how I felt in ~September 2021, so almost exactly two years ago, and it feels reasonably well predictive about how The Big Stupid is going for us - what happens when you combine existing institutional mistrust with the hostile and feverish environment of COVID, followed by the social normalisation of ridiculous ideas. Never underestimate the power of learning it’s all someone else’s fault.

Scientists reserve the right to be wrong. Most observations, most conclusions are at least somewhat tentative. All science is in immediate danger of being overwritten.

And, of course, most of it is complex. Even papers I call ‘very simple’ are not actually simple. This is shorthand for ‘if you’re very familiar with the study area, the analysis methods, and reading papers in general, the methods and results are easy to understand’. Which isn’t simple in any normative sense.

Thus, the mendacious, the stupid, the ridiculous, the robustly fucked, is mixed into one nearly homogenous solution which itself is tipped into a big bucket marked ‘science is hard’ - which also contains the slightly wrong, the moderately misconceived, the important typo screwing up the whole paper, etc.

The sociocultural policing of scientific standards rely heavily on factors which are no longer true , specifically: 

first, that a paper will remain in existence forever in formal paper copy (which is historically what a journal always was);

second, that reputation is a sufficiently valuable currency that scientists will not want to screw around with misinformation or silliness too much;

third, that the research world is very small, and that if you do something truly egregious, you will be remembered by your peers.

Pandemic Science feels like the point where The Bastards have figured out, correctly, that these norms are stone dead. Scientists are proud of their system being trust-based, and it was always porous. And we’ve really, finally opened the gate to The Big Stupid, and let everyone in. Trust is fine when everyone plays the same reputation game as you, but we have new players now, and there is money and love in The Big Stupid.

Also from James Heathers, posted on Mastodon:

New Substack post.

Rescued from the drafts bin of mid-Plague mental horrors - when I genuinely thought that The Big Stupid (the collective incentive to science-grift) had begun.

Not a happy piece of writing, but it happened.

https://techhub.social/@jamesheathers/111133279337316953