1. Guest, 'News In Brief' from w/c 3rd Dec can be read here.
    Dismiss Notice

Article: Scientists Aim To Pull Peer Review Out Of The 17th Century

Discussion in 'Health News and Research unrelated to ME/CFS' started by Indigophoton, Feb 26, 2018.

Tags:
  1. Indigophoton

    Indigophoton Senior Member (Voting Rights)

    Messages:
    848
    Likes Received:
    9,608
    Location:
    UK
    An article discussing the problems with peer review, focusing particularly on the biomedical field, the effect of the pressure to publish to gain career advancement, and a possible alternative approach.
    https://www.npr.org/sections/health...m-to-pull-peer-review-out-of-the-17th-century

    The new approach and the issues are described in detail here, APPRAISE (A Post-Publication Review and Assessment In Science Experiment)
     
    ladycatlover, Barry, JohnM and 8 others like this.
  2. Peter Trewhitt

    Peter Trewhitt Senior Member (Voting Rights)

    Messages:
    423
    Likes Received:
    4,328
    Well most people reading this will be aware of examples of where peer review has failed, and also where journals ignore concerns raised by reviewers.
     
    ladycatlover, Barry, Trish and 2 others like this.
  3. Alvin

    Alvin Senior Member (Voting Rights)

    Messages:
    2,452
    Likes Received:
    12,051
    I very much agree the process is flawed but i don't think
    is a panacea. It may be an incremental improvement at best.

    Also this system is easy to game, "publish" something thats utter bullcrap but appeals to a fringe or reality denying group, their media or even mainstream gets a hold of it, hypes it up on the news and its a millennia of playing whack a mole even when its shot down and deleted. Two examples being vaccines causing autism (which got published then redacted) and homeopathy. Their harmful impacts despite being discredited garbage will probably never be completely undone :mad: :emoji_face_palm:

    Replication may be the only final answer, but a costly and resource intensive solution.
     
    ladycatlover likes this.
  4. Jonathan Edwards

    Jonathan Edwards Senior Member (Voting Rights)

    Messages:
    2,785
    Likes Received:
    30,672
    But that is the situation we have now. 90% of what gets published in biomedical research is already bullshit. The 'proof' that homeopathy worked through water memory was published in Nature and the anti-vax story and PACE in the Lancet. The point is that peer review simply does not work. What works is people pointing out that stuff is bad later - which is exactly what Eisen is suggesting we rely on. The only reason why these bad studies had any credibility for the time they did was because they had the stamp of peer review approval. Take that away and such studies have to stand on their own merit. As things are now someone would point out within 24 hrs that PACE is meaningless or that water memory is not possible when the tablets have no water in them!

    It is interesting to look at the comments on this article. There are clearly still people who feel there has to be some sort of voice of authority. There is no need. You have to learn how to judge papers for yourself in science. There is certainly a need to get authors to write good quality text but at the moment the quality control is going down and down. If you just had to get your head of department's approval for publishing (which at the moment you do not) because each College published all its own people's stuff standard would rocket up. I would not have allowed my juniors to publish poorly written text if it the quality control was seen to be down to me.
     
    TrixieStix, BurnA, Mij and 9 others like this.
  5. Trish

    Trish Moderator Staff Member

    Messages:
    8,644
    Likes Received:
    47,871
    Location:
    UK
    I agree with most of what you say, but what if it's the heads of departments that are publishing crap, as seems to be the case with PACE, SMILE, FITNET etc. where the people doing the research are professors.
     
    BurnA, ballard, Cheshire and 2 others like this.
  6. Jonathan Edwards

    Jonathan Edwards Senior Member (Voting Rights)

    Messages:
    2,785
    Likes Received:
    30,672
    I was only talking about the standard of writing there. A head of department would have to take responsibility for the fact that his juniors could not even construct a proper argument. If the head of department is no good at science itself, then that gets picked up in 24hrs by people like Tom Kindlon posting comments - just as you already can with PLOS One.
     
  7. Trish

    Trish Moderator Staff Member

    Messages:
    8,644
    Likes Received:
    47,871
    Location:
    UK
    Ah, OK. I guess those professors can write coherent English. It is their 'science' that is incoherent.
     
  8. Alvin

    Alvin Senior Member (Voting Rights)

    Messages:
    2,452
    Likes Received:
    12,051
    No argument here

    This is close enough what is happening now and is not working.

    This works when readers and voters want to find the truth, in reality they believe what fits their ideology and reject what does not. Peer review is meant to weed out things but any scientist who thinks its an infallible stamp is an idiot who is not going to wisen up when given more power. :emoji_face_palm:

    People point out lots of alternative facts but those who want to believe do. Giving more power to people to publish more lies means more failure. Its strange i have to tell you this, your the scientist, you should know this already.

    World events show people and even voters are not interested in the truth or know how to figure it out or have the ability to learn. Lets not forget even well meaning people believe taking from the poor and giving to the rich will benefit them even when they are the ones being taken from and that allowing companies to poison the land/water/air wont happen if you stop enforcing environmental laws or that climate change is a hoax...


    We have had this argument before. This only works because your an ethical person, those who have no scruples will happily publish the most baldfaced lie and call it ice cream (or Italian restaurants).
    Giving license to unethical people and believing they won't abuse it is ridiculous

    Peer review is meant to add a few sets of eyes to catch flaws because unfettered publishing did not prevent lies from being published. Its very imperfect but the answer is not to undo the only failsafe available because it sucks, if it was the answer then attempted failsafe would never have been needed in the first place :emoji_face_palm:


    BTW an author's reputation is already on the line, the names of the peer reviewers are not published
     
    ladycatlover likes this.
  9. Valentijn

    Valentijn Not a moderator

    Messages:
    2,275
    Likes Received:
    13,204
    Location:
    Netherlands
    No, it's not what is happening now. The peer review process is still being treated as largely infallible, and post-publication commentary is being delegitimized on that basis. The primary pro-PACE argument is still appeal to authority - that the objections are from crazy/ignorant patients, and the papers were peer-reviewed and published in respected journals. The peer review process is helping authors avoid engaging with the issues being raised post-publication.

    Peer review may have a valid role to play, but it should be recognized as a minor one. Making it a major component is introducing huge delays in the reporting of scientific discoveries - or the disproving of them, etc. And then it's delaying or completely preventing the retraction of errors, or even any scrutiny in the first place.

    Not really. The authors can (and do) claim that errors are not their fault (or cannot exist at all) because of the peer-review process having been involved. They get it past the peer reviewers, and suddenly they're no longer accountable. And peer reviewers are being more frequently named now.
     
  10. Alvin

    Alvin Senior Member (Voting Rights)

    Messages:
    2,452
    Likes Received:
    12,051
    I agree its a minor one, i even argue its very fallible, frankly independent replication is the only true antidote
    Peer review prevents retraction or scrutiny? I just can't agree, its an excuse :emoji_face_palm:

    So authors being scrutinized use every excuse in the book. This is not news, this is standard operating procedure, you may eliminate one pathetic excuse but its ludicrous to believe free reign with even less accountability will cause improvement. :emoji_face_palm:

    So removing one excuse will fix the problem? Excuses are a dime a dozen, its like playing whack a mole. When eliminating peer review doesn't work and they still claim we are making threats perhaps anonymizing authors will be the new fad, especially when even more shoddy research gets published with even less quality control? The excuse will become having to attach their names means they can't pursue the best research which is the real reason medical research is so slow :emoji_rolling_eyes:
    Is the grass is always greener on the other side?
    This seems to be how con men gain followers, blame the imperfect checks and balances that do exist and keep repeating it till people believe it. If a system does not work pretending that eliminating it and going back to what failed will work is not the solution, what you really need to do is make improvements
    Why is giving more power to those willing to abuse it the answer?

    So people who believe in homeopathy and vaccines causing autism would give up those beliefs if peer review is eliminated. No offense but thats laughable :emoji_face_palm:

    Its funny that i'm the one defending peer review, i think its a bad idea. My point is that anarchy publishing won't work better, peer review was brought in as a response to unsuccessful self regulation
    It still doesn't work but forgetting why unsuccessful self regulation is bad and pretending going back to it will self regulate is insane.



    We need a better solution then self regulation or peer review.
     
    Last edited: Feb 27, 2018
    ladycatlover likes this.
  11. Valentijn

    Valentijn Not a moderator

    Messages:
    2,275
    Likes Received:
    13,204
    Location:
    Netherlands
    No one is proposing anarchy, and I think it's extremely likely that processes and standards would remain even if peer review ceases to be the main safeguard. If there's an institutional publishing scheme at all of the Universities, for example, some would certainly be of poor quality. But there could also be ratings of the universities (or other entities) based on the quality of their publications, much like there is of journals now - except the focus could be on quality rather than popularity/sensationalism prioritized by profit-driven journals.

    They're not proposing a wildly different setup, and reviewers would likely face some requirements:
     
    Trish and Invisible Woman like this.
  12. Alvin

    Alvin Senior Member (Voting Rights)

    Messages:
    2,452
    Likes Received:
    12,051
    I don't think shifting the reputation from journal to university is going to make any difference, people already judge universities (especially when deciding which one to apply to) and that has not stopped bad science. How would you change the focus to quality?
    Around here we have an expression, you don't put a fox in charge of the hen house
     
    Last edited: Feb 27, 2018
  13. Valentijn

    Valentijn Not a moderator

    Messages:
    2,275
    Likes Received:
    13,204
    Location:
    Netherlands
    The primary goal of a journal is to make a profit. Some universities also are profit-driven, but that varies between countries and institutions, and they are always driven to improve their reputation. A journal isn't going to be too bothered if it's reputation takes a hit so long as it's still making money in the short-term. A university will be, and it will also then be more directly accountable for the quality of work produced by its own researchers.
     
    Trish and Invisible Woman like this.
  14. Alvin

    Alvin Senior Member (Voting Rights)

    Messages:
    2,452
    Likes Received:
    12,051
    You realize studies published today typically identity the university involved so the bad quality is not sinking them. Identifying them as the publisher is not going to be a revelation or a revolutionary advancement when their name is already on the study.
     
    Last edited: Feb 27, 2018
  15. Valentijn

    Valentijn Not a moderator

    Messages:
    2,275
    Likes Received:
    13,204
    Location:
    Netherlands
    But the universities are generally not paying attention to the quality of studies produced by their employees. If the universities are responsible for publishing those papers, suddenly it's going to be a much bigger problem if there's a controversy about one of their papers.
     
    Indigophoton and Invisible Woman like this.
  16. Alvin

    Alvin Senior Member (Voting Rights)

    Messages:
    2,452
    Likes Received:
    12,051
    Huh? If the university published the paper they will then care their name is on it but don't care now that their name is on bad research?
    Will the extra bolding or having the name on the title page twice instead of once suddenly change how they are regarded?
     
    Last edited: Feb 27, 2018
  17. Arnie Pye

    Arnie Pye Senior Member (Voting Rights)

    Messages:
    1,278
    Likes Received:
    7,237
    Location:
    UK
    One thing I wish would happen with scientific research papers is that authors would do their best to write in as clear a manner as possible using as much standard English as they can, and using statistics that explain and discuss their findings rather than obscure them.

    If I take a look at serious medical research papers from, say, 1960 - 1970, I can understand many of them. There was clearly a desire to explain to and educate their readers rather than obfuscate.

    I realise this is never going to happen though. :(
     
    JohnM, Trish, Valentijn and 1 other person like this.

Share This Page