COMPare Trials, Ben Goldacre et al

Two papers published today on the same subject, first:

COMPare: a prospective cohort study correcting and monitoring 58 misreported trials in real time

Background
Discrepancies between pre-specified and reported outcomes are an important source of bias in trials. Despite legislation, guidelines and public commitments on correct reporting from journals, outcome misreporting continues to be prevalent. We aimed to document the extent of misreporting, establish whether it was possible to publish correction letters on all misreported trials as they were published, and monitor responses from editors and trialists to understand why outcome misreporting persists despite public commitments to address it.

Methods
We identified five high-impact journals endorsing Consolidated Standards of Reporting Trials (CONSORT) (New England Journal of Medicine, The Lancet, Journal of the American Medical Association, British Medical Journal, and Annals of Internal Medicine) and assessed all trials over a six-week period to identify every correctly and incorrectly reported outcome, comparing published reports against published protocols or registry entries, using CONSORT as the gold standard. A correction letter describing all discrepancies was submitted to the journal for all misreported trials, and detailed coding sheets were shared publicly. The proportion of letters published and delay to publication were assessed over 12 months of follow-up. Correspondence received from journals and authors was documented and themes were extracted.

Results
Sixty-seven trials were assessed in total. Outcome reporting was poor overall and there was wide variation between journals on pre-specified primary outcomes (mean 76% correctly reported, journal range 25–96%), secondary outcomes (mean 55%, range 31–72%), and number of undeclared additional outcomes per trial (mean 5.4, range 2.9–8.3). Fifty-eight trials had discrepancies requiring a correction letter (87%, journal range 67–100%). Twenty-three letters were published (40%) with extensive variation between journals (range 0–100%). Where letters were published, there were delays (median 99 days, range 0–257 days). Twenty-nine studies had a pre-trial protocol publicly available (43%, range 0–86%). Qualitative analysis demonstrated extensive misunderstandings among journal editors about correct outcome reporting and CONSORT. Some journals did not engage positively when provided correspondence that identified misreporting; we identified possible breaches of ethics and publishing guidelines.

Conclusions
All five journals were listed as endorsing CONSORT, but all exhibited extensive breaches of this guidance, and most rejected correction letters documenting shortcomings. Readers are likely to be misled by this discrepancy. We discuss the advantages of prospective methodology research sharing all data openly and pro-actively in real time as feedback on critiqued studies. This is the first empirical study of major academic journals’ willingness to publish a cohort of comparable and objective correction letters on misreported high-impact studies. Suggested improvements include changes to correspondence processes at journals, alternatives for indexed post-publication peer review, changes to CONSORT’s mechanisms for enforcement, and novel strategies for research on methods and reporting.​

Open access at https://trialsjournal.biomedcentral.com/articles/10.1186/s13063-019-3173-2

 

Second:

COMPare: Qualitative analysis of researchers’ responses to critical correspondence on a cohort of 58 misreported trials

Background
Discrepancies between pre-specified and reported outcomes are an important and prevalent source of bias in clinical trials. COMPare (Centre for Evidence-Based Medicine Outcome Monitoring Project) monitored all trials in five leading journals for correct outcome reporting, submitted correction letters on all misreported trials in real time, and then monitored responses from editors and trialists. From the trialists’ responses, we aimed to answer two related questions. First, what can trialists’ responses to corrections on their own misreported trials tell us about trialists’ knowledge of correct outcome reporting? Second, what can a cohort of responses to a standardised correction letter tell us about how researchers respond to systematic critical post-publication peer review?

Methods
All correspondence from trialists, published by journals in response to a correction letter from COMPare, was filed and indexed. We analysed the letters qualitatively and identified key themes in researchers’ errors about correct outcome reporting, and approaches taken by researchers when their work was criticised.

Results
Trialists frequently expressed views that contradicted the CONSORT (Consolidated Standards of Reporting Trials) guidelines or made inaccurate statements about correct outcome reporting. Common themes were: stating that pre-specification after trial commencement is acceptable; incorrect statements about registries; incorrect statements around the handling of multiple time points; and failure to recognise the need to report changes to pre-specified outcomes in the trial report. We identified additional themes in the approaches taken by researchers when responding to critical correspondence, including the following: ad hominem criticism; arguing that trialists should be trusted, rather than follow guidelines for trial reporting; appealing to the existence of a novel category of outcomes whose results need not necessarily be reported; incorrect statements by researchers about their own paper; and statements undermining transparency infrastructure, such as trial registers.

Conclusions
Researchers commonly make incorrect statements about correct trial reporting. There are recurring themes in researchers’ responses when their work is criticised, some of which fall short of the scientific ideal. Research on methodological shortcomings is now common, typically in the form of retrospective cohort studies describing the overall prevalence of a problem. We argue that prospective cohort studies which additionally issue correction letters in real time on each individual flawed study—and then follow-up responses from trialists and journals—are more impactful, more informative for those consuming the studies critiqued, more informative on the causes of shortcomings in research, and a better use of research resources.​

https://trialsjournal.biomedcentral.com/articles/10.1186/s13063-019-3172-3#Sec2

 

Twitter thread from Goldacre
https://twitter.com/user/status/1096058701021102080

 

Wasn't the last tremendously exciting thing he got involved in something to do with psychosomatic illness or something?

If we all already knew the system was broke I am not sure what these numbers add. There is also a mammoth in the room from before.

 

Ben is a bit of a 'chameleon' in my view.
Probably some of this rubbed off from his 'early life and experiences at Kings, ......the home of the master chameleon(s)!
He did a cracking lecture at the Adverse Reactions from Medication Conference several years ago.
https://www.april.org.uk/adverse-ps...erences-events-archive/2004-april-conference/

Shame that his consistency to science and scientific rigor has been patchy..... especially in regard to ME and CFS.

 

Goldacre came to Toronto a few years ago to speak - he's quite the energetic presenter/performer and repeatedly asked when the anti-vaxxers in the audience would disrupt his speech and the audience loved it and loved him - at the end during Q and A, he answered a few questions and then again asked where were the anti-vaxxers, my hand shot up with big smile and he called upon me, I disclosed that I wasn't anti-vaxxer but did have question if he was okay with that.

I said something like: I'm so glad you came to Canada to talk about Bad Science (his book iirc) and that there were 100s of 1000s of Canadians who are victims of bad science, and would you comment on the most egregious example of bad science, the PACE trials on ME?

His jovial demeanour changed and he fumbled some lame answer and was waiting for my reply. I didn't, and an awkward silence hung in the air and I could tell he was uncomfortable. He then asked if I was ok with his reply and I said I was surprised that he wouldn't mention such a harmful piece of bad science in his "schtick" and suddenly and unceremoniously the event ended, the lights came up, the moderator declared the show over, and he was whisked off stage.

No doubt it was not how he wanted the show to end. I keep hoping he'll return so I can have another go at him.

 

Oh dear, another vexatious patient who has the temerity to ask a reasonable question! Well done @ScottTriGuy

 

Oh, this is kinda of maybe off topic (but it is about Goldfield), but I just have to remind you all that his Mum is Noosha Fox of Fox. Here's one of her hits - Single Bed.

https://www.youtube.com/watch?v=wdHCKafq-Ew

 

Any chance that was caught on camera, that sounds so awkward it would be classic viewing.

 

I wish.

 

I can see we are speaking from the same hymn sheet. It was on Bengie's site, Sense About Science, that we found the "culprit' 'Jonas" bragging about taking Dr Sarah Myhill down. Well he got his come upance with the combined efforts of Patient /carers and clinicians countering and dealing with the nonsense. Same is happening now it appears. We will win the day of that I am sure.

 

A twitter search for " ben goldacre "PACE" " yields many good past tweets from the vexatious hordes.

https://twitter.com/search?l=&q=ben goldacre "PACE"&src=typd&lang=en

 

I did enjoy reading that post

 

Haha! Nice story, a thing of beauty

 

This might be worth following up?
Home

Welcome to the website of CEP, the Council for Evidence-based Psychiatry.
CEP exists to communicate evidence of the potentially harmful effects of psychiatric drugs to the people and institutions in the UK that can make a difference. The scientific record clearly shows that psychiatric medications, portrayed as safe and effective by areas of the medical profession, often lead to worse outcomes for many patients, particularly when taken long term. Our members include psychiatrists, academics, withdrawal support charities and others who are concerned about the prevalence of the ‘medical model’ and the increasing numbers of prescriptions for psychiatric drugs being given to both adults and children.


http://cepuk.org/members/
( My bolding)

Professor Peter C. Gøtzsche


Professor Peter C Gøtzsche graduated as a Master of Science in biology and chemistry in 1974 and as a physician 1984. He is a specialist in internal medicine; worked with clinical trials and regulatory affairs in the drug industry 1975-1983, and at hospitals in Copenhagen 1984-95. With about 80 others, he helped start The Cochrane Collaboration in 1993 with the founder, Sir Iain Chalmers, and established The Nordic Cochrane Centre the same year. He became professor of Clinical Research Design and Analysis in 2010 at the University of Copenhagen.

Peter has published more than 70 papers in “the big five” (BMJ, Lancet, JAMA, Ann Intern Med and N Engl J Med) and his scientific works have been cited over 15,000 times. Peter is also the author of the following books: Deadly psychiatry and organised denial (2015), Deadly medicines and organised crime: How big pharma has corrupted health care (2013, winner of the British Medical Association’s Annual Book Award in the category Basis of Medicine in 2014), Mammography screening: truth, lies and controversy (2012), Rational diagnosis and treatment: evidence-based clinical decision-making (2007)

Professor Peter C. Gøtzsche is Director of the Nordic Cochrane Centre. Peter has published more than 50 papers in “the big five” (BMJ, Lancet, JAMA, Ann Intern Med and N Engl J Med) and his scientific works have been cited over 10,000 times. He is author of the following books: Rational Diagnosis and Treatment. Evidence-Based Clinical Decision-Making (2007), Mammography Screening: truth, lies and controversy (2012), Deadly Medicines and Organised Crime: How big pharma has corrupted health care (2013).

Peter has an interest in statistics and research methodology. He is a member of several groups publishing guidelines for good reporting of research and has co-authored CONSORT for randomised trials (www.consort-statement.org), STROBE for observational studies (www.strobe-statement.org), PRISMA for systematic reviews and meta-analyses (www.prisma-statement.org), and SPIRIT for trial protocols (www.spirit-statement.org). Peter was one of the editors of the Cochrane Methodology Review Group 1997-2014.

 

For anyone who hasn't seen the quote:

Ben Goldacre said,
November 9, 2007 at 12:38 am
i’m not sure there are very many clear answers, but i have to say i think medically unexplained symptoms and the mechanisms of psychological factors in ill health are officially the most interesting thing in medicine right now, a huge amount of very real distress, little clear knowledge, and i’m totally moving into it as a dayjob, research if not clinical.

more here: BMJ Column – Beware of mentioning psychosocial factors

 

https://twitter.com/user/status/1096343602698039297

 

CONSORT stands for Consolidated Standards of Reporting Trials and encompasses various initiatives developed by the CONSORT Group to alleviate the problems arising from inadequate reporting of randomized controlled trials.

http://www.consort-statement.org/

This should also be fed into the 'NICE' HOTPOT.......

abstract
Overwhelming evidence shows the quality of reporting of randomised controlled trials (RCTs) is not optimal. Without
transparent reporting, readers cannot judge the reliability and validity of trial findings nor extract information for systematic
reviews. Recent methodological analyses indicate that inadequate reporting and design are associated with biased estimates
of treatment effects. Such systematic error is seriously damaging to RCTs, which are considered the gold standard for
evaluating interventions because of their ability to minimise or avoid bias.
A group of scientists and editors developed the CONSORT (Consolidated Standards of Reporting Trials) statement to improve
the quality of reporting of RCTs. It was first published in 1996 and updated in 2001. The statement consists of a checklist
and flow diagram that authors can use for reporting an RCT. Many leading medical journals and major international editorial
groups have endorsed the CONSORT statement. The statement facilitates critical appraisal and interpretation of RCTs.
During the 2001 CONSORT revision, it became clear that explanation and elaboration of the principles underlying the
CONSORT statement would help investigators and others to write or appraise trial reports. A CONSORT explanation and
elaboration article was published in 2001 alongside the 2001 version of the CONSORT statement.
After an expert meeting in January 2007, the CONSORT statement has been further revised and is published as the CONSORT
2010 Statement. This update improves the wording and clarity of the previous checklist and incorporates recommendations
related to topics that have only recently received recognition, such as selective outcome reporting bias.
This explanatory and elaboration document—intended to enhance the use, understanding, and dissemination of the
CONSORT statement—has also been extensively revised. It presents the meaning and rationale for each new and updated
checklist item providing examples of good reporting and, where possible, references to relevant empirical studies. Several
examples of flow diagrams are included.
The CONSORT 2010 Statement, this revised explanatory and elaboration document, and the associated website (www.
consort-statement.org) should be helpful resources to improve reporting of randomised trials.

 

It's really good that this is being looked at - so I commend this study by Goldacre - but there is an issue that always gets overlooked with regard to CONSORT, which is that these are "reporting" guidelines, not "conduct" guidelines.

Of course it is important that trials are reported correctly, but if the next step is then not taken, which is to use that report to assess the quality of the trial, then it is useless. It is assumed by some that simply reporting a trial correctly means that you can assume that it has been done correctly. You can't.

And it's not just outcome reporting. It's participant selection, randomisation, adequate controls and/or placebos, adequate blinding, intervention methods, numerous other things before you get anywhere near the analysis on the pre-specified measures.

fwiw, COMPare would not have picked up on PACE, because they didn't switch primary outcome measures with secondary measures, which is what they were looking for here.