Assessing Randomised Controlled Trials

There are a number of tools available for journals to use to help assess RCTs.
I wondered if anyone here had 'tried them out' for PACE and/or any of the other 'RCT's
by Crawley or Chalder?

I've seen CASP mentioned:
https://casp-uk.net/wp-content/uploads/2018/01/CASP-Randomised-Controlled-Trial-Checklist.pdf

and Consort:
http://www.consort-statement.org/
"
The CONSORT Statement is endorsed by prominent general medical journals, many specialty medical journals, and leading editorial organizations. CONSORT is part of a broader effort, to improve the reporting of different types of health research, and indeed, to improve the quality of research used in decision-making in healthcare."

this is one from the BMJ:
https://bmjopen.bmj.com/content/sup...-2015-008807.DC1/bmjopen-2015-008807supp2.pdf

and this is the one used by NICE:
https://www.nice.org.uk/process/pmg...dology-checklist-randomised-controlled-trials

(do most reviewers at journals use these tools? and do they submit them to the journals or just use them to help make their decisions? @Lucibee )

@Tom Kindlon @Graham @dave30th

 

I'd love to have seen the assessment on PACE: no doubt full of the "although .... the committee decided that .... was acceptable and that bias was kept to a minimum ...."

 

I agree such guidelines can be useful.
I have cited CONSORT guidelines in some of my publications. For example:

 

I would hope that reviewers do not use these 'tools'. I certainly would not. It is a bit like having a tool to provide a jury instructions on whether or not to find someone guilty. Tools like this might possibly be useful for people who are not familiar with trial assessment to remind them of what they need to look out for. For people who are used to assessing trials (which should really include all properly trained physicians who have attended journal clubs throughout their careers) the important issues will be obvious. What may not be obvious are weaknesses relating to a specific trial context (like a trial that trains people to say they are better and then asks them if they are better) but standard tools are not going to pick them out.

Part of the problem with Cochrane has been that they have a tool for assessing quality of evidence from trials. And it is completely useless because it misses the most basic problems - as far as we can see from PACE.

The real problems I think are:
1. A lot of people in biomedical science are actually quite dim, at least in terms of ability to evaluate results. I am not sure what one does about that. Unfortunately the editors in chief of Lancet and BMJ both seem to fall into this category.
2. The establishment prefers to protect its own and cover up embarrassing failures rather than admit they got things wrong.

 

I agree with @Tom Kindlon that CONSORT guidelines are useful, but they don't enable you to assess the quality of trials. They are simply a reporting checklist to make sure that info is reported - ie, have the said what randomisation method they used, what the primary outcome measures were, etc. It doesn't then check that they have used the appropriate ones of either.

The Lancet does use CONSORT in this sense, but just as a box-ticking exercise.

 

This is probably a stupid question, but has anyone pointed this out?
Is there another trial (not ME related) or hypothetical example, that could be used to demonstrate how these things are missed out using the existing tools?

(Just had a quick search and see that the consort diagram was submitted for this 'Early intervention study', O'Dowd/Crawley https://www.s4me.info/threads/odowd-crawley-early-intervention-study.2931/ )

And I see that the PACE authors said that they were compliant with Consort guidelines :
https://www.s4me.info/threads/pace-trial-tsc-and-tmg-minutes-released.3150/page-22#post-88225

Surely something as basic as no (adequate) control group/s in a Randomised Controlled Trial should get flagged up(?)

 

That is exactly what was not flagged up by Larun et al. If you have dim people applying 'tools' you don't necessarily get a classy piece of furniture, or even one that stays up when you sit on it.

And when a president of a Royal College does not understand the basics you get to realise how big a problem dim people are.

I pointed out to Cochrane that whatever evidence grading system they had it obviously was not working. There has been feedback suggesting that they realise that in the PACE case things were not applied properly but I am not sure it sounded as if they really understood the depth of the problem.

 

I've watched this short series of videos of a talk given by Mike Clarke which I found on the Cochrane training website.
"
This resource is a four-part video recording of a talk given by Mike Clarke, the then Director of the UK Cochrane Center, as part of the Oxford MSc in Evidence-Based Healthcare Programme.
The talk is about risks of doing multiple analyses, including looking at different subgroups, which can lead to false positive findings. The ways of reducing these risks are discussed."

the first three are about 15 minutes long each and the last is 6 minutes.

https://training.cochrane.org/resource/mulitiplicity-and-subgroup-analysis-beware

"Mike has 25 years’ experience of the conduct and oversight of randomised trials, systematic reviews and other types of prospective research. Previously, Centre Director of the UK Cochrane Centre. He is currently the inaugural Director of the All Ireland Hub for Trials Methodology Research and the Co-ordinating editor of the Cochrane Methodology Review Group."

as far as I can gather he is now at Queens University Belfast:
https://pure.qub.ac.uk/portal/en/persons/mike-clarke(f6bee3f9-d4fd-4498-b28b-1e52b1dfaf91).html

his contact details are on the last link.

wonder if he is worth contacting?

@dave30th

(as an aside: interesting to see that Oxford do courses in RCTs and Systematic Reviews
https://www.conted.ox.ac.uk/courses/randomized-controlled-trials
https://www.conted.ox.ac.uk/courses/systematic-reviews)

 

From the Cochrane training site:

"Mulitiplicity and subgroup analysis - Beware!
This resource is a four-part video recording of a talk given by Mike Clarke, the then Director of the UK Cochrane Center, as part of the Oxford MSc in Evidence-Based Healthcare Programme.
The talk is about risks of doing multiple analyses, including looking at different subgroups, which can lead to false positive findings. The ways of reducing these risks are discussed.

After using this resource, you should be able to...

• Understand and avoid the risks of using multiple analyses in clinical trials and systematic reviews"

 

I wonderif the Northern ireland groups know about him @Tom Kindlon @Keela Too

Some of his words in the first video could be written directly for the PACE authors.

 

@JoanNI See above. Another person worth contacting re Hope conference?

https://pure.qub.ac.uk/portal/en/persons/mike-clarke(f6bee3f9-d4fd-4498-b28b-1e52b1dfaf91).html

 

Onto it Sally! I can’t read all of this thread but I get the gist. If he replies and is unable to attend the conference, I’ll offer him an interesting info pack and include flyer for David T event in October.
I’ll keep you posted.

 

Might be relevant for the Cochrane review/ IAG thread?

https://www.s4me.info/threads/indep...y-and-me-cfs-2020-led-by-hilda-bastian.13645/