Risk of bias tools in systematic reviews of health interventions: an analysis of PROSPERO-registered protocols - Farrah,Young,Tunis,Zhao Nov 2019

[Sly Saint]

Abstract
Background

Systematic reviews of health interventions are increasingly incorporating evidence outside of randomized controlled trials (RCT). While non-randomized study (NRS) types may be more prone to bias compared to RCT, the tools used to evaluate risk of bias (RoB) in NRS are less straightforward and no gold standard tool exists. The objective of this study was to evaluate the planned use of RoB tools in systematic reviews of health interventions, specifically for reviews that planned to incorporate evidence from RCT and/or NRS.

Methods
We evaluated a random sample of non-Cochrane protocols for systematic reviews of interventions registered in PROSPERO between January 1 and October 12, 2018. For each protocol, we extracted data on the types of studies to be included (RCT and/or NRS) as well as the name and number of RoB tools planned to be used according to study design. We then conducted a longitudinal analysis of the most commonly reported tools in the random sample. Using keywords and name variants for each tool, we searched PROSPERO records by year since the inception of the database (2011 to December 7, 2018), restricting the keyword search to the “Risk of bias (quality) assessment” field.

Results
In total, 471 randomly sampled PROSPERO protocols from 2018 were included in the analysis. About two-thirds (63%) of these planned to include NRS, while 37% restricted study design to RCT or quasi-RCT. Over half of the protocols that planned to include NRS listed only a single RoB tool, most frequently the Cochrane RoB Tool. The Newcastle-Ottawa Scale and ROBINS-I were the most commonly reported tools for NRS (39% and 33% respectively) for systematic reviews that planned to use multiple RoB tools. Looking at trends over time, the planned use of the Cochrane RoB Tool and ROBINS-I seems to be increasing.

Conclusions
While RoB tool selection for RCT was consistent, with the Cochrane RoB Tool being the most frequently reported in PROSPERO protocols, RoB tools for NRS varied widely. Results suggest a need for more education and awareness on the appropriate use of RoB tools for NRS. Given the heterogeneity of study designs comprising NRS, multiple RoB tools tailored to specific designs may be required.

full paper here
https://systematicreviewsjournal.biomedcentral.com/articles/10.1186/s13643-019-1172-8

When protocols were restricted to RCT, the choice of RoB tool was highly consistent, with 85.2% planning to use the Cochrane RoB Tool. A few additional protocols (1.9%) planned to use Cochrane RoB 2 Tool, which was first introduced in 2016 as an update to the original Cochrane RoB Tool [20]; however, the uptake of Cochrane RoB 2 Tool may be underestimated, as authors may not have specified the version number in their protocol.

 

[rvallee]

While non-randomized study (NRS) types may be more prone to bias compared to RCT

It's really bizarre that randomized but non-controlled trials are skipped over. Is it because people just assume they don't happen? They clearly do, in fact they are the norm in BPS (and psychosomatic "research" since forever) and there has not been a single properly controlled BPS trial of ME, CFS or MUS in general, because they cannot be adequately controlled. If the numbers are remotely close to be correct about the frequency of MUS, though given the real explanation of gaps in our medical knowledge rather than unexplainable, then this affects a very sizeable share of current medical research.

This is repeated a few times in the abstract, that it is either non-randomized or a full double-blinded RCT. There are several steps in-between and they are common, in some schools of thought pretty much the only option. There are randomized open label trials that have no possibility of controls and there are also randomized open labels with inadequate controls that then go into lots of speculation about what the inadequate controls represent, things as bizarre as "maybe they've seen news coverage of this and it changed their perception", grasping at any straw in the way.

About two-thirds (63%) of these planned to include NRS, while 37% restricted study design to RCT or quasi-RCT

Those numbers expose a huge flaw in all published research given that meta reviews like Cochrane are held as the gold standard, that it is in fact a garbage-in-garbage-out factory. But I guess we found our non-controlled here, with "quasi" doing a lot of work that it should not be doing. No doubt the Cochrane review on GET would fall in the quasi even though they have absolutely no adequate controls and in addition to being open label are entirely subjective, another flaw that is not addressed and badly needs to. This is very flawed thinking overall.

I don't think this discussion can be had without acknowledging this and it's utterly bizarre that this seems to be skipped over as if it doesn't exist or has a limited impact on published research. I'm wondering if it's worth pointing out to the authors since it seems to be a common failure, almost as if this was taboo but in fact with the way things are going they will only be more frequently used.

That this major elephant in the room isn't even acknowledged by the people focused on talking about the room is not encouraging at all.

 

[alktipping]

rvallee you skipped the part were Cochrane are changing their risk of bias tool to a new even more useless one . soon it will be impossible to trust anything these greedy publishing houses print .

 

[James Morris-Lent]

What is the point of all this? If something is effective, it will prove out in properly designed trials.

This all just seems like a way to (a) get around doing trials capable of producing definitive evidence so that there is wiggle room to claim whatever is convenient by appealing to some official-looking but spurious tools or (b) use said tools to abdicate individual responsibility to critically asses research so as not to run afoul of any benefactors or colleagues.

 

[Snow Leopard]

It's curious that they cite Jadad et al. but don't mention the word "blind" or "blinding" once.

Jadad AR, Moore RA, Carroll D, Jenkinson C, Reynolds DJ, Gavaghan DJ, et al. Assessing the quality of reports of randomized clinical trials: is blinding necessary? Control Clin Trials. 1996;17(1):1–12.

https://www.ncbi.nlm.nih.gov/pubmed/8721797

 

[Sly Saint]

But I guess we found our non-controlled here, with "quasi" doing a lot of work that it should not be doing.

so this method could be described as 'quasi modo'....(?)

eta: 'the bells, the bells'

 

[rvallee]

What is the point of all this? If something is effective, it will prove out in properly designed trials.

This all just seems like a way to (a) get around doing trials capable of producing definitive evidence so that there is wiggle room to claim whatever is convenient by appealing to some official-looking but spurious tools or (b) use said tools to abdicate individual responsibility to critically asses research so as not to run afoul of any benefactors or colleagues.

It's paving the way for more BPS, which cannot stand on its own merits and needs the bar to be lowered below ground. It is largely centered around the UK, where the massive IAPT reform is also failing on its own merits and needs to be propped up because the people responsible can't accept accountability for the predictable debacle. The UK is sort of inflicting its own mistakes on the world because of historical flukes that made it a center of modern medicine.

It's also true that medicine has been running low on progress these past few years, having more to do with increasing costs of incremental progress once the easy stuff has been discovered, but the response has been instead: what if we just allow people to make stuff up about magical psychology? Most of the blame lays in the insulated approach to medicine, where patient input that can bridge the gaps is dismissed as impure, but also a plateau in technology that is bridging us from primitive understanding to all new paradigms. Unfortunately those transitions are usually confused by some people as stagnation.

It's growing pains, a sort of trial by fire attempting to bring back the worst failures of the past, a pre-science approach of merely arguing for random correlations and reversed causality, to make up for more recent failures. The end result will be to spectacularly implode the entire BPS ideology and its peripheral belief systems, but the failure has to be brought to the full before people accept that. It's a shiny "new" toy that has already failed but the hopes are high about magical powers of the mind, about Freud having been right all along.

It's a real problem, that every bit of progress is harder than it used to, but the solution is to take the worst possible way out. The only positive is that in the end it will close the door entirely to this approach, and probably belief in psychosomatic medicine entirely, but not until it has proven itself to be spectacularly incompetent under all the spotlights. Ultimately it's showing that physicians are just as gullible and prone to magical thinking as anyone else, they just require different methods but pseudoscience is about incredible evidence, not about the precise nature of the thing that has no evidence, whether it's humours, fairies, karma or psychogenic explanations.

 

[Sean]

It's paving the way for more BPS, which cannot stand on its own merits and needs the bar to be lowered below ground. It is largely centered around the UK, where the massive IAPT reform is also failing on its own merits and needs to be propped up because the people responsible can't accept accountability for the predictable debacle. The UK is sort of inflicting its own mistakes on the world because of historical flukes that made it a center of modern medicine.

Pretty much the entire story, far as I can tell.