Real-world studies no substitute for RCTs in establishing efficacy (2019)- Gerstein McMurray Holman

[Sly Saint]

Real-world studies no substitute for RCTs in establishing efficacy
January 19, 2019

1
We live in the real world, so it is reasonable to expect that data collected from the real world should help identify effective therapies. Indeed, rapid increases in the availability of registries, electronic health records, and insurance claims, and the ability to access, process, link, and analyse data from these sources at fairly low cost lend support for calls to replace randomised controlled trials (RCTs) with so-called real-world studies to establish the efficacy of a therapy,
2
, particularly for common serious diseases with abundant, easily collected data such as diabetes.
3
This push is driven partly by the need to show payers that therapies are working and are therefore of value when used in the real world.
4
Other driving factors include the industry's wish to reduce costs and time to get results, a mistaken belief that real-world data are somehow more relevant than RCT data for establishing efficacy, and the ease and speed with which registry data can be accessed and publications generated. However, even with the use of sophisticated methods to address various sources of bias,
5
the absence of randomisation
6
precludes protection from confounding and can lead payers and clinicians alike to erroneously infer that a therapy is beneficial or harmful.

https://www.thelancet.com/journals/...&hss_channel=tw-27013292#.XEgvxfcH2L0.twitter



full paper here:
http://sci-hub.tw/https://doi.org/10.1016/S0140-6736(18)32840-X

(retweeted by Michael Sharpe; hmm)

 

[rvallee]

Has anyone ever promoted this as a replacement for RCTs? Ever? Bueller? Anyone?

Both. All of it. This isn't a binary choice where one excludes the other. The most interesting results come from when real-world data conflict with research. It opens up the gaps in research (or in the data) and promotes new research paradigms.

But definitely one motivation in making up a strawman argument would come from people, like Sharpe, who absolutely do not want real-world data to debunk their ideas and reveal the fraud in their own research. SAD!

 

[Jonathan Edwards]

Has anyone ever promoted this as a replacement for RCTs? Ever? Bueller? Anyone?

The paper about real-life data that was flagged up a few days back talked of 'too much weight' being put on RCTs rather than real-life data. That reveals the false analogy with a weighing machine and balance or with a weighted sum. More weight to one thing suggests less weight to the other.

So the implication was there even if covert.

Clearly the proper analysis is different. Science is about putting up theories and trying as many ways as you can to refute them. A convincing fall at any hurdle and your theory is out. If RCTs do not support a hypothesis it is not rescued by some helpful real-life data, unless methodology is in doubt. If methodology was in doubt it was never a gold standard anyway and if it is doubt in both cases you are none the wiser.

It is interesting to see that this stuff is being aired at the moment. My impression is that the little bubble of PACEgate is not as peripheral to the general consciousness as one might have thought.

 

[Jonathan Edwards]

The main message of the paper here seems sound but the last bit looks as if there may be a hidden agenda. There is talk of 'real-life RCTs' being easier and cheaper. There is no mention of blinding in the paper - which is the main source of cost in RCTs (it hugely complicates the administration). Maybe some people are indicating their approval because they too like the idea of cheap and cheerful trials - maybe what have been called 'pragmatic trials' - sound familiar?

 

[Snowdrop]

As a counterpoint there is this paper: https://www.tandfonline.com/doi/full/10.1080/07853890.2018.1453233

"Why all randomised controlled trials produce biased results" Krauss 2018

Seems like what has been seen as the gold standard (RCT) is coming under some scrutiny and there are some who are fighting to maintain the status quo and business as usual.

I don't know anything on this subject but even if RCT are the epitome of science research their limitations and possible abuse ought to be a part of every researchers understanding and there should be gatekeepers to monitor those issues properly.

 

[Jonathan Edwards]

I don't know anything on this subject but even if RCT are the epitome of science research their limitations and possible abuse ought to be a part of every researchers understanding and there should be gatekeepers to monitor those issues properly.

Indeed but that is taken as read. We should al know the limitations - like unblinded trials with subjective endpoints being useless even if randomised, or that 'controlled' has to be adequately controlled and so on.

But the argument here is that bicycles with front and back brakes and a bell are not perfect so why not have ten bikes with no brakes. It is just a political ploy to allow shoddy evidence making.

The gold standard is an adequately controlled randomised double blind trial. Kraus's claim that 'RCTs are not so good' is a con trick - you need the other bits as well. He seems to assume nobody knows what every trainee should know about trials - maybe because he didn't. He almost certainly has some other axe to grind.

 

[alex3619]

I don't know anything on this subject but even if RCT are the epitome of science research their limitations and possible abuse ought to be a part of every researchers understanding and there should be gatekeepers to monitor those issues properly.

Good researchers understand this. Too many do not. Journalists typically do not, but there are exceptions. The general public is usually waiting for the media to decide.

 

[alex3619]

The gold standard is an adequately controlled randomised double blind trial.

The gold standard being the best we have, if done right, but gold standards are usually not perfect.

 

[duncan]

The gold standard being the best we have, if done right, but gold standards are usually not perfect.

Yes. Not only are they not perfect, but their results can be "crafted", as perhaps evidenced by the three notorious Lyme RCTs. Maybe that potential is more reflective of GIGO.

 

[Snowdrop]

Good researchers understand this. Too many do not. Journalists typically do not, but there are exceptions. The general public is usually waiting for the media to decide.

Agreed.

When I keep seeing this issue pop up I can't help but think about how, not so many generations ago many people (more than today - although the trend might be downward) were illiterate. I see a need for better science teaching at the grade school level so that we might have a more science literate society. With luck that might work against all of the con and scams being foisted on us in the name of science.