Investigating the impact of trial retractions on the healthcare evidence ecosystem VITALITY Study I: retrospective cohort study
Chang Xu; Shiqi Fan; Yuan Tian; Fuchen Liu; Luis Furuya-Kanamori; Justin Clark; Chao Zhang; Sheng Li; Lifeng Lin; Haitao Chu; Sheyu Li; Su Golder; Yoon Loke; Sunita Vohra; Paul Glasziou; Suhail A Doi; Hui Liu
OBJECTIVE
To investigate the impact of retracted trials on the production and use of healthcare evidence in the evidence ecosystem.
DESIGN
Retrospective cohort study based on forward citation searching.
DATA SOURCES
Retraction Watch up to 5 November 2024.
STUDY SELECTION
Randomised controlled trials in humans that were retracted for any reason.
METHODS
Forward citation searching via Google Scholar and Scopus was used to identify evidence synthesis research (21 November 2024) that quantitatively incorporated retracted trials. Data were independently extracted by two groups of researchers. The results of meta-analyses were updated after exclusion of the retracted trials. The proportions of meta-analyses that changed direction of the pooled effect and/or the significance of the P value were estimated. A generalised linear mixed model was used to investigate the association between the number of included studies and the impact, measured by odds ratio and 95% confidence interval (CI). The impact of distorted evidence on clinical practice guidelines was also investigated on the basis of citation searching.
RESULTS
The searches identified 1330 retracted trials and 847 systematic reviews that quantitatively synthesised retracted trials, with a total of 3902 meta-analyses that could be replicated. After the potential clustering effects were accounted for, the exclusion of the retracted trials led to a change in the direction of the pooled effect in 8.4% (95% CI 6.8% to 10.1%), in its statistical significance in 16.0% (14.2% to 17.9%), and in both direction and significance in 3.9% (2.5% to 5.2%) and a >50% change in the magnitude of the effect in 15.7% (13.5% to 17.9%). An obvious non-linear association existed between the number of included studies and the impact on the results, with a lower number of studies having higher impact (eg, for 10 studies versus ≥20 studies, change of direction: odds ratio 2.63, 95% CI 1.29 to 5.38; P<0.001). Evidence from 68 systematic reviews with conclusions distorted by retracted trials was used in 157 guideline documents.
CONCLUSION
Retracted trials have a substantial impact on the evidence ecosystem, including evidence synthesis, clinical practice guidelines, and evidence based clinical practice. Evidence generators, synthesisers, and users must pay attention to this problem, and feasible approaches that assist with easier identification and correction of such potential contamination are needed.
STUDY REGISTRATION
Open Science Framework (https://osf.io/7eazq/).
Link | PDF (BMJ) [Open Access]
Chang Xu; Shiqi Fan; Yuan Tian; Fuchen Liu; Luis Furuya-Kanamori; Justin Clark; Chao Zhang; Sheng Li; Lifeng Lin; Haitao Chu; Sheyu Li; Su Golder; Yoon Loke; Sunita Vohra; Paul Glasziou; Suhail A Doi; Hui Liu
OBJECTIVE
To investigate the impact of retracted trials on the production and use of healthcare evidence in the evidence ecosystem.
DESIGN
Retrospective cohort study based on forward citation searching.
DATA SOURCES
Retraction Watch up to 5 November 2024.
STUDY SELECTION
Randomised controlled trials in humans that were retracted for any reason.
METHODS
Forward citation searching via Google Scholar and Scopus was used to identify evidence synthesis research (21 November 2024) that quantitatively incorporated retracted trials. Data were independently extracted by two groups of researchers. The results of meta-analyses were updated after exclusion of the retracted trials. The proportions of meta-analyses that changed direction of the pooled effect and/or the significance of the P value were estimated. A generalised linear mixed model was used to investigate the association between the number of included studies and the impact, measured by odds ratio and 95% confidence interval (CI). The impact of distorted evidence on clinical practice guidelines was also investigated on the basis of citation searching.
RESULTS
The searches identified 1330 retracted trials and 847 systematic reviews that quantitatively synthesised retracted trials, with a total of 3902 meta-analyses that could be replicated. After the potential clustering effects were accounted for, the exclusion of the retracted trials led to a change in the direction of the pooled effect in 8.4% (95% CI 6.8% to 10.1%), in its statistical significance in 16.0% (14.2% to 17.9%), and in both direction and significance in 3.9% (2.5% to 5.2%) and a >50% change in the magnitude of the effect in 15.7% (13.5% to 17.9%). An obvious non-linear association existed between the number of included studies and the impact on the results, with a lower number of studies having higher impact (eg, for 10 studies versus ≥20 studies, change of direction: odds ratio 2.63, 95% CI 1.29 to 5.38; P<0.001). Evidence from 68 systematic reviews with conclusions distorted by retracted trials was used in 157 guideline documents.
CONCLUSION
Retracted trials have a substantial impact on the evidence ecosystem, including evidence synthesis, clinical practice guidelines, and evidence based clinical practice. Evidence generators, synthesisers, and users must pay attention to this problem, and feasible approaches that assist with easier identification and correction of such potential contamination are needed.
STUDY REGISTRATION
Open Science Framework (https://osf.io/7eazq/).
Link | PDF (BMJ) [Open Access]
