Many High-Quality Randomized Controlled Trials in Sports Physical Therapy Are Making False-Positive Claims of Treatment Effect, 2020, Bleakley et al.

Many High-Quality Randomized Controlled Trials in Sports Physical Therapy Are Making False-Positive Claims of Treatment Effect: A Systematic Survey
Chris Bleakley; Jonathan Reijgers; James M. Smoliga

Objective
To examine the risk of false-positive reporting within high-quality randomized controlled trials (RCTs) in the sports physical therapy field.

Design
Cross-sectional. Methods We searched the Physiotherapy Evidence Database for parallel-design, 2-arm RCTs reporting positive treatment effects, based on null-hypothesis significance testing, and scoring greater than 6/10 on the Physiotherapy Evidence Database scale. No restrictions were made on pathology, intervention, or outcome variables. Sixty-two of 212 RCTs reported positive effects in at least 1 outcome variable. We estimated false-positive risk (FPR) with an online calculator, based on number of participants, P value, and effect size. For each study, FPR was estimated using a range of prior probability assumptions: 0.2 (skeptical hypothesis), 0.5, and 0.8 (optimistic hypothesis).

Results
We calculated the FPR associated with 189 statistically significant findings (P<.05) reported across 44 trials. The median FPR was 9% (25th–75th percentile, 2%–24%). Sixty-three percent of statistically significant results (119/189) had an FPR greater than 5%, and 18% (35/189) had an FPR greater than 50%. Changing the prior probability from skeptical to optimistic reduced the median FPR from 29% (25th–75th percentile, 9%–56%) to 2% (25th–75th percentile, 0.6%–7.0%).

Conclusion
High-quality RCTs using null-hypothesis significance testing often overestimated treatment effects. The median FPR was 9%: in 1 in 10 trials, the researchers falsely concluded that there was a treatment effect. Future RCTs in sports physical therapy should be informed by prestudy odds and a minimum FPR estimation.

Link | PDF (Journal of Orthopaedic & Sports Physical Therapy)