OBJECTIVE: To examine the risk of false-positive reporting within high-quality randomized controlled trials (RCTs) in the sports physical therapy field. DESIGN: Cross-sectional. METHODS: We searched the Physiotherapy Evidence Database for parallel-design, 2-arm RCTs reporting positive treatment effects, based on null-hypothesis significance testing, and scoring greater than 6/10 on the Physiotherapy Evidence Database scale. No restrictions were made on pathology, intervention, or outcome variables. Sixty-two of 212 RCTs reported positive effects in at least 1 outcome variable. We estimated false-positive risk (FPR) with an online calculator, based on number of participants, P value, and effect size. For each study, FPR was estimated using a range of prior probability assumptions: 0.2 (skeptical hypothesis), 0.5, and 0.8 (optimistic hypothesis). RESULTS: We calculated the FPR associated with 189 statistically significant findings (P<.05) reported across 44 trials. The median FPR was 9% (25th-75th percentile, 2%-24%). Sixty-three percent of statistically significant results (119/189) had an FPR greater than 5%, and 18% (35/189) had an FPR greater than 50%. Changing the prior probability from skeptical to optimistic reduced the median FPR from 29% (25th-75th percentile, 9%-56%) to 2% (25th-75th percentile, 0.6%-7.0%). CONCLUSION: High-quality RCTs using null-hypothesis significance testing often overestimated treatment effects. The median FPR was 9%: in 1 in 10 trials, the researchers falsely concluded that there was a treatment effect. Future RCTs in sports physical therapy should be informed by prestudy odds and a minimum FPR estimation. J Orthop Sports Phys Ther 2020;50(2):104-109. doi:10.2519/jospt.2020.9264.
|Number of pages||6|
|Journal||Journal of Orthopaedic and Sports Physical Therapy|
|Early online date||31 Jan 2020|
|Publication status||E-pub ahead of print - 31 Jan 2020|
Bibliographical notePublisher Copyright:
Copyright © 2020 Journal of Orthopaedic & Sports Physical Therapy.
Copyright 2020 Elsevier B.V., All rights reserved.
- P values
- false-positive risk
- null-hypothesis significance testing
- randomized controlled trials