Language:

Machine learning reduced workload with minimal risk of missing studies: development and evaluation of a randomized controlled trial classifier for Cochrane Reviews

Journal of clinical epidemiology, 2021-05, Vol.133, p.140-151 [Peer Reviewed Journal]

2020 The Authors ;Copyright © 2020 The Authors. Published by Elsevier Inc. All rights reserved. ;2020. The Authors ;2020 The Authors 2020 ;ISSN: 0895-4356 ;EISSN: 1878-5921 ;DOI: 10.1016/j.jclinepi.2020.11.003 ;PMID: 33171275

Full text available

Citations Cited by

Actions
1. Add to My Research
2. Remove from My Research
3. E-mail
4. Print
5. Permalink
6. Citation
7. EasyBib
8. EndNote
9. RefWorks
10. Delicious
11. Export RIS
12. Export BibTeX

Title:
Machine learning reduced workload with minimal risk of missing studies: development and evaluation of a randomized controlled trial classifier for Cochrane Reviews
Author: Thomas, James ; McDonald, Steve ; Noel-Storr, Anna ; Shemilt, Ian ; Elliott, Julian ; Mavergames, Chris ; Marshall, Iain J.
Subjects: Algorithms ; Automation ; Bibliographic data bases ; Calibration ; Classification ; Classifiers ; Clinical trials ; Cochrane Library ; Confidence intervals ; Cost control ; Crowdsourcing ; Databases, Bibliographic - standards ; Databases, Bibliographic - statistics & numerical data ; Datasets ; Efficiency ; Epidemiology ; Evaluation ; Humans ; Information retrieval ; Information Storage and Retrieval - methods ; Information Storage and Retrieval - standards ; Information Storage and Retrieval - statistics & numerical data ; Learning algorithms ; Machine Learning ; Methods/methodology ; Original ; Production methods ; Randomized controlled trials ; Randomized Controlled Trials as Topic - classification ; Randomized Controlled Trials as Topic - methods ; Randomized Controlled Trials as Topic - standards ; Randomized Controlled Trials as Topic - statistics & numerical data ; Recall ; Reviews ; Searching ; Study classifiers ; Systematic review ; Systematic reviews ; Systematic Reviews as Topic - methods ; Systematic Reviews as Topic - standards ; Validity ; Workflow ; Workload ; Workload - statistics & numerical data ; Workloads
Is Part Of: Journal of clinical epidemiology, 2021-05, Vol.133, p.140-151
Description: This study developed, calibrated, and evaluated a machine learning classifier designed to reduce study identification workload in Cochrane for producing systematic reviews. A machine learning classifier for retrieving randomized controlled trials (RCTs) was developed (the “Cochrane RCT Classifier”), with the algorithm trained using a data set of title–abstract records from Embase, manually labeled by the Cochrane Crowd. The classifier was then calibrated using a further data set of similar records manually labeled by the Clinical Hedges team, aiming for 99% recall. Finally, the recall of the calibrated classifier was evaluated using records of RCTs included in Cochrane Reviews that had abstracts of sufficient length to allow machine classification. The Cochrane RCT Classifier was trained using 280,620 records (20,454 of which reported RCTs). A classification threshold was set using 49,025 calibration records (1,587 of which reported RCTs), and our bootstrap validation found the classifier had recall of 0.99 (95% confidence interval 0.98–0.99) and precision of 0.08 (95% confidence interval 0.06–0.12) in this data set. The final, calibrated RCT classifier correctly retrieved 43,783 (99.5%) of 44,007 RCTs included in Cochrane Reviews but missed 224 (0.5%). Older records were more likely to be missed than those more recently published. The Cochrane RCT Classifier can reduce manual study identification workload for Cochrane Reviews, with a very low and acceptable risk of missing eligible RCTs. This classifier now forms part of the Evidence Pipeline, an integrated workflow deployed within Cochrane to help improve the efficiency of the study identification processes that support systematic review production. •Systematic review processes need to become more efficient.•Machine learning is sufficiently mature for real-world use.•A machine learning classifier was built using data from Cochrane Crowd.•It was calibrated to achieve very high recall.•It is now live and in use in Cochrane review production systems.
Publisher: United States: Elsevier Inc
Language: English
Identifier: ISSN: 0895-4356
EISSN: 1878-5921
DOI: 10.1016/j.jclinepi.2020.11.003
PMID: 33171275
Source: MEDLINE
ProQuest Central

Back to results list


INSPIRE LIBRARY - TON DUC THANG UNIVERSITY	(84-028) 37 755 057	Feedback
19 Nguyen Huu Tho St. Dist.7, HCM	thuvien@tdtu.edu.vn	Feedback

Machine learning reduced workload with minimal risk of missing studies: development and evaluation of a randomized controlled trial classifier for Cochrane Reviews

2020 The Authors ;Copyright © 2020 The Authors. Published by Elsevier Inc. All rights reserved. ;2020. The Authors ;2020 The Authors 2020 ;ISSN: 0895-4356 ;EISSN: 1878-5921 ;DOI: 10.1016/j.jclinepi.2020.11.003 ;PMID: 33171275

Searching Remote Databases, Please Wait