{"title":"Development and evaluation of a model to identify publications on the clinical impact of pharmacist interventions.","authors":"Maxime Thibault, Cynthia Tanguay","doi":"10.1016/j.sapharm.2024.09.004","DOIUrl":null,"url":null,"abstract":"<p><strong>Background: </strong>Pharmacists are increasingly involved in patient care. Pharmacy practice research helps them identify the most clinically meaningful interventions. However, the lack of a widely accepted controlled vocabulary in this field hinders the discovery of this literature.</p><p><strong>Objective: </strong>To compare the performance of a machine learning model with manual literature searches in identifying potentially relevant publications on the clinical impact of pharmacist interventions. To describe the dataset that was built.</p><p><strong>Methods: </strong>An automated PubMed search was performed weekly starting in November 2021. Titles and abstracts were retrieved and independently evaluated by two reviewers to select potentially relevant publications on the clinical impact of pharmacists. A Cohen's kappa score was calculated. Data was collected during an 11-month period to train a machine learning model. It was evaluated prospectively during a 5-month period (predictions were collected without being shown to the reviewers). The performance of the model was compared with manual searches (positive predictive value [PPV] and sensitivity).</p><p><strong>Results: </strong>A transformers-based model was selected. During the prospective evaluation period, 114/1631 (7 %) publications met selection criteria. If the model had been used, 1273/1631 (78 %) would not have needed review. Only 3/114 (3 %) would have been incorrectly excluded. The model showed a PPV of 0.310 and a sensitivity of 0.974. The best manual search showed a PPV of 0.046 and a sensitivity of 0.711. On December 12, 2023, the dataset contained 8607 publications, of which 544 (6 %) met the criteria. The kappa between reviewers was 0.786. The dataset and the model were used to develop a website and a newsletter to share publications (https://impactpharmacy.net).</p><p><strong>Conclusion: </strong>A machine learning model was developed and performs better than manual PubMed searches to identify potentially relevant publications. It represents a considerable workload reduction. This tool can assist pharmacists and other stakeholders in finding evidence that support pharmacists' interventions.</p>","PeriodicalId":48126,"journal":{"name":"Research in Social & Administrative Pharmacy","volume":" ","pages":"1134-1141"},"PeriodicalIF":3.7000,"publicationDate":"2024-12-01","publicationTypes":"Journal Article","fieldsOfStudy":null,"isOpenAccess":false,"openAccessPdf":"","citationCount":"0","resultStr":null,"platform":"Semanticscholar","paperid":null,"PeriodicalName":"Research in Social & Administrative Pharmacy","FirstCategoryId":"3","ListUrlMain":"https://doi.org/10.1016/j.sapharm.2024.09.004","RegionNum":3,"RegionCategory":"医学","ArticlePicture":[],"TitleCN":null,"AbstractTextCN":null,"PMCID":null,"EPubDate":"2024/9/19 0:00:00","PubModel":"Epub","JCR":"Q1","JCRName":"PUBLIC, ENVIRONMENTAL & OCCUPATIONAL HEALTH","Score":null,"Total":0}
引用次数: 0
Abstract
Background: Pharmacists are increasingly involved in patient care. Pharmacy practice research helps them identify the most clinically meaningful interventions. However, the lack of a widely accepted controlled vocabulary in this field hinders the discovery of this literature.
Objective: To compare the performance of a machine learning model with manual literature searches in identifying potentially relevant publications on the clinical impact of pharmacist interventions. To describe the dataset that was built.
Methods: An automated PubMed search was performed weekly starting in November 2021. Titles and abstracts were retrieved and independently evaluated by two reviewers to select potentially relevant publications on the clinical impact of pharmacists. A Cohen's kappa score was calculated. Data was collected during an 11-month period to train a machine learning model. It was evaluated prospectively during a 5-month period (predictions were collected without being shown to the reviewers). The performance of the model was compared with manual searches (positive predictive value [PPV] and sensitivity).
Results: A transformers-based model was selected. During the prospective evaluation period, 114/1631 (7 %) publications met selection criteria. If the model had been used, 1273/1631 (78 %) would not have needed review. Only 3/114 (3 %) would have been incorrectly excluded. The model showed a PPV of 0.310 and a sensitivity of 0.974. The best manual search showed a PPV of 0.046 and a sensitivity of 0.711. On December 12, 2023, the dataset contained 8607 publications, of which 544 (6 %) met the criteria. The kappa between reviewers was 0.786. The dataset and the model were used to develop a website and a newsletter to share publications (https://impactpharmacy.net).
Conclusion: A machine learning model was developed and performs better than manual PubMed searches to identify potentially relevant publications. It represents a considerable workload reduction. This tool can assist pharmacists and other stakeholders in finding evidence that support pharmacists' interventions.
期刊介绍：
Research in Social and Administrative Pharmacy (RSAP) is a quarterly publication featuring original scientific reports and comprehensive review articles in the social and administrative pharmaceutical sciences. Topics of interest include outcomes evaluation of products, programs, or services; pharmacoepidemiology; medication adherence; direct-to-consumer advertising of prescription medications; disease state management; health systems reform; drug marketing; medication distribution systems such as e-prescribing; web-based pharmaceutical/medical services; drug commerce and re-importation; and health professions workforce issues.
求助内容：
应助结果提醒方式：
京公网安备 11010802042870号
