Development and Content Analysis Protocol for Evaluating Artificial Intelligence in Drug-Related Information

IF 2.1 4区医学 Q3 HEALTH CARE SCIENCES & SERVICES

Journal of evaluation in clinical practice Pub Date : 2024-12-16 DOI:10.1111/jep.14276

Dantony Castro Barros de Donato, Guilherme José Aguilar, Lucas Gaspar Ribeiro, Luiz Ricardo Albano dos Santos, Luana Michelly Aparecida Costa dos Santos, Wilbert Dener Lemos Costa, Alan Maicon de Oliveira

{"title":"Development and Content Analysis Protocol for Evaluating Artificial Intelligence in Drug-Related Information","authors":"Dantony Castro Barros de Donato, Guilherme José Aguilar, Lucas Gaspar Ribeiro, Luiz Ricardo Albano dos Santos, Luana Michelly Aparecida Costa dos Santos, Wilbert Dener Lemos Costa, Alan Maicon de Oliveira","doi":"10.1111/jep.14276","DOIUrl":null,"url":null,"abstract":"<div>\n \n \n <section>\n \n <h3> Introduction</h3>\n \n <p>Artificial intelligence (AI) has significant transformative potential across various sectors, particularly in health care. This study aims to develop a protocol for the content analysis of a method designed to assess AI applications in drug-related information, specifically focusing on contraindications, adverse reactions, and drug interactions. By addressing existing challenges, this preliminary research seeks to enhance the safe and reliable integration of AI into healthcare practices.</p>\n </section>\n \n <section>\n \n <h3> Methods</h3>\n \n <p>A study protocol was developed for the creation of the method, followed by an initial content analysis conducted by an expert panel. The method was established in phases: (1) Analysis of drug-related databases and form development; (2) AI configuration; (3) Expert panel review and initial validation.</p>\n </section>\n \n <section>\n \n <h3> Results</h3>\n \n <p>In Phase 1, the Micromedex, UpToDate, and Medscape databases were reviewed to establish terminology and classifications related to contraindications, adverse reactions, and drug interactions, resulting in the development of a questionnaire for the AI. Phase 2 involved configuring the Gemini AI tool to enhance response specificity. In Phase 3, AI responses to 30 questions were validated by an expert panel, yielding a 76.7% agreement rate for appropriateness, while 23.3% were deemed inappropriate, particularly concerning contraindicated drug interactions.</p>\n </section>\n \n <section>\n \n <h3> Conclusion</h3>\n \n <p>This preliminary study demonstrates the potential for using an AI-powered tool to standardize drug-related information retrieval, particularly for contraindications and adverse reactions. While AI responses were generally appropriate, improvements are needed in identifying contraindicated drug interactions. Further research with larger datasets and broader evaluations is required to enhance AI's reliability in healthcare settings.</p>\n </section>\n </div>","PeriodicalId":15997,"journal":{"name":"Journal of evaluation in clinical practice","volume":"31 1","pages":""},"PeriodicalIF":2.1000,"publicationDate":"2024-12-16","publicationTypes":"Journal Article","fieldsOfStudy":null,"isOpenAccess":false,"openAccessPdf":"","citationCount":"0","resultStr":null,"platform":"Semanticscholar","paperid":null,"PeriodicalName":"Journal of evaluation in clinical practice","FirstCategoryId":"3","ListUrlMain":"https://onlinelibrary.wiley.com/doi/10.1111/jep.14276","RegionNum":4,"RegionCategory":"医学","ArticlePicture":[],"TitleCN":null,"AbstractTextCN":null,"PMCID":null,"EPubDate":"","PubModel":"","JCR":"Q3","JCRName":"HEALTH CARE SCIENCES & SERVICES","Score":null,"Total":0}

引用次数: 0

Abstract

Introduction

Artificial intelligence (AI) has significant transformative potential across various sectors, particularly in health care. This study aims to develop a protocol for the content analysis of a method designed to assess AI applications in drug-related information, specifically focusing on contraindications, adverse reactions, and drug interactions. By addressing existing challenges, this preliminary research seeks to enhance the safe and reliable integration of AI into healthcare practices.

Methods

A study protocol was developed for the creation of the method, followed by an initial content analysis conducted by an expert panel. The method was established in phases: (1) Analysis of drug-related databases and form development; (2) AI configuration; (3) Expert panel review and initial validation.

Results

In Phase 1, the Micromedex, UpToDate, and Medscape databases were reviewed to establish terminology and classifications related to contraindications, adverse reactions, and drug interactions, resulting in the development of a questionnaire for the AI. Phase 2 involved configuring the Gemini AI tool to enhance response specificity. In Phase 3, AI responses to 30 questions were validated by an expert panel, yielding a 76.7% agreement rate for appropriateness, while 23.3% were deemed inappropriate, particularly concerning contraindicated drug interactions.

Conclusion

This preliminary study demonstrates the potential for using an AI-powered tool to standardize drug-related information retrieval, particularly for contraindications and adverse reactions. While AI responses were generally appropriate, improvements are needed in identifying contraindicated drug interactions. Further research with larger datasets and broader evaluations is required to enhance AI's reliability in healthcare settings.

查看原文本刊更多论文

导言：人工智能（AI）在各行各业都具有巨大的变革潜力，尤其是在医疗保健领域。本研究旨在为评估人工智能在药物相关信息中应用的方法制定内容分析协议，尤其侧重于禁忌症、不良反应和药物相互作用。通过应对现有挑战，这项初步研究旨在加强将人工智能安全可靠地融入医疗保健实践：方法：为创建该方法制定了研究方案，随后由专家小组进行了初步内容分析。该方法分阶段制定：（1）分析药物相关数据库并开发表格；（2）人工智能配置；（3）专家小组审查和初步验证：在第 1 阶段，对 Micromedex、UpToDate 和 Medscape 数据库进行了审查，以确定与禁忌症、不良反应和药物相互作用相关的术语和分类，从而为人工智能开发了一份问卷。第二阶段包括配置 Gemini AI 工具，以提高回复的特异性。在第 3 阶段，专家小组对 30 个问题的人工智能回答进行了验证，结果显示 76.7% 的回答是恰当的，而 23.3% 的回答被认为是不恰当的，尤其是在禁忌药物相互作用方面：这项初步研究证明了使用人工智能驱动的工具来规范药物相关信息检索的潜力，尤其是在禁忌症和不良反应方面。虽然人工智能的反应总体上是适当的，但在识别药物相互作用禁忌方面还需要改进。要提高人工智能在医疗环境中的可靠性，还需要对更大的数据集和更广泛的评估进行进一步研究。

本文章由计算机程序翻译，如有差异，请以英文原文为准。

求助全文

约1分钟内获得全文求助全文

来源期刊

Journal of evaluation in clinical practice 医学-卫生保健

CiteScore

4.80

自引率

4.20%

发文量

143

审稿时长

3-8 weeks

期刊介绍： The Journal of Evaluation in Clinical Practice aims to promote the evaluation and development of clinical practice across medicine, nursing and the allied health professions. All aspects of health services research and public health policy analysis and debate are of interest to the Journal whether studied from a population-based or individual patient-centred perspective. Of particular interest to the Journal are submissions on all aspects of clinical effectiveness and efficiency including evidence-based medicine, clinical practice guidelines, clinical decision making, clinical services organisation, implementation and delivery, health economic evaluation, health process and outcome measurement and new or improved methods (conceptual and statistical) for systematic inquiry into clinical practice. Papers may take a classical quantitative or qualitative approach to investigation (or may utilise both techniques) or may take the form of learned essays, structured/systematic reviews and critiques.