Pharmacy meets AI: Effect of a drug information activity on student perceptions of generative artificial intelligence

Abstract

Objective

The current study assessed pharmacy students' perceptions about generative AI before and after participation in a ChatGPT-based drug information activity.

Methods

In 2024, students at three colleges of pharmacy completed a baseline and post-activity survey on their perceptions of ChatGPT including its reliability, usefulness, and impact on academic performance and critical thinking. The survey was modified from the TAME-ChatGPT assessment and used a 5-point Likert scale. After the baseline survey, students answered clinically relevant drug information questions on their own using primary or tertiary resources and compared their answers with ChatGPT responses. Independent t-test samples were used to compare baseline and post-activity surveys.

Results

A total of 227 students completed the pre-survey and 203 students completed the post-survey. Students' concerns about the reliability of ChatGPT increased after completing the drug information activity (pre-survey: 3.57 ± 0.96; post-survey: 3.88 ± 1.11; p = 0.002). Students' concerns about reliance on ChatGPT and prevention of critical thinking increased (pre-survey: 3.30 ± 1.34; post-survey: 3.57 ± 1.21; p = 0.031). The following areas decreased after the activity: enthusiasm about ChatGPT as learning and research tool (pre-survey: 3.60 ± 1.02; post-survey: 3.32 ± 1.18; p = 0.008), viewing ChatGPT as an important tool for academic success (pre-survey: 3.40 ± 1.13; post-survey: 3.12 ± 1.23; p = 0.015), and concern regarding being accused of plagiarism when using ChatGPT(pre-survey: 4.12 ± 0.96; post-survey: 3.91 ± 1.10; p = 0.031). Open-ended responses revealed that students largely perceived ChatGPT as unreliable for drug information, citing concerns about accuracy and outdated content. However, some students noted its potential usefulness for non-clinical tasks such as generating ideas, organizing content, or providing general overviews.

Conclusion

After a hands-on ChatGPT-based drug information activity, pharmacy students reported increased concerns about reliability and over-reliance on artificial intelligence-based technology. The results of this study may encourage pharmacy educators to implement classroom activities for active exploration of the benefits and challenges of generative AI.

Contribution to literature

Limited published data describes pharmacy student perceptions of artificial intelligence platforms as a drug information source. There is even less literature with pre- and post-data after implementing an activity in which students gain hands-on experience critiquing an artificial intelligence platform response. Therefore, this study was conducted to evaluate student perceptions after using ChatGPT in the classroom and comparing its performance to their own responses based on information from primary and tertiary literature. The results demonstrate that despite enthusiasm before using ChatGPT, concerns for reliability and hindering thinking increased after a observing the limitations of its performance in answering drug information questions.