Medical students learning about AI – with AI?

Groundbreaking scientific discoveries are often neglected by Medical Schools due to curricular demands or scepticism towards innovations. The use of artificial intelligence (AI) in health care marks a paradigm shift that poses challenges for medical professionals, medical students and society. Today's students are the first generation of medical trainees confronted with this paradigm shift and will have to apply AI in practice in the near future. Therefore, medical students need to know how AI can be applied and how to use its results in an appropriate manner. Based on the current state of research, it is currently unforeseeable which theoretical and practical AI competencies will be required in medical practice. This poses risks, as students are inadequately prepared to administer and reflect on possibilities, limits and ethical-legal challenges of AI in applied medical science.

After the release of openAI's chatbot ChatGPT, we took up the public debate about ethical aspects and potential benefits as well as caveats of the application in our AI course. We used the hype surrounding ChatGPT to reduce students' concerns about AI and at the same time illustrate the potential impact of AI. As a final exercise of our AI course, students were invited to ask the chatbot about structural biases in the use of AI in health care. One exemplary question dealt with the influence of AI on transparency in medical diagnosis. The accuracy of ChatGPT's answers were then reviewed by participants based on ‘traditional’ sources (e.g., textbooks and online sources). Students were supposed to gain practical experience in the use of AI (application competence) on the one hand and learn to examine the answers of AI-applications in a critical manner (appraisal competence) on the other hand. By using this application-oriented final task, we moved away from the lower levels of Bloom's taxonomy¹ and reached an evaluation-oriented meta-level.

As a result, we take away two lessons learned: Through the assessment of the final exercise's results, we gained a unique insight into students' AI reflection skills. For instance, evaluation of ChatGPT's answers was consistently seen as potentially biased, whereas the selection bias of traditional sources such as online search engines or research literature remained unquestioned.

In addition, we found greater student interest in the final exercise compared with the previous cohort. This is reflected in the evaluation of the final assignment, which was rated more positively than in the years before.

It can be stated that the actual use of innovative medical-technological developments can increase the reflection competence of medical students. This is particularly important for AI applications, as these are increasingly reaching clinical practice and are sometimes subject to unfavourable biases that could limit the standard of medical treatment if future physicians are not trained to respond to them.