Effective use of Item Analysis to improve the Reliability and Validity of Undergraduate Medical Examinations: Evaluating the same exam over many years: a different approach.
Nadeem Alam Zubairi, Turki Saad AlAhmadi, Mohamed Hesham Ibrahim, Moustafa Abdelaal Hegazi, Fahad Ussif Gadi
{"title":"Effective use of Item Analysis to improve the Reliability and Validity of Undergraduate Medical Examinations: Evaluating the same exam over many years: a different approach.","authors":"Nadeem Alam Zubairi, Turki Saad AlAhmadi, Mohamed Hesham Ibrahim, Moustafa Abdelaal Hegazi, Fahad Ussif Gadi","doi":"10.12669/pjms.41.3.10693","DOIUrl":null,"url":null,"abstract":"<p><strong>Objective: </strong>MCQ exams are part of end-module assessments in undergraduate medical institutions. Item Analysis (IA) is the best tool to check their reliability and validity. It provides the Reliability Coefficient KR20, Difficulty Index (DI), Discrimination Index (DISC), and Distractor Efficiency (DE). Almost all research papers on IA are based on single exam analysis. We examined the IA of multiple exams of the same module, taken in four years. We aimed to explore the required consistency over the years and the effectiveness of IA-based post-exam measures.</p><p><strong>Methodology: </strong>Item Analysis of eight final MCQ exams of the Pediatric module from 2020-21 to 2023-24, at the Faculty of Medicine in Rabigh, King Abdulaziz University, Saudi Arabia, were included in the study.</p><p><strong>Results: </strong>All exams had KR20 of 90 and above indicating excellent reliability. Difficulty levels were consistent except for a single year. Discriminative ability was maintained over the years. Only 28 out of 800 MCQs had a negative DISC. All exams maintained good DE. Only 15 MCQs over four years had zero DE. The practice of reviewing all Non-Functional Distractors yielded a gradual improvement in exam quality.</p><p><strong>Conclusion: </strong>Besides the IA of individual exams, it is also recommended that IA of the same exam be evaluated over 4-5 years to see consistency and trends towards improvement. It helps in improving the reliability and validity by addressing deficiencies and deviations from the recommended standards.</p>","PeriodicalId":19958,"journal":{"name":"Pakistan Journal of Medical Sciences","volume":"41 3","pages":"810-815"},"PeriodicalIF":1.2000,"publicationDate":"2025-03-01","publicationTypes":"Journal Article","fieldsOfStudy":null,"isOpenAccess":false,"openAccessPdf":"https://www.ncbi.nlm.nih.gov/pmc/articles/PMC11911747/pdf/","citationCount":"0","resultStr":null,"platform":"Semanticscholar","paperid":null,"PeriodicalName":"Pakistan Journal of Medical Sciences","FirstCategoryId":"3","ListUrlMain":"https://doi.org/10.12669/pjms.41.3.10693","RegionNum":4,"RegionCategory":"医学","ArticlePicture":[],"TitleCN":null,"AbstractTextCN":null,"PMCID":null,"EPubDate":"","PubModel":"","JCR":"Q2","JCRName":"MEDICINE, GENERAL & INTERNAL","Score":null,"Total":0}
引用次数: 0
Abstract
Objective: MCQ exams are part of end-module assessments in undergraduate medical institutions. Item Analysis (IA) is the best tool to check their reliability and validity. It provides the Reliability Coefficient KR20, Difficulty Index (DI), Discrimination Index (DISC), and Distractor Efficiency (DE). Almost all research papers on IA are based on single exam analysis. We examined the IA of multiple exams of the same module, taken in four years. We aimed to explore the required consistency over the years and the effectiveness of IA-based post-exam measures.
Methodology: Item Analysis of eight final MCQ exams of the Pediatric module from 2020-21 to 2023-24, at the Faculty of Medicine in Rabigh, King Abdulaziz University, Saudi Arabia, were included in the study.
Results: All exams had KR20 of 90 and above indicating excellent reliability. Difficulty levels were consistent except for a single year. Discriminative ability was maintained over the years. Only 28 out of 800 MCQs had a negative DISC. All exams maintained good DE. Only 15 MCQs over four years had zero DE. The practice of reviewing all Non-Functional Distractors yielded a gradual improvement in exam quality.
Conclusion: Besides the IA of individual exams, it is also recommended that IA of the same exam be evaluated over 4-5 years to see consistency and trends towards improvement. It helps in improving the reliability and validity by addressing deficiencies and deviations from the recommended standards.
期刊介绍：
It is a peer reviewed medical journal published regularly since 1984. It was previously known as quarterly "SPECIALIST" till December 31st 1999. It publishes original research articles, review articles, current practices, short communications & case reports. It attracts manuscripts not only from within Pakistan but also from over fifty countries from abroad.
Copies of PJMS are sent to all the import medical libraries all over Pakistan and overseas particularly in South East Asia and Asia Pacific besides WHO EMRO Region countries. Eminent members of the medical profession at home and abroad regularly contribute their write-ups, manuscripts in our publications. We pursue an independent editorial policy, which allows an opportunity to the healthcare professionals to express their views without any fear or favour. That is why many opinion makers among the medical and pharmaceutical profession use this publication to communicate their viewpoint.
求助内容：
应助结果提醒方式：
京公网安备 11010802042870号
