Testing a test: Analysis of the consistency of a pharmacology exam.

Abstract

In higher education, learning variables are critical for professional activity and they should be properly assessed. Therefore, adequacy of evaluation instruments is very important. Consistency of an exam is one characteristic frequently studied to determine exam reliability. The subject of this study is a Pharmacology test of 70 items, taken by 849 medical students during a Pharmacology course at Medical School in the National University of Mexico in 2008. It explores whether students have been exposed to the same knowledge domain and if consistent execution in all parts of the exam is seen. Exam items were analyzed as independent items and as grouped items (by topic) and tested for consistency with Cronbach's test. Data were processed with the JMP statistical program (SAS Institute Inc.). The coefficient alpha was greater for independent items. We consider that grouping of items into topics tests integration of knowledge; a reasonable degree of consistency found when items are grouped, might favor this conclusion. Information obtained from the analysis of consistency of the exam, should provide elements for feedback during a course. Long term goals of this study include: to develop test instruments useful in formative evaluation and knowledge integration, to characterize the specificity of a Pharmacology exam, to determine degrees of knowledge acquisition and to get feedback about the quality of courses and exams.