Kristina Lindquist, Derek Meeks, Kyle Mefferd, Cheryl Vanier, Terrence W Miller
{"title":"To the Point: Substituting SOAP Notes for Vignettes in Preclinical Assessment Question Stems.","authors":"Kristina Lindquist, Derek Meeks, Kyle Mefferd, Cheryl Vanier, Terrence W Miller","doi":"10.1007/s40670-025-02319-6","DOIUrl":null,"url":null,"abstract":"<p><strong>Introduction: </strong>Vignette-based, multiple-choice questions (MCQs) are common in medical school assessments and licensing exams. Ideally each MCQ assesses a specific objective, but question stems may inadvertently assess reading speed and comprehension by introducing construct-irrelevant variance (CIV). The authors hypothesized that question stems written as a Subjective, Objective, Assessment, and Plan (SOAP) note would address this by reducing CIV resulting in a reduction in time students needed to answer the question.</p><p><strong>Methods: </strong>Vignettes were re-written with the same content presented as a SOAP note. Each of ten exams administered to second-year osteopathic medical students over the academic year possessed four questions in which half the students were tested using a vignette and half the SOAP note version. Data were analyzed using a multi-level, mixed-effects model with question type the main effect; question difficulty, student academic rank, and time during the academic year of exam administration served as covariates. Additional analyses included comparisons of mean difficulty indices and discrimination indices for the two question types.</p><p><strong>Results: </strong>Students averaged 3.77 s longer on each vignette-based question compared to the SOAP note counterparts (<i>p</i> < 0.018). Average difficulty indices and point biserials did not differ between SOAP note- and vignette-based questions. Time per question did not change significantly over the year, nor did it vary significantly among students in different academic quartiles.</p><p><strong>Conclusions: </strong>Reduced CIV in the SOAP note stem sharpens the focus of the question on assessing a specific objective, resulting in students spending less time answering the question.</p>","PeriodicalId":37113,"journal":{"name":"Medical Science Educator","volume":"35 3","pages":"1423-1430"},"PeriodicalIF":1.8000,"publicationDate":"2025-02-15","publicationTypes":"Journal Article","fieldsOfStudy":null,"isOpenAccess":false,"openAccessPdf":"https://www.ncbi.nlm.nih.gov/pmc/articles/PMC12228897/pdf/","citationCount":"0","resultStr":null,"platform":"Semanticscholar","paperid":null,"PeriodicalName":"Medical Science Educator","FirstCategoryId":"1085","ListUrlMain":"https://doi.org/10.1007/s40670-025-02319-6","RegionNum":0,"RegionCategory":null,"ArticlePicture":[],"TitleCN":null,"AbstractTextCN":null,"PMCID":null,"EPubDate":"2025/6/1 0:00:00","PubModel":"eCollection","JCR":"Q2","JCRName":"EDUCATION, SCIENTIFIC DISCIPLINES","Score":null,"Total":0}
引用次数: 0
Abstract
Introduction: Vignette-based, multiple-choice questions (MCQs) are common in medical school assessments and licensing exams. Ideally each MCQ assesses a specific objective, but question stems may inadvertently assess reading speed and comprehension by introducing construct-irrelevant variance (CIV). The authors hypothesized that question stems written as a Subjective, Objective, Assessment, and Plan (SOAP) note would address this by reducing CIV resulting in a reduction in time students needed to answer the question.
Methods: Vignettes were re-written with the same content presented as a SOAP note. Each of ten exams administered to second-year osteopathic medical students over the academic year possessed four questions in which half the students were tested using a vignette and half the SOAP note version. Data were analyzed using a multi-level, mixed-effects model with question type the main effect; question difficulty, student academic rank, and time during the academic year of exam administration served as covariates. Additional analyses included comparisons of mean difficulty indices and discrimination indices for the two question types.
Results: Students averaged 3.77 s longer on each vignette-based question compared to the SOAP note counterparts (p < 0.018). Average difficulty indices and point biserials did not differ between SOAP note- and vignette-based questions. Time per question did not change significantly over the year, nor did it vary significantly among students in different academic quartiles.
Conclusions: Reduced CIV in the SOAP note stem sharpens the focus of the question on assessing a specific objective, resulting in students spending less time answering the question.
期刊介绍:
Medical Science Educator is the successor of the journal JIAMSE. It is the peer-reviewed publication of the International Association of Medical Science Educators (IAMSE). The Journal offers all who teach in healthcare the most current information to succeed in their task by publishing scholarly activities, opinions, and resources in medical science education. Published articles focus on teaching the sciences fundamental to modern medicine and health, and include basic science education, clinical teaching, and the use of modern education technologies. The Journal provides the readership a better understanding of teaching and learning techniques in order to advance medical science education.