Educational Measurement-Issues and Practice最新文献

{"title":"Still Interested in Multidimensional Item Response Theory Modeling? Here Are Some Thoughts on How to Make It Work in Practice","authors":"Terry A. Ackerman, Richard M. Luecht","doi":"10.1111/emip.12645","DOIUrl":"https://doi.org/10.1111/emip.12645","url":null,"abstract":"<p>Given tremendous improvements over the past three to four decades in the computational methods and computer technologies needed to estimate the parameters for higher dimensionality models (Cai, <span>2010a, 2010b</span>, <span>2017</span>), we might expect that MIRT would by now be a widely used array of models and psychometric software tools being used operationally in many educational assessment settings. Perhaps one of the few areas where MIRT has helped practitioners is in the area of understanding Differential Item Functioning (DIF) (Ackerman & Ma, <span>2024</span>; Camilli, <span>1992</span>; Shealy & Stout, <span>1993</span>). Nevertheless, the expectation has not been met nor do there seem to be many operational initiatives to change the <i>status quo</i>.</p><p>Some research psychometricians might lament the lack of large-scale applications of MIRT in the field of educational assessment. However, the simple fact is that MIRT has not lived up to its early expectations nor its potential due to several barriers. Following a discussion of test purpose and metric design issues in the next section, we will examine some of the barriers associated with these topics and provide suggestions for overcoming or completely avoiding them.</p><p>Tests developed for one purpose are rarely of much utility for another purpose. For example, professional certification and licensure tests designed to optimize pass-fail classifications are often not very useful for reporting scores across a large proficiency range—at least not unless the tests are extremely long. Summative, and most interim assessments used in K–12 education, are usually designed to produce reliable total-test scores. The resulting scale scores are summarized as descriptive statistical aggregations of scale scores or other functions of the scores such as classifying students in ordered achievement levels (e.g., Below Basic, Basic, Proficient, Advanced), or in modeling student growth in a subject area as part of an educational accountability system. Some commercially available online “interim” assessments provide limited progress-oriented scores and subscores from on-demand tests. However, the defensible formative utility of most interim assessments remains limited because test development and psychometric analytics follow the summative assessment test design and development paradigm: focusing on maintaining vertically aligned or equated, unidimensional scores scales (e.g., a K–12 math scale).</p><p>The requisite test design and development frameworks for summative tests focus on the relationships between the item responses and the total test score scale (e.g., maximizing item-total score correlations and the conditional reliability within prioritized regions of that score scale).</p><p>Applying MIRT models to most summative or interim assessments makes little sense. The problem is that we continue to allow policymakers to make claims about score interpretations that are not support","PeriodicalId":47345,"journal":{"name":"Educational Measurement-Issues and Practice","volume":"43 4","pages":"93-100"},"PeriodicalIF":2.7,"publicationDate":"2024-12-18","publicationTypes":"Journal Article","fieldsOfStudy":null,"isOpenAccess":false,"openAccessPdf":"https://onlinelibrary.wiley.com/doi/epdf/10.1111/emip.12645","citationCount":null,"resultStr":null,"platform":"Semanticscholar","paperid":"143252772","PeriodicalName":null,"FirstCategoryId":null,"ListUrlMain":null,"RegionNum":4,"RegionCategory":"教育学","ArticlePicture":[],"TitleCN":null,"AbstractTextCN":null,"PMCID":"OA","EPubDate":null,"PubModel":null,"JCR":null,"JCRName":null,"Score":null,"Total":0}

{"title":"Personalizing Assessment: Dream or Nightmare?","authors":"Randy E. Bennett","doi":"10.1111/emip.12652","DOIUrl":"https://doi.org/10.1111/emip.12652","url":null,"abstract":"<p>Over our field's 100-year-plus history, standardization has been a central assumption in test theory and practice. The concept's justification turns on leveling the playing field by presenting all examinees with putatively equivalent experiences. Until relatively recently, our field has accepted that justification almost without question. In this article, I present a case for standardization's antithesis, personalization. Interestingly, personalized assessment has important precedents within the measurement community. As intriguing are some of the divergent ways in which personalization might be realized in practice. Those ways, however, suggest a host of serious issues. Despite those issues, both moral obligation and survival imperative counsel persistence in trying to personalize assessment.</p>","PeriodicalId":47345,"journal":{"name":"Educational Measurement-Issues and Practice","volume":"43 4","pages":"119-125"},"PeriodicalIF":2.7,"publicationDate":"2024-12-04","publicationTypes":"Journal Article","fieldsOfStudy":null,"isOpenAccess":false,"openAccessPdf":"","citationCount":null,"resultStr":null,"platform":"Semanticscholar","paperid":"143248372","PeriodicalName":null,"FirstCategoryId":null,"ListUrlMain":null,"RegionNum":4,"RegionCategory":"教育学","ArticlePicture":[],"TitleCN":null,"AbstractTextCN":null,"PMCID":"","EPubDate":null,"PubModel":null,"JCR":null,"JCRName":null,"Score":null,"Total":0}

{"title":"Measurement Reflections","authors":"John Fremer","doi":"10.1111/emip.12654","DOIUrl":"https://doi.org/10.1111/emip.12654","url":null,"abstract":"","PeriodicalId":47345,"journal":{"name":"Educational Measurement-Issues and Practice","volume":"43 4","pages":"101-103"},"PeriodicalIF":2.7,"publicationDate":"2024-12-03","publicationTypes":"Journal Article","fieldsOfStudy":null,"isOpenAccess":false,"openAccessPdf":"","citationCount":null,"resultStr":null,"platform":"Semanticscholar","paperid":"143248322","PeriodicalName":null,"FirstCategoryId":null,"ListUrlMain":null,"RegionNum":4,"RegionCategory":"教育学","ArticlePicture":[],"TitleCN":null,"AbstractTextCN":null,"PMCID":"","EPubDate":null,"PubModel":null,"JCR":null,"JCRName":null,"Score":null,"Total":0}

{"title":"Using Process Data to Evaluate the Impact of Shortening Allotted Case Time in a Simulation-Based Assessment","authors":"Chunyan Liu,&nbsp;Monica M. Cuddy,&nbsp;Qiwei He,&nbsp;Wenli Ouyang,&nbsp;Cara Artman","doi":"10.1111/emip.12656","DOIUrl":"https://doi.org/10.1111/emip.12656","url":null,"abstract":"<p>The Computer-based Case Simulations (CCS) component of the United States Medical Licensing Examination (USMLE) Step 3 was developed to assess the decision-making and patient-management skills of physicians. Process data can provide deep insights into examinees’ behavioral processes related to completing the CCS assessment task. In this paper, we utilized process data to evaluate the impact of shortening allotted time limit by rescoring the CCS cases based on process data extracted at various timestamps that represented different percentages of the original allotted case time. It was found that examinees’ performance as well as the correlation between original and newly generated scores both tended to decrease as the timestamp condition became stricter. The impact of shortening allotted time limit was found marginally associated with case difficulties, but strongly dependent on the case time intensity under the original time setting.</p>","PeriodicalId":47345,"journal":{"name":"Educational Measurement-Issues and Practice","volume":"43 4","pages":"24-32"},"PeriodicalIF":2.7,"publicationDate":"2024-11-27","publicationTypes":"Journal Article","fieldsOfStudy":null,"isOpenAccess":false,"openAccessPdf":"","citationCount":null,"resultStr":null,"platform":"Semanticscholar","paperid":"143253681","PeriodicalName":null,"FirstCategoryId":null,"ListUrlMain":null,"RegionNum":4,"RegionCategory":"教育学","ArticlePicture":[],"TitleCN":null,"AbstractTextCN":null,"PMCID":"","EPubDate":null,"PubModel":null,"JCR":null,"JCRName":null,"Score":null,"Total":0}

{"title":"What Makes Measurement Important for Education?","authors":"Mark Wilson","doi":"10.1111/emip.12646","DOIUrl":"https://doi.org/10.1111/emip.12646","url":null,"abstract":"<p>This contribution to the Special Issue of <i>EM:IP</i> on the topic of <i>The Past, Present and Future of Educational Measurement</i> concentrates on the present and the future and hence focuses on the goal of improving education. The results of meta-analyses were examined, and it was noted that the largest effect sizes were associated with actual use of formative assessments in classroom settings—hence <i>classroom assessment</i> (in contrast with <i>large-scale assessment</i>). The paper describes micro assessment, which focuses on in-classroom forms of measurement, and then expands this assessment approach to focus on frames beyond that in terms of summative end-of-semester tests (macro). This is followed by a description of how these approaches can be combined using a construct map as the basis for developing and using assessments to span across these two levels in terms of the BEAR Assessment System (BAS). Throughout, this is exemplified using an elementary school program designed to teach students about geometry. Finally, a conclusion summarizes the discussion, and also looks to the future where a meso level of use involves end-of-unit tests.</p>","PeriodicalId":47345,"journal":{"name":"Educational Measurement-Issues and Practice","volume":"43 4","pages":"73-82"},"PeriodicalIF":2.7,"publicationDate":"2024-11-25","publicationTypes":"Journal Article","fieldsOfStudy":null,"isOpenAccess":false,"openAccessPdf":"https://onlinelibrary.wiley.com/doi/epdf/10.1111/emip.12646","citationCount":null,"resultStr":null,"platform":"Semanticscholar","paperid":"143253139","PeriodicalName":null,"FirstCategoryId":null,"ListUrlMain":null,"RegionNum":4,"RegionCategory":"教育学","ArticlePicture":[],"TitleCN":null,"AbstractTextCN":null,"PMCID":"OA","EPubDate":null,"PubModel":null,"JCR":null,"JCRName":null,"Score":null,"Total":0}

{"title":"Growth across Grades and Common Item Grade Alignment in Vertical Scaling Using the Rasch Model","authors":"Sanford R. Student,&nbsp;Derek C. Briggs,&nbsp;Laurie Davis","doi":"10.1111/emip.12639","DOIUrl":"https://doi.org/10.1111/emip.12639","url":null,"abstract":"<p>Vertical scales are frequently developed using common item nonequivalent group linking. In this design, one can use upper-grade, lower-grade, or mixed-grade common items to estimate the linking constants that underlie the absolute measurement of growth. Using the Rasch model and a dataset from Curriculum Associates’ i-Ready Diagnostic in math in grades 3–7, we demonstrate how grade-to-grade mean differences in mathematics proficiency appear much larger when upper-grade linking items are used instead of lower-grade items, with linkings based on a mixture of items falling in between. We then consider salient properties of the three calibrated scales including invariance of the different sets of common items to student grade and item difficulty reversals. These exploratory analyses suggest that upper-grade common items in vertical scaling are more subject to threats to score comparability across grades, even though these items also tend to imply the most growth.</p>","PeriodicalId":47345,"journal":{"name":"Educational Measurement-Issues and Practice","volume":"44 1","pages":"84-95"},"PeriodicalIF":2.7,"publicationDate":"2024-11-25","publicationTypes":"Journal Article","fieldsOfStudy":null,"isOpenAccess":false,"openAccessPdf":"","citationCount":null,"resultStr":null,"platform":"Semanticscholar","paperid":"143424128","PeriodicalName":null,"FirstCategoryId":null,"ListUrlMain":null,"RegionNum":4,"RegionCategory":"教育学","ArticlePicture":[],"TitleCN":null,"AbstractTextCN":null,"PMCID":"","EPubDate":null,"PubModel":null,"JCR":null,"JCRName":null,"Score":null,"Total":0}

{"title":"Educational Measurement: Models, Methods, and Theory","authors":"Lauress L. Wise,&nbsp;Daisy W. Rutstein","doi":"10.1111/emip.12642","DOIUrl":"https://doi.org/10.1111/emip.12642","url":null,"abstract":"<p>This article describes an amazing development of methods and models supporting educational measurement together with a much slower evolution of theory about how and what students learn and how educational measurement best supports that learning. Told from the perspective of someone who has lived through many of these changes, the article provides background on these developments and insights into challenges and opportunities for future development.</p>","PeriodicalId":47345,"journal":{"name":"Educational Measurement-Issues and Practice","volume":"43 4","pages":"83-87"},"PeriodicalIF":2.7,"publicationDate":"2024-11-24","publicationTypes":"Journal Article","fieldsOfStudy":null,"isOpenAccess":false,"openAccessPdf":"","citationCount":null,"resultStr":null,"platform":"Semanticscholar","paperid":"143253240","PeriodicalName":null,"FirstCategoryId":null,"ListUrlMain":null,"RegionNum":4,"RegionCategory":"教育学","ArticlePicture":[],"TitleCN":null,"AbstractTextCN":null,"PMCID":"","EPubDate":null,"PubModel":null,"JCR":null,"JCRName":null,"Score":null,"Total":0}

{"title":"Measurement Must Be Qualitative, then Quantitative, then Qualitative Again","authors":"Andrew D. Ho","doi":"10.1111/emip.12662","DOIUrl":"https://doi.org/10.1111/emip.12662","url":null,"abstract":"<p>Educational measurement is a social science that requires both qualitative and quantitative competencies. Qualitative competencies in educational measurement include developing and applying theories of learning, designing instruments, and identifying the social, cultural, historical, and political contexts of measurement. Quantitative competencies include statistical inference, computational fluency, and psychometric modeling. I review 12 commentaries authored by past presidents of the National Council on Measurement in Education (NCME) published in a special issue prompting them to reflect on the past, present, and future of educational measurement. I explain how a perspective on both qualitative and quantitative competencies yields common themes across the commentaries. These include the appeal and challenge of personalization, the necessity of contextualization, and the value of communication and collaboration. I conclude that elevation of both qualitative and quantitative competencies underlying educational measurement provides a clearer sense of how NCME can advance its mission, “to advance theory and applications of educational measurement to benefit society.”</p>","PeriodicalId":47345,"journal":{"name":"Educational Measurement-Issues and Practice","volume":"43 4","pages":"137-145"},"PeriodicalIF":2.7,"publicationDate":"2024-11-21","publicationTypes":"Journal Article","fieldsOfStudy":null,"isOpenAccess":false,"openAccessPdf":"","citationCount":null,"resultStr":null,"platform":"Semanticscholar","paperid":"143252943","PeriodicalName":null,"FirstCategoryId":null,"ListUrlMain":null,"RegionNum":4,"RegionCategory":"教育学","ArticlePicture":[],"TitleCN":null,"AbstractTextCN":null,"PMCID":"","EPubDate":null,"PubModel":null,"JCR":null,"JCRName":null,"Score":null,"Total":0}

{"title":"Admission Testing in Higher Education: Changing Landscape and Outcomes from Test-Optional Policies","authors":"Wayne Camara","doi":"10.1111/emip.12651","DOIUrl":"https://doi.org/10.1111/emip.12651","url":null,"abstract":"<p>Access to admission tests was greatly restricted during the COVID-19 pandemic resulting in widespread adoption of test-optional policies by colleges and universities. Many institutions adopted such policies on an interim or trial basis, as many others signaled the change would be long term. Several Ivy League institutions and selective public flagship universities have returned to requiring test scores from all applicants citing their own research indicating diversity and ensuring academic success of applicants can be best served by inclusion of test scores in the admissions process. This paper reviews recent research on the impact of test-optional policies on score-sending behaviors of applicants and differential outcomes in college and score sending. Ultimately, test-optional policies are neither the panacea for diversity that proponents suggested nor do they result in a decay of academic outcomes that opponents forecast, but they do have consequences, which colleges will need to weigh going forward.</p>","PeriodicalId":47345,"journal":{"name":"Educational Measurement-Issues and Practice","volume":"43 4","pages":"104-111"},"PeriodicalIF":2.7,"publicationDate":"2024-11-14","publicationTypes":"Journal Article","fieldsOfStudy":null,"isOpenAccess":false,"openAccessPdf":"","citationCount":null,"resultStr":null,"platform":"Semanticscholar","paperid":"143252691","PeriodicalName":null,"FirstCategoryId":null,"ListUrlMain":null,"RegionNum":4,"RegionCategory":"教育学","ArticlePicture":[],"TitleCN":null,"AbstractTextCN":null,"PMCID":"","EPubDate":null,"PubModel":null,"JCR":null,"JCRName":null,"Score":null,"Total":0}

{"title":"Leading ITEMS: A Retrospective on Progress and Future Goals","authors":"Brian C. Leventhal","doi":"10.1111/emip.12661","DOIUrl":"https://doi.org/10.1111/emip.12661","url":null,"abstract":"&lt;p&gt;As this issue marks the conclusion of my tenure as editor of the Instructional Topics in Educational Measurement Series (ITEMS), I take this opportunity to reflect on the progress made during my term and to outline potential future directions for the publication.&lt;/p&gt;&lt;p&gt;First, I extend my gratitude to the National Council on Measurement in Education (NCME) and the publications committee for entrusting me with the role of editor and for their unwavering support of my vision for ITEMS. I am also deeply appreciative of Richard Feinberg, who served as associate editor throughout my tenure, and Zhongmin Cui, editor of &lt;i&gt;Educational Measurement: Issues and Practice&lt;/i&gt; (&lt;i&gt;EM:IP&lt;/i&gt;) for their invaluable collaboration. Additionally, I thank all the authors who contributed modules and the dedicated readership that has engaged with the content.&lt;/p&gt;&lt;p&gt;ITEMS stands as a distinctive publication, bridging the gap between research and education by offering learning modules on both emerging and established practice in educational measurement. I saw the primary objective of ITEMS as to provide accessible learning resources to a diverse audience, including practitioners, students, partners, stakeholders, and the general public. These modules serve various purposes; practitioners may seek to research or expand their skills, students and professors may use them to complement classroom learning, partners and stakeholders may develop foundational knowledge to enhance collaboration with measurement professionals, and the public may gain insights into tests they encounter in their daily lives. Addressing the needs of such a broad audience is challenging, yet it underscores the essential role that ITEMS plays.&lt;/p&gt;&lt;p&gt;Upon assuming the role of editor three years ago, ITEMS had recently transitioned from static articles to interactive digital modules. My efforts focused on furthering this transformation by enhancing the engagement of digital publications and streamlining the development process. Although much of this work occurred behind the scenes, the benefits are evident to learners. The modules are now easily accessible on the NCME website, available in both digital and print formats. Newer modules include downloadable videos for offline use or course integration. Content is now accessible across multiple devices, including computers, phones and tablets. Authors also benefit from the updated development process, which now uses familiar software such as Microsoft PowerPoint or Google Slides. Comprehensive documentation, including timelines, deliverables, and templates, supports authors throughout the development process, allowing them to focus on content creation rather than formatting and logistics.&lt;/p&gt;&lt;p&gt;Reflecting on my tenure, I am proud of the modules published, yet I recognize areas for improvement and future growth. Recruiting authors and maintaining content development posed significant challenges, with some modules remaining incomplete. I am hopeful that th","PeriodicalId":47345,"journal":{"name":"Educational Measurement-Issues and Practice","volume":"43 4","pages":"169"},"PeriodicalIF":2.7,"publicationDate":"2024-11-14","publicationTypes":"Journal Article","fieldsOfStudy":null,"isOpenAccess":false,"openAccessPdf":"https://onlinelibrary.wiley.com/doi/epdf/10.1111/emip.12661","citationCount":null,"resultStr":null,"platform":"Semanticscholar","paperid":"143252692","PeriodicalName":null,"FirstCategoryId":null,"ListUrlMain":null,"RegionNum":4,"RegionCategory":"教育学","ArticlePicture":[],"TitleCN":null,"AbstractTextCN":null,"PMCID":"OA","EPubDate":null,"PubModel":null,"JCR":null,"JCRName":null,"Score":null,"Total":0}