Educational Measurement-Issues and Practice最新文献

{"title":"ITEMS Corner Update: Two Years of Changes to ITEMS","authors":"Brian C. Leventhal","doi":"10.1111/emip.12596","DOIUrl":"https://doi.org/10.1111/emip.12596","url":null,"abstract":"<p>This issue marks the beginning of the final year of my tenure as editor of the <i>Instructional Topics of Educational Measurement Series (ITEMS)</i>. Although I will save a comprehensive reflection until the last issue of the year, I will use this issue to provide an update on the two changes to ITEMS that were made over the past two years in addition to introducing the newest entry to the ITEMS digital library.</p><p>In 2022, I took over a newly created format and process for <i>ITEMS</i> modules—digital interactive teaching modules as opposed to the traditional PDF format. Modules were hosted on a learning management system (LMS), shifting focus from a journal publication to a modernized teaching-focused platform. While the LMS provided ample opportunity for expansion and learning, its form was challenging to navigate given its unique look and feel as a standalone website, separate from the then recently revitalized modern NCME website. Thus, my first significant change to <i>ITEMS</i> was to migrate the <i>ITEMS</i> portal LMS to the NCME website. Learners now go straight to the NCME website and navigate directly to the ITEMS portal without requiring a unique username and password. This makes the <i>ITEMS</i> module library more accessible and makes the NCME website the go-to destination for professional development in educational measurement.</p><p>Prior to 2022, specialized software was necessary to produce digital modules. This software allowed for the inclusion of interactive and nonlinear learning. This process resulted in beautiful, intricate, and pleasing modules for learning, but was complex and costly for development. To continue the process in place would have required producing these modules as a full-time job with a staff—a true testament to the work of the previous editor André Rupp and other volunteers. Fortunately, the <i>ITEMS</i> portal being hosted on the NCME website offered an opportunity to modify the development process while keeping an interactive component with a modernized look and feel.</p><p>I developed a new comprehensive system for creating <i>ITEMS</i> modules. Authors now develop content for four to five sections using pre-made PowerPoint templates, a familiar software to many. Once completed, I insert animations to align with timing of the recorded audio and export them as videos. These videos are hosted on the NCME website and can be downloaded for offline viewing (an extra benefit). Learners interact directly with the videos and content on the website, where module sections may be viewed in any order. In addition to video content, interactive activities that exemplify syntax or case studies are developed to assist with learning. Finally, interactive selected response learning checks are made, allowing learners to check their understanding as they traverse through the module. Datasets, syntax, and other files are hosted on the module page for easy download.</p><p>The <i>ITEMS</i> development process is not l","PeriodicalId":47345,"journal":{"name":"Educational Measurement-Issues and Practice","volume":"43 1","pages":"96"},"PeriodicalIF":2.0,"publicationDate":"2024-02-28","publicationTypes":"Journal Article","fieldsOfStudy":null,"isOpenAccess":false,"openAccessPdf":"https://onlinelibrary.wiley.com/doi/epdf/10.1111/emip.12596","citationCount":null,"resultStr":null,"platform":"Semanticscholar","paperid":"139987487","PeriodicalName":null,"FirstCategoryId":null,"ListUrlMain":null,"RegionNum":4,"RegionCategory":"教育学","ArticlePicture":[],"TitleCN":null,"AbstractTextCN":null,"PMCID":"OA","EPubDate":null,"PubModel":null,"JCR":null,"JCRName":null,"Score":null,"Total":0}

{"title":"On the Cover: High School Coursetaking Sequence Clusters and Postsecondary Enrollment","authors":"Yuan-Ling Liaw","doi":"10.1111/emip.12597","DOIUrl":"https://doi.org/10.1111/emip.12597","url":null,"abstract":"","PeriodicalId":47345,"journal":{"name":"Educational Measurement-Issues and Practice","volume":"43 1","pages":"4"},"PeriodicalIF":2.0,"publicationDate":"2024-02-28","publicationTypes":"Journal Article","fieldsOfStudy":null,"isOpenAccess":false,"openAccessPdf":"","citationCount":null,"resultStr":null,"platform":"Semanticscholar","paperid":"139987388","PeriodicalName":null,"FirstCategoryId":null,"ListUrlMain":null,"RegionNum":4,"RegionCategory":"教育学","ArticlePicture":[],"TitleCN":null,"AbstractTextCN":null,"PMCID":"","EPubDate":null,"PubModel":null,"JCR":null,"JCRName":null,"Score":null,"Total":0}

{"title":"Digital Module 35: Through-Year Assessment","authors":"Nathan Dadey,&nbsp;Brian Gong,&nbsp;Yun-Kyung Kim,&nbsp;Edynn Sato","doi":"10.1111/emip.12595","DOIUrl":"https://doi.org/10.1111/emip.12595","url":null,"abstract":"<div>\u0000 \u0000 <section>\u0000 \u0000 <h3> Module Abstract</h3>\u0000 \u0000 <p><i>Through-year assessments</i> are assessments that are administered in multiple parts and at different times over the course of a school year that also produce summative scores that can be used with state accountability systems (Lorié et al., 2021; Dadey &amp; Gong, 2023). These assessments are alternatively known as instructionally embedded, through-course, or periodic assessments. There are a number of possible through-year assessment models, and they have recently been the subject of much policy interest as they have the potential to inform subsequent instruction, be more closely aligned with and responsive to curricula and instruction, provide more proximal measures of learning, and be a more sensitive measure of student progress or growth than typical year-end summative assessments (Clark &amp; Karvonen, 2021; Gong, 2021; NWEA, 2021; Wise, 2011). More research is needed, however, to substantiate these potential uses.</p>\u0000 </section>\u0000 </div>","PeriodicalId":47345,"journal":{"name":"Educational Measurement-Issues and Practice","volume":"43 1","pages":"97-98"},"PeriodicalIF":2.0,"publicationDate":"2024-02-28","publicationTypes":"Journal Article","fieldsOfStudy":null,"isOpenAccess":false,"openAccessPdf":"","citationCount":null,"resultStr":null,"platform":"Semanticscholar","paperid":"139987433","PeriodicalName":null,"FirstCategoryId":null,"ListUrlMain":null,"RegionNum":4,"RegionCategory":"教育学","ArticlePicture":[],"TitleCN":null,"AbstractTextCN":null,"PMCID":"","EPubDate":null,"PubModel":null,"JCR":null,"JCRName":null,"Score":null,"Total":0}

{"title":"A Workflow for Minimizing Errors in Template-Based Automated Item-Generation Development","authors":"Yanyan Fu","doi":"10.1111/emip.12600","DOIUrl":"10.1111/emip.12600","url":null,"abstract":"<p>The template-based automated item-generation (TAIG) approach that involves template creation, item generation, item selection, field-testing, and evaluation has more steps than the traditional item development method. Consequentially, there is more margin for error in this process, and any template errors can be cascaded to the generated items. Therefore, it is essential to eliminate the source of errors and ensure the quality of the template so items can be problem-free. The article introduces a process to reduce template errors at the early stage of template development, minimize the impact of template errors on generated items, and increase the survival rates of generated items. The article also discusses a statistical method to establish confidence in the quality of the template by systematically examining the quality of the generated items. The proposed method can reduce the review process for some items generated from a template.</p>","PeriodicalId":47345,"journal":{"name":"Educational Measurement-Issues and Practice","volume":"43 2","pages":"30-39"},"PeriodicalIF":2.0,"publicationDate":"2024-02-12","publicationTypes":"Journal Article","fieldsOfStudy":null,"isOpenAccess":false,"openAccessPdf":"","citationCount":null,"resultStr":null,"platform":"Semanticscholar","paperid":"139760582","PeriodicalName":null,"FirstCategoryId":null,"ListUrlMain":null,"RegionNum":4,"RegionCategory":"教育学","ArticlePicture":[],"TitleCN":null,"AbstractTextCN":null,"PMCID":"","EPubDate":null,"PubModel":null,"JCR":null,"JCRName":null,"Score":null,"Total":0}

{"title":"The University of California Was Wrong to Abolish the SAT: Admissions When Affirmative Action Was Banned","authors":"Donald Wittman","doi":"10.1111/emip.12598","DOIUrl":"10.1111/emip.12598","url":null,"abstract":"<p>I study student characteristics and academic performance at the University of California, where consideration of an applicant's ethnicity has been banned since 1996 and SAT scores were used in admitting students to the university until fall 2021. I show the following: (1) SAT scores were more important than high school grades in predicting first-year university GPA; (2) the use of SAT scores alone or with high school grades in determining admission is biased in favor of admitting underrepresented minorities and students who are socioeconomically disadvantaged; (3) SAT scores are more important and high school grades are less important in predicting GPA for underrepresented minorities and/or those students from low-income families than they are for those students who are white and/or from high-income families; and (4) the University of California found ways to admit a significant number of underrepresented minorities despite many of them having low SAT scores.</p>","PeriodicalId":47345,"journal":{"name":"Educational Measurement-Issues and Practice","volume":"43 2","pages":"55-63"},"PeriodicalIF":2.0,"publicationDate":"2024-02-09","publicationTypes":"Journal Article","fieldsOfStudy":null,"isOpenAccess":false,"openAccessPdf":"https://onlinelibrary.wiley.com/doi/epdf/10.1111/emip.12598","citationCount":null,"resultStr":null,"platform":"Semanticscholar","paperid":"139771065","PeriodicalName":null,"FirstCategoryId":null,"ListUrlMain":null,"RegionNum":4,"RegionCategory":"教育学","ArticlePicture":[],"TitleCN":null,"AbstractTextCN":null,"PMCID":"OA","EPubDate":null,"PubModel":null,"JCR":null,"JCRName":null,"Score":null,"Total":0}

{"title":"An Automated Item Pool Assembly Framework for Maximizing Item Utilization for CAT","authors":"Hwanggyu Lim,&nbsp;Kyung (Chris) T. Han","doi":"10.1111/emip.12589","DOIUrl":"10.1111/emip.12589","url":null,"abstract":"<p>Computerized adaptive testing (CAT) has gained deserved popularity in the administration of educational and professional assessments, but continues to face test security challenges. To ensure sustained quality assurance and testing integrity, it is imperative to establish and maintain multiple stable item pools that are consistent in terms of psychometric characteristics and content specifications. This study introduces the Honeycomb Pool Assembly (HPA) framework, an innovative solution for the construction of multiple parallel item pools for CAT that maximizes item utilization in the item bank. The HPA framework comprises two stages—cell assembly and pool assembly—and uses a mixed integer programming modeling approach. An empirical study demonstrated HPA's effectiveness in creating a large number of parallel pools using a real-world high-stakes CAT assessment item bank. The HPA framework offers several advantages, including (a) simultaneous creation of multiple parallel pools, (b) simplification of item pool maintenance, and (c) flexibility in establishing statistical and operational constraints. Moreover, it can help testing organizations efficiently manage and monitor the health of their item banks. Thus, the HPA framework is expected to be a valuable tool for testing professionals and organizations to address test security challenges and maintain the integrity of high-stakes CAT assessments.</p>","PeriodicalId":47345,"journal":{"name":"Educational Measurement-Issues and Practice","volume":"43 1","pages":"39-51"},"PeriodicalIF":2.0,"publicationDate":"2024-02-09","publicationTypes":"Journal Article","fieldsOfStudy":null,"isOpenAccess":false,"openAccessPdf":"","citationCount":null,"resultStr":null,"platform":"Semanticscholar","paperid":"139894957","PeriodicalName":null,"FirstCategoryId":null,"ListUrlMain":null,"RegionNum":4,"RegionCategory":"教育学","ArticlePicture":[],"TitleCN":null,"AbstractTextCN":null,"PMCID":"","EPubDate":null,"PubModel":null,"JCR":null,"JCRName":null,"Score":null,"Total":0}

{"title":"Expected Classification Accuracy for Categorical Growth Models","authors":"Daniel Murphy,&nbsp;Sarah Quesen,&nbsp;Matthew Brunetti,&nbsp;Quintin Love","doi":"10.1111/emip.12599","DOIUrl":"10.1111/emip.12599","url":null,"abstract":"<p>Categorical growth models describe examinee growth in terms of performance-level category transitions, which implies that some percentage of examinees will be misclassified. This paper introduces a new procedure for estimating the classification accuracy of categorical growth models, based on Rudner's classification accuracy index for item response theory–based assessments. Results of a simulation study are presented to provide evidence for the accuracy and validity of the approach. Also, an empirical example is presented to demonstrate the approach using data from the Indiana Student Performance Readiness and Observation of Understanding Tool growth model, which classifies examinees into growth categories used by the Office of Special Education Programs to monitor the progress of preschool children who receive special education services.</p>","PeriodicalId":47345,"journal":{"name":"Educational Measurement-Issues and Practice","volume":"43 2","pages":"64-73"},"PeriodicalIF":2.0,"publicationDate":"2024-01-29","publicationTypes":"Journal Article","fieldsOfStudy":null,"isOpenAccess":false,"openAccessPdf":"","citationCount":null,"resultStr":null,"platform":"Semanticscholar","paperid":"140487311","PeriodicalName":null,"FirstCategoryId":null,"ListUrlMain":null,"RegionNum":4,"RegionCategory":"教育学","ArticlePicture":[],"TitleCN":null,"AbstractTextCN":null,"PMCID":"","EPubDate":null,"PubModel":null,"JCR":null,"JCRName":null,"Score":null,"Total":0}

{"title":"MxML (Exploring the Relationship between Measurement and Machine Learning): Current State of the Field","authors":"Yi Zheng,&nbsp;Steven Nydick,&nbsp;Sijia Huang,&nbsp;Susu Zhang","doi":"10.1111/emip.12593","DOIUrl":"https://doi.org/10.1111/emip.12593","url":null,"abstract":"<p>The recent surge of machine learning (ML) has impacted many disciplines, including educational and psychological measurement (hereafter shortened as <i>measurement</i>). The measurement literature has seen rapid growth in applications of ML to solve measurement problems. However, as we emphasize in this article, it is imperative to critically examine the potential risks associated with involving ML in measurement. The MxML project aims to explore the relationship between measurement and ML, so as to identify and address the risks and better harness the power of ML to serve measurement missions. This paper describes the first study of the MxML project, in which we summarize the state of the field of applications, extensions, and discussions about ML in measurement contexts with a systematic review of the recent 10 years’ literature. We provide a snapshot of the literature in (1) areas of measurement where ML is discussed, (2) types of articles (e.g., applications, conceptual, etc.), (3) ML methods discussed, and (4) potential risks associated with involving ML in measurement, which result from the differences between what measurement tasks need versus what ML techniques can provide.</p>","PeriodicalId":47345,"journal":{"name":"Educational Measurement-Issues and Practice","volume":"43 1","pages":"19-38"},"PeriodicalIF":2.0,"publicationDate":"2024-01-29","publicationTypes":"Journal Article","fieldsOfStudy":null,"isOpenAccess":false,"openAccessPdf":"","citationCount":null,"resultStr":null,"platform":"Semanticscholar","paperid":"139987475","PeriodicalName":null,"FirstCategoryId":null,"ListUrlMain":null,"RegionNum":4,"RegionCategory":"教育学","ArticlePicture":[],"TitleCN":null,"AbstractTextCN":null,"PMCID":"","EPubDate":null,"PubModel":null,"JCR":null,"JCRName":null,"Score":null,"Total":0}

{"title":"Knowledge Integration in Science Learning: Tracking Students' Knowledge Development and Skill Acquisition with Cognitive Diagnosis Models","authors":"Xin Xu,&nbsp;Shixiu Ren,&nbsp;Danhui Zhang,&nbsp;Tao Xin","doi":"10.1111/emip.12592","DOIUrl":"10.1111/emip.12592","url":null,"abstract":"<p>In scientific literacy, knowledge integration (KI) is a scaffolding-based theory to assist students' scientific inquiry learning. To drive students to be self-directed, many courses have been developed based on KI framework. However, few efforts have been made to evaluate the outcome of students' learning under KI instruction. Moreover, finer-grained information has been pursued to better understand students' learning and how it progresses over time. In this article, a normative procedure of building and choosing cognitive diagnosis models (CDMs) and attribute hierarchies was formulated under KI theory. We examined the utility of CDMs for evaluating students' knowledge status in KI learning. The results of the data analysis confirmed an intuitive assumption of the hierarchical structure of KI components. Furthermore, analysis of pre- and posttests using a higher-order, hidden Markov model tracked students' skill acquisition while integrating knowledge. Results showed that students make significant progress after using the web-based inquiry science environment (WISE) platform.</p>","PeriodicalId":47345,"journal":{"name":"Educational Measurement-Issues and Practice","volume":"43 1","pages":"66-82"},"PeriodicalIF":2.0,"publicationDate":"2024-01-24","publicationTypes":"Journal Article","fieldsOfStudy":null,"isOpenAccess":false,"openAccessPdf":"","citationCount":null,"resultStr":null,"platform":"Semanticscholar","paperid":"139587965","PeriodicalName":null,"FirstCategoryId":null,"ListUrlMain":null,"RegionNum":4,"RegionCategory":"教育学","ArticlePicture":[],"TitleCN":null,"AbstractTextCN":null,"PMCID":"","EPubDate":null,"PubModel":null,"JCR":null,"JCRName":null,"Score":null,"Total":0}

{"title":"Measuring Variability in Proctor Decision Making on High-Stakes Assessments: Improving Test Security in the Digital Age","authors":"William Belzak,&nbsp;J. R. Lockwood,&nbsp;Yigal Attali","doi":"10.1111/emip.12591","DOIUrl":"10.1111/emip.12591","url":null,"abstract":"<p>Remote proctoring, or monitoring test takers through internet-based, video-recording software, has become critical for maintaining test security on high-stakes assessments. The main role of remote proctors is to make judgments about test takers' behaviors and decide whether these behaviors constitute rule violations. Variability in proctor decision making, or the degree to which humans/proctors make different decisions about the same test-taking behaviors, can be problematic for both test takers and test users (e.g., universities). In this paper, we measure variability in proctor decision making over time on a high-stakes English language proficiency test. Our results show that (1) proctors systematically differ in their decision making and (2) these differences are trait-like (i.e., ranging from lenient to strict), but (3) systematic variability in decisions can be reduced. Based on these findings, we recommend that test security providers conduct regular measurements of proctors’ judgments and take actions to reduce variability in proctor decision making.</p>","PeriodicalId":47345,"journal":{"name":"Educational Measurement-Issues and Practice","volume":"43 1","pages":"52-65"},"PeriodicalIF":2.0,"publicationDate":"2024-01-24","publicationTypes":"Journal Article","fieldsOfStudy":null,"isOpenAccess":false,"openAccessPdf":"https://onlinelibrary.wiley.com/doi/epdf/10.1111/emip.12591","citationCount":null,"resultStr":null,"platform":"Semanticscholar","paperid":"139587843","PeriodicalName":null,"FirstCategoryId":null,"ListUrlMain":null,"RegionNum":4,"RegionCategory":"教育学","ArticlePicture":[],"TitleCN":null,"AbstractTextCN":null,"PMCID":"OA","EPubDate":null,"PubModel":null,"JCR":null,"JCRName":null,"Score":null,"Total":0}