Educational Measurement-Issues and Practice最新文献

{"title":"Using OpenAI GPT to Generate Reading Comprehension Items","authors":"Ayfer Sayin,&nbsp;Mark Gierl","doi":"10.1111/emip.12590","DOIUrl":"10.1111/emip.12590","url":null,"abstract":"<p>The purpose of this study is to introduce and evaluate a method for generating reading comprehension items using template-based automatic item generation. To begin, we describe a new model for generating reading comprehension items called the text analysis cognitive model assessing inferential skills across different reading passages. Next, the text analysis cognitive model is used to generate reading comprehension items where examinees are required to read a passage and identify the irrelevant sentence. The sentences for the generated passages were created using OpenAI GPT-3.5. Finally, the quality of the generated items was evaluated. The generated items were reviewed by three subject-matter experts. The generated items were also administered to a sample of 1,607 Grade-8 students. The correct options for the generated items produced a similar level of difficulty and yielded strong discrimination power while the incorrect options served as effective distractors. Implications of augmented intelligence for item development are discussed.</p>","PeriodicalId":47345,"journal":{"name":"Educational Measurement-Issues and Practice","volume":"43 1","pages":"5-18"},"PeriodicalIF":2.0,"publicationDate":"2024-01-24","publicationTypes":"Journal Article","fieldsOfStudy":null,"isOpenAccess":false,"openAccessPdf":"https://onlinelibrary.wiley.com/doi/epdf/10.1111/emip.12590","citationCount":null,"resultStr":null,"platform":"Semanticscholar","paperid":"139587888","PeriodicalName":null,"FirstCategoryId":null,"ListUrlMain":null,"RegionNum":4,"RegionCategory":"教育学","ArticlePicture":[],"TitleCN":null,"AbstractTextCN":null,"PMCID":"OA","EPubDate":null,"PubModel":null,"JCR":null,"JCRName":null,"Score":null,"Total":0}

{"title":"Achievement and Growth on English Language Proficiency and Content Assessments for English Learners in Elementary Grades","authors":"Heather M Buzick,&nbsp;Mikyung Kim Wolf,&nbsp;Laura Ballard","doi":"10.1111/emip.12588","DOIUrl":"10.1111/emip.12588","url":null,"abstract":"<p>English language proficiency (ELP) assessment scores are used by states to make high-stakes decisions related to linguistic support in instruction and assessment for English learner (EL) students and for EL student reclassification. Changes to both academic content standards and ELP academic standards within the last decade have resulted in increased academic rigor and language demands. In this study, we explored the association between EL student performance over time on content (English language arts and mathematics) and ELP assessments, generally finding evidence of positive associations. Modeling the simultaneous association between changes over time in both content and ELP assessment performance contributes empirical evidence about the role of language in ELA and mathematics development and provides contextual information to serve as validity evidence for score inferences for EL students.</p>","PeriodicalId":47345,"journal":{"name":"Educational Measurement-Issues and Practice","volume":"43 1","pages":"83-95"},"PeriodicalIF":2.0,"publicationDate":"2024-01-10","publicationTypes":"Journal Article","fieldsOfStudy":null,"isOpenAccess":false,"openAccessPdf":"","citationCount":null,"resultStr":null,"platform":"Semanticscholar","paperid":"139464311","PeriodicalName":null,"FirstCategoryId":null,"ListUrlMain":null,"RegionNum":4,"RegionCategory":"教育学","ArticlePicture":[],"TitleCN":null,"AbstractTextCN":null,"PMCID":"","EPubDate":null,"PubModel":null,"JCR":null,"JCRName":null,"Score":null,"Total":0}

{"title":"Issue Cover","authors":"","doi":"10.1111/emip.12518","DOIUrl":"https://doi.org/10.1111/emip.12518","url":null,"abstract":"","PeriodicalId":47345,"journal":{"name":"Educational Measurement-Issues and Practice","volume":"42 4","pages":""},"PeriodicalIF":2.0,"publicationDate":"2023-12-05","publicationTypes":"Journal Article","fieldsOfStudy":null,"isOpenAccess":false,"openAccessPdf":"https://onlinelibrary.wiley.com/doi/epdf/10.1111/emip.12518","citationCount":null,"resultStr":null,"platform":"Semanticscholar","paperid":"138485186","PeriodicalName":null,"FirstCategoryId":null,"ListUrlMain":null,"RegionNum":4,"RegionCategory":"教育学","ArticlePicture":[],"TitleCN":null,"AbstractTextCN":null,"PMCID":"OA","EPubDate":null,"PubModel":null,"JCR":null,"JCRName":null,"Score":null,"Total":0}

{"title":"ITEMS Corner Update: The Final Three Steps in the Development Process","authors":"Brian C. Leventhal","doi":"10.1111/emip.12586","DOIUrl":"https://doi.org/10.1111/emip.12586","url":null,"abstract":"&lt;p&gt;Throughout 2023, I have detailed each step of the module development process for the &lt;i&gt;Instructional Topics in Educational Measurement Series&lt;/i&gt; (&lt;i&gt;ITEMS&lt;/i&gt;). In the first issue, I outlined the 10 steps necessary to complete a module. In the second issue, I detailed Steps 1–3, which cover outlining the content, developing the content in premade PowerPoint templates, and having the slides reviewed by the editor. In the third issue of the year, I outlined Step 4—recording the audio, Step 5—having the editor polish the module (e.g., animating the content), Step 6—building the activity, and Step 7—building interactive learning checks (i.e., selected response questions designed to check for understanding). In this issue, I elaborate on the final three steps: Step 8—external review, Step 9—building the module on the portal, and Step 10—writing the piece to be published in &lt;i&gt;Educational Measurement: Issues and Practice&lt;/i&gt; (&lt;i&gt;EM:IP&lt;/i&gt;). Following the in-depth explanation of each of these steps, I then introduce the newest module published to the &lt;i&gt;ITEMS&lt;/i&gt; portal (https://www.ncme.org/ITEMSportal).&lt;/p&gt;&lt;p&gt;Authors may opt to have their module externally reviewed (Step 8) prior to recording audio (Step 4) or after the module has been polished (Step 5). Having the module content reviewed prior to recording audio allows for modifying content easily without having to do “double” work (e.g., rerecording audio on slides, reorganizing flow charts). However, many authors find that their bulleted notes for each slide are not sufficient for reviewers to understand the final product. Alternatively, they may opt to have their module sent out for review once it has been editorially polished. This lets reviewers watch the author's “final” product. Because the reviewers may suggest updates, I request authors record audio on each slide. Should an author choose to make a change after review, they then do not have to rerecord an entire 20-minute section of audio. Reviewers are instructed to provide constructive feedback and are given insights about the full process that authors have already worked through (i.e., the ten-step process). It is emphasized that the purpose of &lt;i&gt;ITEMS&lt;/i&gt; is not to present novel cutting-edge research. Rather, it is a publication designed to provide instructional resources on current practices in the field.&lt;/p&gt;&lt;p&gt;After receiving reviewer feedback, authors are provided an opportunity to revise their module. Similar to a manuscript revise and resubmission, authors are asked to respond to each reviewer's comment, articulating how they have addressed each. This serves an additional purpose; specifically, this assists the editor in repolishing the updated module. For example, if audio is rerecorded on a slide, the editor will need to adjust animations and timing. After the editor has made final updates, the author reviews the module to give final approval. Upon receiving approval, the editor then builds the module onto the NCME website &lt;i","PeriodicalId":47345,"journal":{"name":"Educational Measurement-Issues and Practice","volume":"42 4","pages":"81"},"PeriodicalIF":2.0,"publicationDate":"2023-12-05","publicationTypes":"Journal Article","fieldsOfStudy":null,"isOpenAccess":false,"openAccessPdf":"https://onlinelibrary.wiley.com/doi/epdf/10.1111/emip.12586","citationCount":null,"resultStr":null,"platform":"Semanticscholar","paperid":"138485181","PeriodicalName":null,"FirstCategoryId":null,"ListUrlMain":null,"RegionNum":4,"RegionCategory":"教育学","ArticlePicture":[],"TitleCN":null,"AbstractTextCN":null,"PMCID":"OA","EPubDate":null,"PubModel":null,"JCR":null,"JCRName":null,"Score":null,"Total":0}

{"title":"On the Cover: Tell-Tale Triangles of Subscore Value","authors":"Yuan-Ling Liaw","doi":"10.1111/emip.12587","DOIUrl":"https://doi.org/10.1111/emip.12587","url":null,"abstract":"","PeriodicalId":47345,"journal":{"name":"Educational Measurement-Issues and Practice","volume":"42 4","pages":"4"},"PeriodicalIF":2.0,"publicationDate":"2023-12-05","publicationTypes":"Journal Article","fieldsOfStudy":null,"isOpenAccess":false,"openAccessPdf":"","citationCount":null,"resultStr":null,"platform":"Semanticscholar","paperid":"138485202","PeriodicalName":null,"FirstCategoryId":null,"ListUrlMain":null,"RegionNum":4,"RegionCategory":"教育学","ArticlePicture":[],"TitleCN":null,"AbstractTextCN":null,"PMCID":"","EPubDate":null,"PubModel":null,"JCR":null,"JCRName":null,"Score":null,"Total":0}

{"title":"Digital Module 34: Introduction to Multilevel Measurement Modeling","authors":"Mairead Shaw,&nbsp;Jessica K. Flake","doi":"10.1111/emip.12585","DOIUrl":"https://doi.org/10.1111/emip.12585","url":null,"abstract":"<div>\u0000 \u0000 <section>\u0000 \u0000 <h3> Module Abstract</h3>\u0000 \u0000 <p>Clustered data structures are common in many areas of educational and psychological research (e.g., students clustered in schools, patients clustered by clinician). In the course of conducting research, questions are often administered to obtain scores reflecting latent constructs. Multilevel measurement models (MLMMs) allow for modeling measurement (the relationship of test items to constructs) and the relationships between variables in a clustered data structure. Modeling the two concurrently is important for accurately representing the relationships between items and constructs, and between constructs and other constructs/variables. The barrier to entry with MLMMs can be high, with many equations and less-documented software functionality. This module reviews two different frameworks for multilevel measurement modeling: (1) multilevel modeling and (2) structural equation modeling. We demonstrate the entire process in R with working code and available data, from preparing the dataset, through writing and running code, to interpreting and comparing output for the two approaches.</p>\u0000 </section>\u0000 </div>","PeriodicalId":47345,"journal":{"name":"Educational Measurement-Issues and Practice","volume":"42 4","pages":"82"},"PeriodicalIF":2.0,"publicationDate":"2023-12-05","publicationTypes":"Journal Article","fieldsOfStudy":null,"isOpenAccess":false,"openAccessPdf":"https://onlinelibrary.wiley.com/doi/epdf/10.1111/emip.12585","citationCount":null,"resultStr":null,"platform":"Semanticscholar","paperid":"138485188","PeriodicalName":null,"FirstCategoryId":null,"ListUrlMain":null,"RegionNum":4,"RegionCategory":"教育学","ArticlePicture":[],"TitleCN":null,"AbstractTextCN":null,"PMCID":"OA","EPubDate":null,"PubModel":null,"JCR":null,"JCRName":null,"Score":null,"Total":0}

{"title":"The 2024 EM:IP Cover Graphic/Data Visualization Competition","authors":"Yuan-Ling Liaw","doi":"10.1111/emip.12583","DOIUrl":"10.1111/emip.12583","url":null,"abstract":"","PeriodicalId":47345,"journal":{"name":"Educational Measurement-Issues and Practice","volume":"42 4","pages":"5"},"PeriodicalIF":2.0,"publicationDate":"2023-11-13","publicationTypes":"Journal Article","fieldsOfStudy":null,"isOpenAccess":false,"openAccessPdf":"","citationCount":null,"resultStr":null,"platform":"Semanticscholar","paperid":"136347884","PeriodicalName":null,"FirstCategoryId":null,"ListUrlMain":null,"RegionNum":4,"RegionCategory":"教育学","ArticlePicture":[],"TitleCN":null,"AbstractTextCN":null,"PMCID":"","EPubDate":null,"PubModel":null,"JCR":null,"JCRName":null,"Score":null,"Total":0}

{"title":"Comparing Large-Scale Assessments in Two Proctoring Modalities with Interactive Log Data Analysis","authors":"Jinnie Shin,&nbsp;Qi Guo,&nbsp;Maxim Morin","doi":"10.1111/emip.12582","DOIUrl":"10.1111/emip.12582","url":null,"abstract":"<p>With the increased restrictions on physical distancing due to the COVID-19 pandemic, remote proctoring has emerged as an alternative to traditional onsite proctoring to ensure the continuity of essential assessments, such as computer-based medical licensing exams. Recent literature has highlighted the significant impact of different proctoring modalities on examinees’ test experience, including factors like response-time data. However, the potential influence of these differences on test performance has remained unclear. One limitation in the current literature is the lack of a rigorous learning analytics framework to evaluate the comparability of computer-based exams delivered using various proctoring settings. To address this gap, the current study aims to introduce a machine-learning-based framework that analyzes computer-generated response-time data to investigate the association between proctoring modalities in high-stakes assessments. We demonstrated the effectiveness of this framework using empirical data collected from a large-scale high-stakes medical licensing exam conducted in Canada. By applying the machine-learning-based framework, we were able to extract examinee-specific response-time data for each proctoring modality and identify distinct time-use patterns among examinees based on their proctoring modality.</p>","PeriodicalId":47345,"journal":{"name":"Educational Measurement-Issues and Practice","volume":"42 4","pages":"66-80"},"PeriodicalIF":2.0,"publicationDate":"2023-11-02","publicationTypes":"Journal Article","fieldsOfStudy":null,"isOpenAccess":false,"openAccessPdf":"","citationCount":null,"resultStr":null,"platform":"Semanticscholar","paperid":"135934362","PeriodicalName":null,"FirstCategoryId":null,"ListUrlMain":null,"RegionNum":4,"RegionCategory":"教育学","ArticlePicture":[],"TitleCN":null,"AbstractTextCN":null,"PMCID":"","EPubDate":null,"PubModel":null,"JCR":null,"JCRName":null,"Score":null,"Total":0}

{"title":"Foundational Competencies in Educational Measurement","authors":"Terry A. Ackerman,&nbsp;Deborah L. Bandalos,&nbsp;Derek C. Briggs,&nbsp;Howard T. Everson,&nbsp;Andrew D. Ho,&nbsp;Susan M. Lottridge,&nbsp;Matthew J. Madison,&nbsp;Sandip Sinharay,&nbsp;Michael C. Rodriguez,&nbsp;Michael Russell,&nbsp;Alina A. von Davier,&nbsp;Stefanie A. Wind","doi":"10.1111/emip.12581","DOIUrl":"10.1111/emip.12581","url":null,"abstract":"<p>This article presents the consensus of an National Council on Measurement in Education Presidential Task Force on Foundational Competencies in Educational Measurement. Foundational competencies are those that support future development of additional professional and disciplinary competencies. The authors develop a framework for foundational competencies in educational measurement, illustrate how educational measurement programs can help learners develop these competencies, and demonstrate how foundational competencies continue to develop in educational measurement professions. The framework introduces three foundational competency domains: Communication and Collaboration Competencies; Technical, Statistical, and Computational Competencies; and Educational Measurement Competencies. Within the Educational Measurement Competency domain, the authors identify five subdomains: Social, Cultural, Historical, and Political Context; Validity, Validation, and Fairness; Theory and Instrumentation; Precision and Generalization; and Psychometric Modeling.</p>","PeriodicalId":47345,"journal":{"name":"Educational Measurement-Issues and Practice","volume":"43 3","pages":"7-17"},"PeriodicalIF":2.7,"publicationDate":"2023-10-17","publicationTypes":"Journal Article","fieldsOfStudy":null,"isOpenAccess":false,"openAccessPdf":"","citationCount":null,"resultStr":null,"platform":"Semanticscholar","paperid":"136034537","PeriodicalName":null,"FirstCategoryId":null,"ListUrlMain":null,"RegionNum":4,"RegionCategory":"教育学","ArticlePicture":[],"TitleCN":null,"AbstractTextCN":null,"PMCID":"","EPubDate":null,"PubModel":null,"JCR":null,"JCRName":null,"Score":null,"Total":0}

{"title":"Digital Module 33: Fairness in Classroom Assessment: Dimensions and Tensions","authors":"Amirhossein Rasooli","doi":"10.1111/emip.12572","DOIUrl":"10.1111/emip.12572","url":null,"abstract":"<p>Perceptions of fairness are fundamental in building cooperation and trust, undermining conflicts, and gaining legitimacy in teacher-student relationships in classroom assessment. However, perceptions of unfairness in assessment can undermine students’ mental well-being, increase antisocial behaviors, increase psychological disengagement with learning, and threaten the belief in a fair society, fundamental to engaging in civic responsibilities. Despite the crucial role of perceived fairness in assessment, there are widespread experiences of unfairness reported by students internationally. To undermine these widespread unfair experiences, limited explicit education on promoting fairness in assessment is being delivered in graduate, preservice, and in-service training. However, it seems that explicit education is the first step in capacity building for reducing unfair perceptions and related undesirable outcomes. The purpose of this module is thus to share the findings drawn from theoretical and empirical research from various countries to provide a space for further critical reflection on best practices in enhancing fairness in classroom assessment contexts.</p>","PeriodicalId":47345,"journal":{"name":"Educational Measurement-Issues and Practice","volume":"42 3","pages":"82-83"},"PeriodicalIF":2.0,"publicationDate":"2023-09-06","publicationTypes":"Journal Article","fieldsOfStudy":null,"isOpenAccess":false,"openAccessPdf":"https://onlinelibrary.wiley.com/doi/epdf/10.1111/emip.12572","citationCount":null,"resultStr":null,"platform":"Semanticscholar","paperid":"43265276","PeriodicalName":null,"FirstCategoryId":null,"ListUrlMain":null,"RegionNum":4,"RegionCategory":"教育学","ArticlePicture":[],"TitleCN":null,"AbstractTextCN":null,"PMCID":"OA","EPubDate":null,"PubModel":null,"JCR":null,"JCRName":null,"Score":null,"Total":0}