Journal of Educational Measurement最新文献

{"title":"Validation for Personalized Assessments: A Threats-to-Validity Approach","authors":"Sandip Sinharay,&nbsp;Randy E. Bennett,&nbsp;Michael Kane,&nbsp;Jesse R. Sparks","doi":"10.1111/jedm.12434","DOIUrl":"https://doi.org/10.1111/jedm.12434","url":null,"abstract":"<p>Personalized assessments are of increasing interest because of their potential to lead to more equitable decisions about the examinees. However, one obstacle to the widespread use of personalized assessments is the lack of a measurement toolkit that can be used to analyze data from these assessments. This article takes one step toward building such a toolkit by proposing a validation framework for personalized assessments. The framework is built on the threats-to-validity approach. We demonstrate applications of the suggested framework using the AP 3D Art and Design Portfolio examination and a more restrictive culturally relevant assessment as examples.</p>","PeriodicalId":47871,"journal":{"name":"Journal of Educational Measurement","volume":"62 2","pages":"282-310"},"PeriodicalIF":1.4,"publicationDate":"2025-04-23","publicationTypes":"Journal Article","fieldsOfStudy":null,"isOpenAccess":false,"openAccessPdf":"","citationCount":null,"resultStr":null,"platform":"Semanticscholar","paperid":"144524631","PeriodicalName":null,"FirstCategoryId":null,"ListUrlMain":null,"RegionNum":4,"RegionCategory":"心理学","ArticlePicture":[],"TitleCN":null,"AbstractTextCN":null,"PMCID":"","EPubDate":null,"PubModel":null,"JCR":null,"JCRName":null,"Score":null,"Total":0}

{"title":"Using Multiple Maximum Exposure Rates in Computerized Adaptive Testing","authors":"Kylie Gorney,&nbsp;Mark D. Reckase","doi":"10.1111/jedm.12436","DOIUrl":"https://doi.org/10.1111/jedm.12436","url":null,"abstract":"<p>In computerized adaptive testing, item exposure control methods are often used to provide a more balanced usage of the item pool. Many of the most popular methods, including the restricted method (Revuelta and Ponsoda), use a single maximum exposure rate to limit the proportion of times that each item is administered. However, Barrada et al. showed that by using multiple maximum exposure rates, it is possible to obtain an even more balanced usage of the item pool. Therefore, in this paper, we develop four extensions of the restricted method that involve the use of multiple maximum exposure rates. A detailed simulation study reveals that (a) all four of the new methods improve item pool utilization and (b) three of the new methods also improve measurement accuracy. Taken together, these results are highly encouraging, as they reveal that it is possible to improve both types of outcomes simultaneously.</p>","PeriodicalId":47871,"journal":{"name":"Journal of Educational Measurement","volume":"62 2","pages":"360-379"},"PeriodicalIF":1.4,"publicationDate":"2025-04-16","publicationTypes":"Journal Article","fieldsOfStudy":null,"isOpenAccess":false,"openAccessPdf":"https://onlinelibrary.wiley.com/doi/epdf/10.1111/jedm.12436","citationCount":null,"resultStr":null,"platform":"Semanticscholar","paperid":"144524607","PeriodicalName":null,"FirstCategoryId":null,"ListUrlMain":null,"RegionNum":4,"RegionCategory":"心理学","ArticlePicture":[],"TitleCN":null,"AbstractTextCN":null,"PMCID":"OA","EPubDate":null,"PubModel":null,"JCR":null,"JCRName":null,"Score":null,"Total":0}

{"title":"Theory-Driven IRT Modeling of Vocabulary Development: Matthew Effects and the Case for Unipolar IRT","authors":"Qi (Helen) Huang,&nbsp;Daniel M. Bolt,&nbsp;Xiangyi Liao","doi":"10.1111/jedm.12433","DOIUrl":"https://doi.org/10.1111/jedm.12433","url":null,"abstract":"<p>Item response theory (IRT) encompasses a broader class of measurement models than is commonly appreciated by practitioners in educational measurement. For measures of vocabulary and its development, we show how psychological theory might in certain instances support unipolar IRT modeling as a superior alternative to the more traditional bipolar IRT models fit in practice. Although corresponding model choices make unipolar IRT statistically equivalent with bipolar IRT, adopting the unipolar approach substantially alters the resulting metric for proficiency. This shift can have substantial implications for educational research and practices that depend heavily on interval-level score interpretations. As an example, we illustrate through simulation how the perspective of unipolar IRT may account for inconsistencies seen across empirical studies in the observation (or lack thereof) of Matthew effects in reading/vocabulary development (i.e., growth being positively correlated with baseline proficiency), despite theoretical expectations for their presence. Additionally, a unipolar measurement perspective can reflect the anticipated diversification of vocabulary as proficiency level increases. Implications of unipolar IRT representations for constructing tests of vocabulary proficiency and evaluating measurement error are discussed.</p>","PeriodicalId":47871,"journal":{"name":"Journal of Educational Measurement","volume":"62 2","pages":"199-224"},"PeriodicalIF":1.4,"publicationDate":"2025-04-11","publicationTypes":"Journal Article","fieldsOfStudy":null,"isOpenAccess":false,"openAccessPdf":"https://onlinelibrary.wiley.com/doi/epdf/10.1111/jedm.12433","citationCount":null,"resultStr":null,"platform":"Semanticscholar","paperid":"144524868","PeriodicalName":null,"FirstCategoryId":null,"ListUrlMain":null,"RegionNum":4,"RegionCategory":"心理学","ArticlePicture":[],"TitleCN":null,"AbstractTextCN":null,"PMCID":"OA","EPubDate":null,"PubModel":null,"JCR":null,"JCRName":null,"Score":null,"Total":0}

{"title":"Another Look at Yen's Q3: Is .2 an Appropriate Cut-Off?","authors":"Kelsey Nason,&nbsp;Christine DeMars","doi":"10.1111/jedm.12432","DOIUrl":"https://doi.org/10.1111/jedm.12432","url":null,"abstract":"<p>This study examined the widely used threshold of .2 for Yen's Q3, an index for violations of local independence. Specifically, a simulation was conducted to investigate whether Q3 values were related to the magnitude of bias in estimates of reliability, item parameters, and examinee ability. Results showed that Q3 values below the typical cut-off yielded meaningful bias in estimates. Practical implications and limitations are discussed.</p>","PeriodicalId":47871,"journal":{"name":"Journal of Educational Measurement","volume":"62 2","pages":"345-359"},"PeriodicalIF":1.4,"publicationDate":"2025-04-04","publicationTypes":"Journal Article","fieldsOfStudy":null,"isOpenAccess":false,"openAccessPdf":"https://onlinelibrary.wiley.com/doi/epdf/10.1111/jedm.12432","citationCount":null,"resultStr":null,"platform":"Semanticscholar","paperid":"144524843","PeriodicalName":null,"FirstCategoryId":null,"ListUrlMain":null,"RegionNum":4,"RegionCategory":"心理学","ArticlePicture":[],"TitleCN":null,"AbstractTextCN":null,"PMCID":"OA","EPubDate":null,"PubModel":null,"JCR":null,"JCRName":null,"Score":null,"Total":0}

{"title":"Influence of Intersectional Routing Modules between Dimensions on Measurement Precision in Multidimensional Multistage Testing","authors":"Yi-Ling Wu,&nbsp;Yao-Hsuan Huang,&nbsp;Chia-Wen Chen,&nbsp;Po-Hsi Chen","doi":"10.1111/jedm.12430","DOIUrl":"https://doi.org/10.1111/jedm.12430","url":null,"abstract":"<div>\u0000 \u0000 <section>\u0000 \u0000 <p>Multistage testing (MST), a variant of computerized adaptive testing (CAT), differs from conventional CAT in that it is adapted at the module level rather than at the individual item level. Typically, all examinees begin the MST with a linear test form in the first stage, commonly known as the routing stage. In 2020, Han introduced an innovative concept known as Intersectional Routing (ISR), which allows module selection in the first stage of the MST based on the examinees' estimated scores. These scores were predicted using a variety of information, including background data and other correlated latent traits.</p>\u0000 \u0000 <p>In this study, we extend Han's ISR framework to a multidimensional test comprising multiple unidimensional subtests. In a multidimensional test, the correlation coefficients between the latent traits can be estimated by fitting a multidimensional item response theory model. Our extension allows module selection in the first stage of each subtest to consider information from all the other subtests via the known correlation matrix. The results of simulation studies showed that our extension improved the measurement compared with typical MST designs in conditions with moderate intercorrelations across module designs. The practical insights were given in the empirical analysis.</p>\u0000 </section>\u0000 </div>","PeriodicalId":47871,"journal":{"name":"Journal of Educational Measurement","volume":"62 3","pages":"438-467"},"PeriodicalIF":1.6,"publicationDate":"2025-04-03","publicationTypes":"Journal Article","fieldsOfStudy":null,"isOpenAccess":false,"openAccessPdf":"https://onlinelibrary.wiley.com/doi/epdf/10.1111/jedm.12430","citationCount":null,"resultStr":null,"platform":"Semanticscholar","paperid":"145341442","PeriodicalName":null,"FirstCategoryId":null,"ListUrlMain":null,"RegionNum":4,"RegionCategory":"心理学","ArticlePicture":[],"TitleCN":null,"AbstractTextCN":null,"PMCID":"OA","EPubDate":null,"PubModel":null,"JCR":null,"JCRName":null,"Score":null,"Total":0}

{"title":"Using Keystroke Dynamics to Detect Nonoriginal Text","authors":"Paul Deane,&nbsp;Mo Zhang,&nbsp;Jiangang Hao,&nbsp;Chen Li","doi":"10.1111/jedm.12431","DOIUrl":"10.1111/jedm.12431","url":null,"abstract":"<div>\u0000 \u0000 <section>\u0000 \u0000 <p>Keystroke analysis has often been used for security purposes, most often to authenticate users and identify impostors. This paper examines the use of keystroke analysis to distinguish between the behavior of writers who are composing an original text, vs. copying or otherwise reproducing a non-original texts. Recent advances in text generation using large language models makes the use of behavioral cues to identify plagiarism more pressing, since users seeking an advantage on a writing assessment may be able to submit unique AI-generated texts. We examine the use of keystroke log analysis to detect non-original text under three conditions: a laboratory study, where participants were either copying a known text or drafting an original essay, and two studies from operational assessments, where it was possible to identify essays that were non-original by refernece to their content. Our results indicate that it is possible to achieve accuracies inexcess of 94% under ideal conditions where the nature of each writing sessionis known in advance, and greater than 89% in operational conditions where proxies for non-original status, such as similarity to other submitted essays, must be used.</p>\u0000 </section>\u0000 </div>","PeriodicalId":47871,"journal":{"name":"Journal of Educational Measurement","volume":"63 1","pages":""},"PeriodicalIF":1.6,"publicationDate":"2025-04-03","publicationTypes":"Journal Article","fieldsOfStudy":null,"isOpenAccess":false,"openAccessPdf":"","citationCount":null,"resultStr":null,"platform":"Semanticscholar","paperid":"146091158","PeriodicalName":null,"FirstCategoryId":null,"ListUrlMain":null,"RegionNum":4,"RegionCategory":"心理学","ArticlePicture":[],"TitleCN":null,"AbstractTextCN":null,"PMCID":"","EPubDate":null,"PubModel":null,"JCR":null,"JCRName":null,"Score":null,"Total":0}

{"title":"Using Keystroke Dynamics to Detect Nonoriginal Text","authors":"Paul Deane,&nbsp;Mo Zhang,&nbsp;Jiangang Hao,&nbsp;Chen Li","doi":"10.1111/jedm.12431","DOIUrl":"https://doi.org/10.1111/jedm.12431","url":null,"abstract":"<div>\u0000 \u0000 <section>\u0000 \u0000 <p>Keystroke analysis has often been used for security purposes, most often to authenticate users and identify impostors. This paper examines the use of keystroke analysis to distinguish between the behavior of writers who are composing an original text, vs. copying or otherwise reproducing a non-original texts. Recent advances in text generation using large language models makes the use of behavioral cues to identify plagiarism more pressing, since users seeking an advantage on a writing assessment may be able to submit unique AI-generated texts. We examine the use of keystroke log analysis to detect non-original text under three conditions: a laboratory study, where participants were either copying a known text or drafting an original essay, and two studies from operational assessments, where it was possible to identify essays that were non-original by refernece to their content. Our results indicate that it is possible to achieve accuracies inexcess of 94% under ideal conditions where the nature of each writing sessionis known in advance, and greater than 89% in operational conditions where proxies for non-original status, such as similarity to other submitted essays, must be used.</p>\u0000 </section>\u0000 </div>","PeriodicalId":47871,"journal":{"name":"Journal of Educational Measurement","volume":"63 1","pages":""},"PeriodicalIF":1.6,"publicationDate":"2025-04-03","publicationTypes":"Journal Article","fieldsOfStudy":null,"isOpenAccess":false,"openAccessPdf":"","citationCount":null,"resultStr":null,"platform":"Semanticscholar","paperid":"146091041","PeriodicalName":null,"FirstCategoryId":null,"ListUrlMain":null,"RegionNum":4,"RegionCategory":"心理学","ArticlePicture":[],"TitleCN":null,"AbstractTextCN":null,"PMCID":"","EPubDate":null,"PubModel":null,"JCR":null,"JCRName":null,"Score":null,"Total":0}

{"title":"A Comparison of Anchor Selection Strategies for DIF Analysis","authors":"Haeju Lee,&nbsp;Kyung Yong Kim","doi":"10.1111/jedm.12429","DOIUrl":"https://doi.org/10.1111/jedm.12429","url":null,"abstract":"&lt;p&gt;When no prior information of differential item functioning (DIF) exists for items in a test, either the rank-based or iterative purification procedure might be preferred. The rank-based purification selects anchor items based on a preliminary DIF test. For a preliminary DIF test, likelihood ratio test (LRT) based approaches (e.g., all-others-as-anchors: AOAA and one-item-anchor: OIA) and an improved version of Lord's Wald test (i.e., anchor-all-test-all: AATA) have been used in research studies. However, both LRT- and Wald-based procedures often select DIF items as anchor items and as a result, inflate Type &lt;span&gt;&lt;/span&gt;&lt;math&gt;\u0000 &lt;semantics&gt;\u0000 &lt;mi&gt;I&lt;/mi&gt;\u0000 &lt;annotation&gt;${mathrm{{mathrm I}}}$&lt;/annotation&gt;\u0000 &lt;/semantics&gt;&lt;/math&gt; error rates. To overcome this issue, minimum test statistics (&lt;span&gt;&lt;/span&gt;&lt;math&gt;\u0000 &lt;semantics&gt;\u0000 &lt;mrow&gt;\u0000 &lt;mi&gt;Min&lt;/mi&gt;\u0000 &lt;mspace&gt;&lt;/mspace&gt;\u0000 &lt;msup&gt;\u0000 &lt;mi&gt;G&lt;/mi&gt;\u0000 &lt;mn&gt;2&lt;/mn&gt;\u0000 &lt;/msup&gt;\u0000 &lt;/mrow&gt;\u0000 &lt;annotation&gt;${mathrm{Min}};{G^2}$&lt;/annotation&gt;\u0000 &lt;/semantics&gt;&lt;/math&gt;/&lt;span&gt;&lt;/span&gt;&lt;math&gt;\u0000 &lt;semantics&gt;\u0000 &lt;msup&gt;\u0000 &lt;mi&gt;χ&lt;/mi&gt;\u0000 &lt;mn&gt;2&lt;/mn&gt;\u0000 &lt;/msup&gt;\u0000 &lt;annotation&gt;${chi ^2}$&lt;/annotation&gt;\u0000 &lt;/semantics&gt;&lt;/math&gt;) or items with nonsignificant test statistics and large discrimination parameter estimates (&lt;span&gt;&lt;/span&gt;&lt;math&gt;\u0000 &lt;semantics&gt;\u0000 &lt;mi&gt;NonsigMax&lt;/mi&gt;\u0000 &lt;annotation&gt;${mathrm{NonsigMax}}$&lt;/annotation&gt;\u0000 &lt;/semantics&gt;&lt;/math&gt;&lt;i&gt;A&lt;/i&gt;) have been suggested in the literature to select anchor items. Nevertheless, little research has been done comparing combinations of the three anchor selection procedures paired with the two anchor selection criteria. Thus, the performance of the six rank-based strategies was compared in this study in terms of accuracy, power, and Type &lt;span&gt;&lt;/span&gt;&lt;math&gt;\u0000 &lt;semantics&gt;\u0000 &lt;mi&gt;I&lt;/mi&gt;\u0000 &lt;annotation&gt;${mathrm{{mathrm I}}}$&lt;/annotation&gt;\u0000 &lt;/semantics&gt;&lt;/math&gt; error rates. Among the rank-based strategies, the AOAA-based strategies demonstrated greater robustness across various conditions compared to the AATA- and OIA-based strategies. Additionally, the &lt;span&gt;&lt;/span&gt;&lt;math&gt;\u0000 &lt;semantics&gt;\u0000 &lt;mrow&gt;\u0000 &lt;mrow&gt;\u0000 &lt;mi&gt;Min&lt;/mi&gt;\u0000 &lt;mspace&gt;&lt;/mspace&gt;\u0000 &lt;/mrow&gt;\u0000 &lt;msup&gt;\u0000 &lt;mi&gt;G&lt;/mi&gt;\u0000 &lt;mn&gt;2&lt;/mn&gt;\u0000 &lt;/msup&gt;\u0000 &lt;/mrow&gt;\u0000 &lt;annotation&gt;${mathrm{Min;}}{G^2}$&lt;/annotation&gt;\u0000 &lt;/semantics&gt;&lt;/math&gt;/&lt;span&gt;&lt;/span&gt;&lt;math&gt;\u0000 &lt;se","PeriodicalId":47871,"journal":{"name":"Journal of Educational Measurement","volume":"62 2","pages":"311-344"},"PeriodicalIF":1.4,"publicationDate":"2025-03-20","publicationTypes":"Journal Article","fieldsOfStudy":null,"isOpenAccess":false,"openAccessPdf":"https://onlinelibrary.wiley.com/doi/epdf/10.1111/jedm.12429","citationCount":null,"resultStr":null,"platform":"Semanticscholar","paperid":"144525159","PeriodicalName":null,"FirstCategoryId":null,"ListUrlMain":null,"RegionNum":4,"RegionCategory":"心理学","ArticlePicture":[],"TitleCN":null,"AbstractTextCN":null,"PMCID":"OA","EPubDate":null,"PubModel":null,"JCR":null,"JCRName":null,"Score":null,"Total":0}

{"title":"Modeling Missing Response Data in Item Response Theory: Addressing Missing Not at Random Mechanism with Monotone Missing Characteristics","authors":"Jiwei Zhang,&nbsp;Jing Lu,&nbsp;Zhaoyuan Zhang","doi":"10.1111/jedm.12428","DOIUrl":"10.1111/jedm.12428","url":null,"abstract":"<p>Item nonresponses frequently occurs in educational and psychological assessments, and if not appropriately handled, it can undermine the reliability of the results. This study introduces a missing data model based on the missing not at random (MNAR) mechanism, incorporating the monotonic missingness assumption to capture individual-level missingness patterns and behavioral dynamics. In specific, the cumulative number of missing indicators allows to consider the tendency of current item's missingness based on the previous missingnesses, which reduces the number of nuisance parameters for modeling missing data mechanisms. Two Bayesian model evaluation criteria were developed to distinguish between missing at random (MAR) and MNAR mechanisms by imposing specific parameter constraints. Additionally, the study introduces a highly efficient Bayesian slice sampling algorithm to estimate the model parameters. Four simulation studies were conducted to show the performance of the proposed model. The PISA 2015 science data was carried out to further illustrate the application of the proposed approach.</p>","PeriodicalId":47871,"journal":{"name":"Journal of Educational Measurement","volume":"63 1","pages":""},"PeriodicalIF":1.6,"publicationDate":"2025-02-24","publicationTypes":"Journal Article","fieldsOfStudy":null,"isOpenAccess":false,"openAccessPdf":"","citationCount":null,"resultStr":null,"platform":"Semanticscholar","paperid":"146057746","PeriodicalName":null,"FirstCategoryId":null,"ListUrlMain":null,"RegionNum":4,"RegionCategory":"心理学","ArticlePicture":[],"TitleCN":null,"AbstractTextCN":null,"PMCID":"","EPubDate":null,"PubModel":null,"JCR":null,"JCRName":null,"Score":null,"Total":0}

{"title":"The Vulnerability of AI-Based Scoring Systems to Gaming Strategies: A Case Study","authors":"Peter Baldwin,&nbsp;Victoria Yaneva,&nbsp;Kai North,&nbsp;Le An Ha,&nbsp;Yiyun Zhou,&nbsp;Alex J. Mechaber,&nbsp;Brian E. Clauser","doi":"10.1111/jedm.12427","DOIUrl":"https://doi.org/10.1111/jedm.12427","url":null,"abstract":"<p>Recent developments in the use of large-language models have led to substantial improvements in the accuracy of content-based automated scoring of free-text responses. The reported accuracy levels suggest that automated systems could have widespread applicability in assessment. However, before they are used in operational testing, other aspects of their performance warrant examination. In this study, we explore the potential for examinees to inflate their scores by gaming the ACTA automated scoring system. We explore a range of strategies including responding with words selected from the item stem and responding with multiple answers. These responses would be easily identified as incorrect by a human rater but may result in false-positive classifications from an automated system. Our results show that the rate at which these strategies produce responses that are scored as correct varied across items and across strategies but that several vulnerabilities exist.</p>","PeriodicalId":47871,"journal":{"name":"Journal of Educational Measurement","volume":"62 1","pages":"172-194"},"PeriodicalIF":1.4,"publicationDate":"2025-02-20","publicationTypes":"Journal Article","fieldsOfStudy":null,"isOpenAccess":false,"openAccessPdf":"","citationCount":null,"resultStr":null,"platform":"Semanticscholar","paperid":"143688874","PeriodicalName":null,"FirstCategoryId":null,"ListUrlMain":null,"RegionNum":4,"RegionCategory":"心理学","ArticlePicture":[],"TitleCN":null,"AbstractTextCN":null,"PMCID":"","EPubDate":null,"PubModel":null,"JCR":null,"JCRName":null,"Score":null,"Total":0}