Educational and Psychological Measurement最新文献

{"title":"Agreement and Alignment in Binary Rating Tasks: Strategic Convergence as an Equilibrium Outcome.","authors":"Irene Gianeselli","doi":"10.1177/00131644261455307","DOIUrl":"https://doi.org/10.1177/00131644261455307","url":null,"abstract":"<p><p>Agreement between raters does not, by itself, show how that agreement is produced. In binary judgement tasks, raters may classify cases similarly either because they rely on similar decision thresholds or because those thresholds operate under favourable marginal conditions. The present study examined whether convergence in raters' decision thresholds must be imposed exogenously or can emerge endogenously through strategic adjustment. Within an equal-variance signal-detection framework, raters were modelled as choosing thresholds that balance expected classification accuracy against an incentive to align with the rest of the panel. Equilibrium thresholds were obtained by Nash best-response updating, and threshold variance, the Strategic Convergence Index (SCI), and mean pairwise Cohen's <math><mrow><mi>κ</mi></mrow> </math> were examined as functions of alignment pressure and prevalence. Increasing alignment pressure produced a sharp collapse in threshold dispersion and drove SCI towards 1 under the chosen normalisation, whereas <math><mrow><mi>κ</mi></mrow> </math> increased only within a comparatively narrow range. When alignment pressure was held constant, SCI remained effectively unchanged across prevalence conditions, while <math><mrow><mi>κ</mi></mrow> </math> varied substantially. These findings indicate that convergence in decision thresholds can arise as an equilibrium outcome of strategic interaction yet remains analytically distinct from observable agreement. They also clarify the status of Cohen's <math><mrow><mi>κ</mi></mrow> </math> in this setting: it tracks agreement at the level of realised classifications and therefore remains sensitive to marginal conditions, whereas SCI captures convergence in the latent decision structure from which those classifications arise. SCI is thus not proposed as a replacement for agreement coefficients but as a model-based quantity that makes explicit a form of alignment that <math><mrow><mi>κ</mi></mrow> </math> only partially reflects.</p>","PeriodicalId":11502,"journal":{"name":"Educational and Psychological Measurement","volume":" ","pages":"00131644261455307"},"PeriodicalIF":4.4,"publicationDate":"2026-06-25","publicationTypes":"Journal Article","fieldsOfStudy":null,"isOpenAccess":false,"openAccessPdf":"https://www.ncbi.nlm.nih.gov/pmc/articles/PMC13303268/pdf/","citationCount":null,"resultStr":null,"platform":"Semanticscholar","paperid":"148344403","PeriodicalName":null,"FirstCategoryId":null,"ListUrlMain":null,"RegionNum":3,"RegionCategory":"心理学","ArticlePicture":[],"TitleCN":null,"AbstractTextCN":null,"PMCID":"OA","EPubDate":null,"PubModel":null,"JCR":null,"JCRName":null,"Score":null,"Total":0}

{"title":"Interactions Between Termination Criteria and Ability Estimators in Computerized Adaptive Testing.","authors":"Xinyu Liu, David J Weiss","doi":"10.1177/00131644261453945","DOIUrl":"https://doi.org/10.1177/00131644261453945","url":null,"abstract":"<p><p>Computerized adaptive testing (CAT) aims to optimize measurement by tailoring item administration to individual examinees. The efficiency and precision of a CAT heavily depend on the choice of ability ( <math><mrow><mi>θ</mi></mrow> </math> ) estimator and the termination criterion (stopping rule). Prior research suggests these components interact, but comprehensive evaluations across varying item bank characteristics remain limited. This simulation study investigated the interactive effects of four <math><mrow><mi>θ</mi></mrow> </math> estimators (maximum likelihood [MLE], weighted likelihood [WLE], maximum a posteriori [MAP], and expected a posteriori [EAP]) and four termination criteria (fixed-length, standard error of measurement [SEM], minimum information [MI], and change-in-estimate [Δ <math><mrow><mi>θ</mi></mrow> </math> ]) on measurement bias, precision (RMSE), and test length. These combinations were evaluated across low- (100-item) and high-information (500-item) item banks with both flat and peaked information distributions using the three-parameter logistic model. The results demonstrated that the optimal CAT configuration is contingent on item bank size and shape. Across all conditions, WLE emerged as the most robust estimator, effectively neutralizing the boundary estimation issues of MLE and the shrinkage bias characteristic of Bayesian estimators. In high-information banks, the SEM and fixed-length rules yielded the lowest conditional RMSE and bias regardless of bank shape. However, in low-information peaked banks, the strict SEM rule frequently failed to reach precision targets at the θ extremes, resulting in inefficient, maximum-length tests. Under these sparse conditions, the Δθ rule paired with WLE provided a superior balance of accuracy and efficiency by halting administration when precision gains stagnated. Conversely, the MI rule consistently exhibited the highest bias and RMSE. These findings underscore that optimal CAT design is not a one-size-fits-all solution. For high-quality banks, WLE paired with an SEM or fixed-length rule is recommended. For lower-quality banks, practitioners should adopt a <math><mrow><mspace></mspace> <mo>Δ</mo> <mi>θ</mi></mrow> </math> rule or a hybrid SEM approach to prevent inefficient test elongation.</p>","PeriodicalId":11502,"journal":{"name":"Educational and Psychological Measurement","volume":" ","pages":"00131644261453945"},"PeriodicalIF":2.3,"publicationDate":"2026-06-11","publicationTypes":"Journal Article","fieldsOfStudy":null,"isOpenAccess":false,"openAccessPdf":"https://www.ncbi.nlm.nih.gov/pmc/articles/PMC13259887/pdf/","citationCount":null,"resultStr":null,"platform":"Semanticscholar","paperid":"148249775","PeriodicalName":null,"FirstCategoryId":null,"ListUrlMain":null,"RegionNum":3,"RegionCategory":"心理学","ArticlePicture":[],"TitleCN":null,"AbstractTextCN":null,"PMCID":"OA","EPubDate":null,"PubModel":null,"JCR":null,"JCRName":null,"Score":null,"Total":0}

{"title":"Identification and Diagnosis of Misreporting in Surveys.","authors":"Jing Li, Xiao Yang, George Engelhard","doi":"10.1177/00131644261451209","DOIUrl":"https://doi.org/10.1177/00131644261451209","url":null,"abstract":"<p><p>Misreporting and other forms of aberrant responding can undermine the validity of survey-based inferences. Person-level evaluation of aberrant responses is rarely conducted because inspecting individual response patterns is time-intensive. This study proposes an integrated approach for identifying, classifying, and interpreting misfitting response patterns using nonparametric visualizations of person response functions combined with clustering of person response functions. The first step is to calibrate the survey items using an IRT model, such as the Rasch model, to establish an interpretable latent continuum with item-location ordering. Next, person-fit statistics, such as infit and outfit mean square error statistics, are examined, and a smaller subset of response patterns is flagged as misfitting. The third step is to use a nonparametric Hanning procedure to create person response functions, followed by clustering misfitting person response functions using Partitioning Around Medoids (PAM). The advantage of PAM over other clustering methods is that an observed response pattern is identified as a representative case for each cluster. Clusters can then be identified that correspond to an appropriate interpretation for the cluster, such as underreporting, inconsistent reporting, and overreporting patterns. Finally, decisions can be made about how to address aberrant person response patterns. The Household Food Security Survey Module from the U.S. Census is used as an illustration. These visualizations can support transparent data-quality evaluation with the potential for survey improvements.</p>","PeriodicalId":11502,"journal":{"name":"Educational and Psychological Measurement","volume":" ","pages":"00131644261451209"},"PeriodicalIF":2.3,"publicationDate":"2026-06-06","publicationTypes":"Journal Article","fieldsOfStudy":null,"isOpenAccess":false,"openAccessPdf":"https://www.ncbi.nlm.nih.gov/pmc/articles/PMC13242364/pdf/","citationCount":null,"resultStr":null,"platform":"Semanticscholar","paperid":"148204588","PeriodicalName":null,"FirstCategoryId":null,"ListUrlMain":null,"RegionNum":3,"RegionCategory":"心理学","ArticlePicture":[],"TitleCN":null,"AbstractTextCN":null,"PMCID":"OA","EPubDate":null,"PubModel":null,"JCR":null,"JCRName":null,"Score":null,"Total":0}

{"title":"The Aggregated Latent Profile Index: Measuring Person Profile Differentiation Within a Bootstrap-Validated Latent Profile Space.","authors":"Se-Kang Kim","doi":"10.1177/00131644261451487","DOIUrl":"10.1177/00131644261451487","url":null,"abstract":"<p><p>Multivariate assessment profiles contain two conceptually distinct sources of variation: overall level and profile shape. Existing approaches recover some aspects of this structure, but none jointly establishes the replicability of latent pattern dimensions and provides a principled, population-referenced summary of person profile differentiation independent of overall level. We introduce the Aggregated Latent Profile Index (ALPI), a variance-weighted Euclidean distance that quantifies the degree to which an individual's profile departs from a flat population reference within a bootstrap-validated latent profile space. ALPI is derived from the Aggregated Latent Profile Space (ALPS), a framework that combines parallel analysis with bootstrap stability diagnostics - principal angles between subspaces and Tucker's congruence coefficients - to identify replicable latent pattern dimensions, then assembles the retained dimensions into a <i>K</i>-dimensional latent space through singular value decomposition, into which individuals and variables are jointly projected. In a simulation benchmark, ALPS recovered the known four-factor population structure as three replicable ipsatized pattern dimensions, confirming that the pipeline performs as intended when the true structure is known. In an application to normative WAIS-IV data (<i>n</i> = 900), ALPS identified a stable three-dimensional pattern space, and ALPI distinguished individuals with identical Full-Scale IQ - ranging from the 7th to the 85th percentile - on the basis of their ipsatized subtest configurations. ALPI provides assessment researchers and clinicians with a single, measurement-principled index of person profile differentiation that is grounded in replicable latent structure and independent of overall score level.</p>","PeriodicalId":11502,"journal":{"name":"Educational and Psychological Measurement","volume":" ","pages":"00131644261451487"},"PeriodicalIF":4.4,"publicationDate":"2026-06-04","publicationTypes":"Journal Article","fieldsOfStudy":null,"isOpenAccess":false,"openAccessPdf":"https://www.ncbi.nlm.nih.gov/pmc/articles/PMC13236714/pdf/","citationCount":null,"resultStr":null,"platform":"Semanticscholar","paperid":"148197693","PeriodicalName":null,"FirstCategoryId":null,"ListUrlMain":null,"RegionNum":3,"RegionCategory":"心理学","ArticlePicture":[],"TitleCN":null,"AbstractTextCN":null,"PMCID":"OA","EPubDate":null,"PubModel":null,"JCR":null,"JCRName":null,"Score":null,"Total":0}

{"title":"The Anonymous Collection of Longitudinal Data: An Evaluation of Self-Generated Identification Codes and Methodological Challenges.","authors":"Jodi B A McKibben, Kimberly Bucklin, Jordan Salafia","doi":"10.1177/00131644261453059","DOIUrl":"https://doi.org/10.1177/00131644261453059","url":null,"abstract":"<p><p>Self-generated identification codes (SGICs) are commonly used in longitudinal research studies to anonymously link data collected from a participant across time points. However, the lack of research evaluating the methods of developing SGICs is concerning, given that low match rates of participants across time points are common, resulting in significant data loss over time and diminishing statistical power. The current study aims to fill this gap by evaluating the effectiveness and reliability of a newly developed SGIC. Participants (<i>n</i> = 135) completed an SGIC during two sessions, 8 weeks apart. Researchers informed the experimental group that the second survey included sensitive information and the control group that the study focused solely on the SGIC. Although no sensitive information was requested, this allowed analysis of how the perception that sensitive information was being collected might affect the consequent matching of the SGIC. On average, more than 91.9% (91.5% experimental group, 92.2% control group) of participants had at least 10 correct SGIC element matches. Upon examining perfect matches, only 40.0% of participants matched on all 12 elements across the two sessions. Multivariate logistic analyses revealed that the trust rating was a significant predictor of group membership with higher trust ratings being associated with control group membership (odds ratio [OR] = 0.541, 95% confidence interval [CI] = [0.32, 0.91], <i>p</i> = .021) and trustworthiness was a significant predictor of matching on all 12 SGIC elements (OR = 4.975, 95% CI = [1.70, 14.55], <i>p</i> = .003). This study demonstrates that SGICs can function effectively under the right conditions, but reliability depends on the structure of the code itself and on participants' perceptions of trustworthiness, anonymity, and engagement. As researchers continue to seek methods that balance participant privacy with data accuracy, the field must move toward empirically grounded standards.</p>","PeriodicalId":11502,"journal":{"name":"Educational and Psychological Measurement","volume":" ","pages":"00131644261453059"},"PeriodicalIF":2.3,"publicationDate":"2026-06-02","publicationTypes":"Journal Article","fieldsOfStudy":null,"isOpenAccess":false,"openAccessPdf":"https://www.ncbi.nlm.nih.gov/pmc/articles/PMC13233592/pdf/","citationCount":null,"resultStr":null,"platform":"Semanticscholar","paperid":"148197716","PeriodicalName":null,"FirstCategoryId":null,"ListUrlMain":null,"RegionNum":3,"RegionCategory":"心理学","ArticlePicture":[],"TitleCN":null,"AbstractTextCN":null,"PMCID":"OA","EPubDate":null,"PubModel":null,"JCR":null,"JCRName":null,"Score":null,"Total":0}

{"title":"Beyond One-Size-Fits-All: A Differential Sensitivity Framework for Machine Learning-Based Detection of Anomalous Survey Responses.","authors":"Cody Ding","doi":"10.1177/00131644261448404","DOIUrl":"https://doi.org/10.1177/00131644261448404","url":null,"abstract":"<p><p>Anomalous survey responses, including random, careless, extreme, acquiescent, straightline, and alternating responding, threaten the validity of survey-based research. Machine learning (ML) algorithms offer flexible, model-agnostic alternatives to traditional detection methods, yet their relative effectiveness across anomaly types remains poorly understood. This study evaluated 11 unsupervised anomaly detection algorithms spanning four paradigms (distance-based, density-based, reconstruction-based, and tree/boundary-based) against six simulated anomaly types embedded in a realistic survey dataset (<i>N</i> = 3,000). Results revealed pronounced differential sensitivity: globally deviant patterns (random, extreme, alternating) were universally detectable, whereas careless and acquiescent responding required reconstruction- or boundary-based methods, and straightline responding resisted detection by all algorithms (maximum area under the receiver operating characteristic curve [AUC-ROC] < .70). No single algorithm dominated across all types. These findings argue for multimethod approaches combining ML algorithms with traditional response quality indicators, and provide a framework for selecting detection methods based on anticipated anomaly types.</p>","PeriodicalId":11502,"journal":{"name":"Educational and Psychological Measurement","volume":" ","pages":"00131644261448404"},"PeriodicalIF":2.3,"publicationDate":"2026-06-01","publicationTypes":"Journal Article","fieldsOfStudy":null,"isOpenAccess":false,"openAccessPdf":"https://www.ncbi.nlm.nih.gov/pmc/articles/PMC13226289/pdf/","citationCount":null,"resultStr":null,"platform":"Semanticscholar","paperid":"148155849","PeriodicalName":null,"FirstCategoryId":null,"ListUrlMain":null,"RegionNum":3,"RegionCategory":"心理学","ArticlePicture":[],"TitleCN":null,"AbstractTextCN":null,"PMCID":"OA","EPubDate":null,"PubModel":null,"JCR":null,"JCRName":null,"Score":null,"Total":0}

{"title":"Cumulant-Based Approaches for Testing the Assumption of Independent Errors in Non-Gaussian Parallel and Congeneric Measures.","authors":"Wolfgang Wiedermann, Dexin Shi","doi":"10.1177/00131644261444671","DOIUrl":"10.1177/00131644261444671","url":null,"abstract":"<p><p>In classical test theory, independence of measurement errors constitutes a central assumption when estimating the reliability of measures. Furthermore, it is well known that this assumption cannot be tested with standard methods that rely on second-order moments (variances, covariances). The present study, therefore, explores properties of non-Gaussian parallel and congeneric measures (i.e., when observed scores deviate from the Gaussian distribution) with respect to their capabilities of identifying violations of the error independence assumption. We show that, under non-Gaussianity and inequality of hidden confounding effects, third and fourth cumulant-based test statistics can be derived which enable researchers to detect non-independent error structures. We describe identifiability conditions under which the proposed test statistics can be expected to have adequate statistical power in parallel and congeneric measures and present results of Monte-Carlo simulation experiments. Results suggest that third-order tests adequately protect the nominal significance level. However, fourth-order tests can produce inflated Type I error rates, in particular, when error variances are unequal. In general, the power to detect non-independent errors increases with the sample size, the magnitude of non-Gaussianity, the degree of inequality of hidden confounding effects, and the degree of error non-independence. A real-world data example is presented for illustrative purposes. The current study presents important insights for developing statistical methods to detect error non-independence. However, these methods also rest on crucial assumptions, emphasizing the severity of the issue of statistically detecting non-independent measurement errors.</p>","PeriodicalId":11502,"journal":{"name":"Educational and Psychological Measurement","volume":" ","pages":"00131644261444671"},"PeriodicalIF":4.4,"publicationDate":"2026-05-19","publicationTypes":"Journal Article","fieldsOfStudy":null,"isOpenAccess":false,"openAccessPdf":"https://www.ncbi.nlm.nih.gov/pmc/articles/PMC13186936/pdf/","citationCount":null,"resultStr":null,"platform":"Semanticscholar","paperid":"147987072","PeriodicalName":null,"FirstCategoryId":null,"ListUrlMain":null,"RegionNum":3,"RegionCategory":"心理学","ArticlePicture":[],"TitleCN":null,"AbstractTextCN":null,"PMCID":"OA","EPubDate":null,"PubModel":null,"JCR":null,"JCRName":null,"Score":null,"Total":0}

{"title":"Factor Scores in Small Samples: Recommendations and Solutions.","authors":"Christian L L Strauss","doi":"10.1177/00131644261441852","DOIUrl":"10.1177/00131644261441852","url":null,"abstract":"<p><p>Simultaneous estimation of structural and measurement models in structural equation modeling (SEM) is not always tenable in small samples. In such cases, it may be necessary or advantageous to obtain scores. Current scoring recommendations draw predominantly from simulations with sample sizes greater than <i>N</i> = 200. This paper extends these recommendations to small <i>N</i>, directly comparing factor scores to sum scores. In addition, scores computed from an essentially tau-equivalent factor model are introduced as an alternative scoring option aimed at balancing the competing benefits of sum scores and factor scores. Findings largely suggest that factor scores from an essentially tau-equivalent factor model are advantageous when considering convergence and stability, even when not supported by the data, so long as departures from their assumptions are not substantial. They are obtainable when congeneric factor models fail to converge and have similar correlations with true scores compared with typical factor scores in samples at or less than <i>N</i> = 200.</p>","PeriodicalId":11502,"journal":{"name":"Educational and Psychological Measurement","volume":" ","pages":"00131644261441852"},"PeriodicalIF":4.4,"publicationDate":"2026-05-15","publicationTypes":"Journal Article","fieldsOfStudy":null,"isOpenAccess":false,"openAccessPdf":"https://www.ncbi.nlm.nih.gov/pmc/articles/PMC13179303/pdf/","citationCount":null,"resultStr":null,"platform":"Semanticscholar","paperid":"147962568","PeriodicalName":null,"FirstCategoryId":null,"ListUrlMain":null,"RegionNum":3,"RegionCategory":"心理学","ArticlePicture":[],"TitleCN":null,"AbstractTextCN":null,"PMCID":"OA","EPubDate":null,"PubModel":null,"JCR":null,"JCRName":null,"Score":null,"Total":0}

{"title":"Signposts on the Path From Nominal to Ordinal Scales: Moving From a Discrete to a Continuous View.","authors":"Roza Nalbandyan, Joshua B Gilbert, Vithor R Franco, Benjamin W Domingue","doi":"10.1177/00131644261440556","DOIUrl":"https://doi.org/10.1177/00131644261440556","url":null,"abstract":"<p><p>Polytomous item response data are typically classified as either nominal or ordinal, but this binary distinction may oversimplify their true structure. In this paper, we reframe the nominal-ordinal distinction as a continuum and introduce six empirical indices to quantify the degree of category ordering in item response data. Through extensive simulations with various item response theory (IRT) models and applications to 245 empirical datasets, we evaluate the indices' sensitivity, computational efficiency, and interpretability across diverse measurement contexts. Our findings show that two parametric indices-Mean Difference between Slope Parameters (Index 5) and Arctangent of Paired Category Ratios (Index 6)-are particularly robust and informative, even with low-frequency categories. These indices offer a practical tool for assessing whether and how item categories align with ordinal assumptions, supporting more accurate measurement and model selection. We conclude that treating ordering as a continuum, rather than a binary property, provides deeper insights for psychometric practice and strengthens the connection between empirical response patterns and their theoretical representations.</p>","PeriodicalId":11502,"journal":{"name":"Educational and Psychological Measurement","volume":" ","pages":"00131644261440556"},"PeriodicalIF":2.3,"publicationDate":"2026-05-09","publicationTypes":"Journal Article","fieldsOfStudy":null,"isOpenAccess":false,"openAccessPdf":"https://www.ncbi.nlm.nih.gov/pmc/articles/PMC13157476/pdf/","citationCount":null,"resultStr":null,"platform":"Semanticscholar","paperid":"147873434","PeriodicalName":null,"FirstCategoryId":null,"ListUrlMain":null,"RegionNum":3,"RegionCategory":"心理学","ArticlePicture":[],"TitleCN":null,"AbstractTextCN":null,"PMCID":"OA","EPubDate":null,"PubModel":null,"JCR":null,"JCRName":null,"Score":null,"Total":0}

{"title":"Assessing the Unconditional and Conditional External Validity of Noncognitive Test Scores: A Unifying Model-Based Proposal.","authors":"Pere J Ferrando, Fabia Morales-Vives, Silvia Duran-Bonavila, David Navarro-González","doi":"10.1177/00131644261440168","DOIUrl":"https://doi.org/10.1177/00131644261440168","url":null,"abstract":"<p><p>Evidence of external validity based on individual score estimates is still relevant in many psychometric applications. From a model-based perspective, however, the topic appears to have been rather neglected in recent decades. Thus, in structural equation modelling (SEM), this evidence is sought to be obtained structurally, bypassing the scoring stage. And, in item response theory (IRT), the score interest mostly focuses on internal properties. Taking this state of affairs into account, this paper develops and proposes a model-based approach, intended for noncognitive measures, that combines SEM and IRT developments, and which allows a detailed assessment of the external validity of a class of score estimates to be carried out. The starting point is a general extended model that also includes the relevant external variables. From this general model, four well-known extended IRT models can be derived and fitted at the structural level. Next, on the basis of the structural results, a series of unconditional (population-dependent) and conditional (population-independent) indices that describe the model-implied relation between the score estimates and each external variable are developed and proposed. The practical relevance of the proposal is discussed mainly around three applications: assessing model appropriateness, obtaining point and interval prediction estimates at the individual level, and shortening a test while optimizing the external validity of the resulting version. The functioning of the proposal is illustrated using a real-data example.</p>","PeriodicalId":11502,"journal":{"name":"Educational and Psychological Measurement","volume":" ","pages":"00131644261440168"},"PeriodicalIF":2.3,"publicationDate":"2026-04-30","publicationTypes":"Journal Article","fieldsOfStudy":null,"isOpenAccess":false,"openAccessPdf":"https://www.ncbi.nlm.nih.gov/pmc/articles/PMC13133035/pdf/","citationCount":null,"resultStr":null,"platform":"Semanticscholar","paperid":"147812434","PeriodicalName":null,"FirstCategoryId":null,"ListUrlMain":null,"RegionNum":3,"RegionCategory":"心理学","ArticlePicture":[],"TitleCN":null,"AbstractTextCN":null,"PMCID":"OA","EPubDate":null,"PubModel":null,"JCR":null,"JCRName":null,"Score":null,"Total":0}