Journal of Educational and Behavioral Statistics最新文献

{"title":"A Case Study of Nonresponse Bias Analysis in Educational Assessment Surveys","authors":"Yajuan Si, R. Little, Ya Mo, N. Sedransk","doi":"10.3102/10769986221141074","DOIUrl":"https://doi.org/10.3102/10769986221141074","url":null,"abstract":"Nonresponse bias is a widely prevalent problem for data on education. We develop a ten-step exemplar to guide nonresponse bias analysis (NRBA) in cross-sectional studies and apply these steps to the Early Childhood Longitudinal Study, Kindergarten Class of 2010–2011. A key step is the construction of indices of nonresponse bias based on proxy pattern-mixture models for survey variables of interest. A novel feature is to characterize the strength of evidence about nonresponse bias contained in these indices, based on the strength of the relationship between the characteristics in the nonresponse adjustment and the key survey variables. Our NRBA improves the existing methods by incorporating both missing at random and missing not at random mechanisms, and all analyses can be done straightforwardly with standard statistical software.","PeriodicalId":48001,"journal":{"name":"Journal of Educational and Behavioral Statistics","volume":"48 1","pages":"271 - 295"},"PeriodicalIF":2.4,"publicationDate":"2021-04-09","publicationTypes":"Journal Article","fieldsOfStudy":null,"isOpenAccess":false,"openAccessPdf":"","citationCount":null,"resultStr":null,"platform":"Semanticscholar","paperid":"46105939","PeriodicalName":null,"FirstCategoryId":null,"ListUrlMain":null,"RegionNum":3,"RegionCategory":"心理学","ArticlePicture":[],"TitleCN":null,"AbstractTextCN":null,"PMCID":"","EPubDate":null,"PubModel":null,"JCR":null,"JCRName":null,"Score":null,"Total":0}

{"title":"Introduction to JEBS Special Issue on NAEP Linked Aggregate Scores","authors":"D. McCaffrey, S. Culpepper","doi":"10.3102/10769986211001480","DOIUrl":"https://doi.org/10.3102/10769986211001480","url":null,"abstract":"The Stanford Education Data Archive (SEDA) was created by Sean Reardon, Andrew Ho, Demetra Kalogrides, and their colleagues using annual state summative test score data retrieved from the EDFacts Restricted-Use Files and publicly available NAEP data from the National Center for Education Statistics. SEDA provides test score data on a common scale across all states for mathematics and reading language arts for students in Grades 3 through 8 for almost all schools, districts, and counties in the United States. An online tool (edopportu nity.org) allows users to visually compare schools and districts from anywhere in the country. Data also include various covariates at each of these levels, and all the data can be downloaded for free for analysis. These data have the potential to be a very valuable resource for researchers, educators, policy makers, and possibly even the general public. The catch is that there is no common standardized test administered to students in Grades 3 through 8 in all schools and school districts in all states. NAEP is only administered in a relatively small sample of schools in each state and only to students in Grades 4 and 8 and only every other year. The school data in SEDA are derived from the annual tests administered by each state in accordance with federal regulations. Reardon, Ho, Kalogrides, and colleagues start with aggregate data of the numbers of students in each school or district meeting various performance levels on their state standardized tests. State tests are on different scales and test somewhat different content. They also use different cutoffs for performance levels that are not common across states. Reardon, Ho, Kalogrides, and colleagues convert these frequencies to means and standard deviations for the scores in each school or district using the Heteroskedastic Ordered Probit model that was developed into a series of papers in JEBS (Lockwood et al., 2018; Reardon et al., 2017; Shear & Reardon, 2021). They then link these means and standard deviations to the NAEP scale using methods described in Reardon et al. (2021). Reardon, Ho, Kalogrides, and colleagues stitched together a collection of methods to create a national data source of Journal of Educational and Behavioral Statistics 2021, Vol. 46, No. 2, pp. 135–137 DOI: 10.3102/10769986211001480 Article reuse guidelines: sagepub.com/journals-permissions © 2021 AERA. https://journals.sagepub.com/home/jeb","PeriodicalId":48001,"journal":{"name":"Journal of Educational and Behavioral Statistics","volume":"46 1","pages":"135 - 137"},"PeriodicalIF":2.4,"publicationDate":"2021-04-01","publicationTypes":"Journal Article","fieldsOfStudy":null,"isOpenAccess":false,"openAccessPdf":"","citationCount":null,"resultStr":null,"platform":"Semanticscholar","paperid":"46561773","PeriodicalName":null,"FirstCategoryId":null,"ListUrlMain":null,"RegionNum":3,"RegionCategory":"心理学","ArticlePicture":[],"TitleCN":null,"AbstractTextCN":null,"PMCID":"","EPubDate":null,"PubModel":null,"JCR":null,"JCRName":null,"Score":null,"Total":0}

{"title":"Validation Methods for Aggregate-Level Test Scale Linking: A Rejoinder","authors":"Andrew D. Ho, Sean F. Reardon, Demetra Kalogrides","doi":"10.3102/1076998621994540","DOIUrl":"https://doi.org/10.3102/1076998621994540","url":null,"abstract":"In this issue, Reardon, Kalogrides, and Ho developed precision-adjusted random effects models to estimate aggregate-level linking error, for populations and subpopulations, for averages and progress over time. We are grateful to past editor Dan McCaffrey for selecting our paper as the focal article for a set of commentaries from our colleagues Daniel Bolt, Mark Davison, Alina von Davier, Tim Moses, and Neil Dorans. These commentaries reinforce important cautions and identify promising directions for future research. In this rejoinder, we clarify aspects of our originally proposed method. (1) Validation methods provide evidence of benefits and risks that different experts may weigh differently for different purposes. (2) Our proposed method differs from “standard mapping” procedures using the National Assessment of Educational Progress not only by using a linear (vs. equipercentile) link but also by targeting direct validity evidence about counterfactual aggregate scores. (3) Multilevel approaches that assume common score scales across states are indeed a promising next step for validation, and we hope that states enable researchers to use more of their common-core-era consortium test data for this purpose. Finally, we apply our linking method to an extended panel of data from 2009 to 2017 to show that linking recovery has remained stable.","PeriodicalId":48001,"journal":{"name":"Journal of Educational and Behavioral Statistics","volume":"46 1","pages":"209 - 218"},"PeriodicalIF":2.4,"publicationDate":"2021-04-01","publicationTypes":"Journal Article","fieldsOfStudy":null,"isOpenAccess":false,"openAccessPdf":"","citationCount":null,"resultStr":null,"platform":"Semanticscholar","paperid":"49049001","PeriodicalName":null,"FirstCategoryId":null,"ListUrlMain":null,"RegionNum":3,"RegionCategory":"心理学","ArticlePicture":[],"TitleCN":null,"AbstractTextCN":null,"PMCID":"","EPubDate":null,"PubModel":null,"JCR":null,"JCRName":null,"Score":null,"Total":0}

{"title":"A Rating Scale Mixture Model to Account for the Tendency to Middle and Extreme Categories","authors":"R. Colombi, S. Giordano, G. Tutz","doi":"10.3102/1076998621992554","DOIUrl":"https://doi.org/10.3102/1076998621992554","url":null,"abstract":"A mixture of logit models is proposed that discriminates between responses to rating questions that are affected by a tendency to prefer middle or extremes of the scale regardless of the content of the item (response styles) and purely content-driven preferences. Explanatory variables are used to characterize the content-driven way of answering as well as the tendency to middle or extreme categories. The proposed model is extended to account for the presence of response styles in the case of several items, and the association among responses is described, both when they are content driven or dictated by response styles. In addition, stochastic orderings, related to the tendency to select middle or extreme categories, are introduced and investigated. A simulation study describes the effectiveness of the proposed model, and an application to a questionnaire on attitudes toward ethnic minorities illustrates the applicability of the modeling approach.","PeriodicalId":48001,"journal":{"name":"Journal of Educational and Behavioral Statistics","volume":"46 1","pages":"682 - 716"},"PeriodicalIF":2.4,"publicationDate":"2021-03-31","publicationTypes":"Journal Article","fieldsOfStudy":null,"isOpenAccess":false,"openAccessPdf":"","citationCount":null,"resultStr":null,"platform":"Semanticscholar","paperid":"45838221","PeriodicalName":null,"FirstCategoryId":null,"ListUrlMain":null,"RegionNum":3,"RegionCategory":"心理学","ArticlePicture":[],"TitleCN":null,"AbstractTextCN":null,"PMCID":"","EPubDate":null,"PubModel":null,"JCR":null,"JCRName":null,"Score":null,"Total":0}

{"title":"Detecting Noneffortful Responses Based on a Residual Method Using an Iterative Purification Process","authors":"Yue Liu, Hongyun Liu","doi":"10.3102/1076998621994366","DOIUrl":"https://doi.org/10.3102/1076998621994366","url":null,"abstract":"The prevalence and serious consequences of noneffortful responses from unmotivated examinees are well-known in educational measurement. In this study, we propose to apply an iterative purification process based on a response time residual method with fixed item parameter estimates to detect noneffortful responses. The proposed method is compared with the traditional residual method and noniterative method with fixed item parameters in two simulation studies in terms of noneffort detection accuracy and parameter recovery. The results show that when severity of noneffort is high, the proposed method leads to a much higher true positive rate with a small increase of false discovery rate. In addition, parameter estimation is significantly improved by the strategies of fixing item parameters and iteratively cleansing. These results suggest that the proposed method is a potential solution to reduce the impact of data contamination due to severe low test-taking effort and to obtain more accurate parameter estimates. An empirical study is also conducted to show the differences in the detection rate and parameter estimates among different approaches.","PeriodicalId":48001,"journal":{"name":"Journal of Educational and Behavioral Statistics","volume":"46 1","pages":"717 - 752"},"PeriodicalIF":2.4,"publicationDate":"2021-03-29","publicationTypes":"Journal Article","fieldsOfStudy":null,"isOpenAccess":false,"openAccessPdf":"","citationCount":null,"resultStr":null,"platform":"Semanticscholar","paperid":"44643360","PeriodicalName":null,"FirstCategoryId":null,"ListUrlMain":null,"RegionNum":3,"RegionCategory":"心理学","ArticlePicture":[],"TitleCN":null,"AbstractTextCN":null,"PMCID":"","EPubDate":null,"PubModel":null,"JCR":null,"JCRName":null,"Score":null,"Total":0}

{"title":"Item Characteristic Curve Asymmetry: A Better Way to Accommodate Slips and Guesses Than a Four-Parameter Model?","authors":"Xiangyi Liao, D. Bolt","doi":"10.3102/10769986211003283","DOIUrl":"https://doi.org/10.3102/10769986211003283","url":null,"abstract":"Four-parameter models have received increasing psychometric attention in recent years, as a reduced upper asymptote for item characteristic curves can be appealing for measurement applications such as adaptive testing and person-fit assessment. However, applications can be challenging due to the large number of parameters in the model. In this article, we demonstrate in the context of mathematics assessments how the slip and guess parameters of a four-parameter model may often be empirically related. This observation also has a psychological explanation to the extent that both asymptote parameters may be manifestations of a single item complexity characteristic. The relationship between lower and upper asymptotes motivates the consideration of an asymmetric item response theory model as a three-parameter alternative to the four-parameter model. Using actual response data from mathematics multiple-choice tests, we demonstrate the empirical superiority of a three-parameter asymmetric model in several standardized tests of mathematics. To the extent that a model of asymmetry ultimately portrays slips and guesses not as purely random but rather as proficiency-related phenomena, we argue that the asymmetric approach may also have greater psychological plausibility.","PeriodicalId":48001,"journal":{"name":"Journal of Educational and Behavioral Statistics","volume":"46 1","pages":"753 - 775"},"PeriodicalIF":2.4,"publicationDate":"2021-03-29","publicationTypes":"Journal Article","fieldsOfStudy":null,"isOpenAccess":false,"openAccessPdf":"","citationCount":null,"resultStr":null,"platform":"Semanticscholar","paperid":"49036230","PeriodicalName":null,"FirstCategoryId":null,"ListUrlMain":null,"RegionNum":3,"RegionCategory":"心理学","ArticlePicture":[],"TitleCN":null,"AbstractTextCN":null,"PMCID":"","EPubDate":null,"PubModel":null,"JCR":null,"JCRName":null,"Score":null,"Total":0}

{"title":"Monitoring Item Performance With CUSUM Statistics in Continuous Testing","authors":"Yi-Hsuan Lee, C. Lewis","doi":"10.3102/1076998621994563","DOIUrl":"https://doi.org/10.3102/1076998621994563","url":null,"abstract":"In many educational assessments, items are reused in different administrations throughout the life of the assessments. Ideally, a reused item should perform relatively similarly over time. In reality, an item may become easier with exposure, especially when item preknowledge has occurred. This article presents a novel cumulative sum procedure for detecting item preknowledge in continuous testing where data for each reused item may be obtained from small and varying sample sizes across administrations. Its performance is evaluated with simulations and analytical work. The approach is effective in detecting item preknowledge quickly with group size at least 10 and is easy to implement with varying item parameters. In addition, it is robust to the ability estimation error introduced in the simulations.","PeriodicalId":48001,"journal":{"name":"Journal of Educational and Behavioral Statistics","volume":"46 1","pages":"611 - 648"},"PeriodicalIF":2.4,"publicationDate":"2021-03-08","publicationTypes":"Journal Article","fieldsOfStudy":null,"isOpenAccess":false,"openAccessPdf":"","citationCount":null,"resultStr":null,"platform":"Semanticscholar","paperid":"48504334","PeriodicalName":null,"FirstCategoryId":null,"ListUrlMain":null,"RegionNum":3,"RegionCategory":"心理学","ArticlePicture":[],"TitleCN":null,"AbstractTextCN":null,"PMCID":"","EPubDate":null,"PubModel":null,"JCR":null,"JCRName":null,"Score":null,"Total":0}

{"title":"Jenss–Bayley Latent Change Score Model With Individual Ratio of the Growth Acceleration in the Framework of Individual Measurement Occasions","authors":"Jin Liu","doi":"10.3102/10769986221099919","DOIUrl":"https://doi.org/10.3102/10769986221099919","url":null,"abstract":"Longitudinal data analysis has been widely employed to examine between-individual differences in within-individual changes. One challenge of such analyses is that the rate-of-change is only available indirectly when change patterns are nonlinear with respect to time. Latent change score models (LCSMs), which can be employed to investigate the change in rate-of-change at the individual level, have been developed to address this challenge. We extend an existing LCSM with the Jenss–Bayley growth curve and propose a novel expression for change scores that allows for (1) unequally spaced study waves and (2) individual measurement occasions around each wave. We also extend the existing model to estimate the individual ratio of the growth acceleration (that largely determines the trajectory shape and is viewed as the most important parameter in the Jenss–Bayley model). We present the proposed model by a simulation study and a real-world data analysis. Our simulation study demonstrates that the proposed model can estimate the parameters unbiasedly and precisely and exhibit target confidence interval coverage. The simulation study also shows that the proposed model with the novel expression for the change scores outperforms the existing model. An empirical example using longitudinal reading scores shows that the model can estimate the individual ratio of the growth acceleration and generate individual rate-of-change in practice. We also provide the corresponding code for the proposed model.","PeriodicalId":48001,"journal":{"name":"Journal of Educational and Behavioral Statistics","volume":"47 1","pages":"507 - 543"},"PeriodicalIF":2.4,"publicationDate":"2021-02-27","publicationTypes":"Journal Article","fieldsOfStudy":null,"isOpenAccess":false,"openAccessPdf":"","citationCount":null,"resultStr":null,"platform":"Semanticscholar","paperid":"47673583","PeriodicalName":null,"FirstCategoryId":null,"ListUrlMain":null,"RegionNum":3,"RegionCategory":"心理学","ArticlePicture":[],"TitleCN":null,"AbstractTextCN":null,"PMCID":"","EPubDate":null,"PubModel":null,"JCR":null,"JCRName":null,"Score":null,"Total":0}

{"title":"Estimating Difference-Score Reliability in Pretest–Posttest Settings","authors":"Zhengguo Gu, W. Emons, K. Sijtsma","doi":"10.3102/1076998620986948","DOIUrl":"https://doi.org/10.3102/1076998620986948","url":null,"abstract":"Clinical, medical, and health psychologists use difference scores obtained from pretest–posttest designs employing the same test to assess intraindividual change possibly caused by an intervention addressing, for example, anxiety, depression, eating disorder, or addiction. Reliability of difference scores is important for interpreting observed change. This article compares the well-documented traditional method and the unfamiliar, rarely used item-level method for estimating difference-score reliability. We simulated data under various conditions that are typical of change assessment in pretest–posttest designs. The item-level method had smaller bias and greater precision than the traditional method and may be recommended for practical use.","PeriodicalId":48001,"journal":{"name":"Journal of Educational and Behavioral Statistics","volume":"46 1","pages":"592 - 610"},"PeriodicalIF":2.4,"publicationDate":"2021-02-15","publicationTypes":"Journal Article","fieldsOfStudy":null,"isOpenAccess":false,"openAccessPdf":"","citationCount":null,"resultStr":null,"platform":"Semanticscholar","paperid":"46183979","PeriodicalName":null,"FirstCategoryId":null,"ListUrlMain":null,"RegionNum":3,"RegionCategory":"心理学","ArticlePicture":[],"TitleCN":null,"AbstractTextCN":null,"PMCID":"","EPubDate":null,"PubModel":null,"JCR":null,"JCRName":null,"Score":null,"Total":0}

{"title":"Statistical Power for Estimating Treatment Effects Using Difference-in-Differences and Comparative Interrupted Time Series Estimators With Variation in Treatment Timing","authors":"Peter Z. Schochet","doi":"10.3102/10769986211070625","DOIUrl":"https://doi.org/10.3102/10769986211070625","url":null,"abstract":"This article develops new closed-form variance expressions for power analyses for commonly used difference-in-differences (DID) and comparative interrupted time series (CITS) panel data estimators. The main contribution is to incorporate variation in treatment timing into the analysis. The power formulas also account for other key design features that arise in practice: autocorrelated errors, unequal measurement intervals, and clustering due to the unit of treatment assignment. We consider power formulas for both cross-sectional and longitudinal models and allow for covariates. An illustrative power analysis provides guidance on appropriate sample sizes. The key finding is that accounting for treatment timing increases required sample sizes. Further, DID estimators have considerably more power than standard CITS and ITS estimators. An available Shiny R dashboard performs the sample size calculations for the considered estimators.","PeriodicalId":48001,"journal":{"name":"Journal of Educational and Behavioral Statistics","volume":"47 1","pages":"367 - 405"},"PeriodicalIF":2.4,"publicationDate":"2021-02-12","publicationTypes":"Journal Article","fieldsOfStudy":null,"isOpenAccess":false,"openAccessPdf":"","citationCount":null,"resultStr":null,"platform":"Semanticscholar","paperid":"41400358","PeriodicalName":null,"FirstCategoryId":null,"ListUrlMain":null,"RegionNum":3,"RegionCategory":"心理学","ArticlePicture":[],"TitleCN":null,"AbstractTextCN":null,"PMCID":"","EPubDate":null,"PubModel":null,"JCR":null,"JCRName":null,"Score":null,"Total":0}