Divya Varier, Marvin G. Powell, Stephanie Dodman, Samantha T. Ives, Elizabeth DeMulder, Jenice L. View
{"title":"Use and Usefulness of Assessments to Inform Instruction: Developing a K-12 Classroom Teacher Assessment Practice Measure","authors":"Divya Varier, Marvin G. Powell, Stephanie Dodman, Samantha T. Ives, Elizabeth DeMulder, Jenice L. View","doi":"10.1080/10627197.2024.2316907","DOIUrl":"https://doi.org/10.1080/10627197.2024.2316907","url":null,"abstract":"Considerable literature is devoted to teachers’ assessment use to support teaching and learning. The study examined the factor structure of a measure of teachers’ assessment use along the assessmen...","PeriodicalId":46209,"journal":{"name":"Educational Assessment","volume":null,"pages":null},"PeriodicalIF":1.5,"publicationDate":"2024-02-13","publicationTypes":"Journal Article","fieldsOfStudy":null,"isOpenAccess":false,"openAccessPdf":"","citationCount":null,"resultStr":null,"platform":"Semanticscholar","paperid":"139762025","PeriodicalName":null,"FirstCategoryId":null,"ListUrlMain":null,"RegionNum":0,"RegionCategory":"","ArticlePicture":[],"TitleCN":null,"AbstractTextCN":null,"PMCID":"","EPubDate":null,"PubModel":null,"JCR":null,"JCRName":null,"Score":null,"Total":0}
Jennifer Randall, Mya Poe, Maria Elena Oliveri, David Slomp
{"title":"Justice-Oriented, Antiracist Validation: Continuing to Disrupt White Supremacy in Assessment Practices","authors":"Jennifer Randall, Mya Poe, Maria Elena Oliveri, David Slomp","doi":"10.1080/10627197.2023.2285047","DOIUrl":"https://doi.org/10.1080/10627197.2023.2285047","url":null,"abstract":"Traditional validation approaches fail to account for the ways oppressive systems (e.g. racism, radical nationalism) impact the test design and development process. To disrupt this legacy of white ...","PeriodicalId":46209,"journal":{"name":"Educational Assessment","volume":null,"pages":null},"PeriodicalIF":1.5,"publicationDate":"2023-11-22","publicationTypes":"Journal Article","fieldsOfStudy":null,"isOpenAccess":false,"openAccessPdf":"","citationCount":null,"resultStr":null,"platform":"Semanticscholar","paperid":"138516536","PeriodicalName":null,"FirstCategoryId":null,"ListUrlMain":null,"RegionNum":0,"RegionCategory":"","ArticlePicture":[],"TitleCN":null,"AbstractTextCN":null,"PMCID":"","EPubDate":null,"PubModel":null,"JCR":null,"JCRName":null,"Score":null,"Total":0}
{"title":"Measuring Item Influence for Diagnostic Classification Models","authors":"D. Jurich, Matthew J. Madison","doi":"10.1080/10627197.2023.2244411","DOIUrl":"https://doi.org/10.1080/10627197.2023.2244411","url":null,"abstract":"ABSTRACT Diagnostic classification models (DCMs) are psychometric models that provide probabilistic classifications of examinees on a set of discrete latent attributes. When analyzing or constructing assessments scored by DCMs, understanding how each item influences attribute classifications can clarify the meaning of the measured constructs, facilitate appropriate construct representation, and identify items contributing minimal utility. In cases of short assessments, common in the DCM literature, item influence becomes paramount as individual items can have a disproportionate impact on, or entirely determine, classification. This study proposes four indices to quantify item influence and distinguishes them from other available item and test measures. We use simulation methods to evaluate and provide guidelines for interpreting each index, followed by a real data application to illustrate their use in practice. We discuss theoretical considerations regarding when influence presents a psychometric concern and other practical concerns such as how the indices function when reducing influence imbalance.","PeriodicalId":46209,"journal":{"name":"Educational Assessment","volume":null,"pages":null},"PeriodicalIF":1.5,"publicationDate":"2023-08-14","publicationTypes":"Journal Article","fieldsOfStudy":null,"isOpenAccess":false,"openAccessPdf":"","citationCount":null,"resultStr":null,"platform":"Semanticscholar","paperid":"42614213","PeriodicalName":null,"FirstCategoryId":null,"ListUrlMain":null,"RegionNum":0,"RegionCategory":"","ArticlePicture":[],"TitleCN":null,"AbstractTextCN":null,"PMCID":"","EPubDate":null,"PubModel":null,"JCR":null,"JCRName":null,"Score":null,"Total":0}
{"title":"English Learners and Constructed-Response Science Test Items Challenges and Opportunities","authors":"T. Noble, C. Wells, Ann Rosebery","doi":"10.1080/10627197.2023.2226387","DOIUrl":"https://doi.org/10.1080/10627197.2023.2226387","url":null,"abstract":"ABSTRACT This article reports on two quantitative studies of English learners’ (ELs) interactions with constructed-response items from a Grade 5 state science test. Study 1 investigated the relationships between the constructed-response item-level variables of English Reading Demand, English Writing Demand, and Background Knowledge Demand and the performance of ELs vs. non-ELs on those items. English Writing Demand was the strongest predictor of Differential Item Functioning favoring non-ELs over ELs for constructed-response items. In Study 2, we investigated the student-level variable of English language proficiency level and found that lower English language proficiency was related to greatly increased odds of omitting a response to a constructed-response item, even when controlling for science proficiency. These findings challenge the validity of scores on constructed-response test items as measures of ELs’ science proficiency.","PeriodicalId":46209,"journal":{"name":"Educational Assessment","volume":null,"pages":null},"PeriodicalIF":1.5,"publicationDate":"2023-07-07","publicationTypes":"Journal Article","fieldsOfStudy":null,"isOpenAccess":false,"openAccessPdf":"","citationCount":null,"resultStr":null,"platform":"Semanticscholar","paperid":"47443213","PeriodicalName":null,"FirstCategoryId":null,"ListUrlMain":null,"RegionNum":0,"RegionCategory":"","ArticlePicture":[],"TitleCN":null,"AbstractTextCN":null,"PMCID":"","EPubDate":null,"PubModel":null,"JCR":null,"JCRName":null,"Score":null,"Total":0}
{"title":"Examining the Question-Behavior Effect in Low-Stakes Testing Contexts: A Cheap Strategy to Increase Examinee Effort","authors":"S. Finney, M. McFadden","doi":"10.1080/10627197.2023.2222588","DOIUrl":"https://doi.org/10.1080/10627197.2023.2222588","url":null,"abstract":"ABSTRACT Questioning individuals about future behavior influences the subsequent performance of that behavior. This question-behavior effect may be enhanced when questions include positive self-identity prompts. To evaluate the effectiveness of this intervention in a testing context, we randomly assigned examinees to one of three conditions prior to completing a low-stakes test: answering five questions regarding intended effort prior to the test, answering five questions regarding intended effort that referenced positive self-identity prior to the test, or a control condition. We then administered two multiple-choice tests and collected two measures of test-taking effort: self-reported effort and response time effort. Answering questions about effort prior to completing the tests resulted in higher self-reported effort and perceived test importance. Moreover, in the question conditions, significantly fewer examinees were identified to be filtered from the dataset due to low effort. The easy strategy of asking examinees to report their intended effort appears to increase test-taking effort.","PeriodicalId":46209,"journal":{"name":"Educational Assessment","volume":null,"pages":null},"PeriodicalIF":1.5,"publicationDate":"2023-06-19","publicationTypes":"Journal Article","fieldsOfStudy":null,"isOpenAccess":false,"openAccessPdf":"","citationCount":null,"resultStr":null,"platform":"Semanticscholar","paperid":"46037789","PeriodicalName":null,"FirstCategoryId":null,"ListUrlMain":null,"RegionNum":0,"RegionCategory":"","ArticlePicture":[],"TitleCN":null,"AbstractTextCN":null,"PMCID":"","EPubDate":null,"PubModel":null,"JCR":null,"JCRName":null,"Score":null,"Total":0}
{"title":"Identifying Problem-Solving Solution Patterns Using Network Analysis of Operation Sequences and Response Times","authors":"Maoxin Zhang, Björn Andersson","doi":"10.1080/10627197.2023.2222585","DOIUrl":"https://doi.org/10.1080/10627197.2023.2222585","url":null,"abstract":"ABSTRACT Process data from educational assessments enhance the understanding of how students answer cognitive items. However, effectively making use of these data is challenging. We propose an approach to identify solution patterns from operation sequences and response times by generating networks from process data and defining network features that extract essential information from them. With these features, we group respondents to a problem-solving task from PISA 2012 using Gaussian mixture models. The results indicate the presence of two and four clusters for groups defined by failure and success on the task, respectively. We interpret the clusters as less-able, low-effort, adaptable, back-and-forth, deliberate, and trial-and-error clusters by considering the cluster-specific feature statistics. The proposed approach sheds light on students’ problem-solving mental processes, which can aid item development and facilitate individualized feedback to students. The method is applicable to many computer-based problems, but a limitation is that the feature definitions can be task-dependent.","PeriodicalId":46209,"journal":{"name":"Educational Assessment","volume":null,"pages":null},"PeriodicalIF":1.5,"publicationDate":"2023-06-16","publicationTypes":"Journal Article","fieldsOfStudy":null,"isOpenAccess":false,"openAccessPdf":"","citationCount":null,"resultStr":null,"platform":"Semanticscholar","paperid":"41526853","PeriodicalName":null,"FirstCategoryId":null,"ListUrlMain":null,"RegionNum":0,"RegionCategory":"","ArticlePicture":[],"TitleCN":null,"AbstractTextCN":null,"PMCID":"","EPubDate":null,"PubModel":null,"JCR":null,"JCRName":null,"Score":null,"Total":0}
{"title":"Public Opinion About National Large-Scale Student Assessment: A Case of NAPLAN","authors":"Jihyun Lee, Jung‐Sook Lee, N. Ellis","doi":"10.1080/10627197.2023.2215977","DOIUrl":"https://doi.org/10.1080/10627197.2023.2215977","url":null,"abstract":"ABSTRACT This study examines public opinion about the National Assessment Program – Literacy and Numeracy (NAPLAN) in Australia, based on an online survey conducted with a sample of New South Wales (NSW) residents (n = 2,017). Our study participants did recognize the benefits of NAPLAN despite their opinion generally trending toward negative viewpoints of the testing program. The views about the positive aspects of NAPLAN varied across socio-demographic groups, but there were more similar views concerning the negative views of NAPLAN. As predicted by the social-cultural and institutional performance models, those living in metropolitan areas and those from high SES groups tended to possess stronger positive views. Political party preference was a strong predictor of the perceptions about NAPLAN. Overall, this study offers possible explanations for the underlying mechanisms explaining sub-group differences in attitudes toward large-scale standardized national testing.","PeriodicalId":46209,"journal":{"name":"Educational Assessment","volume":null,"pages":null},"PeriodicalIF":1.5,"publicationDate":"2023-05-21","publicationTypes":"Journal Article","fieldsOfStudy":null,"isOpenAccess":false,"openAccessPdf":"","citationCount":null,"resultStr":null,"platform":"Semanticscholar","paperid":"41772940","PeriodicalName":null,"FirstCategoryId":null,"ListUrlMain":null,"RegionNum":0,"RegionCategory":"","ArticlePicture":[],"TitleCN":null,"AbstractTextCN":null,"PMCID":"","EPubDate":null,"PubModel":null,"JCR":null,"JCRName":null,"Score":null,"Total":0}
{"title":"A Comparison of Methodologies for Scaling Longitudinal Social-Emotional Survey Responses","authors":"J. Soland, Megan Kuhfeld, Brennan Register","doi":"10.1080/10627197.2023.2213432","DOIUrl":"https://doi.org/10.1080/10627197.2023.2213432","url":null,"abstract":"ABSTRACT Much of what we know about how children develop is based on survey data. In order to estimate growth across time and, thereby, better understand that development, short survey scales are typically administered at repeated timepoints. Before estimating growth, those repeated measures must be put onto the same scale. Yet, little research examines how scaling decisions affect comparisons of growth derived from survey item responses. In this study, we use a sample of 174,669 students in grades 7 through 12 who took the same self-efficacy and social awareness surveys for four years. We use those survey item responses to construct scales using different approaches, then compare the resultant scores to see how inferences about changes over time during adolescence might shift dependent on scaling. While we find that conclusions about average trends are quite consistent by scaling approach, specific quantifications of change like effect sizes can differ by scaling method.","PeriodicalId":46209,"journal":{"name":"Educational Assessment","volume":null,"pages":null},"PeriodicalIF":1.5,"publicationDate":"2023-05-18","publicationTypes":"Journal Article","fieldsOfStudy":null,"isOpenAccess":false,"openAccessPdf":"","citationCount":null,"resultStr":null,"platform":"Semanticscholar","paperid":"46872564","PeriodicalName":null,"FirstCategoryId":null,"ListUrlMain":null,"RegionNum":0,"RegionCategory":"","ArticlePicture":[],"TitleCN":null,"AbstractTextCN":null,"PMCID":"","EPubDate":null,"PubModel":null,"JCR":null,"JCRName":null,"Score":null,"Total":0}
{"title":"Assessing Mathematical Higher-Order Thinking Skills: An Analysis of Turkish University Entrance Examinations","authors":"Utkun Aydın, B. Birgili","doi":"10.1080/10627197.2023.2202311","DOIUrl":"https://doi.org/10.1080/10627197.2023.2202311","url":null,"abstract":"ABSTRACT Internationally, mathematics education reform has been directed toward characterizing educational goals that go beyond topic/content/skill descriptions and develop students’ problem solving. The Revised Bloom’s Taxonomy and MATH (Mathematical Assessment Task Hierarchy) Taxonomy characterize such goals. University entrance examinations have been seen as one way of accomplishing these goals and influence learning, teaching, and assessment in mathematics. The present study analyzed mathematics items (N = 1077) in Turkish university entrance examinations in 1998-2013 and objectives (N = 621) in mathematics curricula in 2005, 2011, and 2013 to determine the extent to which they represent the dimensions/categories of these taxonomies and the degree to which items are aligned with objectives in terms of reflecting the dimensions/categories of these taxonomies. The findings reveal that the items demand, to a large extent, automated computational skills; this is also evident in the relevant mathematics curricula. Implications for practice are discussed and could play a role in reforming assessment.","PeriodicalId":46209,"journal":{"name":"Educational Assessment","volume":null,"pages":null},"PeriodicalIF":1.5,"publicationDate":"2023-04-17","publicationTypes":"Journal Article","fieldsOfStudy":null,"isOpenAccess":false,"openAccessPdf":"","citationCount":null,"resultStr":null,"platform":"Semanticscholar","paperid":"48128926","PeriodicalName":null,"FirstCategoryId":null,"ListUrlMain":null,"RegionNum":0,"RegionCategory":"","ArticlePicture":[],"TitleCN":null,"AbstractTextCN":null,"PMCID":"","EPubDate":null,"PubModel":null,"JCR":null,"JCRName":null,"Score":null,"Total":0}
{"title":"It Ain’t Near ‘Bout Fair: Re-Envisioning the Bias and Sensitivity Review Process from a Justice-Oriented Antiracist Perspective","authors":"Jennifer Randall","doi":"10.1080/10627197.2023.2223924","DOIUrl":"https://doi.org/10.1080/10627197.2023.2223924","url":null,"abstract":"ABSTRACT In a justice-oriented antiracist assessment process, attention to the disruption of white supremacy must occur at every stage – from construct articulation to score reporting. An important step in the assessment development process is the item review stage often referred to as Bias/Fairness and Sensitivity Review. I argue that typical approaches to the item and test review process miss the opportunity to actively disrupt white supremacist and racist logics – in other words, to be anti-racist. Using Critical Race and Critical Whiteness Theory as a frame, this paper challenges the field to re-envision the purpose and outcomes of the bias and sensitivity review process by (a) identifying common themes and/or recommendations found in bias and sensitivity guidelines that, even if unintentionally, center whiteness and/or the paradigm of white dominant culture; and (b) recommending a set of bias and sensitivity principles that promote an antiracist approach to assessment design, specifically item review.","PeriodicalId":46209,"journal":{"name":"Educational Assessment","volume":null,"pages":null},"PeriodicalIF":1.5,"publicationDate":"2023-04-03","publicationTypes":"Journal Article","fieldsOfStudy":null,"isOpenAccess":false,"openAccessPdf":"","citationCount":null,"resultStr":null,"platform":"Semanticscholar","paperid":"48704460","PeriodicalName":null,"FirstCategoryId":null,"ListUrlMain":null,"RegionNum":0,"RegionCategory":"","ArticlePicture":[],"TitleCN":null,"AbstractTextCN":null,"PMCID":"","EPubDate":null,"PubModel":null,"JCR":null,"JCRName":null,"Score":null,"Total":0}