Sabine Meinck, Jörg-Henrik Heine, Julia Mang, Gabriel Nagy
{"title":"Bias risks in ILSA related to non-participation: evidence from a longitudinal large-scale survey in Germany (PISA Plus)","authors":"Sabine Meinck, Jörg-Henrik Heine, Julia Mang, Gabriel Nagy","doi":"10.1007/s11092-023-09422-5","DOIUrl":"https://doi.org/10.1007/s11092-023-09422-5","url":null,"abstract":"<p>This study uses evidence from a longitudinal survey (PISA Plus, Germany) to examine the potential of bias in international large-scale assessments (ILSAs). In PISA Plus, participation was mandatory at the first measurement point, but voluntary at the second measurement point. The study provides evidence for relevant selection bias regarding student competencies and background variables when participation is voluntary. Sample dropout at the second measurement point was related to characteristics such as family background, achievement in mathematics, reading and science, and other student and school demographic variables at both the student and school levels, with lower performing students and those with less favorable background characteristics having higher dropout frequencies, from which higher dropout probabilities of such students can be inferred. We further contrast the possibilities for addressing non-response through weight adjustments in longitudinal surveys with those in cross-sectional surveys. Considering our results, we evaluate and confirm the validity and appropriateness of strict participation rate requirements in ILSAs. Likely magnitudes of bias in cross-sectional studies in varying scenarios are illustrated. Accordingly, if combined participation rates drop below 70%, a difference of at least one-fifth of a standard deviation in an achievement score between non-respondents and participants leads to relevant bias. When participation drops below 50%, even a very small difference (one-tenth of a standard deviation) will cause non-negligible bias. Finally, we conclude that the stringent participation rate requirements established in most ILSAs are fully valid, reasonable, and important since they ensure a relatively low risk of biased results.</p>","PeriodicalId":46725,"journal":{"name":"Educational Assessment Evaluation and Accountability","volume":"26 1","pages":""},"PeriodicalIF":3.9,"publicationDate":"2023-12-06","publicationTypes":"Journal Article","fieldsOfStudy":null,"isOpenAccess":false,"openAccessPdf":"","citationCount":null,"resultStr":null,"platform":"Semanticscholar","paperid":"138545528","PeriodicalName":null,"FirstCategoryId":null,"ListUrlMain":null,"RegionNum":3,"RegionCategory":"教育学","ArticlePicture":[],"TitleCN":null,"AbstractTextCN":null,"PMCID":"","EPubDate":null,"PubModel":null,"JCR":null,"JCRName":null,"Score":null,"Total":0}
{"title":"Principals’ attitudes towards on-site school evaluations: contribution to sustained school reform","authors":"Pablo Bezem","doi":"10.1007/s11092-023-09418-1","DOIUrl":"https://doi.org/10.1007/s11092-023-09418-1","url":null,"abstract":"<p>Test-based accountability offers schools limited guidance on informing sustained reforms, whereas on-site school evaluations provide broader and more nuanced evidence. In order to transform inspection feedback into long-term reforms, school leaders’ attitudes towards these evaluations are critical. Relying on organizational change theory, this study examines principals’ beliefs regarding school inspection feedback effectiveness and appropriateness as well as their readiness to implement changes. A large US district that has experimented with such school inspection serves as the case study. The results demonstrate the principals’ strong positive attitudes towards inspection, and these attitudes are associated with sustained change. But, because the accountability framework did not center on inspection, principals were ambivalent about implementing specific changes. This study highlights the tensions involved in incorporating on-site, expert evaluations in test-based accountability systems.</p>","PeriodicalId":46725,"journal":{"name":"Educational Assessment Evaluation and Accountability","volume":"1 1","pages":""},"PeriodicalIF":3.9,"publicationDate":"2023-11-22","publicationTypes":"Journal Article","fieldsOfStudy":null,"isOpenAccess":false,"openAccessPdf":"","citationCount":null,"resultStr":null,"platform":"Semanticscholar","paperid":"138516348","PeriodicalName":null,"FirstCategoryId":null,"ListUrlMain":null,"RegionNum":3,"RegionCategory":"教育学","ArticlePicture":[],"TitleCN":null,"AbstractTextCN":null,"PMCID":"","EPubDate":null,"PubModel":null,"JCR":null,"JCRName":null,"Score":null,"Total":0}
Luis Alejandro Lopez-Agudo, Claudia Prieto Latorre, Oscar David Marcenaro-Gutierrez
{"title":"Grade retention in Spain: the right way?","authors":"Luis Alejandro Lopez-Agudo, Claudia Prieto Latorre, Oscar David Marcenaro-Gutierrez","doi":"10.1007/s11092-023-09421-6","DOIUrl":"https://doi.org/10.1007/s11092-023-09421-6","url":null,"abstract":"","PeriodicalId":46725,"journal":{"name":"Educational Assessment Evaluation and Accountability","volume":"59 12","pages":"0"},"PeriodicalIF":0.0,"publicationDate":"2023-10-23","publicationTypes":"Journal Article","fieldsOfStudy":null,"isOpenAccess":false,"openAccessPdf":"","citationCount":null,"resultStr":null,"platform":"Semanticscholar","paperid":"135365518","PeriodicalName":null,"FirstCategoryId":null,"ListUrlMain":null,"RegionNum":3,"RegionCategory":"教育学","ArticlePicture":[],"TitleCN":null,"AbstractTextCN":null,"PMCID":"","EPubDate":null,"PubModel":null,"JCR":null,"JCRName":null,"Score":null,"Total":0}
Cherry Zin Oo, Dennis Alonzo, Ria Asih, Giovanni Pelobillo, Rex Lim, Nang Mo Hline San, Sue O’Neill
{"title":"Implementing school-based assessment reforms to enhance student learning: a systematic review","authors":"Cherry Zin Oo, Dennis Alonzo, Ria Asih, Giovanni Pelobillo, Rex Lim, Nang Mo Hline San, Sue O’Neill","doi":"10.1007/s11092-023-09420-7","DOIUrl":"https://doi.org/10.1007/s11092-023-09420-7","url":null,"abstract":"","PeriodicalId":46725,"journal":{"name":"Educational Assessment Evaluation and Accountability","volume":"256 1","pages":"0"},"PeriodicalIF":0.0,"publicationDate":"2023-10-18","publicationTypes":"Journal Article","fieldsOfStudy":null,"isOpenAccess":false,"openAccessPdf":"","citationCount":null,"resultStr":null,"platform":"Semanticscholar","paperid":"135888592","PeriodicalName":null,"FirstCategoryId":null,"ListUrlMain":null,"RegionNum":3,"RegionCategory":"教育学","ArticlePicture":[],"TitleCN":null,"AbstractTextCN":null,"PMCID":"","EPubDate":null,"PubModel":null,"JCR":null,"JCRName":null,"Score":null,"Total":0}
{"title":"Accountability in the evaluation of teacher effectiveness: views of teachers and administrators","authors":"Xumei Fan","doi":"10.1007/s11092-023-09419-0","DOIUrl":"https://doi.org/10.1007/s11092-023-09419-0","url":null,"abstract":"","PeriodicalId":46725,"journal":{"name":"Educational Assessment Evaluation and Accountability","volume":"45 1","pages":"0"},"PeriodicalIF":0.0,"publicationDate":"2023-10-12","publicationTypes":"Journal Article","fieldsOfStudy":null,"isOpenAccess":false,"openAccessPdf":"","citationCount":null,"resultStr":null,"platform":"Semanticscholar","paperid":"136014381","PeriodicalName":null,"FirstCategoryId":null,"ListUrlMain":null,"RegionNum":3,"RegionCategory":"教育学","ArticlePicture":[],"TitleCN":null,"AbstractTextCN":null,"PMCID":"","EPubDate":null,"PubModel":null,"JCR":null,"JCRName":null,"Score":null,"Total":0}
{"title":"School closure policies and student reading achievement: evidence across countries","authors":"Alec I. Kennedy, Rolf Strietholt","doi":"10.1007/s11092-023-09415-4","DOIUrl":"https://doi.org/10.1007/s11092-023-09415-4","url":null,"abstract":"Abstract The COVID-19 pandemic disrupted education worldwide as educational systems made the decision to close schools to contain the spread of the virus. The duration of school closures varied greatly internationally. In this study, we use international variation in school closure policies to examine the effects of school closures on student achievement. Specifically, we use representative trend data from more than 300,000 students in 29 countries to examine whether the length of school closures is related to changes in student achievement before and after the outbreak of COVID-19. We observe a significant and substantial negative effect of school closures on student reading achievement. This school closure effect remains even after controlling for measures of pandemic severity such as infection rates, vaccination policies, and a measure of lockdown stringency. The estimated effect implies that a year of school closures corresponds roughly to the loss of a little more than half a school year of learning. This effect is even more pronounced for socioeconomically disadvantaged students and those without home computer access.","PeriodicalId":46725,"journal":{"name":"Educational Assessment Evaluation and Accountability","volume":"21 1","pages":"0"},"PeriodicalIF":0.0,"publicationDate":"2023-09-30","publicationTypes":"Journal Article","fieldsOfStudy":null,"isOpenAccess":false,"openAccessPdf":"","citationCount":null,"resultStr":null,"platform":"Semanticscholar","paperid":"136341938","PeriodicalName":null,"FirstCategoryId":null,"ListUrlMain":null,"RegionNum":3,"RegionCategory":"教育学","ArticlePicture":[],"TitleCN":null,"AbstractTextCN":null,"PMCID":"","EPubDate":null,"PubModel":null,"JCR":null,"JCRName":null,"Score":null,"Total":0}
{"title":"Correction to: Principals may inflate teacher evaluation scores to achieve important goals","authors":"Eli Jones, Christi Bergin, Bridget Murphy","doi":"10.1007/s11092-023-09417-2","DOIUrl":"https://doi.org/10.1007/s11092-023-09417-2","url":null,"abstract":"","PeriodicalId":46725,"journal":{"name":"Educational Assessment Evaluation and Accountability","volume":"137 1","pages":"0"},"PeriodicalIF":0.0,"publicationDate":"2023-09-27","publicationTypes":"Journal Article","fieldsOfStudy":null,"isOpenAccess":false,"openAccessPdf":"","citationCount":null,"resultStr":null,"platform":"Semanticscholar","paperid":"135537280","PeriodicalName":null,"FirstCategoryId":null,"ListUrlMain":null,"RegionNum":3,"RegionCategory":"教育学","ArticlePicture":[],"TitleCN":null,"AbstractTextCN":null,"PMCID":"","EPubDate":null,"PubModel":null,"JCR":null,"JCRName":null,"Score":null,"Total":0}
{"title":"Exclusion rates from international large-scale assessments: an analysis of 20 years of IEA data","authors":"Umut Atasever, John Jerrim, Sabine Tieck","doi":"10.1007/s11092-023-09416-3","DOIUrl":"https://doi.org/10.1007/s11092-023-09416-3","url":null,"abstract":"Abstract Cross-national comparisons of educational achievement rely upon each participating country collecting nationally representative data. While obtaining high response rates is a key part of reaching this goal, other potentially important factors may also be at play. This paper focuses on one such issue—exclusion rates—which has received relatively little attention in the academic literature. Using data from 20 years of international large-scale assessment data, we find there to be modest variation in exclusion rates across countries and that there has been a relatively small increase in exclusion rates in some over time. We also demonstrate how exclusion rates tend to be higher in studies of primary students than in studies of secondary students. Finally, while there seems to be little relationship between exclusion rates and response rates, there is a weak negative association between the level of exclusions and test performance. We conclude by discussing how information about exclusions—and other similar issues—might be more clearly communicated to non-specialist audiences.","PeriodicalId":46725,"journal":{"name":"Educational Assessment Evaluation and Accountability","volume":"80 1","pages":"0"},"PeriodicalIF":0.0,"publicationDate":"2023-09-21","publicationTypes":"Journal Article","fieldsOfStudy":null,"isOpenAccess":false,"openAccessPdf":"","citationCount":null,"resultStr":null,"platform":"Semanticscholar","paperid":"136235650","PeriodicalName":null,"FirstCategoryId":null,"ListUrlMain":null,"RegionNum":3,"RegionCategory":"教育学","ArticlePicture":[],"TitleCN":null,"AbstractTextCN":null,"PMCID":"","EPubDate":null,"PubModel":null,"JCR":null,"JCRName":null,"Score":null,"Total":0}
{"title":"Gender differences in item nonresponse in the PISA 2018 student questionnaire","authors":"Kseniia Marcq, J. Braeken","doi":"10.1007/s11092-023-09412-7","DOIUrl":"https://doi.org/10.1007/s11092-023-09412-7","url":null,"abstract":"","PeriodicalId":46725,"journal":{"name":"Educational Assessment Evaluation and Accountability","volume":" ","pages":""},"PeriodicalIF":3.9,"publicationDate":"2023-08-24","publicationTypes":"Journal Article","fieldsOfStudy":null,"isOpenAccess":false,"openAccessPdf":"","citationCount":null,"resultStr":null,"platform":"Semanticscholar","paperid":"42982171","PeriodicalName":null,"FirstCategoryId":null,"ListUrlMain":null,"RegionNum":3,"RegionCategory":"教育学","ArticlePicture":[],"TitleCN":null,"AbstractTextCN":null,"PMCID":"","EPubDate":null,"PubModel":null,"JCR":null,"JCRName":null,"Score":null,"Total":0}
{"title":"Analyses of quality assurance and development in education","authors":"G. Skedsmo, S. Huber","doi":"10.1007/s11092-023-09414-5","DOIUrl":"https://doi.org/10.1007/s11092-023-09414-5","url":null,"abstract":"","PeriodicalId":46725,"journal":{"name":"Educational Assessment Evaluation and Accountability","volume":"35 1","pages":"309 - 312"},"PeriodicalIF":3.9,"publicationDate":"2023-08-01","publicationTypes":"Journal Article","fieldsOfStudy":null,"isOpenAccess":false,"openAccessPdf":"","citationCount":null,"resultStr":null,"platform":"Semanticscholar","paperid":"42459824","PeriodicalName":null,"FirstCategoryId":null,"ListUrlMain":null,"RegionNum":3,"RegionCategory":"教育学","ArticlePicture":[],"TitleCN":null,"AbstractTextCN":null,"PMCID":"","EPubDate":null,"PubModel":null,"JCR":null,"JCRName":null,"Score":null,"Total":0}