Fouad Abd-El-Khalick, Ryan Summers, Jeanne L. Brunner, Jeremy Belarmino, John Myers
{"title":"Development of VAScoR: A rubric to qualify and score responses to the views of nature of science (VNOS) questionnaire","authors":"Fouad Abd-El-Khalick, Ryan Summers, Jeanne L. Brunner, Jeremy Belarmino, John Myers","doi":"10.1002/tea.21916","DOIUrl":null,"url":null,"abstract":"<p>We report on the development of a rubric to reliably qualify and score responses to the <i>Views of Nature of Science Questionnaire</i> (VNOS): The <i>VNOS Analysis and Scoring Rubric</i> (VAScoR). The <i>VAScoR</i> is designed to (a) provide systematic guidance for the qualitative analysis, and score assignment to nuanced categories, of <i>VNOS</i> responses, (b) explicitly scaffold qualitative inferencing and standardize score assignment to substantially lessen the burden of, and variance in, analyzing and scoring the <i>VNOS</i>, and (c) improve the viability and meaningfulness of cross-study comparisons drawing on <i>VNOS</i> data. The rubric adopted the <i>VNOS's</i> consensus NOS framework and further delineated core and related elements across 10 target NOS aspects. The <i>VAScoR's</i> reliability was examined in two studies that drew on <i>VNOS</i> questionnaires completed by 185 preservice secondary science teachers (58% female; 126 undergraduate and 59 graduate students) enrolled over several years in a combined undergraduate and graduate licensure program in a large U.S. Midwestern university. In Study I, <i>VAScoR</i> analyses of 86 <i>VNOS</i> questionnaires undertaken by a single author were used to examine the rubric's intra-rater reliability, which resulted in a robust Cronbach's alpha value of 0.81. In Study II, analyses by four authors of a randomly generated, overlapping set of 18 questionnaires were used to examine inter-rater reliability, which was supported with substantial consensus among raters as indicated by a Cohen's kappa of 0.71. Further evidence for the <i>VAScoR's</i> inter-rater reliability was indicated by moderate to strong consistency among four raters with an overall Pearson's correlation coefficient of 0.82, and coefficient values ranging from 0.77 to 0.89 for six possible rater pairings.</p>","PeriodicalId":48369,"journal":{"name":"Journal of Research in Science Teaching","volume":null,"pages":null},"PeriodicalIF":3.6000,"publicationDate":"2023-11-20","publicationTypes":"Journal Article","fieldsOfStudy":null,"isOpenAccess":false,"openAccessPdf":"","citationCount":"0","resultStr":null,"platform":"Semanticscholar","paperid":null,"PeriodicalName":"Journal of Research in Science Teaching","FirstCategoryId":"95","ListUrlMain":"https://onlinelibrary.wiley.com/doi/10.1002/tea.21916","RegionNum":1,"RegionCategory":"教育学","ArticlePicture":[],"TitleCN":null,"AbstractTextCN":null,"PMCID":null,"EPubDate":"","PubModel":"","JCR":"Q1","JCRName":"EDUCATION & EDUCATIONAL RESEARCH","Score":null,"Total":0}
引用次数: 0
Abstract
We report on the development of a rubric to reliably qualify and score responses to the Views of Nature of Science Questionnaire (VNOS): The VNOS Analysis and Scoring Rubric (VAScoR). The VAScoR is designed to (a) provide systematic guidance for the qualitative analysis, and score assignment to nuanced categories, of VNOS responses, (b) explicitly scaffold qualitative inferencing and standardize score assignment to substantially lessen the burden of, and variance in, analyzing and scoring the VNOS, and (c) improve the viability and meaningfulness of cross-study comparisons drawing on VNOS data. The rubric adopted the VNOS's consensus NOS framework and further delineated core and related elements across 10 target NOS aspects. The VAScoR's reliability was examined in two studies that drew on VNOS questionnaires completed by 185 preservice secondary science teachers (58% female; 126 undergraduate and 59 graduate students) enrolled over several years in a combined undergraduate and graduate licensure program in a large U.S. Midwestern university. In Study I, VAScoR analyses of 86 VNOS questionnaires undertaken by a single author were used to examine the rubric's intra-rater reliability, which resulted in a robust Cronbach's alpha value of 0.81. In Study II, analyses by four authors of a randomly generated, overlapping set of 18 questionnaires were used to examine inter-rater reliability, which was supported with substantial consensus among raters as indicated by a Cohen's kappa of 0.71. Further evidence for the VAScoR's inter-rater reliability was indicated by moderate to strong consistency among four raters with an overall Pearson's correlation coefficient of 0.82, and coefficient values ranging from 0.77 to 0.89 for six possible rater pairings.
期刊介绍:
Journal of Research in Science Teaching, the official journal of NARST: A Worldwide Organization for Improving Science Teaching and Learning Through Research, publishes reports for science education researchers and practitioners on issues of science teaching and learning and science education policy. Scholarly manuscripts within the domain of the Journal of Research in Science Teaching include, but are not limited to, investigations employing qualitative, ethnographic, historical, survey, philosophical, case study research, quantitative, experimental, quasi-experimental, data mining, and data analytics approaches; position papers; policy perspectives; critical reviews of the literature; and comments and criticism.