Validating the Use of LMS-Derived Rubric Structural Features to Facilitate Automated Measurement of Rubric Quality

Proceedings of the 9th International Conference on Learning Analytics & Knowledge Pub Date : 2019-03-04 DOI:10.1145/3303772.3303829

Philip Arcuria, W. Morgan, T. Fikes

{"title":"Validating the Use of LMS-Derived Rubric Structural Features to Facilitate Automated Measurement of Rubric Quality","authors":"Philip Arcuria, W. Morgan, T. Fikes","doi":"10.1145/3303772.3303829","DOIUrl":null,"url":null,"abstract":"Rubrics are widely used throughout postsecondary education as means for aiding in instruction and evaluation. However, despite their broad global adoption, very little is known about the quality of rubrics in use. We develop two measures to assess the quality of rubrics: (1) a checklist identifying criteria of high-quality rubrics based on analytic rubric design best practices and (2) a set of LMS-derived features that are hypothesized to represent structural components that are, in general, necessary but not sufficient for high quality rubrics. The validity of using the feature-generated scores as proxies for identifying rubric quality is evaluated through several means. First, the feature-generated scores are calculated for a set of external exemplary rubrics of known high quality. Second, the feature-scores for a subset of internal rubrics are compared to average human rater scores of rubric quality based on the checklist. We discuss the results, practical applications, and a larger research program surrounding the feature-generated scores.","PeriodicalId":382957,"journal":{"name":"Proceedings of the 9th International Conference on Learning Analytics & Knowledge","volume":"18 1","pages":"0"},"PeriodicalIF":0.0000,"publicationDate":"2019-03-04","publicationTypes":"Journal Article","fieldsOfStudy":null,"isOpenAccess":false,"openAccessPdf":"","citationCount":"3","resultStr":null,"platform":"Semanticscholar","paperid":null,"PeriodicalName":"Proceedings of the 9th International Conference on Learning Analytics & Knowledge","FirstCategoryId":"1085","ListUrlMain":"https://doi.org/10.1145/3303772.3303829","RegionNum":0,"RegionCategory":null,"ArticlePicture":[],"TitleCN":null,"AbstractTextCN":null,"PMCID":null,"EPubDate":"","PubModel":"","JCR":"","JCRName":"","Score":null,"Total":0}

引用次数: 3

Abstract

Rubrics are widely used throughout postsecondary education as means for aiding in instruction and evaluation. However, despite their broad global adoption, very little is known about the quality of rubrics in use. We develop two measures to assess the quality of rubrics: (1) a checklist identifying criteria of high-quality rubrics based on analytic rubric design best practices and (2) a set of LMS-derived features that are hypothesized to represent structural components that are, in general, necessary but not sufficient for high quality rubrics. The validity of using the feature-generated scores as proxies for identifying rubric quality is evaluated through several means. First, the feature-generated scores are calculated for a set of external exemplary rubrics of known high quality. Second, the feature-scores for a subset of internal rubrics are compared to average human rater scores of rubric quality based on the checklist. We discuss the results, practical applications, and a larger research program surrounding the feature-generated scores.

查看原文本刊更多论文

验证使用lms衍生的毛坯结构特征来促进毛坯质量的自动测量

在整个高等教育中，作为辅助教学和评估的手段，标准被广泛使用。然而，尽管它们在全球范围内被广泛采用，但人们对使用中的红字的质量知之甚少。我们开发了两种方法来评估规则的质量:(1)基于分析规则设计最佳实践确定高质量规则标准的清单;(2)一组lms衍生的特征，假设这些特征代表了通常情况下对高质量规则必要但不充分的结构组件。使用特征生成的分数作为识别标题质量的代理的有效性通过几种方法进行了评估。首先，为一组已知高质量的外部示例规则计算特征生成的分数。其次，将内部规则子集的特征分数与基于检查表的规则质量的平均人类评分分数进行比较。我们讨论了结果，实际应用，以及围绕特征生成分数的更大的研究计划。

本文章由计算机程序翻译，如有差异，请以英文原文为准。

求助全文

约1分钟内获得全文求助全文

来源期刊

Proceedings of the 9th International Conference on Learning Analytics & Knowledge

自引率

0.00%

发文量