Transforming Everyday Information into Practical Analytics with Crowdsourced Assessment Tasks

Abstract

Educators use a wide variety of data to inform their practices. Examples of these data include forms of information that are commonplace in schools, such as student work and paper-based artifacts. One limitation in these situations is that there are less efficient ways to process such everyday varieties of information into analytics that are more usable and practical for educators. To explore how to address this constraint, we describe two sets of design experiments that utilize crowdsourced tasks for scoring open-ended assessments. Developing crowdsourced systems and their resulting analytics introduced a variety of challenges, such as attending to the expertise and learning of the crowd. In this paper, we describe the potential efficacy of design decisions such as screening the crowd, providing multimedia instruction, and asking the crowd to explain their answers. We also explore the potential of crowdsourcing as a learning opportunity for those participating in the collective tasks. Our work offers key design implications for leveraging crowdsourcing to process educational data in ways that are relevant to educators, while offering learning experiences for the crowd.