I don't have time! But keep me in the loop: Co-designing requirements for a learning analytics cockpit with teachers

IF 5.1 2区教育学 Q1 EDUCATION & EDUCATIONAL RESEARCH

Journal of Computer Assisted Learning Pub Date : 2024-05-06 DOI:10.1111/jcal.12997

Onur Karademir, Daniele Di Mitri, Jan Schneider, Ioana Jivet, Jörn Allmang, Sebastian Gombert, Marcus Kubsch, Knut Neumann, Hendrik Drachsler

{"title":"I don't have time! But keep me in the loop: Co-designing requirements for a learning analytics cockpit with teachers","authors":"Onur Karademir, Daniele Di Mitri, Jan Schneider, Ioana Jivet, Jörn Allmang, Sebastian Gombert, Marcus Kubsch, Knut Neumann, Hendrik Drachsler","doi":"10.1111/jcal.12997","DOIUrl":null,"url":null,"abstract":"<div>\n \n \n <section>\n \n <h3> Background</h3>\n \n <p>Teacher dashboards can help secondary school teachers manage online learning activities and inform instructional decisions by visualising information about class learning. However, when designing teacher dashboards, it is not trivial to choose which information to display, because not all of the vast amount of information retrieved from digital learning environments is useful for teaching. Information elicited from formative assessment (FA), though, is a strong predictor for student performance and can be a useful data source for effective teacher dashboards. Especially in the secondary education context, FA and feedback on FA, have been extensively studied and shown to positively affect student learning outcomes. Moreover, secondary teachers struggle to make sense of the information displayed in dashboards and decide on pedagogical actions, such as providing feedback to students.</p>\n </section>\n \n <section>\n \n <h3> Objectives</h3>\n \n <p>To facilitate the provision of feedback for secondary school teachers via a teacher dashboard, this study identifies requirements for designing a Learning Analytics Cockpit (LA Cockpit), that is, (1) a teacher dashboard that provides teachers with visualisations of results from formative assessment (FA) and (2) a feedback system that supports teachers in providing feedback to students.</p>\n </section>\n \n <section>\n \n <h3> Methods</h3>\n \n <p>This study was conducted in the context of STEM classes and is based on semi-structured co-design interviews with German secondary school teachers. In these interviews, we first explored challenges teachers encountered in monitoring students' learning and providing feedback. Second, in the ideation phase, teachers were asked to define features an LA Cockpit for FA should have. Finally, in the evaluation phase, we provided teachers with a design template for an LA Cockpit, the LAC_Template, which was built upon our previous work and feedback theory, and asked them to evaluate and improve it. Further design requirements were derived based on the evaluation of the LAC_Template and teachers' suggestions for improvement.</p>\n </section>\n \n <section>\n \n <h3> Results</h3>\n \n <p>We derived 16 requirements for designing an LA Cockpit for FA in secondary schools. Findings from the interviews indicated that the feedback system of an LA Cockpit should address teachers' time limitations in giving students individualised feedback. It should therefore be designed to minimise the steps required to deliver feedback. To reduce workload, teachers requested an automated reminder to send feedback, but with the ability to adjust feedback to the learning context. Such a semi-automated feedback system can help teachers support students individually but also underline the importance of actively involving teachers in the feedback loop and giving them control when using such technologies in secondary school practice. A challenge for future teacher dashboard designs could be to find a balance between technology and teacher control that utilises the strengths of both in a beneficial combination.</p>\n </section>\n </div>","PeriodicalId":48071,"journal":{"name":"Journal of Computer Assisted Learning","volume":"40 6","pages":"2681-2699"},"PeriodicalIF":5.1000,"publicationDate":"2024-05-06","publicationTypes":"Journal Article","fieldsOfStudy":null,"isOpenAccess":false,"openAccessPdf":"https://onlinelibrary.wiley.com/doi/epdf/10.1111/jcal.12997","citationCount":"0","resultStr":null,"platform":"Semanticscholar","paperid":null,"PeriodicalName":"Journal of Computer Assisted Learning","FirstCategoryId":"95","ListUrlMain":"https://onlinelibrary.wiley.com/doi/10.1111/jcal.12997","RegionNum":2,"RegionCategory":"教育学","ArticlePicture":[],"TitleCN":null,"AbstractTextCN":null,"PMCID":null,"EPubDate":"","PubModel":"","JCR":"Q1","JCRName":"EDUCATION & EDUCATIONAL RESEARCH","Score":null,"Total":0}

引用次数: 0

Abstract

Background

Teacher dashboards can help secondary school teachers manage online learning activities and inform instructional decisions by visualising information about class learning. However, when designing teacher dashboards, it is not trivial to choose which information to display, because not all of the vast amount of information retrieved from digital learning environments is useful for teaching. Information elicited from formative assessment (FA), though, is a strong predictor for student performance and can be a useful data source for effective teacher dashboards. Especially in the secondary education context, FA and feedback on FA, have been extensively studied and shown to positively affect student learning outcomes. Moreover, secondary teachers struggle to make sense of the information displayed in dashboards and decide on pedagogical actions, such as providing feedback to students.

Objectives

To facilitate the provision of feedback for secondary school teachers via a teacher dashboard, this study identifies requirements for designing a Learning Analytics Cockpit (LA Cockpit), that is, (1) a teacher dashboard that provides teachers with visualisations of results from formative assessment (FA) and (2) a feedback system that supports teachers in providing feedback to students.

Methods

This study was conducted in the context of STEM classes and is based on semi-structured co-design interviews with German secondary school teachers. In these interviews, we first explored challenges teachers encountered in monitoring students' learning and providing feedback. Second, in the ideation phase, teachers were asked to define features an LA Cockpit for FA should have. Finally, in the evaluation phase, we provided teachers with a design template for an LA Cockpit, the LAC_Template, which was built upon our previous work and feedback theory, and asked them to evaluate and improve it. Further design requirements were derived based on the evaluation of the LAC_Template and teachers' suggestions for improvement.

Results

We derived 16 requirements for designing an LA Cockpit for FA in secondary schools. Findings from the interviews indicated that the feedback system of an LA Cockpit should address teachers' time limitations in giving students individualised feedback. It should therefore be designed to minimise the steps required to deliver feedback. To reduce workload, teachers requested an automated reminder to send feedback, but with the ability to adjust feedback to the learning context. Such a semi-automated feedback system can help teachers support students individually but also underline the importance of actively involving teachers in the feedback loop and giving them control when using such technologies in secondary school practice. A challenge for future teacher dashboard designs could be to find a balance between technology and teacher control that utilises the strengths of both in a beneficial combination.

Abstract Image

查看原文本刊更多论文

我没有时间！但请随时与我联系：与教师共同设计学习分析驾驶舱的要求

背景教师仪表盘可以帮助中学教师管理在线学习活动，并通过可视化的班级学习信息为教学决策提供依据。然而，在设计教师仪表盘时，选择显示哪些信息并非易事，因为从数字化学习环境中获取的大量信息并非都对教学有用。不过，从形成性评价（FA）中获取的信息对学生的成绩有很强的预测作用，可以成为有效的教师仪表盘的有用数据源。特别是在中学教育中，形成性评价和对形成性评价的反馈已被广泛研究，并被证明对学生的学习成绩有积极影响。此外，中学教师很难理解仪表板中显示的信息，也很难决定教学行动，如向学生提供反馈。为了促进通过教师仪表板向中学教师提供反馈，本研究确定了设计学习分析驾驶舱（LA Cockpit）的要求，即：（1）教师仪表板，为教师提供形成性评价（FA）结果的可视化；（2）反馈系统，支持教师向学生提供反馈。在访谈中，我们首先探讨了教师在监督学生学习和提供反馈时遇到的挑战。其次，在构思阶段，我们要求教师定义 FA LA Cockpit 应具备的功能。最后，在评估阶段，我们向教师提供了一个洛杉矶驾驶舱设计模板（LAC_Template），该模板建立在我们之前的工作和反馈理论基础之上，并要求教师对其进行评估和改进。根据对 LAC_Template 的评估和教师提出的改进建议，我们得出了进一步的设计要求。访谈结果表明，LA Cockpit 的反馈系统应解决教师在向学生提供个性化反馈时时间有限的问题。因此，该系统的设计应尽量减少反馈所需的步骤。为了减少工作量，教师要求自动提醒发送反馈，但能够根据学习情况调整反馈。这种半自动化的反馈系统可以帮助教师对学生进行个别辅导，但同时也强调了在中学实践中使用此类技术时，让教师积极参与反馈环路并给予他们控制权的重要性。未来教师仪表板设计所面临的一个挑战可能是如何在技术和教师控制之间找到一个平衡点，利用两者的优势进行有益的结合。

本文章由计算机程序翻译，如有差异，请以英文原文为准。

求助全文

约1分钟内获得全文求助全文

来源期刊

Journal of Computer Assisted Learning EDUCATION & EDUCATIONAL RESEARCH-

CiteScore

9.70

自引率

6.00%

发文量

116

期刊介绍： The Journal of Computer Assisted Learning is an international peer-reviewed journal which covers the whole range of uses of information and communication technology to support learning and knowledge exchange. It aims to provide a medium for communication among researchers as well as a channel linking researchers, practitioners, and policy makers. JCAL is also a rich source of material for master and PhD students in areas such as educational psychology, the learning sciences, instructional technology, instructional design, collaborative learning, intelligent learning systems, learning analytics, open, distance and networked learning, and educational evaluation and assessment. This is the case for formal (e.g., schools), non-formal (e.g., workplace learning) and informal learning (e.g., museums and libraries) situations and environments. Volumes often include one Special Issue which these provides readers with a broad and in-depth perspective on a specific topic. First published in 1985, JCAL continues to have the aim of making the outcomes of contemporary research and experience accessible. During this period there have been major technological advances offering new opportunities and approaches in the use of a wide range of technologies to support learning and knowledge transfer more generally. There is currently much emphasis on the use of network functionality and the challenges its appropriate uses pose to teachers/tutors working with students locally and at a distance. JCAL welcomes: -Empirical reports, single studies or programmatic series of studies on the use of computers and information technologies in learning and assessment -Critical and original meta-reviews of literature on the use of computers for learning -Empirical studies on the design and development of innovative technology-based systems for learning -Conceptual articles on issues relating to the Aims and Scope