Programmable Questions in Edgar

Abstract

Automated programming assessment systems (APAS) are a valuable tool that is growing in popularity, particularly in the field of computer science education. They can provide quick and objective assessment and feedback to the programming assignments – those that receive source code as a response. Most APASs treat code as a black box and employ dynamic analysis to assess code. Dynamic analysis is straightforward, easily implemented, explainable, and works well in most situations. Edgar is a comprehensive, state of the art APAS, that has been used daily and has evolved for the past six years. This paper examines the pipeline used by Edgar to assess programming questions and presents our enhancements to the traditional dynamic analysis programmable templates and scripts. Templates enable customized question texts based on the programmable model, so that each student can receive personalized variation of the question. Personalized questions are a great way to fight potential academic dishonesty. Scripts are plugged into the assessment pipeline after the dynamic analysis and can override the default grade by examining some other aspect of the program. We also offer our thoughts on upcoming plans to include generic static analysis as we move closer to a unified assessment pipeline.