PRIORITY: An Intelligent Problem Indicator Repository

We report the design, development and deployment of PRIORITY, an intelligent portal aimed at reducing the workload of instructors, tutors and teaching assistants in large programming courses of creating lab, assignment and exam problems every week. PRIORITY offers a scalable, user friendly and indexed repository of problems that can be queried to retrieve problems related to a particular programming concept, say for loops. PRIORITY accomplishes this by casting problem retrieval as a multi-label learning problem and using solving it using novel feature selection and AI-techniques. We also report the results of an A/B test and user survey, both conducted while PRIORITY was being used to offer a CS1 course taught at IIT Kanpur with over 500 students. PRIORITY has been in deployment at IIT Kanpur for almost 2 years now and our experience thus far suggests that it not only presents a valuable tool for course administrators, but also opens up several intriguing problems at the intersection of programming instruction, pedagogy, machine learning, semi-supervised learning and information retrieval. Code for PRIORITY is available at https://github.com/purushottamkar/priority/