Investigating Failure Patterns in Machine Learning-based Object Detection Tasks in Software Development Courses

Object detection, one of the popular tasks in computer vision, is to find all objects of interest in an image and determine their category and location. When people use deep learning frameworks to implement object detection networks, defects are often caused by human-introduced faults. These defects may cause different types of failures. Exploring frequent failure patterns in object detection programs can help developers detect and fix defects more effectively and efficiently. Therefore, we conducted an empirical study on failure patterns in deep learning-based object detection programs submitted in university software development courses. By exploring 101 submissions of a Yolov4 object detection task completed by 104 students, we found the most frequent 13 failure patterns in these submissions and six types of root causes of these failures. To help students and entry-level software engineers avoid possible faults in object detection programs, 13 concrete suggestions that belong to six classes are given in this paper. These results can reveal some basic laws of failures and mistakes in the development of deep learning-based object detection programs and provide guidances to assist students and entry-level developers in improving their skills in developing object detection programs.