Are Automatic Bug Report Summarizers Missing the Target?

Abstract

Bug reports can be lengthy due to long descriptions and long conversation threads. Automatic summarization of the text in a bug report can reduce the time spent by software project members on understanding the content of a bug report. Quality of the bug report summaries have been historically evaluated using human-created gold-standard summaries. However, we believe this is not a good practice for two reasons. First, we observed high disagreement levels in the annotated summaries and the number of annotators to create gold-standard summaries was lower than the established value for stable annotation. We believe that creating a fixed summary length of 25% of the word count of the corresponding bug report is not suitable for every time when a person refers to a bug report. Therefore, we propose an automatic sentence annotation method and an interface to customize the presented summary.