Visualizing the Results of Field Testing

Abstract

Field testing of software is necessary to find potential user problems before market deployment. The large number of users involved in field testing along with the variety of problems reported by them increases the complexity of managing the field testing process. However, most field testing processes are monitored using ad-hoc techniques and simple metrics (e.g., the number of reported problems). Deeper analysis and tracking of field testing results is needed. This paper introduces visualization techniques which provide a global view of the field testing results. The techniques focus on the relation between users and their reported problems. The visualizations help identify general patterns to locate the problems. For example, the technique identifies groups of users with similar problem profiles. Such knowledge helps reduce the number of needed users since we can pick representative users. We demonstrate our proposed techniques using the field testing results for four releases of a large scale enterprise application used by millions of users worldwide.