Reliability Assessment of Mass-Market Software: Insights from Windows Vista®

Abstract

Assessing the reliability of mass-market software (MMS), such as the Windowsreg operating system, presents many challenges. In this paper, we share insights gained from the Windows Vistareg and Windows Vistareg SP1 operating systems. First, we find that the automated reliability monitoring approach, which periodically reports reliability status, provides higher quality data and requires less effort compared to other approaches available today. We describe one instance in detail: the Windows reliability analysis component, and illustrate its advantages using data from Windows Vista. Second, we show the need to account for usage scenarios during reliability assessments. For pre-release versions of Windows Vista and Vista SP1, usage scenarios differ by 2-4X for Microsoft internal and external samples; corresponding reliability assessments differ by 2-3X. Our results help motivate and guide further research in reliability assessment.