Mining until it hurts: automatic extraction of usability issues from online reviews compared to traditional usability evaluation

Large amounts of data available on the web, for example reviews, tweets, and forum postings, contain user narratives on interaction with products. Finding usability issues in such user narratives offers an interesting alternative to traditional usability testing. To leverage such data for identifying usability issues, we (I) devise a methodology for building automated extraction tools for usability issues; (II) perform empirical assessment of such tools by training a number of classifiers to extract sentences describing usability issues for two digital cameras and a children's tablet; (III) perform quantitative and qualitative comparisons between the usability issues identified by the classifiers and those identified and assessed by two traditional methods: heuristic evaluation and think aloud testing. Our results show that it is possible to build and train algorithms for extracting actionable usability issues, but raise serious concerns about the practical future prospects for supplementing traditional evaluation methods with automated extraction algorithms.