Competitive testing: issues and methodology

Abstract

The purpose of this Special Interest Group is to provide a forum for Usability professionals with an interest in performing Competitive Tests to discuss issues and exchange advice. There is very little information regarding appropriate methodology or guidelines for performing Competitive Tests published on this topic. This Special Interest Group will provide an opportunity for individuals whose work involves the performance or review of competitive tests to share tips and techniques and will serve as an avenue for those interested in competitive testing to gain insight on the differences between competitive and diagnostic usability tests. In addition, it will allow the members of CHI to discuss the option of adopting standardized methodologies and metrics for performing competitive usability tests. The organizers of this Special Interest Group are three Usability Professionals with experience planning and conducting Competitive Tests. The majority of the meeting will be devoted to an informal discussion of the Competitive Testing experiences of both the organizers and other meeting participants. Issues that we intend to discuss include: goals of competitive testing, how competitive testing differs from other forms of usability testing, legal and licensing issues, who performs the various tasks, and when to hire a consultant. Emphasis will be placed upon identifying useful techniques, potential problems and solutions to these problems. The final portion of the meeting will be reserved for a brainstorming session on a set of basic and commonly agreed upon techniques and standards for organizing, designing, and performing competitive tests and reporting test results. Guidelines for competitive usability testing are necessary to assist individuals who are unfamiliar with competitive testing requirements, and as a resource to ensure lack of bias when designing and running a Competitive Test. It is also hoped that the creation of this rough set of guidelines would encourage further discussion of the creation of a standardized Competitive Testing Methodology that members of CHI and other Human Factors Professionals could refer to. The guidelines produced by this brainstorming session will be distributed to SIG participants after CHI via E-mail or regular mail. Used with permission of Oracle Corporation and American Institutes for Research