A Proposed Quality Metric for Ideation Effectiveness

Abstract

A suitable quality metric is essential to improving ideation effectiveness. Many proposed quality metrics struggle to adequately capture this critical, subjective concept in a reliable and efficient way. This paper shows our development and testing of a quality metric that is meaningful, repeatable, and efficient. This quality metric is a weighted sum of quality dimensions adapted from the literature. The weighting factors for each dimension are adjusted to the specific ideation problem, and we present here a systematic method to quickly determine these weightings by experimental means. We demonstrate repeatability of the quality metric through interrater reliability, we show meaningfulness by comparing with raters’ intuitive interpretation of quality, and we demonstrate efficiency in the rating process. These initial findings show the quality metric has great promise and merits additional testing and refinement in future work.