Interpreting Measures of Rarity and Novelty: Investigating Correlations Between Relative Infrequency and Perceived Ratings

Central to the struggle in design ideation research is the quantification of abstract and qualitative measures. Among these measures, creativity, originality, and novelty are some of the most subjective and generally disagreed upon constructs. In recent years in the design community, novelty has primarily been measured with two distinct styles of metrics: relative infrequency and perceived ratings. Relative infrequency captures how rare an idea is within an idea set for an objective representation of novelty, while ratings quantify the perceptions of appropriate judges for an intuitive understanding of novelty. This paper investigates the convergent validity between these two styles through the implementation of three previously published methods. Moderate convergent validity is shown between a measure of relative infrequency and perceived ratings leading to clear and meaningful recommendations for metrics in future research. However, the degree of disagreement warrants differentiation between the two styles regarding terminology and analysis. This study will aid in the research and interpretation of future studies of creativity in ideation.