Evaluating Crowdsourced Design Concepts With Machine Learning

Abstract

Automation has enabled design of increasingly complex products, services, and systems. Advanced technology enables designers to automate repetitive tasks in earlier design phases, even high level conceptual ideation. One particularly repetitive task in ideation is to process the large concept sets that can be developed through crowdsourcing. This paper introduces a method for filtering, categorizing, and rating large sets of design concepts. It leverages unsupervised machine learning (ML) trained on open source databases. Input design concepts are written in natural language. The concepts are not pre-tagged, structured or processed in any way which requires human intervention. Nor does the approach require dedicated training on a sample set of designs. Concepts are assessed at the sentence level via a mixture of named entity tagging (keywords) through contextual sense recognition and topic tagging (sentence topic) through probabilistic mapping to a knowledge graph. The method also includes a filtering strategy, the introduction of two metrics, and a selection strategy for assessing design concepts. The metrics are analogous to the design creativity metrics novelty, level of detail, and a selection strategy. To test the method, four ideation cases were studied; over 4,000 concepts were generated and evaluated. Analyses include: asymptotic convergence analysis; a predictive industry case study; and a dominance test between several approaches to selection of high ranking concepts. Notably, in a series of binary comparisons between concepts that were selected from the entire set by a time limited human versus those with the highest ML metric scores, the ML selected concepts were dominant.