Measuring the Direction of Innovation: Frontier Tools in Unassisted Machine Learning

Abstract

Understanding the factors affecting the direction of innovation is a central aim of research in the economics and strategic management of innovation. Progress on this topic has been inhibited by difficulties in measuring the location and movement of innovation in ideas space. To date, most efforts at measuring the direction of innovation rely on curated taxonomies, such as technology classes and keyword approaches, which either adapt slowly or are subject to gaming, and early generations of text analysis, which provide information on the similarity of sets of words, but not on the number of paths or direction of change. Relative to these, recent advances in machine learning offer promising paths forward. In this paper, we introduce and explore a particular approach based on an unassisted machine learning technique, Hierarchical Dirichlet Process (HDP), that flexibly generates categories from a corpus of text and enables calculations of the distance across knowledge categories and movement in ideas space. We apply our algorithm to the corpus of USPTO patent abstracts from the period 2000-2018 and demonstrate that, relative to the USPTO taxonomy of patent classes, our algorithm provides a leading indicator of shift in innovation topics and enables a more precise analysis of movement in ideas space. Working with such measures is important because it enables more accurate estimates of the direction of innovation and, hence, of economic actors’ responses to competitive environments and public policies. We share our algorithm, which can be applied to other innovation text corpora, as well as the patent data and measures we develop, with the aim of facilitating additional inquiries regarding the direction of innovation.