Evolutions of semantic consistency in research topic via contextualized word embedding

Abstract

Topic evolution has been studied extensively in the field of the science of science. This study first analyzes topic evolution pattern from topics’ semantic consistency in the semantic vector space, and explore its possible causes. Specifically, we extract papers in the computer science field from Microsoft Academic Graph as our dataset. We propose a novel method for encoding a topic with numerous Contextualized Word Embeddings (CWE), in which the title and abstract fields of papers studying the topic is taken as its context. Subsequently, we employ three geometric metrics to analyze topics’ semantic consistency over time, from which the influence of the anisotropy of CWE is excluded. The K-Means clustering algorithm is employed to identify four general evolution patterns of semantic consistency, that is, semantic consistency increases (IM), decreases (DM), increases first and then decreases (Inverted U-shape), and decreases first and then increases (U-shape). We also find that research methods tend to show DM and U-shape, but research questions tend to be IM and Inverted U-shape. Finally, we further utilize the regression analysis to explore whether and, if so, how a series of key features of a topic affect its semantic consistency. Importantly, semantic consistency of a topic varies inversely with the semantic similarity between the topic and other topics. Overall, this study sheds light on the evolution law of topics, and helps researchers to understand these patterns from a geometric perspective.