Change point detection in text data

Q1 Mathematics

Behaviormetrika Pub Date : 2023-10-11 DOI:10.1007/s41237-023-00207-0

Axel Preis, Stefanie Schwaar

引用次数: 0

Abstract

Abstract The analysis of text data using artificial intelligence and statistical methods has become increasingly important in recent years. One application is the automatic assignment of documents. For this purpose, a classification model is trained on the basis of historical data. If the structure of the texts to be classified changes over time, the quality of the classification will decrease. Change point detection algorithms can counteract this. Such algorithms automatically detect changes in the structure of the texts and indicate that the trained classification model has to be adapted. However, the undesired influence of the length of the document needs to be handled when modeling the text data. We present a multinomial change-point model detecting changes in text structures. The results are supported by simulation studies.

查看原文本刊更多论文

文本数据中的更改点检测

近年来，使用人工智能和统计方法对文本数据进行分析变得越来越重要。一个应用程序是文档的自动分配。为此，在历史数据的基础上训练分类模型。如果要分类的文本的结构随着时间的推移而改变，分类的质量就会下降。变化点检测算法可以抵消这一点。这种算法自动检测文本结构的变化，并指示训练好的分类模型必须进行调整。但是，在对文本数据建模时，需要处理文档长度的不良影响。我们提出了一个多项变化点模型来检测文本结构的变化。结果得到了仿真研究的支持。

本文章由计算机程序翻译，如有差异，请以英文原文为准。

求助全文

约1分钟内获得全文求助全文

来源期刊

Behaviormetrika Mathematics-Analysis

CiteScore

5.10

自引率

0.00%

发文量

期刊介绍： Behaviormetrika is issued twice a year to provide an international forum for new theoretical and empirical quantitative approaches in data science. When Behaviormetrika was launched in 1974, the journal advocated data science, as an interdisciplinary field that included the use of statistical methods to extract meaningful knowledge from data in its various forms: structured or unstructured. Behaviormetrika is the oldest journal addressing the topic of data science. The first editor-in-chief of Behaviormetrika, Dr. Chikio Hayashi, described data science in this way:“Data science is not only a synthetic concept to unify statistics, data analysis, and their related methods; it also comprises its results. Data science is intended to analyze and understand actual phenomena with ‘data.’ In other words, the aim of data science is to reveal the features or the hidden structure of complicated natural, human, and social phenomena using data from a different perspective from the established or traditional theory and method.” Behaviormetrika is a fully refereed international journal, which publishes original research papers, notes, and review articles. Subject areas suitable for publication include but are not limited to the following methodologies and fields. Methodologies Data scienceMathematical statisticsSurvey methodologiesArtificial intelligence Information theoryMachine learning Knowledge discovery in databases (KDD)Graphical modelsComputer scienceAlgorithms FieldsMedicinePsychologyEducationEconomicsMarketingSocial scienceSociologyPolitical sciencePolicy scienceCognitive scienceBrain science