通过向前和向后计数不同单词数量的过程进行文本分割

IF 0.7 2区 文学 0 LANGUAGE & LINGUISTICS
Berhane Abebe, Mikhail Chebunin, Artyom Kovalevskii
{"title":"通过向前和向后计数不同单词数量的过程进行文本分割","authors":"Berhane Abebe, Mikhail Chebunin, Artyom Kovalevskii","doi":"10.1080/09296174.2023.2275342","DOIUrl":null,"url":null,"abstract":"ABSTRACTThe paper is developing a new statistical approach to automatic partitioning of texts into parts belonging to different authors. It is based on the analysis of processes that counts the number of different words forward and backward. The theoretical study of the processes is based on the assumptions of an elementary probability model with a change point. We prove consistence of our statistical estimate of the point of concatenation in the case when the concatenated texts have different Zipf exponents. This method is being tested on the Brown corpus and also on newspaper texts in different languages. Testing shows a good estimate of the concatenation point. This method can be used in parallel with other text segmentation methods. AcknowledgmentsThe authors like to thank anonymous referees for their helpful and constructive comments and suggestions.Disclosure statementNo potential conflict of interest was reported by the author(s).Data availability statementWe used texts from open sources.Additional informationFundingThe work was supported by the Siberian Branch, Russian Academy of Sciences [FWNF-2022-0010].","PeriodicalId":45514,"journal":{"name":"Journal of Quantitative Linguistics","volume":"79 19","pages":"0"},"PeriodicalIF":0.7000,"publicationDate":"2023-11-12","publicationTypes":"Journal Article","fieldsOfStudy":null,"isOpenAccess":false,"openAccessPdf":"","citationCount":"0","resultStr":"{\"title\":\"Text Segmentation Via Processes that Count the Number of Different Words Forward and Backward\",\"authors\":\"Berhane Abebe, Mikhail Chebunin, Artyom Kovalevskii\",\"doi\":\"10.1080/09296174.2023.2275342\",\"DOIUrl\":null,\"url\":null,\"abstract\":\"ABSTRACTThe paper is developing a new statistical approach to automatic partitioning of texts into parts belonging to different authors. It is based on the analysis of processes that counts the number of different words forward and backward. The theoretical study of the processes is based on the assumptions of an elementary probability model with a change point. We prove consistence of our statistical estimate of the point of concatenation in the case when the concatenated texts have different Zipf exponents. This method is being tested on the Brown corpus and also on newspaper texts in different languages. Testing shows a good estimate of the concatenation point. This method can be used in parallel with other text segmentation methods. AcknowledgmentsThe authors like to thank anonymous referees for their helpful and constructive comments and suggestions.Disclosure statementNo potential conflict of interest was reported by the author(s).Data availability statementWe used texts from open sources.Additional informationFundingThe work was supported by the Siberian Branch, Russian Academy of Sciences [FWNF-2022-0010].\",\"PeriodicalId\":45514,\"journal\":{\"name\":\"Journal of Quantitative Linguistics\",\"volume\":\"79 19\",\"pages\":\"0\"},\"PeriodicalIF\":0.7000,\"publicationDate\":\"2023-11-12\",\"publicationTypes\":\"Journal Article\",\"fieldsOfStudy\":null,\"isOpenAccess\":false,\"openAccessPdf\":\"\",\"citationCount\":\"0\",\"resultStr\":null,\"platform\":\"Semanticscholar\",\"paperid\":null,\"PeriodicalName\":\"Journal of Quantitative Linguistics\",\"FirstCategoryId\":\"1085\",\"ListUrlMain\":\"https://doi.org/10.1080/09296174.2023.2275342\",\"RegionNum\":2,\"RegionCategory\":\"文学\",\"ArticlePicture\":[],\"TitleCN\":null,\"AbstractTextCN\":null,\"PMCID\":null,\"EPubDate\":\"\",\"PubModel\":\"\",\"JCR\":\"0\",\"JCRName\":\"LANGUAGE & LINGUISTICS\",\"Score\":null,\"Total\":0}","platform":"Semanticscholar","paperid":null,"PeriodicalName":"Journal of Quantitative Linguistics","FirstCategoryId":"1085","ListUrlMain":"https://doi.org/10.1080/09296174.2023.2275342","RegionNum":2,"RegionCategory":"文学","ArticlePicture":[],"TitleCN":null,"AbstractTextCN":null,"PMCID":null,"EPubDate":"","PubModel":"","JCR":"0","JCRName":"LANGUAGE & LINGUISTICS","Score":null,"Total":0}
引用次数: 0

摘要

摘要本文提出了一种新的统计方法,用于将文本自动划分为属于不同作者的部分。它基于对进程的分析,向前和向后计算不同单词的数量。该过程的理论研究是基于一个具有变化点的初等概率模型的假设。我们证明了在连接文本具有不同Zipf指数的情况下,我们对连接点的统计估计的一致性。这种方法正在布朗语料库和不同语言的报纸文本上进行测试。测试显示了对连接点的一个很好的估计。该方法可以与其他文本分割方法并行使用。作者要感谢匿名审稿人提供的有帮助和建设性的意见和建议。披露声明作者未报告潜在的利益冲突。数据可用性声明我们使用了来自开放资源的文本。本研究得到了俄罗斯科学院西伯利亚分院的支持[FWNF-2022-0010]。
本文章由计算机程序翻译,如有差异,请以英文原文为准。
Text Segmentation Via Processes that Count the Number of Different Words Forward and Backward
ABSTRACTThe paper is developing a new statistical approach to automatic partitioning of texts into parts belonging to different authors. It is based on the analysis of processes that counts the number of different words forward and backward. The theoretical study of the processes is based on the assumptions of an elementary probability model with a change point. We prove consistence of our statistical estimate of the point of concatenation in the case when the concatenated texts have different Zipf exponents. This method is being tested on the Brown corpus and also on newspaper texts in different languages. Testing shows a good estimate of the concatenation point. This method can be used in parallel with other text segmentation methods. AcknowledgmentsThe authors like to thank anonymous referees for their helpful and constructive comments and suggestions.Disclosure statementNo potential conflict of interest was reported by the author(s).Data availability statementWe used texts from open sources.Additional informationFundingThe work was supported by the Siberian Branch, Russian Academy of Sciences [FWNF-2022-0010].
求助全文
通过发布文献求助,成功后即可免费获取论文全文。 去求助
来源期刊
CiteScore
2.90
自引率
7.10%
发文量
7
期刊介绍: The Journal of Quantitative Linguistics is an international forum for the publication and discussion of research on the quantitative characteristics of language and text in an exact mathematical form. This approach, which is of growing interest, opens up important and exciting theoretical perspectives, as well as solutions for a wide range of practical problems such as machine learning or statistical parsing, by introducing into linguistics the methods and models of advanced scientific disciplines such as the natural sciences, economics, and psychology.
×
引用
GB/T 7714-2015
复制
MLA
复制
APA
复制
导出至
BibTeX EndNote RefMan NoteFirst NoteExpress
×
提示
您的信息不完整,为了账户安全,请先补充。
现在去补充
×
提示
您因"违规操作"
具体请查看互助需知
我知道了
×
提示
确定
请完成安全验证×
copy
已复制链接
快去分享给好友吧!
我知道了
右上角分享
点击右上角分享
0
联系我们:info@booksci.cn Book学术提供免费学术资源搜索服务,方便国内外学者检索中英文文献。致力于提供最便捷和优质的服务体验。 Copyright © 2023 布克学术 All rights reserved.
京ICP备2023020795号-1
ghs 京公网安备 11010802042870号
Book学术文献互助
Book学术文献互助群
群 号:481959085
Book学术官方微信