{"title":"通过向前和向后计数不同单词数量的过程进行文本分割","authors":"Berhane Abebe, Mikhail Chebunin, Artyom Kovalevskii","doi":"10.1080/09296174.2023.2275342","DOIUrl":null,"url":null,"abstract":"ABSTRACTThe paper is developing a new statistical approach to automatic partitioning of texts into parts belonging to different authors. It is based on the analysis of processes that counts the number of different words forward and backward. The theoretical study of the processes is based on the assumptions of an elementary probability model with a change point. We prove consistence of our statistical estimate of the point of concatenation in the case when the concatenated texts have different Zipf exponents. This method is being tested on the Brown corpus and also on newspaper texts in different languages. Testing shows a good estimate of the concatenation point. This method can be used in parallel with other text segmentation methods. AcknowledgmentsThe authors like to thank anonymous referees for their helpful and constructive comments and suggestions.Disclosure statementNo potential conflict of interest was reported by the author(s).Data availability statementWe used texts from open sources.Additional informationFundingThe work was supported by the Siberian Branch, Russian Academy of Sciences [FWNF-2022-0010].","PeriodicalId":45514,"journal":{"name":"Journal of Quantitative Linguistics","volume":null,"pages":null},"PeriodicalIF":0.7000,"publicationDate":"2023-11-12","publicationTypes":"Journal Article","fieldsOfStudy":null,"isOpenAccess":false,"openAccessPdf":"","citationCount":"0","resultStr":"{\"title\":\"Text Segmentation Via Processes that Count the Number of Different Words Forward and Backward\",\"authors\":\"Berhane Abebe, Mikhail Chebunin, Artyom Kovalevskii\",\"doi\":\"10.1080/09296174.2023.2275342\",\"DOIUrl\":null,\"url\":null,\"abstract\":\"ABSTRACTThe paper is developing a new statistical approach to automatic partitioning of texts into parts belonging to different authors. It is based on the analysis of processes that counts the number of different words forward and backward. The theoretical study of the processes is based on the assumptions of an elementary probability model with a change point. We prove consistence of our statistical estimate of the point of concatenation in the case when the concatenated texts have different Zipf exponents. This method is being tested on the Brown corpus and also on newspaper texts in different languages. Testing shows a good estimate of the concatenation point. This method can be used in parallel with other text segmentation methods. AcknowledgmentsThe authors like to thank anonymous referees for their helpful and constructive comments and suggestions.Disclosure statementNo potential conflict of interest was reported by the author(s).Data availability statementWe used texts from open sources.Additional informationFundingThe work was supported by the Siberian Branch, Russian Academy of Sciences [FWNF-2022-0010].\",\"PeriodicalId\":45514,\"journal\":{\"name\":\"Journal of Quantitative Linguistics\",\"volume\":null,\"pages\":null},\"PeriodicalIF\":0.7000,\"publicationDate\":\"2023-11-12\",\"publicationTypes\":\"Journal Article\",\"fieldsOfStudy\":null,\"isOpenAccess\":false,\"openAccessPdf\":\"\",\"citationCount\":\"0\",\"resultStr\":null,\"platform\":\"Semanticscholar\",\"paperid\":null,\"PeriodicalName\":\"Journal of Quantitative Linguistics\",\"FirstCategoryId\":\"1085\",\"ListUrlMain\":\"https://doi.org/10.1080/09296174.2023.2275342\",\"RegionNum\":2,\"RegionCategory\":\"文学\",\"ArticlePicture\":[],\"TitleCN\":null,\"AbstractTextCN\":null,\"PMCID\":null,\"EPubDate\":\"\",\"PubModel\":\"\",\"JCR\":\"0\",\"JCRName\":\"LANGUAGE & LINGUISTICS\",\"Score\":null,\"Total\":0}","platform":"Semanticscholar","paperid":null,"PeriodicalName":"Journal of Quantitative Linguistics","FirstCategoryId":"1085","ListUrlMain":"https://doi.org/10.1080/09296174.2023.2275342","RegionNum":2,"RegionCategory":"文学","ArticlePicture":[],"TitleCN":null,"AbstractTextCN":null,"PMCID":null,"EPubDate":"","PubModel":"","JCR":"0","JCRName":"LANGUAGE & LINGUISTICS","Score":null,"Total":0}
Text Segmentation Via Processes that Count the Number of Different Words Forward and Backward
ABSTRACTThe paper is developing a new statistical approach to automatic partitioning of texts into parts belonging to different authors. It is based on the analysis of processes that counts the number of different words forward and backward. The theoretical study of the processes is based on the assumptions of an elementary probability model with a change point. We prove consistence of our statistical estimate of the point of concatenation in the case when the concatenated texts have different Zipf exponents. This method is being tested on the Brown corpus and also on newspaper texts in different languages. Testing shows a good estimate of the concatenation point. This method can be used in parallel with other text segmentation methods. AcknowledgmentsThe authors like to thank anonymous referees for their helpful and constructive comments and suggestions.Disclosure statementNo potential conflict of interest was reported by the author(s).Data availability statementWe used texts from open sources.Additional informationFundingThe work was supported by the Siberian Branch, Russian Academy of Sciences [FWNF-2022-0010].
期刊介绍:
The Journal of Quantitative Linguistics is an international forum for the publication and discussion of research on the quantitative characteristics of language and text in an exact mathematical form. This approach, which is of growing interest, opens up important and exciting theoretical perspectives, as well as solutions for a wide range of practical problems such as machine learning or statistical parsing, by introducing into linguistics the methods and models of advanced scientific disciplines such as the natural sciences, economics, and psychology.