Too Noisy at the Bottom: Why Gries’ (2008, 2020) Dispersion Measures Cannot Identify Unbiased Distributions of Words

IF 0.7 2区文学 0 LANGUAGE & LINGUISTICS

Journal of Quantitative Linguistics Pub Date : 2023-02-02 DOI:10.1080/09296174.2023.2172711

Robert N. Nelson

{"title":"Too Noisy at the Bottom: Why Gries’ (2008, 2020) Dispersion Measures Cannot Identify Unbiased Distributions of Words","authors":"Robert N. Nelson","doi":"10.1080/09296174.2023.2172711","DOIUrl":null,"url":null,"abstract":"ABSTRACT Gries (2008, 2021) defined two dispersion measures able to alert corpus analysts to words that have a problematically limited distribution. Gries (2010, 2022) posited that these measures may additionally be relevant to language development research, as the learnability of a pattern may be predicted by the evenness of its distribution in corpora. However, both measures work by comparing vectors of observed and expected frequencies in partitioned corpora and this method cannot determine that a word is evenly distributed because it cannot distinguish the random noise inherent to an unbiased process from substantial non-random bias. An additional concern with the 2008 measure is raised: the 2008 measure is Manhattan distance scaled to the unit interval and, as such, it is extremely sensitive to the number of corpus parts because this choice sets the dimensionality of the measure space. In sum, this short analysis presents evidence that these measures should not be used to declare a pattern evenly distributed as neither can tell the difference between statistical noise and systematic bias.","PeriodicalId":45514,"journal":{"name":"Journal of Quantitative Linguistics","volume":"30 1","pages":"153 - 166"},"PeriodicalIF":0.7000,"publicationDate":"2023-02-02","publicationTypes":"Journal Article","fieldsOfStudy":null,"isOpenAccess":false,"openAccessPdf":"","citationCount":"0","resultStr":null,"platform":"Semanticscholar","paperid":null,"PeriodicalName":"Journal of Quantitative Linguistics","FirstCategoryId":"98","ListUrlMain":"https://doi.org/10.1080/09296174.2023.2172711","RegionNum":2,"RegionCategory":"文学","ArticlePicture":[],"TitleCN":null,"AbstractTextCN":null,"PMCID":null,"EPubDate":"","PubModel":"","JCR":"0","JCRName":"LANGUAGE & LINGUISTICS","Score":null,"Total":0}

引用次数: 0

Abstract

ABSTRACT Gries (2008, 2021) defined two dispersion measures able to alert corpus analysts to words that have a problematically limited distribution. Gries (2010, 2022) posited that these measures may additionally be relevant to language development research, as the learnability of a pattern may be predicted by the evenness of its distribution in corpora. However, both measures work by comparing vectors of observed and expected frequencies in partitioned corpora and this method cannot determine that a word is evenly distributed because it cannot distinguish the random noise inherent to an unbiased process from substantial non-random bias. An additional concern with the 2008 measure is raised: the 2008 measure is Manhattan distance scaled to the unit interval and, as such, it is extremely sensitive to the number of corpus parts because this choice sets the dimensionality of the measure space. In sum, this short analysis presents evidence that these measures should not be used to declare a pattern evenly distributed as neither can tell the difference between statistical noise and systematic bias.

查看原文本刊更多论文

底部噪音太大：为什么Gries（20082020）的分散度量不能识别单词的无偏分布

摘要Gries（20082021）定义了两种分散度量，能够提醒语料库分析师注意分布有限的单词。Gries（20102022）认为，这些措施可能与语言发展研究有关，因为模式的可学习性可以通过其在语料库中的分布均匀性来预测。然而，这两种测量方法都是通过比较分割语料库中观察到的频率和预期频率的向量来工作的，并且这种方法不能确定一个词是均匀分布的，因为它不能区分无偏过程固有的随机噪声和实质上的非随机偏误。对2008度量提出了另一个担忧：2008度量是缩放到单位区间的曼哈顿距离，因此，它对语料库部分的数量非常敏感，因为这种选择设置了度量空间的维度。总之，这篇简短的分析提供了证据，证明这些措施不应用于宣布均匀分布的模式，因为它们都不能区分统计噪声和系统偏差。

本文章由计算机程序翻译，如有差异，请以英文原文为准。

求助全文

约1分钟内获得全文求助全文

来源期刊

Journal of Quantitative Linguistics Multiple-

CiteScore

2.90

自引率

7.10%

发文量

期刊介绍： The Journal of Quantitative Linguistics is an international forum for the publication and discussion of research on the quantitative characteristics of language and text in an exact mathematical form. This approach, which is of growing interest, opens up important and exciting theoretical perspectives, as well as solutions for a wide range of practical problems such as machine learning or statistical parsing, by introducing into linguistics the methods and models of advanced scientific disciplines such as the natural sciences, economics, and psychology.