Forging Better Axes: Evaluating and Improving the Reliability of Semantic Dimensions in Word Embeddings

IF 5.3 2区社会学 Q1 SOCIAL SCIENCES, MATHEMATICAL METHODS

Sociological Methods & Research Pub Date : 2026-09-18 DOI:10.1177/00491241261467930

Andrei Boutyline, Elle Johnston

{"title":"Forging Better Axes: Evaluating and Improving the Reliability of Semantic Dimensions in Word Embeddings","authors":"Andrei Boutyline, Elle Johnston","doi":"10.1177/00491241261467930","DOIUrl":null,"url":null,"abstract":"Word embeddings are a powerful tool for measuring cultural meaning using large text corpora. In sociology, applications commonly estimate the meanings of concepts by projecting keywords onto axes constructed from pairs of opposing anchor keywords. At present, there are no established metrics for assessing axis reliability from embedding data. We investigate three potentially relevant forms of reliability: parallelism between offsets, synonymy between same-endpoint anchors, and antonymy between opposed anchors. Using an original <mml:math xmlns:mml=\"http://www.w3.org/1998/Math/MathML\" display=\"inline\" overflow=\"scroll\"> <mml:mi>N</mml:mi> <mml:mo>=</mml:mo> <mml:mn>1</mml:mn> <mml:mo>,</mml:mo> <mml:mn>750</mml:mn> </mml:math> survey of respondents’ ratings of several hundred terms along 36 semantic dimensions plus existing ratings for 3 more, we find that parallelism robustly outperforms the other metrics in predicting the accuracy of 39,000 axes that operationalize these dimensions. Our decomposition of the parallelism–accuracy relationship between and within latent dimensions and anchor sets further shows that, while reliability is robustly related to axis accuracy at all levels, it provides a relatively weak guide for selecting specific keywords. Instead, the primary utility of parallelism may lie at higher levels of methodological decision-making: strongly identifying the most empirically tractable semantic dimension and, secondarily, suggesting the best embedding model for an anchor set.","PeriodicalId":21849,"journal":{"name":"Sociological Methods & Research","volume":"27 1","pages":""},"PeriodicalIF":5.3000,"publicationDate":"2026-09-18","publicationTypes":"Journal Article","fieldsOfStudy":null,"isOpenAccess":false,"openAccessPdf":"","citationCount":"0","resultStr":null,"platform":"Semanticscholar","paperid":null,"PeriodicalName":"Sociological Methods & Research","FirstCategoryId":"90","ListUrlMain":"https://doi.org/10.1177/00491241261467930","RegionNum":2,"RegionCategory":"社会学","ArticlePicture":[],"TitleCN":null,"AbstractTextCN":null,"PMCID":null,"EPubDate":"","PubModel":"","JCR":"Q1","JCRName":"SOCIAL SCIENCES, MATHEMATICAL METHODS","Score":null,"Total":0}

引用次数: 0

Abstract

Word embeddings are a powerful tool for measuring cultural meaning using large text corpora. In sociology, applications commonly estimate the meanings of concepts by projecting keywords onto axes constructed from pairs of opposing anchor keywords. At present, there are no established metrics for assessing axis reliability from embedding data. We investigate three potentially relevant forms of reliability: parallelism between offsets, synonymy between same-endpoint anchors, and antonymy between opposed anchors. Using an original N = 1 , 750 survey of respondents’ ratings of several hundred terms along 36 semantic dimensions plus existing ratings for 3 more, we find that parallelism robustly outperforms the other metrics in predicting the accuracy of 39,000 axes that operationalize these dimensions. Our decomposition of the parallelism–accuracy relationship between and within latent dimensions and anchor sets further shows that, while reliability is robustly related to axis accuracy at all levels, it provides a relatively weak guide for selecting specific keywords. Instead, the primary utility of parallelism may lie at higher levels of methodological decision-making: strongly identifying the most empirically tractable semantic dimension and, secondarily, suggesting the best embedding model for an anchor set.

查看原文本刊更多论文

锻造更好的轴：评价和提高词嵌入中语义维度的可靠性

词嵌入是使用大型文本语料库测量文化意义的有力工具。在社会学中，应用程序通常通过将关键词投射到由对立锚关键词对构建的轴上来估计概念的含义。目前，还没有建立的指标来评估轴的可靠性从嵌入数据。我们研究了三种可能相关的可靠性形式：偏移量之间的平行性，相同端点锚之间的同义性，以及对立锚之间的反义性。使用原始N = 1,750的调查，调查对象沿着36个语义维度对数百个术语进行评级，加上现有的3个语义维度的评级，我们发现并行性在预测实现这些维度的39,000个轴的准确性方面明显优于其他指标。我们对潜在维度和锚集之间和内部的平行度-精度关系的分解进一步表明，尽管可靠性在各个层面上都与轴精度有很强的相关性，但它对选择特定关键词的指导作用相对较弱。相反，并行性的主要用途可能在于更高层次的方法论决策：强烈识别最经验上可处理的语义维度，其次，为锚定集建议最佳嵌入模型。

本文章由计算机程序翻译，如有差异，请以英文原文为准。

求助全文

约1分钟内获得全文求助全文

来源期刊

Sociological Methods & Research Multiple-

CiteScore

16.30

自引率

3.20%

发文量

期刊介绍： Sociological Methods & Research is a quarterly journal devoted to sociology as a cumulative empirical science. The objectives of SMR are multiple, but emphasis is placed on articles that advance the understanding of the field through systematic presentations that clarify methodological problems and assist in ordering the known facts in an area. Review articles will be published, particularly those that emphasize a critical analysis of the status of the arts, but original presentations that are broadly based and provide new research will also be published. Intrinsically, SMR is viewed as substantive journal but one that is highly focused on the assessment of the scientific status of sociology. The scope is broad and flexible, and authors are invited to correspond with the editors about the appropriateness of their articles.