{"title":"Self supervised learning and the poverty of the stimulus","authors":"Csaba Veres , Jennifer Sampson","doi":"10.1016/j.datak.2023.102208","DOIUrl":null,"url":null,"abstract":"<div><p><em>Diathesis alternations</em> are the possible expressions of the arguments of verbs in different, systematically related subcategorization frames. Semantically similar verbs such as <em>spill</em> and <em>spray</em> can behave differently with respect to the alternations they can participate in. For example one can “spill/spray water on the plant”, but while one can “spray the plant with water”, it is odd to say “spill the plant with water”. “Spray” is a verb which can alternate between syntactic frames while “spill” is not alternating. How human speakers learn the difference between such verbs is not clearly understood, because the primary linguistic data (PLD) they receive does not appear sufficient to infer the knowledge required for adult competence. More generally the poverty of the stimulus (POS) hypothesis states that the PLD is not sufficient for a learner to infer full adult competence of language. That is, learning relies on prior constraints introduced by the language faculty. We tested state-of-the-art machine learning models trained by self supervision, and found some evidence that they could in fact learn the correct pattern of acceptability judgement in the <em>locative alternation</em>. However, we argued that this was partially a result of fine-tuning which introduced <em>negative evidence</em> into the learning data, which facilitated <em>shortcut learning</em>. Large language models (LLMs) cannot learn some linguistic facts from normal language data, but they can compensate to some extent by learning spurious correlated features when negative feedback is introduced during the training cycle.</p></div>","PeriodicalId":55184,"journal":{"name":"Data & Knowledge Engineering","volume":null,"pages":null},"PeriodicalIF":2.7000,"publicationDate":"2023-09-01","publicationTypes":"Journal Article","fieldsOfStudy":null,"isOpenAccess":false,"openAccessPdf":"","citationCount":"0","resultStr":null,"platform":"Semanticscholar","paperid":null,"PeriodicalName":"Data & Knowledge Engineering","FirstCategoryId":"94","ListUrlMain":"https://www.sciencedirect.com/science/article/pii/S0169023X2300068X","RegionNum":3,"RegionCategory":"计算机科学","ArticlePicture":[],"TitleCN":null,"AbstractTextCN":null,"PMCID":null,"EPubDate":"","PubModel":"","JCR":"Q3","JCRName":"COMPUTER SCIENCE, ARTIFICIAL INTELLIGENCE","Score":null,"Total":0}
引用次数: 0
Abstract
Diathesis alternations are the possible expressions of the arguments of verbs in different, systematically related subcategorization frames. Semantically similar verbs such as spill and spray can behave differently with respect to the alternations they can participate in. For example one can “spill/spray water on the plant”, but while one can “spray the plant with water”, it is odd to say “spill the plant with water”. “Spray” is a verb which can alternate between syntactic frames while “spill” is not alternating. How human speakers learn the difference between such verbs is not clearly understood, because the primary linguistic data (PLD) they receive does not appear sufficient to infer the knowledge required for adult competence. More generally the poverty of the stimulus (POS) hypothesis states that the PLD is not sufficient for a learner to infer full adult competence of language. That is, learning relies on prior constraints introduced by the language faculty. We tested state-of-the-art machine learning models trained by self supervision, and found some evidence that they could in fact learn the correct pattern of acceptability judgement in the locative alternation. However, we argued that this was partially a result of fine-tuning which introduced negative evidence into the learning data, which facilitated shortcut learning. Large language models (LLMs) cannot learn some linguistic facts from normal language data, but they can compensate to some extent by learning spurious correlated features when negative feedback is introduced during the training cycle.
期刊介绍:
Data & Knowledge Engineering (DKE) stimulates the exchange of ideas and interaction between these two related fields of interest. DKE reaches a world-wide audience of researchers, designers, managers and users. The major aim of the journal is to identify, investigate and analyze the underlying principles in the design and effective use of these systems.