Context-biased vs. structure-biased disambiguation of relative clauses in large language models

Abstract

This work investigates the processing behavior of large language models (LLMs) in sentences involving ambiguous relative clauses (RCs). We are particularly interested in unravelling attachment preferences of LLMs in disambiguating RCs (complementizer phrases CPs modifying a genitival phrase), which are either semantically (context-biased) or syntactically (structure-biased) associated to one of the preceding NP referents. A low interpretation of the RC occurs when it is joined to the local NP (low attachment). A high interpretation is provided when the RC modifies the distant NP (high attachment). We create a small dataset of parallel low- and high-attachment sentences. We use zero-shot prompting to evaluate a set of LLMs based on insights from psycholinguistic experiments. Our results show variability in the performance of some models that favor low attachment (semantically-related meanings in the CP) while other models can resolve ambiguity by choosing high-attachment (structure-biased CPs). The findings are discussed in light of directing future experimental studies to consider a comparative paradigm encompassing both multi-modal LLMs and human subjects.