Arti-‘fickle’ intelligence: using LLMs as a tool for inference in the political and social sciences

Abstract

To promote the scientific use of large language models (LLMs), we suggest that researchers in the political and social sciences refocus on the scientific goal of inference. We suggest that this refocus will improve the accumulation of shared scientific knowledge about these tools and their uses in the social sciences. We discuss the challenges and opportunities related to scientific inference with LLMs, using validation of model output as an illustrative case for discussion. We then propose a set of guidelines related to establishing the failure and success of LLMs when completing particular tasks and discuss how to make inferences from these observations. Large language models are increasingly important in social science research. The authors provide guidance on how best to validate and use these models as rigorous tools to further scientific inference.