Probing Numeracy and Logic of Language Models of Code

Machine learning techniques have found a widespread use in the software engineering community. In particular, language models (LMs) trained on code form the backbone of a majority of these applications, spanning tasks such as code completion, summarization, refactoring, execution prediction, and test generation. These tasks require reasoning about both the syntax and semantics of code. Recent work has shown that language models learn to capture the syntactic properties of code, but it is unclear to what extent they can reason about the semantics of code. In this work, we explore the ability of 3 language models of code to reason about a specific kind of semantics: numerical and logical properties of code. We propose several probing tasks to test the numerical and logical reasoning abilities of these models. We find that the models we explore - CodeBERT, GraphCodeBERT and CodeGen do indeed learn many numerical and logical properties of code, such as finding maximum in a list of numbers, comparing numbers, evaluating boolean expressions and representing numbers. They do not perform as well on complex tasks such as evaluating arithmetic expressions and substituting variables in such expressions. Our results indicate that while these models hold promise, there is a lot of room for improvement of their numeric and logical reasoning abilities.