Quantitative methods in syntax/semantics research: A response to Sprouse and Almeida (2013)

Sprouse and Almeida (S&A) present quantitative results that suggest that intuitive judgments utilised in syntax research are generally correct in two-condition comparisons: the sentence type that is presented as “good/grammatical” is usually rated better than the sentence type that is presented as “bad/ungrammatical” in controlled experiments. Although these evaluations of intuitive relative judgments are valuable, they do not justify the use of nonquantitative linguistic methods. We argue that objectivity is a universal value in science that should be adopted by linguistics. In addition, the reliability measures that S&A report are not sufficient for developing sophisticated linguistic theories. Furthermore, quantitative methods yield two additional benefits: consistency of judgments across many pairs of judgments; and an understanding of the relative effect sizes across sets of judgments. We illustrate these points with an experiment that demonstrates five clear levels of acceptability. Finally, we observe that S&A's experiments—where only two authors evaluated 10 years' worth of journal articles and one standard textbook within a few months—further emphasise one of our critical original points: conducting behavioural experiments is in many respects easy and fast with the advent of online research tools like Amazon's Mechanical Turk. Given the current ease of performing quantitative experiments (using a platform like Mechanical Turk) and the clear limitations of not doing so, linguistic hypotheses should be evaluated quantitatively whenever it is feasible.