Who’s testing and who’s being tested: A cautionary tale of citizen science within applied animal behaviour science

Abstract

Citizen science has become a popular means to collect data in many research fields, including animal behaviour science. Using this approach has numerous potential benefits (e.g. larger sample sizes, decreased strain on resources, increasing the public’s engagement in science), but there are also issues and assumptions relating to data quality that should be considered. The aim of this study was to assess the reliability of a battery of game-like tests developed to assess behavioural flexibility in dogs. The battery was administered to samples of shelter dogs and owned dogs who had remained in homes long-term. From the two populations, four test groups were created: shelter dogs assessed by the principal investigator (n=85), shelter dogs assessed by shelter staff/volunteers (n=34), long-term owned dogs assessed by the principal investigator (n=21), and long-term dogs assessed by their owners (n=29). Shelter staff/volunteers and dog owners participated as “citizen scientists”. It is accepted that using less skilled “citizen” researchers increases the error within the data, but often claimed that this is offset by larger sample sizes. This implies that the increased error is random and not systematic. Whether who (citizen versus trained researcher) tested the group in a given context was associated with overall test item outcomes was evaluated. In two of the tests, the Alone Time Test and the Three-Toy Test, tester was found to be associated with outcomes. In the Alone Time Test, tester was associated with four items in the shelter dog samples and with three in the long-term owned dog samples. In the Three Toy Test, a series of subsequent items were contingent on the first, and the outcome of that initial item was related to tester. These results demonstrate that it is unwise to assume that the increased error from citizen science work is largely random and thus evened out by the use of a large sample of researchers. Unexpected systematic error might arise within citizen science projects so controls need to be introduced to test for these effects, so that unsound assumptions are not made. There may also be relatively unique factors, beyond researcher objectivity, to consider when using a citizen science approach to study domestic dogs, such as the bidirectional influence on behaviour of any emotional bond between the owner or shelter staff/volunteer and the dog being tested. These results highlight the need for quality checks and preliminary analysis to ensure the identification of any relevant tester effects.