Ten steps toward a better personality science - how quality may be rewarded more in research evaluation

Abstract

In February 2020, the Personality and Diagnostics Chapter (DPPD) of the German Psychological Society (DGPs) tasked this group of authors with outlining what should be considered “good personality science”, as a positive vision of how to improve the credibility of research in the field. We argue in favor of working toward greater consensus about (1) shared, important research goals, (2) standardized use of terminology, (3) standardized measurement practices, (4) standardized ways of pre-processing and analyzing data, and (5) shared views of the current state of theory and knowledge. All of these should help streamline the field considerably. We also argue in favor of (6) theory formalization, (7) pre-registration requirements for any confirmatory claims, (8) valuing replication attempts more (e.g., by reserving a quota of journal space for them), (9) planning for informative (e.g., well-powered) studies, and (10) making data, code, and materials open to the public by default. The current, quantity-based incentive structures in academia clearly stand in the way of implementing many of these practices, resulting in a research literature with sometimes questionable utility and/or integrity. As a solution, we propose a quality-based reward scheme that explicitly weights published research by its good science merits. Adoption of such a reward scheme may incur a significant decline in overall publication numbers, hopefully resulting in (a) an improved signal-to-noise ratio in the literature, and (b) more efficient allocation of resources (e.g., time) by researchers, who would be enabled to read more of what is being published, and to review each other’s work more carefully. Scientists need to be increasingly rewarded for doing good work, not just lots of work.