How much habituation is enough? An R application for ad hoc content-related limits of acceptance in behavioral research.

Abstract

Evaluating test-retest reliability is of crucial importance for clinical decision-making and an integral component of studies in empirical sciences. In psychology, classical test theory focused on reliability coefficients, but research has shown that intraclass correlation coefficients and Cronbach's alpha have limited value in quantifying individual measurement error sources appropriately. Preliminary assumptions such as parallel readings and independency of error sources are violated by practice/habituation effects, i.e., test-retest systematic errors. Although awareness of the need for additional measurement quantifications to properly assess test-retest agreement has increased, many such quantifications are insufficiently implemented. Furthermore, one of the most important agreement analyses introduced by Bland and Altman in 1986 has interpretation limitations, as the limits of agreement are descriptive and not linked to content-based clinical decision-making, oftentimes causing inappropriate usage. To counteract these limitations and improve the implementation of agreement analyses in addition to more commonly used reliability coefficients, we introduce a software package that allows predefining acceptable measurement limits (limits of acceptance) to evaluate the sufficiency of task habituation to gauge individual error sources. The package implements a comprehensive test-retest evaluation in order to improve transparency of data quality reporting in psychological and cognitive research.