Introducing the Validation of Data Quality Indicators Through Re-Classification: The example of SQP and pretest surveys

Abstract

The present study introduces the concept of validating data quality indicators through re-classification. We use the term re-classification to mean the evaluation of how well an indicator detects the quality of different versions of a survey question for which the quality is known a priori. We illustrate its application with two examples. In both, we make use of 12 questions from prior experiments that manipulated text features of questions to create ‘low’ and ‘high’ quality versions of each question. In the first example, we coded each question version in SQP 2.1 to obtain indicators of validity, reliability, and quality. We compared these indicators between the two versions of each question to assess whether the SQP outcomes were sensitive to text features. In the second example, we used a pretest survey to obtain three indicators of survey quality: response latencies, item nonresponse, and consistency over time. Again, we compared these indicators between question versions to assess whether the indicators were sensitive to text features. We give recommendations for applying re-classification and an outlook for future research opportunities.