Pre-Testing Establishment Surveys: Moving Beyond the Lab

Abstract

Cognitive interviewing is a widely used method for ensuring that survey questions produce valid data. In household surveys, cognitive interviewing is typically conducted by experienced survey methodologists in laboratory settings. Agencies with laboratories can conduct multiple cognitive interviews in a day using small staffs of experienced methodologists, thus reducing costs and expediting the research process. However, when populations are hard to reach, methodologists must travel to participants. In these situations, cognitive interviewing can be a very time- consuming and expensive method. In surveys of establishments, cognitive interviewing is almost always conducted outside of a laboratory setting. Researchers must travel to multiple establishment sites that are not always geographically close to one another. At the United States Department of Agriculture’s National Agricultural Statistics Service (NASS), cognitive interviewing has been limited due to the rural nature and geographic distribution of our survey universe, namely, farm and ranch operators. In 2014, NASS expanded its cognitive interviewing program by training production staff in state and regional offices to conduct cognitive interviews, under the direction of survey methodologists at NASS’s headquarters office. This paper discusses how the expanded cognitive interview training program was planned and developed; describes experiences and lessons learned; and outlines NASS’s plan to continue using its nontraditional cognitive interviewing program.