Message Evaluation Equivalence in Pre Versus Postproduction Tobacco Education Ads

Introduction

The U.S. Food and Drug Administration’s The Real Cost campaigns inform the public about the harms of tobacco use. The U.S. Food and Drug Administration has followed a 2-part approach to advertise formative testing to ensure sufficient audience receptivity: preproduction qualitative testing and postproduction quantitative testing. This process provides safeguards for ad performance but course corrections to message selection or development after the ads are already fully produced can be time and resource intensive. This study assesses equivalence in message evaluations between pre and postproduction campaign ads.

Methods

Data were collected in 2023 from an online panel survey of 804 youth aged 13–17 years, who currently use or are susceptible to use E-cigarettes or cigarettes. Participants were randomly assigned to a set of 4 preproduction or postproduction The Real Cost campaign ads. Participants completed a series of message evaluation measures after each ad. Authors tested for equivalence of mean message evaluation scores between the pre and postproduction versions of each ad using 2 one-sided t-tests and estimated concordance correlation coefficients. Data were analyzed in 2024.

Results

Aggregate ad performance scores between pre and postproduction versions were equivalent. Patterns varied for individual ads, with most demonstrating equivalence on some but not all message evaluation measures. Concordance correlation coefficients for pre and postproduction ads were positive and moderate in size.

Conclusions

Results suggest that message evaluation assessments of preproduction ads are likely a good indicator of message evaluation assessments for postproduction ads. Public health practitioners may consider incorporating quantitative copy testing earlier in the formative testing process to identify the most promising ad concepts.