Diagnosticity of perceived message effectiveness in campaign message pretesting: multilevel analysis of the between-message correlation and message-pair standing comparisons

Abstract

Whether perceived message effectiveness (PME) can be diagnostic for the differences of actual message effect (AME) in campaign message pretesting and how the diagnosticity of PME should be tested have been controversial. To address these issues, we conducted a survey involving 19 campaign messages (N = 760) and statistically analyzed the multilevel relationships among the between-message, within-message, and disaggregated across-message PME–AME correlations. From our analysis, we advocate for a multilevel analysis of between-message PME–AME correlation as the optimal method for testing PME’s diagnosticity. We also evaluated O’Keefe’s (2018) message-pair standing comparison method and suggested using statistical significance tests to examine PME differences between messages. The between-message PME–AME correlations were strongly positive (greater than 0.824), and PME standing corresponded with AME standing in 94.5% of the message-pairs. Our findings confirm that PME is diagnostic for AME.