The reliability and validity of problem generation tests: A meta-analysis with implications for problem finding and creativity

Abstract

Problem finding (PF) is a very important part of the creative process. This suggests that careful measurement is necessary. Paper and pencil Problem Generation (PG) tests were developed nearly 40 years ago in an attempt to assess the potential for PF. The reliability and validity of these instruments have yet to be statistically examined with meta-analytic methods. This meta-analysis examined 19 previous empirical investigations of PG tests to examine internal reliability (k = 43, N = 2029), convergent validity (k = 125, N = 2573), and discriminant validity (k = 26, N = 2145). Analyses indicated that the overall random-effects weighted mean reliability was α = 0.816, t(42) = 26.419, p < .001 (95 % CI: 0.786, 0.842), indicating good internal consistency. A second analysis produced a random-effects weighted mean validity coefficient, which indicated moderate agreement between PG tests and other creativity measures, such as divergent thinking and creative achievement (r = 0.463, t(124) = 13.994, p < .001 (95 % CI: 0.406, 0.518). The mean correlation among the scores produced by PG tests (i.e., fluency, flexibility, and originality) was 0.590, t(25) = 4.714, p < .001 (95 % CI: 0.364, 0.750), which indicates a moderate level of discriminant validity of the indices. Data were heterogeneous in all three analyses, but moderator analyses did not explain significant amounts of variation. This is the first study to examine the reliability and validity of the PG tests using meta-analytic methods. These showed that the test is, in several ways, a valid and reliable tool for assessing PF ability.