Examining the Idea Density and Semantic Distance of Responses Given by AI to Tests of Divergent Thinking

Abstract

Research suggests that generative AI (GAI) responds to divergent thinking (DT) prompts with multiple ideas, some of which seem to be original. The present investigation administered 55 DT tasks to three GAI services (Bard, GPT 3.5, and GPT 4.0). Instead of examining individual responses, an Idea Density algorithm was used to assess the output. This algorithm quantifies the ideas within responses, controlling for the number of words. A subset of the DT tests administered to the GAI were also scored for Semantic Distance, which estimates originality. Results indicated that the three GAI models differed in the Idea Density of the output. There were also significant differences between Realistic and Nonrealistic DT tasks. As has been the case in human samples, directions given when the GAI received the prompts also had a significant impact, with more Idea Density following directions that explicitly prompted original responses. Adjusted scores removed all verbiage in the output, which did not actually address the questions conveyed by the prompts. These corrected scores shared approximately 50% of the variance with the uncorrected “raw” responses, implying that the typical output of GAI is not always relevant. This was interpreted in the context of the standard definition of creativity, which emphasizes effectiveness, as well as originality.