A Multimetric Approach for Evaluation of ChatGPT-Generated Text Summaries

Abstract

This article investigates the summarization capabilities of ChatGPT, a language model seen as effectively shortening texts, employing a hypothesis-generating and explorative approach. Using a specific prompt, the study examines the expected lengths of generated summaries across various input word counts (IWC). A shortening ratio is introduced to describe these relationships, with identified dependencies on IWCs between 100 and 400 words. The study also explores coherence comparisons, highlighting that the ChatGPT-generated text is often evaluated as more coherent than the original. The article introduces a multimetric approach for the evaluation and discusses dependencies of best case summaries on different input word counts, providing insights into the model's performance characteristics.