Automatic test case generation using natural language processing: A systematic mapping study

Abstract

Context:

Artificial intelligence (AI) has made significant progress in recent years, which has motivated its use in many disciplines and industrial domains, including software engineering, especially in the testing process, where many research efforts have been made. These studies focus on the automatic test case generation using natural language processing (NLP), an emerging branch of AI. Despite these efforts, the literature lacks a structured and systematic approach, since reported mappings and systematic literature reviews have limitations in their scope.

Objective:

This study aims to systematically organize and synthesize the existing literature to establish the state of the art in the automatic generation of test cases using NLP.

Methodology:

We conducted systematic mapping following Kai Petersen’s methodology, exploring five databases. The initial search yielded 1262 articles, of which 61 were selected. 16 thematic questions and 4 non-thematic questions were posed.

Results:

The findings reveal an increase in the number of articles published in journals starting in 2022. Among the most reported NLP techniques are POS tagging, dependency parsing and tokenization, implemented with tools such as Stanford Core NLP and NLTK. The reported approaches mostly achieved a medium level of automation, using natural and formal language requirements as main inputs. Only 9 articles explicitly mention the use of test case design techniques, such as boundary value analysis, equivalent class partitioning, state transition and decision tables.

Conclusions:

We systematically identified and organized the reported primary studies on the automatic or semi-automatic generation of software test cases applying NLP.