Extensive mutation for testing of word sense disambiguation models

Abstract

Context:

Word sense disambiguation (WSD) models are extensively utilized in various translation and question-answering systems. Assessing the WSD capability of these models aids in their improvement and enhances their dependability. Recently, researchers have introduced the concept of “mutation” to induce WSD errors in machine translation systems to evaluate their WSD ability.

Objective:

Inspired by the recent research, this study aims to extend types of mutations and check their potential application in testing WSD models to check whether these mutations can effectively provoke WSD errors.

Method:

We have designed and implemented nine innovative types of mutations focusing on words, phrases, and sentence structure for the sentence in WSD testing. Based on these extensive mutations, we have proposed a WSD testing framework that utilizes large language models to prompt sentence mutations and assess the disambiguation capability of WSD models.

Results:

In our research, we have conducted experiments using five widely recognized test sets for WSD tasks under five widely used WSD models. The experimental results show that (a) our testing framework can produce correct mutations for nine proposed mutations, and (b) the newly developed mutations have been shown to successfully trigger a substantial number of factual and unique WSD errors.

Conclusions:

The new types of mutations we designed can effectively be applied in mutation-based WSD testing. This suggests that by exploring more types of mutations, more WSD errors can be triggered.