Metamorphic testing for textual and visual entailment: A unified framework for model evaluation and explanation

Abstract

Context:

Textual entailment (TE) and visual entailment (VE) serve as the basis for a broad spectrum of tasks in natural language processing and vision–language modeling. However, although being extensively studied, both TE and VE models exhibit several quality issues. Additionally, their black-box nature hampers the understanding of their behaviors, making it unclear why the model fails to correctly predict entailment relationships. Consequently, there is a pressing need for methods that can effectively evaluate and explain TE and VE models.

Objective:

This study aims to develop a unified approach for detecting and interpreting failures, in both TE and VE models.

Methods:

We propose a metamorphic testing-based approach for evaluating and explaining both TE and VE models. The central aspect of our approach lies in the proposed three metamorphic relations, which are generic to both TE and VE, and also preserve specific associations among relevant inputs. The proposed approach conducts metamorphic testing to detect failures in TE and VE models. When a failure is revealed, it further performs a post-hoc analysis within the relevant group of inputs to identify information that is critical for the detected failure.

Results:

Experimental results demonstrate the effectiveness of the proposed approach in failure detection and also confirm its potential to provide useful information to pinpoint the root causes of detected failures.

Conclusion:

This study presents a general metamorphic testing approach for both TE and VE. It also demonstrates that, with specifically designed metamorphic relations, metamorphic testing can serve as an effective basis for model explanation.