MT-Boost: A metamorphic-testing based training method for enhancing the robustness of deep neural network classifiers

Context:

In metamorphic testing (MT), a set of metamorphic relations (MRs) are identified to verify whether or not a trained deep neural network (DNN) can produce consistent performance when specific transformations are applied to its input. Most DNNs trained with existing methods often perform poorly with respect to MRs, thereby indicating that these DNNs are not robust.

Objective:

To improve DNN’s performance in the context of MT, a set of defined MRs is used to generate training inputs to retrain a DNN model. Our main objective is to develop a method to balance a DNN’s accuracy and robustness with less time consumption and having the capability to cater to multiple MRs.

Methods:

In this paper, we introduce our regularization-based method (known as MT-Boost), which uses reinforcement learning to search for the best way of using MRs to generate inputs and express them as loss function regularizers. When developing MT-Boost, we transform the robustness-improving problem into a reinforcement-learning agent’s training problem.

Results:

MT-Boost is evaluated on eight DNN models with four popular datasets. MT-Boost achieves the largest robustness improvement for each model and maintains relatively high accuracy performance when compared with seven other baseline methods. Our sensitivity analysis also shows the high stability performance of MT-Boost across four reinforcement-learning algorithms and other hyperparameters.

Conclusion:

Experimental results show that MT-Boost is effective and efficient for improving DNN’s robustness.