Automatic Evaluation of Machine Generated Feedback For Text and Image Data

Abstract

In this paper, a novel system, ‘AutoEvaINet,’ has been developed for evaluating machine-generated feedback in response to multimodal input containing text and images. A new metric, ‘Automatically Evaluated Relevance Score’ (AER Score), has also been defined to automatically compute the similarity between human-generated comments and machine-generatedfeedback. The AutoEvalNet's architecture comprises a pre-trained feedback synthesis model and the proposed feedback evaluation model. It uses an ensemble of Bidirectional Encoder Representations from Transformers (BERT) and Global Vectors for Word Representation (GloVe) models to generate the embeddings of the ground-truth comment and machine-synthesized feedback using which the similarity score is calculated. The experiments have been performed on the MMFeed dataset. The generated feedback has been evaluated automatically using the AER score and manually by having the human users evaluate the feedbackfor relevance to the input and ground-truth comments. The values of the AER score and human evaluation scores are in line, affirming the AER score's applicability as an automatic evaluation measure for machine-generated text instead of human evaluation.