Facial expression analysis using convolutional neural network for drug-naive and chronic schizophrenia.

Abstract

Objective: Facial images have been shown to convey mental conditions as clinical symptoms. This study aimed to use facial images to detect patients with drug-naive schizophrenia (DN-SCZ) or chronic schizophrenia (C-SCZ) from healthy controls (HCs), and to investigate differences in facial expressions among these 3 groups, as well as relationships between facial expressions and psychiatric symptoms.

Methods: We recruited 45 DN-SCZ patients, 106 C-SCZ patients and 101 HCs for the study, and videotaped their facial expressions through a fixed experimental paradigm. The video data was converted to facial images and divided into two sets: one for training a group classification-convolutional neural network (CNN) with the classification of DN-SCZ patient, C-SCZ patient and HC as output, and the other for evaluating classification results of the group classification-CNN. Subsequently, we extracted and evaluated 300 labeled facial images for each basic facial expression. These labeled images were employed to train separate facial expression-CNNs for each group (DN-SCZ, C-SCZ, and HCs). All facial images from the videos were then processed by their facial expression-CNNs to output the most probable facial expressions. The psychiatric symptoms were assessed using the Positive and Negative Syndrome Scale. Statistical analyses were conducted on the predicted facial expressions to identify differences among the groups, and to examine relationships between the predicted facial expressions and the clinical data of DN/C-SCZ patients.

Results: The group classification-CNN achieved an accuracy of 90.99% in correctly classifying participants based on facial images. The 3 facial expression-CNNs achieved accuracies of 95.95%, 87.23%, and 92.11% in predicting 8 basic facial expressions within the 3 groups. Facial images of HCs were rated higher in valence, arousal and attractiveness, but lower in deviation from normal face than those of DN/C-SCZ patients. Happy images of DN-SCZ patients were rated lower in valence and arousal than those of C-SCZ patients, while their angry images were rated higher in arousal, attractiveness and deviation from normal images than those of C-SCZ patients. Within the fixed experimental paradigm, DN-SCZ patients exhibited sadder, more surprised expressions, while displaying fewer happy, angry and disgusted expressions, statistical metrics of their fearful and angry expressions were correlated with their total positive symptom score and total general psychopathology score, respectively. C-SCZ patients exhibited happier, more content, angry and neutral expressions, while showing fewer surprised expressions, no significant relationships were observed between their facial expressions and clinical data.

Conclusions: Facial expressions can potentially serve as indicative signs for detecting DN-SCZ and C-SCZ patients. There are objective differences in certain facial expressions among the 3 groups, and certain facial expressions in DN-SCZ patients are associated with some of their psychiatric symptoms.