LMTformer: facial depression recognition with lightweight multi-scale transformer from videos

Abstract

Depression will become the most common mental disorder worldwide by 2030. A number of models based on deep learning are proposed to help the clinicians to assess the severity of depression. However, two issues remain unresolved: (1) few studies have not considered to encode multi-scale facial behaviors. (2) the current studies have the high computational complexity to hinder the proposed architecture in clinical application. To mitigate the above issues, an end-to-end, lightweight, multi-scale transformer based architecture, termed LMTformer, for sequential video-based depression analysis (SVDA), is proposed. In LMTformer, which consists of the three models: coarse-grained feature extraction (CFE) block, light multi-scale transformer (LMST), final Beck Depression Inventory–II (BDI–II) predictor (FBP). In CFE, coarse-grained features are extracted for LMST. In LMST, a multi-scale transformer is proposed to model the potential local and global features at the different receptive field. In addition, multi-scale global feature aggregation (MSGFA) is also proposed to model the global features. For FBP, two fully connected layers are used. Our novel architecture LMTformer is evaluated on the AVEC2013/AVEC2014 depression databases, and the former dataset with a root mean square error (RMSE) of 7.75 and a mean absolute error (MAE) of 6.12 for AVEC2013, and a RMSE of 7.97 and a MAE of 6.05 for AVEC2014. On the LMVD dataset, we obtain the best performances with F1-score of 82.74%. Additionally, the model represents the excellent computational complexity while only need 0.95M parameters and 1.1G floating-point operations per second (FLOPs). Code will be available at: https://github.com/helang818/LMTformer/.