PH-CBAM: A Parallel Hybrid CBAM Network with Multi-Feature Extraction for Facial Expression Recognition
Abstract
:1. Introduction
- The network design incorporates a multimodal extraction approach to capture diverse facial features during initial image feature extraction. By utilizing depth-separable convolution and attention modules, the model reduces parameter count and computational burden, making it suitable for resource-constrained environments without compromising performance.
- Through the combination of CBAM attention and sub-channel attention, the proposed model can automatically identify facial expressions. Parallel Hybrid attention enhances focus on critical facial information, improving key feature extraction for precise expression classification.
- Integrating residual networks and the MISH activation function, the network structure effectively addresses issues like gradient vanishing and zero negative gradient during residual transmission. This integration enhances information retention, suppresses irrelevant data, and facilitates information flow between crucial image details and the network model.
2. Methods
2.1. Multi-Feature Extraction Module
2.2. MISH Activation Function
2.3. Parallel Hybrid Attention
2.3.1. Channel Attention
2.3.2. Spatial Attention
2.3.3. Parallel Hybrid Attention Module
2.4. The Proposed Model
3. Experiment
3.1. Dataset Details
3.2. Result
3.2.1. The Performance of the Proposed Model
3.2.2. Ablation Experiments
4. Discussion and Conclusions
Author Contributions
Funding
Data Availability Statement
Conflicts of Interest
References
- Pan, X.; Ying, G.; Chen, G.; Li, H.; Li, W. A deep spatial and tempo-ral aggregation framework for video-based facial expression recognition. IEEE Access 2019, 7, 48807–48815. [Google Scholar] [CrossRef]
- Pham, T.-D.; Duong, M.-T.; Ho, Q.-T.; Lee, S.; Hong, M.-C. CNN-Based Facial Expression Recognition with Simultaneous Consideration of Inter-Class and Intra-Class Variations. Sensors 2023, 23, 9658. [Google Scholar] [CrossRef] [PubMed]
- Gaddam, D.K.R.; Ansari, M.D.; Vuppala, S.; Gunjan, V.K.; Sati, M.M. Human facial emotion detection using deep learning. In Lecture Notes in Electrical Engineering. In Proceedings of the ICDSMLA 2020: 2nd International Conference on Data Science, Machine Learning and Applications, Pune, India, 21–22 November 2020; pp. 1417–1427. [Google Scholar]
- Hossain, S.; Umer, S.; Rout, R.K.; Tanveer, M. Fine-grained image analysis for facial expression recognition using deep convolutional neural networks with bilinear pooling. Appl. Soft Comput. 2023, 134, 109997. [Google Scholar] [CrossRef]
- Tamantini, C.; di Luzio, F.S.; Hromei, C.D.; Cristofori, L.; Croce, D.; Cammisa, M.; Cristofaro, A.; Marabello, M.V.; Basili, R.; Zollo, L. Integrating physical and cognitive interaction capabilities in a robot-aided rehabilitation platform. IEEE Syst. 2023, 17, 1–12. [Google Scholar] [CrossRef]
- Poulose, A.; Reddy, C.S.; Kim, J.H.; Han, D.S. Foreground Extraction Based Facial Emotion Recognition Using Deep Learning Xception Model. In Proceedings of the 2021 Twelfth International Conference on Ubiquitous and Future Networks (ICUFN), Jeju Island, Republic of Korea, 17–20 August 2021; pp. 356–360. [Google Scholar]
- Zhu, X.; Ye, S.; Zhao, L.; Dai, Z. Hybrid attention cascade network for facial expression recognition. Sensors 2021, 21, 2003. [Google Scholar] [CrossRef]
- Cheng, Y.; Kong, D. CSINet: Channel–Spatial Fusion Networks for Asymmetric Facial Expression Recognition. Symmetry 2024, 16, 471. [Google Scholar] [CrossRef]
- Alonazi, M.; Alshahrani, H.J.; Alotaibi, F.A.; Maray, M.; Alghamdi, M.; Sayed, A. Automated Facial Emotion Recognition Using the Pelican Optimization Algorithm with a Deep Convolutional Neural Network. Electronics 2023, 12, 4608. [Google Scholar] [CrossRef]
- Huang, C. Combining convolutional neural networks for emotion recognition. In Proceedings of the 2017 IEEE MIT Undergraduate Research Technology Conference (URTC), Cambridge, MA, USA, 3–5 November 2017; pp. 1–4. [Google Scholar]
- Rajan, V.; Brutti, A.; Cavallaro, A. Is cross-attention preferable to self-attention for multi-modal emotion recognition. In Proceedings of the ICASSP 2022–2022 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), Virtual, 22–27 May 2022; pp. 4693–4697. [Google Scholar]
- Zhuang, X.; Liu, F.; Hou, J.; Hao, J.; Cai, X. Transformer-based interactive multi-modal attention network for video sentiment detection. Neural Process. Lett. 2022, 54, 1943–1960. [Google Scholar] [CrossRef]
- Kuhnke, F.; Rumberg, L.; Ostermann, J. Twostream aural-visual affect analysis in the wild. In Proceedings of the 15th IEEE International Conference on Automatic Face and Gesture Recognition (FG 2020), Buenos Aires, Argentina, 16–20 November 2020; pp. 600–605. [Google Scholar]
- Savchenko, A.V.; Savchenko, L.V.; Makarov, I. Classifying emotions and engagement in online learning based on a single facial expression recognition neural network. IEEE Trans. Affect. Comput. 2022, 13, 2132–2143. [Google Scholar] [CrossRef]
- Wang, Z.; Zeng, F.; Liu, S.; Zeng, B. OAENet: Oriented attention ensemble for accurate facial expression recognition. Pattern Recognit. 2021, 112, 107694. [Google Scholar] [CrossRef]
- Huang, Z.Y.; Chiang, C.C.; Chen, J.H.; Chen, Y.C.; Chung, H.L.; Cai, Y.P.; Hsu, H.C. A study on computer vision for facial emotion recognition. Sci. Rep. 2023, 13, 8425. [Google Scholar] [CrossRef] [PubMed]
- Vats, A.; Chadha, A. Facial Expression Recognition using Squeeze and Excitation-powered Swin Transformers. arXiv 2023, arXiv:2301.10906. [Google Scholar]
- Kollias, D. Abaw: Learning from synthetic data & multi-task learning challenges. In Proceedings of the European Conference on Computer Vision, Tel Aviv, Israel, 23–27 October 2022; pp. 157–172. [Google Scholar]
- Yao, L.; He, S.; Su, K.; Shao, Q. Facial expression recognition based on spatial and channel attention mechanisms. Wirel. Pers. Commun. 2022, 125, 1483–1500. [Google Scholar] [CrossRef]
- Liu, Z.; Lin, Y.; Cao, Y.; Hu, H.; Wei, Y.; Zhang, Z.; Lin, S.; Guo, B. Swin transformer: Hierarchical vision transformer using shifted windows. In Proceedings of the IEEE/CVF International Conference on Computer Vision (ICCV), Montreal, BC, Canada, 11–17 October 2021; pp. 10012–10022. [Google Scholar]
- Wen, Z.; Lin, W.; Wang, T.; Xu, G. Distract your attention: Multi-head cross attention network for facial expression recognition. Biomimetics 2023, 8, 199. [Google Scholar] [CrossRef] [PubMed]
- Zhang, X.; Yan, C. E-MobileNeXt: Face expression recognition model based on improved MobileNeXt. Optoelectron. Lett. 2024, 20, 122–128. [Google Scholar] [CrossRef]
- Mish, M.D. A self regularized non-monotonic activation function. arXiv 2019, arXiv:1908.08681. [Google Scholar]
- Guo, M.H.; Xu, T.X.; Liu, J.J.; Liu, Z.N.; Jiang, P.T.; Mu, T.J.; Zhang, S.H.; Martin, R.R.; Cheng, M.M.; Hu, S.M. Attention mechanisms in computer vision: A survey. Comput. Vis. Media 2022, 8, 331–368. [Google Scholar] [CrossRef]
- Zhu, X.; Cheng, D.; Zhang, Z.; Lin, S.; Dai, J. An empirical study of spatial attention mechanisms in deep networks. In Proceedings of the IEEE/CVF International Conference on Computer Vision, Seoul, Republic of Korea, 27 October–2 November 2019; pp. 6688–6697. [Google Scholar]
- Woo, S.; Park, J.; Lee, J.Y.; Kweon, I.S. Cbam: Convolutional block attention module. In Proceedings of the European Conference on Computer Vision (ECCV), Munich, Germany, 8–14 September 2018; pp. 3–19. [Google Scholar]
- Lucey, P.; Cohn, J.F.; Kanade, T.; Saragih, J.; Ambadar, Z.; Matthews, I. The extended cohn-kanade dataset (ck+): A complete dataset for action unit and emotion-specified expression. In Proceedings of the 2010 IEEE Computer Society Conference on Computer Vision and Pattern Recognition-Workshops, San Francisco, CA, USA, 13–18 June 2010; pp. 94–101. [Google Scholar]
- Zhang, X.; Chen, Z.; Wei, Q. Research and application of facial expression recognition based on attention mechanism. In Proceedings of the 2021 IEEE Asia-Pacific Conference on Image Processing, Electronics and Computers (IPEC), Dalian, China, 14–16 April 2021; pp. 282–285. [Google Scholar]
- Zhang, H.; Su, W.; Yu, J.; Wang, Z. Identity–expression dual branch network for facial expression recognition. IEEE Trans. Cogn. Dev. Syst. 2020, 13, 898–911. [Google Scholar] [CrossRef]
- Belmonte, R.; Allaert, B.; Tirilly, P.; Bilasco, I.M.; Djeraba, C.; Sebe, N. Impact of facial landmark localization on facial expression recognition. IEEE Trans. Affect. Comput. 2021, 14, 1267–1279. [Google Scholar] [CrossRef]
- Zhu, Q.; Zhuang, H.; Zhao, M.; Xu, S.; Meng, R. A study on expression recognition based on improved mobilenetV2 network. Sci. Rep. 2024, 14, 8121. [Google Scholar] [CrossRef]
- Sidhom, O.; Ghazouani, H.; Barhoumi, W. Three-phases hybrid feature selection for facial expression recognition. J. Supercomput. 2024, 80, 8094–8128. [Google Scholar] [CrossRef]
- Mukhopadhyay, M.; Dey, A.; Kahali, S. A deep-learning-based facial expression recognition method using textural features. Neural Comput. Appl. 2023, 35, 6499–6514. [Google Scholar] [CrossRef]
- Jiang, B.; Li, N.; Cui, X.; Liu, W.; Yu, Z.; Xie, Y. Research on Facial Expression Recognition Algorithm Based on Lightweight Transformer. Information 2024, 15, 321. [Google Scholar] [CrossRef]
Number of Layers | Parameter Quantity | Flops | |
---|---|---|---|
3 | 91.85% | 0.14 M | 70.95 M |
4 | 95.11% | 0.52 M | 88.52 M |
5 | 97.13% | 1.98 M | 108.77 M |
6 | 95.10% | 3.48 M | 116.44 M |
Dataset | Anger | Disgust | Fear | Happiness | Sadness | Surprise | Neutral | Contempt |
---|---|---|---|---|---|---|---|---|
CK+ | 45 | 59 | 25 | 69 | 28 | 83 | 593 | 18 |
BigFer2013 | 5202 | 755 | 5142 | 14,685 | 6345 | 4379 | 13,066 | |
Fer2013 | 4953 | 547 | 5121 | 8989 | 6077 | 4002 | 6198 |
Dataset | Validation Accuracy | Average Precision | Recall | F1 Score |
---|---|---|---|---|
FER2013 | 68.82 | 68.21 | 68.39 | 68.22 |
Bigfer 2013 | 72.31 | 71.82 | 71.66 | 71.69 |
CK+ | 97.13 | 96.16 | 95.11 | 95.20 |
Validation Accuracy | Average Precision | Recall | F1 Score | Flops | Params | T-Test | |
---|---|---|---|---|---|---|---|
removing all attention | 67.37 | 67.38 | 67.05 | 67.08 | 106.51 M | 1.92 M | 0.0004 |
RELU | 66.62 | 66.57 | 66.38 | 66.35 | 108.26 M | 1.98 M | 0.0006 |
CBAM | 68.07 | 67.55 | 67.50 | 67.40 | 106.81 M | 1.95 M | 0.0093 |
Split-channel attention | 67.92 | 67.67 | 67.79 | 67.62 | 108.47 M | 1.96 M | 0.0075 |
Ours | 68.82 | 68.21 | 68.39 | 68.22 | 108.77 M | 1.98 M |
Approach | Dataset | Accuracy (%) |
---|---|---|
CBAM [28] | CK+ | 95.1 |
IE-DBN [29] | CK+ | 96.02 |
CCFS+SVM [30] | CK+ | 96.05 |
Improved MobilenetV2 [31] | CK+ | 95.96 |
Model by Sidhom O et al. [32] | Fer2013 | 66.1 |
Self-Cure Net [33] | Fer2013 | 66.17 |
Improved MobileViT [34] | Fer2013 | 62.2 |
Improved MobilenetV2 [31] | Fer2013 | 68.62 |
Ours | CK+ | 97.13 |
Ours | Fer2013 | 68.82 |
Disclaimer/Publisher’s Note: The statements, opinions and data contained in all publications are solely those of the individual author(s) and contributor(s) and not of MDPI and/or the editor(s). MDPI and/or the editor(s) disclaim responsibility for any injury to people or property resulting from any ideas, methods, instructions or products referred to in the content. |
© 2024 by the authors. Licensee MDPI, Basel, Switzerland. This article is an open access article distributed under the terms and conditions of the Creative Commons Attribution (CC BY) license (https://creativecommons.org/licenses/by/4.0/).
Share and Cite
Liao, L.; Wu, S.; Song, C.; Fu, J. PH-CBAM: A Parallel Hybrid CBAM Network with Multi-Feature Extraction for Facial Expression Recognition. Electronics 2024, 13, 3149. https://doi.org/10.3390/electronics13163149
Liao L, Wu S, Song C, Fu J. PH-CBAM: A Parallel Hybrid CBAM Network with Multi-Feature Extraction for Facial Expression Recognition. Electronics. 2024; 13(16):3149. https://doi.org/10.3390/electronics13163149
Chicago/Turabian StyleLiao, Liefa, Shouluan Wu, Chao Song, and Jianglong Fu. 2024. "PH-CBAM: A Parallel Hybrid CBAM Network with Multi-Feature Extraction for Facial Expression Recognition" Electronics 13, no. 16: 3149. https://doi.org/10.3390/electronics13163149