A Multi-CNN Fusion Approach for Improved Facial Expression Recognition
Main Article Content
Abstract
Facial expression recognition (FER) is crucial in expressing a human's emotional state. Emotions and expressions on the human face are information that computers and deep learning can recognize. FER is a current research topic due to the recent advancement and use of human-computer interaction systems. The recognition of facial expressions is challenging for current deep learning models due to the variable brightness, background, pose, etc. of the face images. This work presents an improved learning method based on a feature fusion convolutional neural network (CNN) to recognize seven facial expressions. First, we trained three different base model structures and then fused the features of the pre-trained base models 1 and 2 from the final fully connected layers to obtain a fusion network. Second, the fusion network was trained and used with the third pre-trained base model for performance evaluation. We applied the max-score fusion and mean-score fusion techniques between the fusion network and the third pre-trained base model to predict the output class. Our results indicated that the proposed method outperforms the base models in all metrics and achieves a classification accuracy of 69.03% on the facial expression dataset.
Downloads
Article Details
Section
How to Cite
References
Ahmed, A., Yousif, H., Kays, R. and He, Z., 2019. Semantic region of interest and species classification in the deep neural network feature domain. Ecological Informatics, 52, pp. 57–68. https://doi.org/10.1016/j.ecoinf.2019.05.006
Alkababji, A. M. and Abd, S. R., 2021. Half-face-based recognition using principal component analysis. Indonesian Journal of Electrical Engineering and Computer Science, 22, pp. 1404–1410. https://doi.org/10.11591/ijeecs.v22.i3.pp1404-1410
Aly, M., 2025. Revolutionizing online education: Advanced facial expression recognition for real-time student progress tracking via deep learning model. Multimedia Tools and Applications, 84, pp. 12575–12614. https://doi.org/10.1007/s11042-024-19392-5
Avanija, J., Madhavi, K. R., Sunitha, G., Sangapu, S. C., and Raju, S., 2022. Facial expression recognition using convolutional neural network. In Proceedings of 2022 First International Conference on Artificial Intelligence Trends and Pattern Recognition (ICAITPR). pp. 1–7. IEEE. https://doi.org/10.1109/icaitpr51569.2022.9844221
Cao, J., Hu, C., Kong, L. and Yu, Z., 2023. Expression recognition based on a multi-level multi-model fusion deep convolutional neural network. Highlights in Science, Engineering and Technology, 34, pp. 232–237. https://doi.org/10.54097/hset.v34i.5477
Cavallo, F., Semeraro, F., Fiorini, L., Magyar, G., Sinčák, P. and Dario, P., 2018. Emotion modelling for social robotics applications: a review. Journal of Bionic Engineering, 15, pp. 185–203. https://doi.org/10.1007/s42235-018-0015-y
Chouhayebi, H., Mahraz, M. A., Riffi, J. and Tairi, H., 2024. A dynamic fusion of features from deep learning and the HOG-TOP algorithm for facial expression recognition. Multimedia Tools and Applications, 83, pp. 32993–33017. https://doi.org/10.1007/s11042-023-16779-8
Dhall, A., Asthana, A., Goecke, R. and Gedeon, T., 2011. Emotion recognition using PHOG and LPQ features. In Proceedings of 2011 IEEE International Conference on Automatic Face & Gesture Recognition (FG). pp. 878–883. IEEE. https://doi.org/10.1109/fg.2011.5771366
Díaz, A. A. O., Tamayo, S. C., De Oliveira, D. N. and Abensur, G. A., 2024. Models for real-time emotion classification: FER-2013 dataset. In Proceedings of Intelligent Systems Conference. Springer Nature Switzerland. pp. 289–304. https://doi.org/10.1007/978-3-031-66431-1_19
Ge, H., Zhu, Z., Dai, Y., Wang, B. and Wu, X., 2022. Facial expression recognition based on deep learning. Computer Methods and Programs in Biomedicine, 215, P. 106621. https://doi.org/10.1016/j.cmpb.2022.106621
He, K., Zhang, X., Ren, S. and Sun, J., 2016. Deep residual learning for image recognition. In Proceedings of the IEEE conference on computer vision and pattern recognition (CVPR). pp. 770–778. https://doi.org/10.1109/cvpr.2016.90
Ji, Y., Hu, Y., Yang, Y., Shen, F. and Shen, H. T., 2019. Cross-domain facial expression recognition via an intra-category common feature and inter-category distinction feature fusion network. Neurocomputing, 333, pp. 231–239. https://doi.org/10.1016/j.neucom.2018.12.037
Krizhevsky, A., Sutskever, I. and Hinton, G. E., 2017. ImageNet classification with deep convolutional neural networks. Communications of the ACM, 60, pp. 84–90. https://doi.org/10.1145/3065386
Lan, X., Xue, J., Qi, J., Jiang, D., Lu, K. and Chua, T.-S., 2025. Expllm: Towards chain of thought for facial expression recognition. IEEE Transactions on Multimedia, 27, pp. 3069-3081. https://doi.org/10.1109/tmm.2025.3557704
Liang, D., Liang, H., Yu, Z. and Zhang, Y., 2020. Deep convolutional BiLSTM fusion network for facial expression recognition. The Visual Computer, 36, pp. 499–508. https://doi.org/10.1007/s00371-019-01636-3
Liu, W., Song, C., Wang, Y. and Jia, L., 2012. Facial expression recognition based on gabor features and sparse representation. In Proceedings of 2012 12th International Conference on Control Automation Robotics & Vision (ICARCV). pp. 1402–1406. IEEE. https://doi.org/10.1109/ICARCV.2012.6485394
Mollahosseini, A., Chan, D. and Mahoor, M. H., 2016. Going deeper in facial expression recognition using deep neural networks. In Proceedings of 2016 IEEE Winter Conference on Applications of Computer Vision (WACV). pp. 1–10. IEEE. https://doi.org/10.1109/WACV.2016.7477450
Ni, R., Yang, B., Zhou, X., Cangelosi, A. and Liu, X., 2022. Facial expression recognition through cross-modality attention fusion. IEEE Transactions on Cognitive and Developmental Systems, 15, pp. 175–185. https://doi.org/10.1109/tcds.2022.3150019
Niu, B., Gao, Z. and Guo, B., 2021. Facial expression recognition with LBP and ORB features. Computational Intelligence and Neuroscience, 2021, P. 8828245. https://doi.org/10.1155/2021/8828245
Pierre, C. and Aaron, C., FER-2013 Dataset. Available at: https://www.kaggle.com/c/challenges-in-representation-learning-facial-expression-recognition-challenge/data (Accessed: 17 May 2026).
Revina, I. M. and Emmanuel, W. S., 2021. A survey on human face expression recognition techniques. Journal of King Saud University-Computer and Information Sciences, 33, pp. 619–628. https://doi.org/10.1016/j.jksuci.2018.09.002
Russakovsky, O., Deng, J., Su, H., Krause, J., Satheesh, S., Ma, S., Huang, Z., Karpathy, A., Khosla, A. and Bernstein, M., 2015. Image net large scale visual recognition challenge. International Journal of Computer Vision, 115, pp. 211–252. https://doi.org/10.1007/s11263-015-0816-y
Salloum, S. A., Alomari, K. M., Alfaisal, A. M., Aljanada, R. A. and Basiouni, A., 2025. Emotion recognition for enhanced learning: using AI to detect students’ emotions and adjust teaching methods. Smart Learning Environments, 12, P. 21. https://doi.org/10.1186/s40561-025-00374-5
Shan, C., Gong, S. and Mcowan, P. W., 2009. Facial expression recognition based on local binary patterns: A comprehensive study. Image and Vision Computing, 27, pp. 803–816. https://doi.org/10.1016/j.imavis.2008.08.005
Shao, J. and Qian, Y., 2019. Three convolutional neural network models for facial expression recognition in the wild. Neurocomputing, 355, pp. 82–92. https://doi.org/10.1016/j.neucom.2019.05.005
Sharma, P., Esengönül, M., Khanal, S. R., Khanal, T. T., Filipe, V. and Reis, M. J., 2018. Student concentration evaluation index in an e-learning context using facial emotion analysis. In Proceedings of International Conference on Technology and Innovation in Learning, Teaching and Education. pp. 529–538. Springer. https://doi.org/10.1007/978-3-030-20954-4_40
Sun, N., Li, Q., Huan, R., Liu, J. and Han, G., 2019. Deep spatial-temporal feature fusion for facial expression recognition in static images. Pattern Recognition Letters, 119, pp. 49–61. https://doi.org/10.1016/j.patrec.2017.10.022
Szegedy, C., Liu, W., Jia, Y., Sermanet, P., Reed, S., Anguelov, D., Erhan, D., Vanhoucke, V. and Rabinovich, A., 2015. Going deeper with convolutions. In Proceedings of the IEEE conference on computer vision and pattern recognition (CVPR). pp. 1–9. https://doi.org/10.1109/CVPR.2015.7298594
Tan, M. and Le, Q., 2019. EfficientNet: rethinking model scaling for convolutional neural networks. In Proceedings of International Conference on Machine Learning (ICML). pp. 6105–6114. PMLR. https://doi.org/10.48550/arXiv.1905.11946
Tang, X., Gong, Y., Xiao, Y., Xiong, J. and Bao, L., 2025. Facial expression recognition for probing students’ emotional engagement in science learning. Journal of Science Education and Technology, 34, pp. 13–30. https://doi.org/10.1007/s10956-024-10143-7
Tang, Y., Zhang, X. M. and Wang, H., 2018. Geometric-convolutional feature fusion based on learning propagation for facial expression recognition. IEEE Access, 6, pp. 42532–42540. https://doi.org/10.1109/access.2018.2858278
Wang, W., Xu, K., Niu, H. and Miao, X., 2020. Emotion recognition of students based on facial expressions in online education based on the perspective of computer simulation. Complexity, 2020, P. 4065207. https://doi.org/10.1155/2020/4065207
Xie, S. and Hu, H., 2018. Facial expression recognition using hierarchical features with deep comprehensive multipatches aggregation convolutional neural networks. IEEE Transactions on Multimedia, 21, pp. 211–220. https://doi.org/10.1109/TMM.2018.2844085
Xu, L., Yang, A., Fei, M. and Zhou, W., 2018. Brief technical analysis of facial expression recognition. In Proceedings of International Conference on Intelligent Computing for Sustainable Energy and Environment. pp. 302–310. Springer. https://doi.org/10.1007/978-981-13-2384-3_28
Yu, M., Zheng, H., Peng, Z., Dong, J. and Du, H., 2020. Facial expression recognition based on a multi-task global-local network. Pattern Recognition Letters, 131, pp. 166–171. https://doi.org/10.1016/j.patrec.2020.01.016
Zahara, L., Musa, P., Wibowo, E. P., Karim, I. and Musa, S. B., 2020. The facial emotion recognition (FER-2013) dataset for prediction system of micro-expressions face using the convolutional neural network (CNN) algorithm based Raspberry Pi. In Proceedings of 2020 Fifth International Conference on Informatics and Computing (ICIC). pp. 1–9. IEEE. https://doi.org/10.1109/ICIC50835.2020.9288560
Zhu, M. and Wei, L., 2024. A multi-channel convolutional neural network based on attention mechanism fusion for facial expression recognition. Applied Mathematics and Nonlinear Sciences, 9, P. 2. https://doi.org/10.2478/amns.2023.1.00084