Citation :

Sunny Bagga, Hemant Makwana, "Facial Expression Recognition System Using a Hybrid CNN and LSTM Model," International Journal of Engineering Trends and Technology (IJETT), vol. 74, no. 1, pp. 317-332, 2026. Crossref, https://doi.org/10.14445/22315381/IJETT-V74I1P125

Abstract

Facial expression analysis represents one of the most challenging and exciting problems in the fields of machine applications and human–computer interaction. The recognition of facial expressions has long been an important domain of academic interest. While various identification processes behind the recognition activities are mainly based on the identification of emotional attributes, intra-class variation can considerably inhibit the identification of the same. It is observed that static imagery does not capture the facial traits quite well. In the given paper, the authors present a novel architecture based on neutral terms for categorizing seven basic expressions: happiness, anger, neutrality, fear, disgust, sadness, and Surprise. FER2013 is used to present comprehensive experimental results. The proposed design is based on CNN and LSTM. For improving the accuracy of the FER system, a hybrid approach involving CNN and LSTM is presented. This hybrid approach involves two steps: first, CNN learns on the FER2013 dataset for the extraction of visual features, after which LSTM is used in modeling the temporal dependencies between sequences of images and their corresponding emotions. The outputs of the architecture are evaluated by using a confusion matrix and are compared with other relevant architectures. Experimental results on publicly available datasets indicate that the method proposed in this work outperforms modern techniques.

Keywords

Convolution Neural Network (CNN), Deep Learning, Facial Expression Recognition, Long Short-Term Memory (LSTM).

References

[1] Chengxu Liang et al., “Facial Expression Recognition using LBP and CNN Networks Integrating Attention Mechanism,” 2023 Asia Symposium on Image Processing (ASIP), Tianjin, China, pp. 1-6, 2023.
[CrossRef] [Google Scholar] [Publisher Link]

[2] Farshad Safavi, Kulin Patel, and Ramana Kumar Vinjamuri, “Towards Efficient Deep Learning Models for Facial Expression Recognition using Transformers,” 2023 IEEE 19^th International Conference on Body Sensor Networks (BSN), Boston, MA, USA,  pp. 1-4, 2023.
[CrossRef] [Google Scholar] [Publisher Link]

[3] Hadeel Mohammed, Mohammed Nasser Hussain, and Faiz Al Alawy, “Facial Expression Recognition: Machine Learning Algorithms and Feature Extraction Techniques,” Al-Iraqia Journal of Scientific Engineering Research, vol. 2, no. 2, pp. 23-28, 2023.
[CrossRef] [Google Scholar] [Publisher Link]

[4] M. Priya et al., “Automatic Emotion Detection using SVM-Based Optimal Kernel Function,” 2024 11^th International Conference on Computing for Sustainable Global Development (INDIACom), New Delhi, India, pp. 1607-1611, 2024.
[CrossRef] [Google Scholar] [Publisher Link]

[5] Qisheng Wang, Xiaofei Yan, and Yanqiu Wang, “Research on Deep Learning-based Facial Expression Recognition and its Application in Online Learning State Monitoring,” 3^rd International Conference on Advanced Algorithms and Signal Image Processing (AASIP 2023), Kuala Lumpur, Malaysia, vol. 12799, pp. 459-464, 2023.
[CrossRef] [Google Scholar] [Publisher Link]            

[6] Rasha Friji et al., “Geometric Deep Neural Network using Rigid and Non-Rigid Transformations for Landmark-Based Human Behavior Analysis,” IEEE Transactions on Pattern Analysis and Machine Intelligence, vol. 45, no. 11, pp. 13314-13327, 2023.
[CrossRef] [Google Scholar] [Publisher Link]

[7] Huimin Xiao, Xinghao Wang, and Qiang Xing, “Facial Expression Recognition in the Wild based on Convolutional Neural Network and Graph Convolutional Network,” Fourth International Conference on Signal Processing and Computer Science (SPCS 2023), Guilin, China, vol. 12970, pp. 976-981, 2023.
[CrossRef] [Google Scholar] [Publisher Link]

[8] Wissam J. Baddar, Sangmin Lee, and Yong Man Ro, “On-the-Fly Facial Expression Prediction using LSTM Encoded Appearance-Suppressed Dynamics,” IEEE Transactions on Affective Computing, vol. 13, no. 1, pp. 159-174, 2022.
[CrossRef] [Google Scholar] [Publisher Link]

[9] Yikang Guo et al., “A Personalized Spatial-Temporal Cold Pain Intensity Estimation Model based on Facial Expression,” IEEE Journal of Translational Engineering in Health and Medicine, vol. 9, pp. 1-8, 2021.
[CrossRef] [Google Scholar] [Publisher Link]

[10] Mayuri Arul Vinayakam Rajasimman et al., “Robust Facial Expression Recognition using an Evolutionary Algorithm with a Deep Learning Model,” Applied Sciences, vol. 13, no. 1, pp. 1-20, 2022.
[CrossRef] [Google Scholar] [Publisher Link]

[11] Yanisa Mahayossanunt et al., “Explainable Depression Detection based on Facial Expression using LSTM on Attentional Intermediate Feature Fusion with Label Smoothing,” Sensors, vol. 23, no. 23, pp. 1-16, 2023.
[CrossRef] [Google Scholar] [Publisher Link]

[12] Rupali Gill, Aditi Moudgil, and Palak Bajaj, “Hybrid Approach for Emotion Recognition using CNLSTM in Video Expressions,” 2022 10^th International Conference on Reliability, Infocom Technologies and Optimization (Trends and Future Directions) (ICRITO), Noida, India, pp. 1-5, 2022.
[CrossRef] [Google Scholar] [Publisher Link]

[13] Mustafa Can Gursesli et al., “Facial Emotion Recognition (FER) through Custom Lightweight CNN Model: Performance Evaluation in Public Datasets,” IEEE Access, vol. 12, pp. 45543-45559, 2024.
[CrossRef] [Google Scholar] [Publisher Link]

[14] Jasthi Vivek Vardhan, Yelavarti Kalyan Chakravarti, and Annam Jitin Chand, “Emotion Recognition by Facial Expressions and Speech using Deep Learning,” 2024 Fourth International Conference on Advances in Electrical, Computing, Communication and Sustainable Technologies (ICAECT), Bhilai, India, pp. 1-7, 2024.
[CrossRef] [Google Scholar] [Publisher Link]

[15] Yue Luo et al., “Design of Facial Expression Recognition Algorithm based on CNN Model,” 2023 IEEE 3^rd International Conference on Power, Electronics and Computer Applications (ICPECA), Shenyang, China, pp. 580-583, 2023.
[CrossRef] [Google Scholar] [Publisher Link]

[16] Nongmeikapam Thoiba Singh et al., “Comparative Analysis of Traditional Machine Learning and Deep Learning Techniques for Facial Expression Recognition,” 2023 14^th International Conference on Computing Communication and Networking Technologies (ICCCNT), Delhi, India, pp. 1-7, 2023.
[CrossRef] [Google Scholar] [Publisher Link]

[17] Luu-Ngoc Do et al., “Deep Neural Network-Based Fusion Model for Emotion Recognition using Visual Data,” The Journal of Supercomputing, vol. 77, no. 10, pp. 10773-10790, 2021.
[CrossRef] [Google Scholar] [Publisher Link]

[18] Li Muyao, and Ding Weili, “Facial Expression Recognition based on FPN and LSTM,” 2023 IEEE 5^th International Conference on Power, Intelligent Computing and Systems (ICPICS), Shenyang, China, pp. 762-767, 2023.
[CrossRef] [Google Scholar] [Publisher Link]

[19] Md. Mohsin Kabir et al., “Facial Expression Recognition using CNN-LSTM Approach,” 2021 International Conference on Science & Contemporary Technologies (ICSCT), Dhaka, Bangladesh, pp. 1-6, 2021.
[CrossRef] [Google Scholar] [Publisher Link]

[20] Saikat Das et al., “IoT based Framework Design for Automated Human Emotion Recognition,” 2023 2^nd International Conference on Ambient Intelligence in Health Care (ICAIHC), Bhubaneswar, India, pp. 1-6, 2023.
[CrossRef] [Google Scholar] [Publisher Link]

[21] Angalakuditi Hindu, and Biswajit Bhowmik, “An IoT-enabled Stress Detection Scheme using Facial Expression,” 2022 IEEE 19^th India Council International Conference (INDICON), Kochi, India, pp. 1-6, 2022.
[CrossRef] [Google Scholar] [Publisher Link]

[22] Jinyu Luo et al., “Facial Expression Recognition using Machine Learning Models in FER2013,” 2021 IEEE 3^rd International Conference on Frontiers Technology of Information and Computer (ICFTIC), Greenville, SC, USA, pp. 231-235, 2021.
[CrossRef] [Google Scholar] [Publisher Link]

[23] Uma Chinta, and Adham Atyabi, “A Framework Pipeline to Address Imbalanced Class Distribution Problem in Real-World Datasets,” 2023 IEEE 13^th Annual Computing and Communication Workshop and Conference (CCWC), Las Vegas, NV, USA, pp. 746-753, 2023.
[CrossRef] [Google Scholar] [Publisher Link]

[24] Deepak Raj, and Md. Abdul Wassay, “Facial Emotional Recognition using Convolutional Neural Network,” 2023 2^nd International Conference for Innovation in Technology (INOCON), Bangalore, India, pp. 1-5, 2023.
[CrossRef] [Google Scholar] [Publisher Link]

[25] Özay Ezerceli, and M. Taner Eskil, “Convolutional Neural Network (CNN) Algorithm based Facial Emotion Recognition (FER) System for FER-2013 Dataset,” 2022 International Conference on Electrical, Computer, Communications and Mechatronics Engineering (ICECCME), Maldives, pp. 1-6, 2022.
[CrossRef] [Google Scholar] [Publisher Link]

[26] K. Kousalya et al., “Group Emotion Detection using Convolutional Neural Network,” 2022 OPJU International Technology Conference on Emerging Technologies for Sustainable Development (OTCON), Raigarh, Chhattisgarh, India, pp. 1-6, 2022.
[CrossRef] [Google Scholar] [Publisher Link]

[27] Aicha Nouisser, Ramzi Zouari, and Monji Kherallah, “Enhanced MobileNet and Transfer Learning for Facial Emotion Recognition,” 2022 International Arab Conference on Information Technology (ACIT), Abu Dhabi, United Arab Emirates, pp. 1-5, 2022.
[CrossRef] [Google Scholar] [Publisher Link]

[28] Dorfell Parra, and Carlos Camargo, “Design Methodology for Single-Channel CNN-Based FER Systems,” 2023 6^th International Conference on Information and Computer Technologies (ICICT), Raleigh, NC, USA, pp. 89-94, 2023.
[CrossRef] [Google Scholar] [Publisher Link]

[29] Tinuk Agustin, Moch Hari Purwidiantoro, and Mochammad Luthfi Rahmadi, “Enhancing Facial Expression Recognition through Ensemble Deep Learning,” 2023 5^th International Conference on Cybernetics and Intelligent System (ICORIS), Pangkalpinang, Indonesia, pp. 1-6, 2023.
[CrossRef] [Google Scholar] [Publisher Link]

[30] Mj Alben Richards et al., “Facial Expression Recognition using Convolutional Neural Network,” 2023 International Conference on Bio Signals, Images, and Instrumentation (ICBSII), Chennai, India, pp. 1-5, 2023.
[CrossRef] [Google Scholar] [Publisher Link]

[31] Ankit S. Vyas, Harshadkumar B. Prajapati, and Vipul K. Dabhi, “Survey on Face Expression Recognition using CNN,” 2019 5^th International Conference on Advanced Computing & Communication Systems (ICACCS), Coimbatore, India, pp. 102-106, 2019.
[CrossRef] [Google Scholar] [Publisher Link]

[32] Xinpeng Zhang, “Face Expression Recognition based on Convolutional Neural Networks,” 2022 International Conference on Cloud Computing, Big Data Applications and Software Engineering (CBASE), Suzhou, China, pp. 254-258, 2022.
[CrossRef] [Google Scholar] [Publisher Link]

[33] Christian Białek, Andrzej Matiolański, and Michał Grega, “An Efficient Approach to Face Emotion Recognition with Convolutional Neural Networks,” Electronics, vol. 12, no. 12, pp. 1-21, 2023.
[CrossRef] [Google Scholar] [Publisher Link]

[34] Shan Li, and Weihong Deng, “Deep Facial Expression Recognition: A Survey,” IEEE Transactions on Affective Computing, vol. 13, no. 3, pp. 1195-1215, 2022.
[CrossRef] [Google Scholar] [Publisher Link]

[35] Ceren Özkara, and Pınar Oğuz Ekim, “Real-Time Facial Emotion Recognition for Visualization Systems,” 2022 Innovations in Intelligent Systems and Applications Conference (ASYU), Antalya, Turkey, pp. 1-5, 2022.
[CrossRef] [Google Scholar] [Publisher Link]

[36] M.N. Kavitha, and A. RajivKannan, “Hybrid Convolutional Neural Network and Long Short-Term Memory Approach for Facial Expression Recognition,” Intelligent Automation & Soft Computing, vol. 35, no. 1, pp. 689-704, 2023.
[CrossRef] [Google Scholar] [Publisher Link]

[37] Greg Van Houdt, Carlos Mosquera, and Gonzalo Nápoles, “A Review on the Long Short-Term Memory Model,” Artificial Intelligence Review, vol. 53, no. 8, pp. 5929-5955, 2020.
[CrossRef] [Google Scholar] [Publisher Link]

[38] Harya Widiputra, Adele Mailangkay, and Elliana Gautama, “Multivariate CNN-LSTM Model for Multiple Parallel Financial Time-Series Prediction,” Complexity, vol. 2021, no. 1, pp. 1-14, 2021.
[CrossRef] [Google Scholar] [Publisher Link]

[39] Chen Jia, Chu Li Li, and Zhou Ying, “Facial Expression Recognition based on the Ensemble Learning of CNNs,” 2020 IEEE International Conference on Signal Processing, Communications and Computing (ICSPCC), Macau, China, pp. 1-5, 2020.
[CrossRef] [Google Scholar] [Publisher Link]

[40] Muhamad Dwisnanto Putro et al., “Multi-View Facial Emotion Recognition using Attention-Based Convolutional Network for Human-Machine Interaction,” 2025 International Conference on Information Management and Technology (ICIMTech), Bandung, Jawa Barat, Indonesia, pp. 512-517, 2025.
[CrossRef] [Publisher Link]

[41] Roberto Pecoraro, Valerio Basile, and Viviana Bono, “Local Multi-Head Channel Self-Attention for Facial Expression Recognition,” Information, vol. 13, no. 9, pp. 1-17, 2022. [CrossRef] [Google Scholar] [Publisher Link]

[42] Sunil S. Harakannanavar et al., “Enhanced Facial Emotion Recognition using Deep Learning Techniques: A Comprehensive Analysis and Implementation,” 2024 International Conference on Distributed Systems, Computer Networks and Cybersecurity (ICDSCNC), Bengaluru, India, pp. 1-6, 2024.
[CrossRef] [Google Scholar] [Publisher Link]

[43] Wenqian Yan et al., “Optimizing Facial Expression Recognition: A One-Class Classification Approach using ResNet18 and CBAM,” 2024 3^rd International Conference on Computer Technologies (ICCTech), Bali, Indonesia, pp. 1-5, 2024.
[CrossRef] [Google Scholar] [Publisher Link]

[44] S. Vignesh et al., “A Novel Facial Emotion Recognition Model using Segmentation VGG-19 Architecture,” International Journal of Information Technology, vol. 15, no. 4, pp. 1777-1787, 2023.
[CrossRef] [Google Scholar] [Publisher Link]

[45] Luan Pham et al., “Facial Expression Recognition using Residual Masking Network,” 2020 25^th International Conference on Pattern Recognition (ICPR), Milan, Italy, pp. 4513-4519, 2021.
[CrossRef] [Google Scholar] [Publisher Link]

[46] Arnab Kumar Roy et al., “ResEmoteNet: Bridging Accuracy and Loss Reduction in Facial Emotion Recognition,” IEEE Signal Processing Letters, vol. 32, pp. 491-495, 2025.
[CrossRef] [Google Scholar] [Publisher Link]

[47] Neha Gahlan, and Divyashikha Sethia, “Emotion Recognition from Facial Expressions using Deep Recurrent Attention Network,” 2024 16^th International Conference on Communication Systems & Networks (COMSNETS), Bengaluru, India, pp. 86-91, 2024.
[CrossRef] [Google Scholar] [Publisher Link]

[48] Rifky, Panca Dewi Pamungkasari, and Endah Tri Esti Handayani, “Performance Analysis of Facial Expression Recognition by Using Geometry Augmentation and Random Oversampling for CNN Model,” 2024 10^th International Conference on Smart Computing and Communication (ICSCC), Bali, Indonesia, pp. 614-618, 2024.
[CrossRef] [Google Scholar] [Publisher Link]