An Aggregated Optical Flow Vectors for Micro Expression Recognition Using Spatio-Temporal Binary Pattern Coding

An Aggregated Optical Flow Vectors for Micro Expression Recognition Using Spatio-Temporal Binary Pattern Coding

© 2021 by IJETT Journal
Volume-69 Issue-11
Year of Publication : 2021
Authors : Sammaiah Seelothu, Dr. K. Venugopal Rao
DOI :  10.14445/22315381/IJETT-V69I11P230

How to Cite?

Sammaiah Seelothu, Dr. K. Venugopal Rao, "An Aggregated Optical Flow Vectors for Micro Expression Recognition Using Spatio-Temporal Binary Pattern Coding," International Journal of Engineering Trends and Technology, vol. 69, no. 11, pp. 236-247, 2021. Crossref,

Micro Expressions (MEs) are unique facial expressions when individual experiences an emotion but intentionally tries to hide their genuine emotion. MEs are involuntary and spontaneous, and their recognition has gained a significant research interest due to their potential applications. However, Micro Expression Recognition (MER) is an arduous task due to its short duration, subtle and local movements of faces. This paper proposes an effective descriptor called Composite Local Binary Pattern on Three Orthogonal Planes (CLBP-TOP) for Micro Expressions Recognition to address these problems. We also propose a novel Aggregated Optical Flow Vectors (AOFVs) Computation mechanism where the neighbour optical flows in a particular period are aggregated to measure motion intensities. Based on these motion intensities, we compute a weight matrix, and it is multiplied with CLBP-TOP histogram features to get weighted histogram features. For classification purposes, we employ the Support Vector Machine (SVM) algorithm. Extensive experimental evaluation of the CASME II dataset shows that our proposed approach significantly improves recognition accuracy and shows superior performance than the state-of-art methods.

MER, feature extraction, aggregated optical flow vectors, Composite local binary pattern, accuracy.

[1] I. Cohen , N. Sebe , A. Garg , L.S. Chen , T.S. Huang , Facial expression recognition from video sequences: temporal and static modeling, Comput. Vision ImageUnderst. 91 (1) (2003) 160–187.
[2] E. Vural , M. Bartlett , G. Littleworth , M. Cetin , A. Cecil , J. Movellan , Discrimination of moderate and acute drowsiness based on spontaneous facial expressions, in: 20th International Conference on Pattern Recognition (ICPR), IEEE, (2010) 3874–3877 .
[3] J. Whitehill, M. Bartlett, J. Movellan, Automatic facial expression recognition for intelligent tutoring systems, in: IEEE Computer Society Conference on Computer Vision and Pattern Recognition Workshops (CVPRW`08), (2008) 1–6 .
[4] T.A. Russell, E. Chu, M.L. Phillips, A pilot study to investigate the effectivenessof emotion recognition remediation in schizophrenia using the micro-expressiontraining tool, Br. J. Clin. Psychol. 45 (4) (2006) 579–583.
[5] X. Ben, P. Zhang, R. Yan, M. Yang, G. Ge, Gait recognition and micro-expressionrecognition based on maximum margin projection with tensor representation,Neural Comput. Appl. 27 (8) (2016) 2629– 2646.
[6] L. Su, MD. Levine,High-stakes deception detection based on facial expressions, in: 22nd IEEE International Conference on Pattern Recognition (ICPR), (2014) 2519–2524.
[7] E.A. Haggard, K.S. Isaacs, Micro-momentary facial expressions as indicators of ego-mechanisms in psychotherapy, in: Methods of Research in Psychotherapy, Springer, (1966) 154–165.
[8] P. Ekman, W.V. Friesen, Non-verbal leakage and clues to deception, Psychiatry 32 (1) (1969) 88–106.
[9] P. Ekman, M. O`Sullivan, and M. G. Frank, A few can catch a liar,``Psychol. Sci., 10(3) (1999) 263_266.
[10] M. G. Frank, C. J. Maccario, and V. Govindaraju, Behavior and security, in Protecting Airline Passengers in the Age of Terrorism. Santa Barbara,CA, USA: ABC-CLIO, LLC, (2009) 86_106.
[11] M. O`Sullivan, M. G. Frank, C. M. Hurley, and J. Tiwana, Police liedetection accuracy: The effect of lie scenario, Law Hum. Behav., 33(6) (2009) 530.
[12] X. Li, X. Hong, A. Moilanen, X. Huang, T. P_ster, G. Zhao, andM. Pietikäinen, Towards reading hidden emotions: A comparative studyof spontaneous micro-expression spotting and recognition methods, IEEETrans. Affect. Comput., 9(4) 563_577, Oct./Dec. 2017.
[13] M. Frank, M. Herbasz, K. Sink, A. Keller, and C. Nolan, ``I see how youfeel: Training laypeople and professionals to recognise meeting emotions,``in Proc. Annu. Meeting Int. Commun. Assoc. Sheraton, New York, NY,USA, (2009).
[14] Yee-Hui Oh, John See, Anh Cat Le Ngo, Raphael C. W. Phan and Vishnu M. Baskaran, A Survey of Automatic Facial Micro- Expression Analysis: Databases, Methods, and Challenges, Frontiers in Psychology, 9 (2018) 11-21.
[15] G. Zhao and M. Pietikäinen, Dynamic texture recognition using localbinary patterns with an application to facial expressions, IEEE Trans.Pattern Anal. Mach. Intell., 29(6) (2007) 915_928.
[16] W. J. Yan, X. Li, S. J. Wang, G. Zhao, Y. J. Liu, Y. H. Chen, X. Fu, CASME II: An improved spontaneous micro-expression database and the baseline evaluation, PLoS One 9 (1) (2014) e86041.
[17] S. J. Wang, W. J. Yan, X. Li, G. Zhao, C. G. Zhou, X. Fu, M. Yang, J. Tao, Micro expression recognition using color spaces, IEEE Trans. Image Process. 24 (12) (2015) 6034–6047.
[18] X. Huang, S. J. Wang, G. Zhao, M. Piteikainen, Facial microexpression recognition using spatiotemporal local binary pattern with integral projection, in: Proceedings of the IEEE International Conference on Computer Vision Workshops, (2015) 1–9.
[19] Zeileis, Achim; Hornik, Kurt; Murrell, Paul (2009). Escaping RGBland: Selecting Colors for Statistical Graphics, Computational Statistics & Data Analysis. 53 (9) (2009)3259–3270.
[20] I. P. Adegun and V. Hima Bindu, Facial micro-expression recognition: A machine learning approach, Scientific African, 8 (2020) e00465
[21] Yanliang Zhang , Hanxiao Jiang, Xingwang Li, Bing Lu, Khaled M. Rabie, and Ateeq Ur Rehman, A New Framework Combining Local- RegionDivision and Feature Selection forMicro-Expressions Recognition", IEEE Access, 8 (2020) 94499-94509.
[22] N. Dalal and B. Triggs, Histograms of oriented gradients for human detection, in Proc. IEEE Comput. Soc. Conf. Comput. Vis. Pattern Recognit., Jun. (2005) 886-893.
[23] M. R. Guidera, A. E. Qadi, M. R. Lrit, and M. E. Hassouni, A novel method for image categorisation based on histogram oriented gradient and support vector machine, in Proc. Int. Conf. Electr. Inf. Technol. (ICEIT) (2017) 1-5.
[24] Xiaobai Li, Xiaopeng Hong, Antti Moilanen, Xiaohua Huang, Tomas Pfister, Guoying Zhao, and Matti Pietik¨ainen, Reading Hidden Emotions: Spontaneous Micro-expression Spotting and Recognition, arXiv:1511.00423v1 [cs.CV] 2 Nov (2015).
[25] K. Kira and L. A. Rendell, A Practical Approach to Feature Selection, in Proc. 9th Int. Workshop Mach. Learn. (ML), Dec. (1992) 249-256.
[26] Guo, Y.; Xue, C.; Wang, Y.; Yu, M., "Micro-expression recognition based on CBP-TOP feature with ELM, Optik2015, 126, 4446–4451.
[27] Wang, Y.; See, J.; Phan, W.; Oh, Y.H. LBP with Six Intersection Points: Reducing Redundant Information in LBP-TOP for Microexpression Recognition. In Asian Conference on Computer Visio; Springer: Cham, Switzerland, (2014) 525–537.
[28] Lu G., Yang C., Yang W., Yan J., Micro-expression recognition based on LBP-TOP features, Journal of Nanjing Institute of Posts and Telecommunications, 37(6) (2017) 1-7
[29] Yandan Wang, John See, Raphael C. W. Phan, Yee-Hui Oh, Efficient Spatio-Temporal Local Binary Patterns for Spontaneous Facial Micro-Expression Recognition, PLoS ONE 10(5): e0124674.
[30] X. Huang, G. Zhao, X. Hong, W. Zheng, M. Pietikäinen, Spontaneous facial micro-expressionanalysis using spatiotemporal completed local quantised patterns,Neuro-computing 175 (2016) 564–578.
[31] Y.-J. Liu, J.-K. Zhang, W.-J. Yan, S.-J. Wang, G. Zhao, X. Fu, A main directionalmean optical flow feature for spontaneous microexpression recognition, IEEETrans. Affect. Comput. 7 (4) (2016) 299–310.
[32] S. Happy, A. Routray, Fuzzy histogram of optical flow orientations formicro-expression recognition, IEEE Trans. Affect. Comput. (2017).
[33] S.-T. Liong, J. See, K. Wong, R.C.-W. Phan, Less is more: Microexpression recognition from video using apex frame, Signal Process., Image Commun. 62 (2018) 82–92.
[34] Benjamin Allaert, Isaac Ronald Ward, Ioan Marius Bilasco, ChaabaneDjerba and Mohammed Bennamoun, Optical Flow Techniques for Facial Expression analysis - a Practical Evaluation Study, arXiv:1904.11592v2 [cs.CV] 4 Jan 2021.
[35] Junjie Wu, Jianfeng Xu, Deyu Lin and Min Tu, Optical Flow Filtering-Based Micro-Expression Recognition Method, Electronics (2020), 9, 2056.
[36] M.J. Black, P. Anandan, The robust estimation of multiple motions: Parametric and piecewise-smooth flow fields, Computer Vision and Image Understanding, 63 (1) (1996) 75–104.
[37] T. Ojala, M. Pietikäinen, and T. Mäenpää, Multi-resolution grayscale and rotation invariant texture classi_cation with local binary patterns, IEEE Trans. Pattern Anal. Mach. Intell., 24(7) (2002) 971_987.
[38] T. Ojala, M. Pietikäinen, and D. Harwood, A comparative study of texture measures with classification based on featured distributions,`` Pattern Recognit., 29(1) (1996) 51_59.
[39] Chengyu Guo, Jingyun Liang, Geng Zhan, Zhong Liu, Matti Pietikäinen, and Li Liu, Extended Local Binary Patterns for Efficient and Robust Spontaneous Facial Micro-Expression Recognition, IEEE Access, 7(2019) 174517-174530.
[40] Petr Husa`k, Jan C? each, Jir?´? Matas, Spotting Facial Micro- Expressions In the Wild, 22nd Computer Vision Winter Workshop Nicole M. Artner, Ines Janusch, Walter G. Kropatsch (eds.) Retz, Austria, February (2017) 6–8.
[41] H. Lu, K. Kpalma, J. Ronsin, Motion descriptors for microexpression recognition, Signal Process., Image Commun. 67 (2018) 108–117.
[42] A. K. Davison, W. Merghani, and M. H. Yap, Objective classes for micro-facial expression recognition,`` J. Imag., 4(10) (2018) 119.