Facial Expression Recognition from Video Sequence Using Self Organizing Feature Map

Walid Amin  Mahmoud; Jane Jaleel  Stephan; Anmar Abdel Wahab Razzak Razzak

doi:10.36371/port.2020.2.2

Authors

Walid Amin Mahmoud
Jane Jaleel Stephan
Anmar Abdel Wahab Razzak Razzak

Abstract References Downloads Cite Statistics Share Information

Automatic analysis of facial expressions is rapidly becoming an area of intense interest in computer vision and artificial intelligence research communities. In this paper an approach is presented for facial expression recognition of the six basic prototype expressions (i.e., joy, surprise, anger, sadness, fear, and disgust) based on Facial Action Coding System (FACS). The approach utilizes the topological ordering patterns produced by Kohonen Self Organizing Map, in which implemented on expression image sequence for each prototype facial expression. The map will compute the topological relationship between the particular expression sequences, starting from the neutral expression to peak. This method tried to find a topological ordering pattern (shape) for each expression; it will not require any pre-processing tedious work such as normalization. The only requirement is that, image background must be kept constant, also with non-rigid head motion. The feature extraction phase had been performed by this method, while the classification phase done by especially designed procedures for shape and direction finding to recognize the pattern of the shape, thereafter the type of the expression also backpropagation neural network is implemented for the classification task. An average recognition rate of 88.7% was achieved for six basic expressions, where different databases had been used for the test of the method.

Keywords:

Facial expression recognition, Video-frame, Self-Organizing Maps (SOM), Image sequences.

[1] Liu, C., Hirota, K., Ma, J., Jia, Z., & Dai, Y. (2021). Facial Expression Recognition Using Hybrid Features of Pixel and Geometry. IEEE Access, 9, 18876–18889. https://doi.org/10.1109/ACCESS.2021.3054332

[2] Chen, J., Chen, Z., Chi, Z., & Fu, H. (2018). Facial Expression Recognition in Video with Multiple Feature Fusion. IEEE Transactions on Affective Computing, 9(1), 38–50. https://doi.org/10.1109/TAFFC.2016.2593719

[3] Dulguerov, P., Marchal, F., Wang, D., & Gysin, C. (1999). Review of objective topographic facial nerve evaluation methods. American Journal of Otology, 20(5), 672–678.

[4] Kuo, C. M., Lai, S. H., & Sarkis, M. (2018). A compact deep learning model for robust facial expression recognition. In IEEE Computer Society Conference on Computer Vision and Pattern Recognition Workshops (Vol. 2018-June, pp. 2202–2210). IEEE Computer Society. https://doi.org/10.1109/CVPRW.2018.00286

[5] Koenen, R., & Pereira, F. (2000). MPEG-7: A standardized description of audiovisual content. Signal Processing: Image Communication, 16(1), 5–13. https://doi.org/10.1016/S0923-5965(00)00014-X

[6] Rajendra Kurup, A., Ajith, M., & Martínez Ramón, M. (2019). Semi-supervised facial expression recognition using reduced spatial features and Deep Belief Networks. Neurocomputing, 367, 188–197. https://doi.org/10.1016/j.neucom.2019.08.029

[7] Fasel, B., & Luettin, J. (2003). Automatic facial expression analysis: A survey. Pattern Recognition. Elsevier Ltd. https://doi.org/10.1016/S0031-3203(02)00052-3

[8] Li, S., & Deng, W. (2020). Deep Facial Expression Recognition: A Survey. IEEE Transactions on Affective Computing. https://doi.org/10.1109/TAFFC.2020.2981446

[9] Ekman, P., & Friesen, W. V. (1971). Constants across cultures in the face and emotion. Journal of Personality and Social Psychology, 17(2), 124–129. https://doi.org/10.1037/h0030377

[10] Li, S., & Deng, W. (2020, November 16). Deep facial expression recognition: a survey. Journal of Image and Graphics. Editorial and Publishing Board of JIG. https://doi.org/10.11834/jig.200233

[11] Ekman, P. (2004). Emotional and Conversational Nonverbal Signals. In Language, Knowledge, and Representation (pp. 39–50). Springer Netherlands. https://doi.org/10.1007/978-1-4020-2783-3_3

[12] Kohonen, T., Huang, T., & Schroeder, M. (2012). Self-Organizing Maps. Berlin, Heidelberg: Springer Berlin / Heidelberg.

[13] Kohonen, T. (1982). Self-organized formation of topologically correct feature maps. Biological Cybernetics, 43(1), 59–69. https://doi.org/10.1007/BF00337288

[14] Padgett, C., & Cottrell, G. (1997). Representing face images for emotion classification. In Advances in Neural Information Processing Systems (pp. 894–900). Neural information processing systems foundation.

[15] Bartlett, M.-S. (1999). Face image analysis by unsupervised learning and redundancy reduction. Dissertation-Abstracts-International:-Section-b:-The-Sciences-and-Engineering.

[16] Pantic, M., & Rothkrantz, L. J. M. (2000). Expert system for automatic analysis of facial expressions. Image and Vision Computing, 18(11), 881–905. https://doi.org/10.1016/S0262-8856(00)00034-2

[17] Zhao J, Kearney G: Classifying Facial Emotions by Backpropagation Neural Networks with Fuzzy Inputs. Proceedings of Conference on Neural Information Processing 1996, 1: 454–457.

[18] Lisetti, C. L., & Rumelhart, D. E. (1998). Facial Expression Recognition using a Neural Network. In Proceedings of the 11th International Flairs Conference (FLAIRS-98). Menlo Park, CA: AAAI Press.

[19] Kobayashi, H., & Hara, F. (1997). Facial interaction between animated 3D face robot and human beings. In Proceedings of the IEEE International Conference on Systems, Man and Cybernetics (Vol. 4, pp. 3732–3737). IEEE. https://doi.org/10.1109/icsmc.1997.633250

[20] Yoneyama, M., Iwano, Y., Ohtake, A., & Shirai, K. (1997). Facial expressions recognition using discrete Hopfield neural networks. In IEEE International Conference on Image Processing (Vol. 1, pp. 117–120). IEEE Comp Soc. https://doi.org/10.1109/icip.1997.647398

[21] Sanby, L., Todd, I., & Keet, M. C. (2016). Comparing the Template-Based Approach to GF: the case of Afrikaans (pp. 50–53). Association for Computational Linguistics (ACL). https://doi.org/10.18653/v1/w16-3510

[22] Tang, Y., Zhang, X., Hu, X., Wang, S., & Wang, H. (2021). Facial Expression Recognition Using Frequency Neural Network. IEEE Transactions on Image Processing, 30, 444–457. https://doi.org/10.1109/TIP.2020.3037467

[23] Zhang, Q., Li, H., Li, M., & Ding, L. (2020). Feature extraction of face image based on LBP and 2-D Gabor wavelet transform. Mathematical Biosciences and Engineering, 17(2), 1578–1592. https://doi.org/10.3934/mbe.2020082

[24] Lien, J. J., Cohn, J. F., Kanade, T., & Li, C. C. (1998). Automated facial expression recognition based on FACS action units. In Proceedings - 3rd IEEE International Conference on Automatic Face and Gesture Recognition, FG 1998 (pp. 390–395). IEEE Computer Society. https://doi.org/10.1109/AFGR.1998.670980

[25] Donate, G., Bartlett, M. S., Hager, J. C., Ekman, P., & Sejnowski, T. J. (1999). Classifying facial actions. IEEE Transactions on Pattern Analysis and Machine Intelligence, 21(10), 974–989. https://doi.org/10.1109/34.799905

[26] Fasel, B., & Luettin, J. (2000). Recognition of asymmetric facial action unit activities and intensities. Proceedings - International Conference on Pattern Recognition, 15(1), 1100–1103. https://doi.org/10.1109/icpr.2000.905664

[27] Lisa, Y. T., Arun, B., Sharat, H., Andrew, P., & Bolle, R. (2003). Real World Real-time Automatic Recognition of Facial Expressions. Most.

[28] Lien, J.-J. J. (2013). AUTOMATIC RECOGNITION OF FACIAL EXPRESSIONS USING HIDDEN MARKOV MODELS AND ESTIMATION OF EXPRSSION INTENSITY. Journal of Chemical Information and Modeling.

[29] Hasan, H., AL.Jouhar, W., & Alwan, M. (2012). 3-D Face Recognition Using Improved 3D Mixed Transform. International Journal Of Biometrics And Bioinformatics, 6(1), 278-290. Retrieved from https://citeseerx.ist.psu.edu/viewdoc/download?doi=10.1.1.740.1016&rep=rep1&type=pdf

[30] Kanade, T., Cohn, J. F., & Tian, Y. (2000). Comprehensive database for facial expression analysis. In Proceedings - 4th IEEE International Conference on Automatic Face and Gesture Recognition, FG 2000 (pp. 46–53). IEEE Computer Society. https://doi.org/10.1109/AFGR.2000.840611

[31] Wallhoff, F. (2021). Facial Expressions and Emotion Database. Retrieved 10 November 2021, from https://www.jade-hs.de/team/frank-wallhoff/databases/

[32] Mahmod, W., Stephan, J., & Razzak, A. (2020). Facial Expression Recognition Using Fast Walidlet Hybrid Transform. Journal Port Science Research, 59–69. https://doi.org/10.36371/port.2020.3.4

PDF

Mahmoud, W. A. ., Stephan, . J. J. ., & Razzak, A. A. W. R. (2021). Facial Expression Recognition from Video Sequence Using Self Organizing Feature Map. Journal Port Science Research, 4(2), 53–68. https://doi.org/10.36371/port.2020.2.2