Facial Expression Recognition from Video Sequence Using Self Organizing Feature Map
Automatic analysis of facial expressions is rapidly becoming an area of intense interest in computer vision and artificial intelligence research communities. In this paper an approach is presented for facial expression recognition of the six basic prototype expressions (i.e., joy, surprise, anger, sadness, fear, and disgust) based on Facial Action Coding System (FACS). The approach utilizes the topological ordering patterns produced by Kohonen Self Organizing Map, in which implemented on expression image sequence for each prototype facial expression. The map will compute the topological relationship between the particular expression sequences, starting from the neutral expression to peak. This method tried to find a topological ordering pattern (shape) for each expression; it will not require any pre-processing tedious work such as normalization. The only requirement is that, image background must be kept constant, also with non-rigid head motion. The feature extraction phase had been performed by this method, while the classification phase done by especially designed procedures for shape and direction finding to recognize the pattern of the shape, thereafter the type of the expression also backpropagation neural network is implemented for the classification task. An average recognition rate of 88.7% was achieved for six basic expressions, where different databases had been used for the test of the method.
Keywords:Facial expression recognition, Video-frame, Self-Organizing Maps (SOM), Image sequences.
 Liu, C., Hirota, K., Ma, J., Jia, Z., & Dai, Y. (2021). Facial Expression Recognition Using Hybrid Features of Pixel and Geometry. IEEE Access, 9, 18876–18889. https://doi.org/10.1109/ACCESS.2021.3054332
 Chen, J., Chen, Z., Chi, Z., & Fu, H. (2018). Facial Expression Recognition in Video with Multiple Feature Fusion. IEEE Transactions on Affective Computing, 9(1), 38–50. https://doi.org/10.1109/TAFFC.2016.2593719
 Dulguerov, P., Marchal, F., Wang, D., & Gysin, C. (1999). Review of objective topographic facial nerve evaluation methods. American Journal of Otology, 20(5), 672–678.
 Kuo, C. M., Lai, S. H., & Sarkis, M. (2018). A compact deep learning model for robust facial expression recognition. In IEEE Computer Society Conference on Computer Vision and Pattern Recognition Workshops (Vol. 2018-June, pp. 2202–2210). IEEE Computer Society. https://doi.org/10.1109/CVPRW.2018.00286
 Koenen, R., & Pereira, F. (2000). MPEG-7: A standardized description of audiovisual content. Signal Processing: Image Communication, 16(1), 5–13. https://doi.org/10.1016/S0923-5965(00)00014-X
 Rajendra Kurup, A., Ajith, M., & Martínez Ramón, M. (2019). Semi-supervised facial expression recognition using reduced spatial features and Deep Belief Networks. Neurocomputing, 367, 188–197. https://doi.org/10.1016/j.neucom.2019.08.029
 Fasel, B., & Luettin, J. (2003). Automatic facial expression analysis: A survey. Pattern Recognition. Elsevier Ltd. https://doi.org/10.1016/S0031-3203(02)00052-3
 Li, S., & Deng, W. (2020). Deep Facial Expression Recognition: A Survey. IEEE Transactions on Affective Computing. https://doi.org/10.1109/TAFFC.2020.2981446
 Ekman, P., & Friesen, W. V. (1971). Constants across cultures in the face and emotion. Journal of Personality and Social Psychology, 17(2), 124–129. https://doi.org/10.1037/h0030377
 Li, S., & Deng, W. (2020, November 16). Deep facial expression recognition: a survey. Journal of Image and Graphics. Editorial and Publishing Board of JIG. https://doi.org/10.11834/jig.200233
 Ekman, P. (2004). Emotional and Conversational Nonverbal Signals. In Language, Knowledge, and Representation (pp. 39–50). Springer Netherlands. https://doi.org/10.1007/978-1-4020-2783-3_3
 Kohonen, T., Huang, T., & Schroeder, M. (2012). Self-Organizing Maps. Berlin, Heidelberg: Springer Berlin / Heidelberg.
 Kohonen, T. (1982). Self-organized formation of topologically correct feature maps. Biological Cybernetics, 43(1), 59–69. https://doi.org/10.1007/BF00337288
 Padgett, C., & Cottrell, G. (1997). Representing face images for emotion classification. In Advances in Neural Information Processing Systems (pp. 894–900). Neural information processing systems foundation.
 Bartlett, M.-S. (1999). Face image analysis by unsupervised learning and redundancy reduction. Dissertation-Abstracts-International:-Section-b:-The-Sciences-and-Engineering.
 Pantic, M., & Rothkrantz, L. J. M. (2000). Expert system for automatic analysis of facial expressions. Image and Vision Computing, 18(11), 881–905. https://doi.org/10.1016/S0262-8856(00)00034-2
 Zhao J, Kearney G: Classifying Facial Emotions by Backpropagation Neural Networks with Fuzzy Inputs. Proceedings of Conference on Neural Information Processing 1996, 1: 454–457.
 Lisetti, C. L., & Rumelhart, D. E. (1998). Facial Expression Recognition using a Neural Network. In Proceedings of the 11th International Flairs Conference (FLAIRS-98). Menlo Park, CA: AAAI Press.
 Kobayashi, H., & Hara, F. (1997). Facial interaction between animated 3D face robot and human beings. In Proceedings of the IEEE International Conference on Systems, Man and Cybernetics (Vol. 4, pp. 3732–3737). IEEE. https://doi.org/10.1109/icsmc.1997.633250
 Yoneyama, M., Iwano, Y., Ohtake, A., & Shirai, K. (1997). Facial expressions recognition using discrete Hopfield neural networks. In IEEE International Conference on Image Processing (Vol. 1, pp. 117–120). IEEE Comp Soc. https://doi.org/10.1109/icip.1997.647398
 Sanby, L., Todd, I., & Keet, M. C. (2016). Comparing the Template-Based Approach to GF: the case of Afrikaans (pp. 50–53). Association for Computational Linguistics (ACL). https://doi.org/10.18653/v1/w16-3510
 Tang, Y., Zhang, X., Hu, X., Wang, S., & Wang, H. (2021). Facial Expression Recognition Using Frequency Neural Network. IEEE Transactions on Image Processing, 30, 444–457. https://doi.org/10.1109/TIP.2020.3037467
 Zhang, Q., Li, H., Li, M., & Ding, L. (2020). Feature extraction of face image based on LBP and 2-D Gabor wavelet transform. Mathematical Biosciences and Engineering, 17(2), 1578–1592. https://doi.org/10.3934/mbe.2020082
 Lien, J. J., Cohn, J. F., Kanade, T., & Li, C. C. (1998). Automated facial expression recognition based on FACS action units. In Proceedings - 3rd IEEE International Conference on Automatic Face and Gesture Recognition, FG 1998 (pp. 390–395). IEEE Computer Society. https://doi.org/10.1109/AFGR.1998.670980
 Donate, G., Bartlett, M. S., Hager, J. C., Ekman, P., & Sejnowski, T. J. (1999). Classifying facial actions. IEEE Transactions on Pattern Analysis and Machine Intelligence, 21(10), 974–989. https://doi.org/10.1109/34.799905
 Fasel, B., & Luettin, J. (2000). Recognition of asymmetric facial action unit activities and intensities. Proceedings - International Conference on Pattern Recognition, 15(1), 1100–1103. https://doi.org/10.1109/icpr.2000.905664
 Lisa, Y. T., Arun, B., Sharat, H., Andrew, P., & Bolle, R. (2003). Real World Real-time Automatic Recognition of Facial Expressions. Most.
 Lien, J.-J. J. (2013). AUTOMATIC RECOGNITION OF FACIAL EXPRESSIONS USING HIDDEN MARKOV MODELS AND ESTIMATION OF EXPRSSION INTENSITY. Journal of Chemical Information and Modeling.
 Hasan, H., AL.Jouhar, W., & Alwan, M. (2012). 3-D Face Recognition Using Improved 3D Mixed Transform. International Journal Of Biometrics And Bioinformatics, 6(1), 278-290. Retrieved from https://citeseerx.ist.psu.edu/viewdoc/download?doi=10.1.1.740.1016&rep=rep1&type=pdf
 Kanade, T., Cohn, J. F., & Tian, Y. (2000). Comprehensive database for facial expression analysis. In Proceedings - 4th IEEE International Conference on Automatic Face and Gesture Recognition, FG 2000 (pp. 46–53). IEEE Computer Society. https://doi.org/10.1109/AFGR.2000.840611
 Wallhoff, F. (2021). Facial Expressions and Emotion Database. Retrieved 10 November 2021, from https://www.jade-hs.de/team/frank-wallhoff/databases/
 Mahmod, W., Stephan, J., & Razzak, A. (2020). Facial Expression Recognition Using Fast Walidlet Hybrid Transform. Journal Port Science Research, 59–69. https://doi.org/10.36371/port.2020.3.4
Copyright (c) 2021 Journal Port Science Research
This work is licensed under a Creative Commons Attribution-ShareAlike 4.0 International License.
- Published: 2021-06-10
- Issue: Vol. 4 No. 2 (2021): Transaction on Engineering, Technology and their Applications
- Section: Articles