Overview of facial expression recognition technology

doi:10.12267/j.issn.2096-5931.2022.08.013

Abstract

Abstract:

Facial expression recognition is a research direction involving knowledge in many fields such as machine learning, image processing, psychology, etc., and its application scenarios are also quite broad. Recently, the study of facial expression recognition technology by emotion calculation has attracted attention. Facial expression recognition tasks generally consist of four parts: obtaining facial images, image preprocessing, feature extraction, and feature classification. Facial expression images are generally obtained directly from related data sets. Therefore, the step of image preprocessing is first introduced, followed by a sequential introduction of traditional research methods and deep learning methods in feature extraction and feature classification. Finally, a comprehensive elaboration of the data sets, development trends and challenges related to facial expression recognition is described.

Key words: artificial intelligence, facial expression recognition, feature extraction, feature classification

CLC Number:

TP391.41

LIU Jianhua, TANG Lei. Overview of facial expression recognition technology[J]. Information and Communications Technology and Policy, 2022, 48(8): 89-96.

Add to citation manager EndNote|Ris|BibTeX

URL:

http://ictp.caict.ac.cn/EN/10.12267/j.issn.2096-5931.2022.08.013

http://ictp.caict.ac.cn/EN/Y2022/V48/I8/89

Figures/Tables 1

References 49

[1]	MEHRABIAN A, RUSSELL JA. An Approach to Environmental Psychology[M]. Cambridge: MITPress, 1974.
[2]	EKMAN R. What the face reveals: basic and applied studies of spontaneous expression using the Facial Action Coding System (FACS)[M]. Oxford University Press, USA, 1997.
[3]	EKMAN P, FRIESEN W V. Facial action coding system[J]. Environmental Psychology & Nonverbal Behavior, 1978.
[4]	XIONG X, DE LA TORRE F. Supervised descent method and its applications to face alignment[C]// Proceedings of the IEEE conference on computer vision and pattern recognition, 2013: 532-539.
[5]	ZHU X, RAMANAN D. Face detection, pose estimation, and landmark localization in the wild[C]// 2012 IEEE conference on computer vision and pattern recognition. IEEE, 2012: 2879-2886.
[6]	ASTHANA A, ZAFEIRIOU S, Cheng S, et al. Robust discriminative response map fitting with constrained local models[C]// Proceedings of the IEEE conference on computer vision and pattern recognition, 2013: 3444-3451.
[7]	KING DE. Dlib-ml: A machine learning toolkit[J]. The Journal of Machine Learning Research, 2009, 10: 1755-1758.
[8]	ZHANG K, ZHANG Z, LI Z, et al. Joint face detection and alignment using multitask cascaded convolutional networks[J]. IEEE Signal Processing Letters, 2016, 23(10): 1499-1503. doi: 10.1109/LSP.2016.2603342 URL
[9]	ALP GULER R, TRIGEORGIS G, ANTONAKOS E, et al. Densereg: fully convolutional dense shape regression in-the-wild[C]// Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, 2017: 6799-6808.
[10]	HU P, RAMANAN D. Finding tiny faces[C]// Proceedings of the IEEE conference on computer vision and pattern recognition, 2017: 951-959.
[11]	SHIN M, KIM M, KWON DS. Baseline CNN structure analysis for facial expression recognition[C]// 2016 25th IEEE International Symposium on Robot and Human Interactive Communication (RO-MAN). IEEE, 2016: 724-729.
[12]	LI J, LAM EY. Facial expression recognition using deep neural networks[C]// 2015 IEEE International Conference on Imaging Systems and Techniques (IST). IEEE, 2015: 1-6.
[13]	EBRAHIMI KAHOU S, MICHALSKI V, KONDA K, et al. Recurrent neural networks for emotion recognition in video[C]// Proceedings of the 2015 ACM on International Conference on Multimodal Interaction, 2015: 467-474.
[14]	BARGAL S A, BARSOUM E, FERRER CC, et al. Emotion recognition in the wild from videos using images[C]// Proceedings of the 18th ACM International Conference on Multimodal Interaction, 2016: 433-436.
[15]	PITALOKA DA, WULANDARI A, BASARUDDIN T, et al. Enhancing CNN with preprocessing stage in automatic emotion recognition[J]. Procedia computer science, 2017, 116: 523-529. doi: 10.1016/j.procs.2017.10.038 URL
[16]	KUO CM, LAI S H, SARKIS M. A compact deep learning model for robust facial expression recognition[C]// Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition Workshops, 2018: 2121-2129.
[17]	YIN X, YU X, SOHN K, et al. Towards large-pose face frontalization in the wild[C]// Proceedings of the IEEE international conference on computer vision, 2017: 3990-3999.
[18]	HUANG R, ZHANG S, LI T, et al. Beyond face rotation: Global and local perception gan for photorealistic and identity preserving frontal view synthesis[C]// Proceedings of the IEEE International Conference on Computer Vision, 2017: 2439-2448.
[19]	TRAN L, YIN X, LIU X. Disentangled representation learning gan for pose-invariant face recognition[C]// Proceedings of the IEEE conference on computer vision and pattern recognition, 2017: 1415-1424.
[20]	周书仁, 梁昔明, 朱灿, 等. 基于ICA和HMM 的表情识别[J]. 中国图象图形学报, 2008, 13(12): 2321-2328.
[21]	周书仁. 人脸表情识别算法分析与研究[D]. 长沙: 中南大学, 2009.
[22]	KYPEROUNTAS M, TEFAS A, PITAS I. Salient feature and reliable classifier selection for facial expression classification[J]. Pattern Recognition, 2010, 43(3): 972-986. doi: 10.1016/j.patcog.2009.07.007 URL
[23]	GU W, XIANG C, VENKATESH YV, et al. Facial expression recognition using radial encoding of local Gabor features and classifier synthesis[J]. Pattern Recognition, 2012, 45(1): 80-91. doi: 10.1016/j.patcog.2011.05.006 URL
[24]	COHN JF, ZLOCHOWER AJ, LIEN JJ, et al. Feature-point tracking by optical flow discriminates subtle differences in facial expression[C]// Proceedings Third IEEE International Conference on Automatic Face and Gesture Recognition. IEEE, 1998: 396-401.
[25]	ANDERSON K, MCOWAN PW. A real-time automated system for the recognition of human facial expressions[J]. IEEE Transactions on Systems, Man, and Cybernetics, Part B (Cybernetics), 2006, 36(1): 96-105. doi: 10.1109/TSMCB.2005.854502 URL
[26]	TSALAKANIDOU F, MALASSIOTIS S. Real-time 2D+3D facial action and expression recognition[J]. Pattern Recognition, 2010, 43(5): 1763-1775. doi: 10.1016/j.patcog.2009.12.009 URL
[27]	HINTON GE, OSINDERO S, Teh YW. A fast learning algorithm for deep belief nets[J]. Neural computation, 2006, 18(7): 1527-1554. doi: 10.1162/neco.2006.18.7.1527 URL
[28]	LIU P, HAN S, MENG Z, et al. Facial expression recognition via a boosted deep belief network[C]// Proceedings of the IEEE conference on computer vision and pattern recognition. 2014: 1805-1812.
[29]	LIU M, LI S, SHAN S, et al. Au-inspired deep networks for facial expression feature learning[J]. Neurocomputing, 2015, 159: 126-136. doi: 10.1016/j.neucom.2015.02.011 URL
[30]	HINTON GE, SALAKHUTDINOV RR. Reducing the dimensionality of data with neural networks[J]. Science, 2006, 313(5786): 504-507. doi: 10.1126/science.1127647 URL
[31]	ZENG N, ZHANG H, SONG B, et al. Facial expression recognition via learning deep sparse autoencoders[J]. Neurocomputing, 2018, 273: 643-649. doi: 10.1016/j.neucom.2017.08.043 URL
[32]	WU Y, QIU W. Facial expression recognition based on improved local ternary pattern and stacked auto-encoder[C]// AIP Conference Proceedings. AIP Publishing LLC, 2017, 1864(1): 020131.
[33]	SUN B, LI L, ZHOU G, et al. Facial expression recognition in the wild based on multimodal texture features[J]. Journal of Electronic Imaging, 2016, 25(6): 061407. doi: 10.1117/1.JEI.25.6.061407 URL
[34]	SUN B, LI L, ZHOU G, et al. Combining multimodal features within a fusion network for emotion recognition in the wild[C]// Proceedings of the 2015 ACM on International Conference on Multimodal Interaction, 2015: 497-502.
[35]	GIRSHICK R, DONAHUE J, DARRELL T, et al. Rich feature hierarchies for accurate object detection and semantic segmentation[C]// Proceedings of the IEEE conference on computer vision and pattern recognition, 2014: 580-587.
[36]	CUI R, LIU M, LIU M. Facial expression recognition based on ensemble of mulitple CNNs[C]// Chinese Conference on Biometric Recognition. Springer, Cham, 2016: 511-518.
[37]	孙晓, 潘汀, 任福继. 基于 ROI-KNN 卷积神经网络的面部表情识别[J]. 自动化学报, 2016, 42(6): 883-891.
[38]	WANG X, WANG X, NI Y. Unsupervised domain adaptation for facial expression recognition using generative adversarial networks[J]. Computational Intelligence and Neuroscience, 2018:1-10.
[39]	LAI YH, LAI SH. Emotion-preserving representation learning via generative adversarial network for multi-view facial expression recognition[C]// 2018 13th IEEE International Conference on Automatic Face & Gesture Recognition (FG 2018). IEEE, 2018: 263-270.
[40]	ZHANG F, ZHANG T, MAO Q, et al. Joint pose and expression modeling for facial expression recognition[C]// Proceedings of the IEEE conference on computer vision and pattern recognition, 2018: 3359-3368.
[41]	ZHANG Y, JI Q. Active and dynamic information fusion for facial expression understanding from image sequences[J]. IEEE Transactions on pattern analysis and machine intelligence, 2005, 27(5): 699-714. doi: 10.1109/TPAMI.2005.93 URL
[42]	WANG TH, LIEN JJJ. Facial expression recognition system based on rigid and non-rigid motion separation and 3D pose estimation[J]. Pattern Recognition, 2009, 42(5): 962-977. doi: 10.1016/j.patcog.2008.09.035 URL
[43]	徐文晖, 孙正兴. 面向视频序列表情分类的LSVM算法[J]. 计算机辅助设计与图形学学报, 2009, 21(4): 542-548.
[44]	徐琴珍, 章品正, 裴文江, 等. 基于混淆交叉支撑向量机树的自动面部表情分类方法[J]. 中国图象图形学报, 2008, 13(7):1329-1334.
[45]	New handbook of methods in nonverbal behavior research[M]. Oxford University Press, 2008.
[46]	LUCEY P, COHN JF, MATTHEWS I, et al. Automatically detecting pain in video through facial action units[J]. IEEE Transactions on Systems, Man, and Cybernetics, Part B (Cybernetics), 2010, 41(3): 664-674. doi: 10.1109/TSMCB.2010.2082525 URL
[47]	DU S, TAO Y, MARTINEZ A M. Compound facial expressions of emotion[J]. Proceedings of the National Academy of Sciences, 2014, 111(15): E1454-E1462.
[48]	LI Y, ZENG J, SHAN S, et al. Occlusion aware facial expression recognition using cnn with attention mechanism[J]. IEEE Transactions on Image Processing, 2018, 28(5): 2439-2450. doi: 10.1109/TIP.2018.2886767 URL
[49]	HE K, ZHANG X, REN S, et al. Deep residual learning for image recognition[C]// Proceedings of the IEEE conference on computer vision and pattern recognition, 2016: 770-778.