3D FACIAL LANDMARK-BASED DECEPTION DETECTION IN VIDEO USING GRU MODEL

Amira Himyari; Israa Hadi

doi:10.30572/2018/KJE/160211

Authors

Amira Himyari Master’s degree Student, The University of Babylon, College of Information Technology, Software Department https://orcid.org/0009-0007-3553-3498
Israa Hadi The University of Babylon, College of Information Technology, Software Department

DOI:

https://doi.org/10.30572/2018/KJE/160211

Keywords:

Facial micro expression, Facial action coding system, Mediapipe face mesh model, Feature selection-based multivariate mutual information method, Gated recurrent unit (GRU) model

Abstract

Deception detection is an interdisciplinary field that has researchers from psychology, criminology, and computer science. We propose the automated detection of deception based on facial micro expressions which occur spontaneously in response to the attempt to mask the inner emotion. It has received significant attention as an indicator of deceit, it reveals the genuine emotions that are concealed. In this paper, we first proposed a 3D 478 Mediapipe Face Mesh Model to extract facial landmarks that reflect facial micro expression, this is contrary to the traditional method, which relies on human judgment and the use of devices to detect facial micro expression. Second, a feature selection-based multivariate mutual information method was proposed to select facial landmarks that are most related to the deceptive cues and have critical influence on the classification task. Finally, a gated recurrent unit model was trained to predict deceptive behavior on a real-life trial dataset. The model successfully achieved 97% accuracy, outperforming other state-of-the-art methods.

Downloads

Download data is not yet available.

References

Alaa, R., Hussein, E. and Al-libawy, H. (2024) ‘OBJECT DETECTION ALGORITHMS IMPLEMENTATION ON EMBEDDED DEVICES: CHALLENGES AND SUGGESTED SOLUTIONS’, Kufa Journal of Engineering, 15(3), pp. 148–169.

Alaskar, H. (2023) ‘Hybrid Metaheuristics with Deep Learning Enabled Automated Deception Detection and Classification of Facial Expressions.’, Computers, Materials & Continua, 75(3).

Batina, L. et al. (2011) ‘Mutual information analysis: a comprehensive study’, Journal of Cryptology, 24(2), pp. 269–291.

Chen, F. et al. (2015) ‘2D facial landmark model design by combining key points and inserted points’, Expert Systems with Applications, 42(21), pp. 7858–7868.

D’Ulizia, A. et al. (2023) ‘Detecting Deceptive Behaviours through Facial Cues from Videos: A Systematic Review’, Applied Sciences, 13(16), p. 9188.

Dinges, L. et al. (2024) ‘Exploring facial cues: automated deception detection using artificial intelligence’, Neural Computing and Applications, pp. 1–27.

Farah, M.J. et al. (2014) ‘Functional MRI-based lie detection: scientific and societal challenges’, Nature Reviews Neuroscience, 15(2), pp. 123–131.

Islam, S. et al. (2021) ‘Non-invasive deception detection in videos using machine learning techniques’, in 2021 5th International Conference on Electrical Engineering and Information Communication Technology (ICEEICT). IEEE, pp. 1–6.

Jabberi, M. et al. (2023) ‘68 landmarks are efficient for 3D face alignment: what about more? 3D face alignment method applied to face recognition’, Multimedia Tools and Applications, 82(27), pp. 41435–41469.

Jakubowska, J. and Białecka‐Pikul, M. (2020) ‘A new model of the development of deception: Disentangling the role of false‐belief understanding in deceptive ability’, Social Development, 29(1), pp. 21–40.

Kang, J. et al. (2024) ‘Deception Detection Algorithm Based on Global and Local Feature Fusion with Multi-head Attention’, in 2024 3rd International Conference on Image Processing and Media Computing (ICIPMC). IEEE, pp. 162–168.

Khan, W. et al. (2021) ‘Deception in the eyes of deceiver: A computer vision and machine learning based automated deception detection’, Expert Systems with Applications, 169, p. 114341.

Kulkarni, A., Chong, D. and Batarseh, F.A. (2020) ‘Foundations of data imbalance and solutions for a data democracy’, in Data democracy. Elsevier, pp. 83–106.

Levine, T.R. (2014) ‘Truth-default theory (TDT) a theory of human deception and deception detection’, Journal of Language and Social Psychology, 33(4), pp. 378–392.

Li, S.Z. et al. (no date) Handbook of Face Recognition Third Edition. Available at: https://doi.org/https://doi.org/10.1007/978-3-031-43567-6.

Lugaresi, C. et al. (2019) ‘Mediapipe: A framework for perceiving and processing reality’, in Third workshop on computer vision for AR/VR at IEEE computer vision and pattern recognition (CVPR).

Luque, A. et al. (2019) ‘The impact of class imbalance in classification performance metrics based on the binary confusion matrix’, Pattern Recognition, 91, pp. 216–231.

Martinez, B. et al. (2017) ‘Automatic analysis of facial actions: A survey’, IEEE transactions on affective computing, 10(3), pp. 325–347.

Mohammed, H., Kareem, S. and Mohammed, A. (2022) ‘A COMPARATIVE EVALUATION OF DEEP LEARNING METHODS IN DIGITAL IMAGE CLASSIFICATION’, Kufa Journal of Engineering, 13(4), pp. 53–69.

Monaro, M. et al. (2022) ‘Detecting deception through facial expressions in a dataset of videotaped interviews: A comparison between human judges and machine learning models’, Computers in Human Behavior, 127, p. 107063.

Nam, B. et al. (2023) ‘FacialCueNet: unmasking deception-an interpretable model for criminal interrogation using facial expressions’, Applied Intelligence, 53(22), pp. 27413–27427.

Nguyen, H.B., Xue, B. and Andreae, P. (2016) ‘Mutual information for feature selection: estimation or counting?’, Evolutionary Intelligence, 9, pp. 95–110.

Nikbin, S. and Qu, Y. (2024) ‘A Study on the Accuracy of Micro Expression Based Deception Detection with Hybrid Deep Neural Network Models’, European Journal of Electrical Engineering and Computer Science, 8(3), pp. 14–20.

Pérez-Rosas, V. et al. (2015) ‘Verbal and nonverbal clues for real-life deception detection’, in Proceedings of the 2015 conference on empirical methods in natural language processing, pp. 2336–2346.

Prome, S.A. et al. (2024) ‘Deception detection using ML and DL techniques: A systematic review’, Natural Language Processing Journal, p. 100057.

Pudjihartono, N. et al. (2022) ‘A review of feature selection methods for machine learning-based disease risk prediction’, Frontiers in Bioinformatics, 2, p. 927312.

Rengasamy, D. et al. (2020) ‘Deep learning with dynamically weighted loss function for sensor-based prognostics and health management’, Sensors, 20(3), p. 723.

Şen, M.U. et al. (2020) ‘Multimodal deception detection using real-life trial data’, IEEE Transactions on Affective Computing, 13(1), pp. 306–319.

Shen, X. et al. (2021) ‘Catching a liar through facial expression of fear’, Frontiers in Psychology, 12, p. 675097.

Stathopoulos, A. et al. (2023) ‘Deception Detection in Videos Using Robust Facial Features with Attention Feedback’, in Handbook of Dynamic Data Driven Applications Systems: Volume 2. Springer, pp. 725–741.

Tsuchiya, K., Hatano, R. and Nishiyama, H. (2023) ‘Detecting deception using machine learning with facial expressions and pulse rate’, Artificial Life and Robotics, 28(3), pp. 509–519.

Venkatesh, S., Ramachandra, R. and Bours, P. (2019) ‘Robust algorithm for multimodal deception detection’, in 2019 IEEE Conference on Multimedia Information Processing and Retrieval (MIPR). IEEE, pp. 534–537.

Verma, M. et al. (2019) ‘LEARNet: Dynamic imaging network for micro expression recognition’, IEEE Transactions on Image Processing, 29, pp. 1618–1627.

Xing, Y. and Xiao, C. (2019) ‘A GRU model for aspect level sentiment analysis’, in Journal of Physics: Conference Series. IOP Publishing, p. 32042.

Yildirim, S., Chimeumanu, M.S. and Rana, Z.A. (2023) ‘The influence of micro-expressions on deception detection’, Multimedia Tools and Applications, 82(19), pp. 29115–29133.