Automated deception detection systems are now crucial in many fields, such as commerce, medicine, education, law enforcement, and national security, in this digital era. These systems are needed because human interviewers have limitations that can lead to false accusations and ineffective detection. To address this, researchers from Tokyo University of Science have proposed a machine-learning approach that combines facial expressions and pulse rate data to improve deception detection.
The researchers aim to develop a fair and reliable system that can assist in interviews with crime victims, suspects, and individuals with mental health issues. They believe that precise suspect classification is essential to avoid misidentifications and uphold ethical and legal considerations. They suggest a human-in-the-loop approach to ensure ethical compliance while enabling widespread applications in crucial decision-making processes.
Previous studies have explored deception detection using different methods. Some studies used multi-modal information from videos or focused on differences in valences and arousal between truthful and deceptive speakers. Another study used non-verbal behavior cues such as facial micro-movements, changes in gaze, and blink rates. However, these studies had limitations, such as unnatural role-playing approaches for data collection.
In contrast, the Tokyo University of Science study introduces a natural approach where subjects freely improvise deceptive behaviors to enhance accuracy. The researchers used machine learning, specifically the Random Forest (RF) technique, to create a deception detection model that integrates facial expressions and pulse rate data. They collected data from four male graduate students having discussions while making deceptive statements. They recorded facial expressions using a web camera and measured pulse rates using a smartwatch during the interviews.
The process involved standard machine learning steps, including data collection, labeling, feature extraction, preprocessing, and classification. Facial landmarks were extracted from recorded videos using the OpenFace library, and various facial features and pulse rate were derived from these landmarks. Preprocessing steps were performed to remove missing values, filter outliers, and balance positive and negative cases.
The Random Forest (RF) model was trained and evaluated using 10-fold cross-validation, with performance metrics used to assess its effectiveness. Remarkably, the method showed similar performance during actual remote job interviews compared to cross-validation results, proving its real-world applicability. The analysis of feature importance highlighted specific facial features, pulse rate, and gaze and head movements as significant indicators of deception.
Overall, this research provides a practical and promising approach to detecting deceptive statements in remote interviews using machine learning and facial feature analysis. The proposed method showed promising accuracy and F1 scores for different subjects. However, further studies are needed to handle multi-class classification and include psychological assessments for a more comprehensive analysis.
Despite the limitations in dataset size, this research lays the foundation for interviewers interested in using automatic deception detection systems while emphasizing the importance of ethical considerations and legal compliance.