DEVELOPMENT OF FRAMEWORK FOR FACIAL EMOTION RECOGNITION

Abstract

Human ideas and sentiments are mirrored in facial expressions. Facial emotion recognition (FER) is a crucial type of visual data that can be utilized to deduce a person’s emotional state. It gives the spectator a plethora of social cues, such as the viewer’s focus of attention, emotion, motivation, and intention. It is considered a powerful instrument for silent communication. AI-based facial recognition systems can be deployed at different areas like bus stations, railway stations, airports, or stadiums to help security forces identify potential threats. In this thesis, different aspects of facial emotion recognition are explored and addressed. A facial emotion recognition (FER) has significantly advanced over the past few decades, but issues related to occlusion robustness and pose invariance, are relatively less encountered, specifically in uncontrolled environments. Therefore, a novel framework is presented in this thesis that can capture geometric facial features from occluded and pose variant images and classify the facial emotions accurately. Facial emotion recognition has garnered significant interest in recent years due to its wide-ranging applications in human-computer interaction (HCI), affective computing, and psychological research. In this thesis, a capsule neural network-based model is introduced which incorporates an attention mechanism to improve the classification performance. More specifically, an empirical study is performed on the effectiveness of various attention mechanisms in the context of FER. Considering this aspect, four attention techniques are explored and compared: channel attention, spatial attention, CBAM attention, self-attention, and multi-head attention (MHA). The experimental results show the distinct impacts of each attention mechanism on improving the recognition performance of different facial expressions. Along with the basic emotions addressed above, there are complex expressions which are made up of two basic emotions like “Happily Disgusted”, “Happily Sur prised”, “Sadly Surprised”, etc. Compound emotion provides a richer understanding of the human emotional state by capturing the nuanced combinations of the basic seven emotions. To recognize compound emotion, this thesis contributes by present vii ing a lightweight Swin Transformer model (LSwin-CBAM) for compound emotion recognition. Facial expression recognition systems can have advanced applications like public safety, information management and retrieval, social media, psychological studies, and patient care. Addressing its application in driver safety by recognizing the driver’s emotions can enhance driving comfort, safety, and the adoption of intel ligent vehicles. This thesis presents a Vision Transformer-based framework capable of recognizing driver emotions across various resolutions. It offers an improved solu tion for emotion recognition, even with small datasets. In this thesis, comprehensive experiments have been carried out on several FER datasets, including RAF-DB, Af fectNet, EmotioNet, CK+, JAFFE, D3S, and KMU-FED to show the effectiveness of the proposed frameworks.