EMOTION-AWARE MULTIMODAL SARCASM DETECTION USING DEEP LEARNING TECHNIQUES

Abstract

Sarcasm detection in digital communication has become increasingly challenging as users employ sophisticated combinations of textual and visual elements to convey ironic meaning. Traditional text-based approaches and existing multimodal methods struggle to capture the emotional incongruity that is fundamental to sarcastic expressions, repre- senting a significant gap in current multimodal sarcasm detection research. This thesis proposes an emotion-aware multimodal framework for sarcasm detection that systematically integrates emotional features from both textual and visual modal- ities. The approach enhances existing deep learning architectures, specifically Bidirec- tional Long Short-Term Memory (BiLSTM) networks and Graph Convolutional Networks (GCNs), with emotion recognition capabilities utilizing DistilRoBERTa for textual emo- tion analysis and computer vision techniques for visual emotion recognition. The frame- work was primarily developed and optimized using the MMSD2.0 dataset, followed by comprehensive cross-dataset evaluation on MMSD Original and MEMOTION datasets to assess generalization capabilities. Experimental results demonstrate that systematic integration of emotional features from both modalities significantly improves sarcasm detection performance. The BiLSTM- based emotion-aware architecture achieves the best overall performance on MMSD2.0 with 83.12% accuracy, 81.10% precision, 85.07% recall, and 83.04% F1-score, while the GCN-based approach achieves competitive results with 81.29% accuracy and 81.97% F1- score. Cross-dataset evaluation reveals robust generalization capabilities, with the BiL- STM model maintaining 79.86% accuracy on MMSD Original and 80.45% accuracy on MEMOTION, demonstrating effective transferability of emotion-aware features across dif- ferent data distributions and even different domains. These findings establish the first em- pirical evidence for the effectiveness of dual-modality emotion integration in multimodal sarcasm detection, providing a robust foundation for future research in emotion-aware approaches to understanding digital communication nuances.