A HYBRID DEEP LEARNING AND TRANSFORMER BASED MODEL FOR MULTIMODAL SARCASAM DETECTION

Abstract

In the past ten years social media has transformed how we communicate online and also has seen an increase in the use of casual, vague, at times sarcastic writing. While sarcasm is the most complex one can interpret, and with words going against images is more complicated today's sentiment analysis systems. For the above purpose which is to improve the above we formulated a Multimodal Sarcasm Detection system in the present piece of work which is designed to detect sarcasm from the whole text and also visual elements. The methods used were: RoBERTa (Robustly Optimized BERT Pretraining Approach) for embedding text semantically, a Bidirectional Long Term Short Term Memory (BiLSTM) network for processing sequential context data for words in sentences and a pre learned ResNet50 convolutional neural network (CNN) for features extraction at higher order of visual information. Here we present what we term Emotion Heads: special projection networks, designed for each modality, that map text and visual representation to a what we may call emotion aware feature space before fusing modalities together. In fact, this kind of architecture generates the same situation as it typically generates the model of what’s characteristically sarcastic: the occurrence of a cross modal emotional incongruity. The Multimodal Sarcasm Detection (MMSD) dataset contains social media image-text pairs, and experiments were run on this data set using a preprocessing pipeline that includes use of the subword tokenization process from the RoBERTa tokenizer; standardized normalization for images based on ImageNet; and trained for 3 epochs using the AdamW optimizer with a learning rate of 2e-5. The best validation accuracy (96.36%) and the Best validation F1-score (96.37%) was determined through this training. The test accuracy and weighted F1-score in the test set were 93.97% and 94.00% respectively, respectively, for its tested performance. The single-modal baselines and the two previous-state-of-the-art multimodal approaches were considerably outperformed by the model. Additionally, the test-set ROC-AUC was greater than 0.95, indicating good discriminating power. The F1-scores for both classes are around 0.94 per class, showing that the model performs well without being biased for any particular class, a usual problem with class imbalance for sarcasm detection.