DEVELOPMENT OF MODEL FOR MULTIMODAL SARCASM DETECTION

Abstract

Sarcasm detection is used to single out natural language statements where intended meaning differs from what the surface meaning implies. A number of tasks in natural language processing areas like analysis of sentiments as well as mining of opinions use sarcasm detection underneath. Many of the key research in the area of sarcasm detection primarily focus on only text-based input. In the present day scenario , there has been a sudden explosion in the amount of multimodal data mainly due to social media. As a result of that, users these days are not just limited to text while expressing themselves , but also make heavy use of visuals like in images and videos. The objective of this research work is to incorporate multimodal data so as to enhance the performance of present sarcasm detection algorithms. Multiple methods that leverage data in the form of image and text, and also as a combination of both, have been presented thus far.We present a unique architecture which works on the Robustly Optimized BERT pre-training approach or RoBERTa which is nothing but facebook modified version of well known model BERT having co-attention layer on top for including the incongruity in the context between attributes of the image and input text. Using Gated Recurrent Unit(GRU), the text-based features are extracted and the features of the image are retrieved by conditioning the image through Feature-wise Linearly Modulated ResNet Blocks. These combined with CLS token from the model RoBERTa are used to obtain the final predictions. In the results, we show performance enhancements from the proposed model when it is stacked against the other latest models with competitive results which make use of the same publicly available Twitter dataset.