EVALUATING LARGE LANGUAGE MODEL ARCHITECTURES FOR SENTIMENT ANALYSIS ACROSS MULTIPLE DATASETS

Abstract

What is now referred to as LLMs and their rapid growth have touched every sphere of Natural Language Processing. Therefore, sentiment analysis continues to be an appli cation worth understanding as it provides insights into opinions, emotions, and attitudes being expressed in textual data. However, with the existence of LLMs, interpretation is still a challenge when it comes to subtle language use, complex linguistic phenomena such as sarcasm, and blatant issues of biases within models. The objective of this thesis is to analyse and assess the performance of different state-of-the-art LLMs in sentiment analysis on various datasets and under different sentiment paradigms. A full experimental setup for such analysis was developed, which uses a handful of major LLMs, including proprietary ones such as GPT-4o for benchmarking the cutting-edge performance and also opensource variants like Llama 3, Mistral Large/Mixtral-8x22B, Falcon LLM, and XLM-RoBERTa for their accessibility, transparency, and customizability. The experi ments covered a variety of sentiment analysis tasks such as those for binary classification (IMDb Movie Reviews, SST-2), aspect-based sentiment analysis (MAMS-for-ABSA), mul tilingual sentiment analysis (Multilingual Amazon Reviews Corpus), and tests for harder cases such as sarcasm detection. To achieve this, a comprehensive experimental frame work was developed, utilizing a selection of prominent LLMs, including proprietary models such as GPT-4o for benchmarking against cutting-edge performance, and open-source al ternatives like Llama 3, Mistral Large/Mixtral-8x22B, Falcon LLM, and XLM-RoBERTa for their accessibility, transparency, and customizability. Experiments were conducted on a range of sentiment analysis tasks, with binary classification (IMDb Movie Reviews, SST-2), aspect-based sentiment analysis (MAMS-for-ABSA), multilingual sentiment anal ysis (Multilingual Amazon Reviews Corpus), and challenging scenarios such as sarcasm detection (Twitter Sentiment Analysis Datasets). The research explored the efficacy of zero-shot, few-shot, and fine-tuning approaches, emphasizing the critical role of prompt engineering in optimizing LLM performance. For sentiment analysis, older deep learning iv architectures like Convolutional Neural Networks (CNNs) and Recurrent Neural Networks (RNNs - including LSTMs) primarily focused on capturing local patterns (CNNs) or se quential dependencies (RNNs) within text.