A MULTIMODAL DEEP LEARNING SYSTEM FOR ENHANCED EMOTION DETECTION

Abstract

Detecting emotions from multimodal data is increasingly significant in various do mains such as mental health support, customer experience enhancement, and human computer interaction. As digital communication grows, understanding the emotional tone of conversational content has become essential for building responsive and human centric applications. This study explores a multimodal deep learning system that aims to improve the accuracy and class-fairness of emotion detection by fusing visual, textual, and audio modalities. In-depth analysis is conducted on a multimodal architecture that combines ResNet 18 for visual features with a hybrid text stream incorporating RoBERTa, FastText, and the NRC Emotion Lexicon. Audio is selectively re-routed through OpenAI Whisper transcription to augment the text stream. The model is evaluated on the IEMOCAP dataset—a widely used benchmark for multimodal emotion recognition—chosen for its diversity in conversational language and emotion categories. A three-layer MLP classifier on the 928-dimensional fused descriptor achieves 86.48% accuracy and Macro F1-score of 0.8690, a statistically significant improvement (Z=16.94, p < 0.05) over unimodal baselines. The aim of this research work is to demonstrate that pre-fusion linguistic enrich ment paired with class-fair evaluation closes a substantial portion of the gap left open by prior IEMOCAP multimodal work. The results of experiments indicate that pre fusion enrichment of the linguistic features is more effective than complex fusion schemes based on a transformer network. This study not only unveils the potential of hybrid feature engineering but also reveals where more can be gained, the keys to even more sophisticated and sensitive multimodal emotion recognition system.