EMOTION RECOGNITION USING FEDERATED PARADIGM BASED ON PHYSIOLOGICAL SIGNALS

Abstract

Emotions are complex psychological states that involve physiological arousal, cognitive interpretation, and behavioural expression, influencing how individuals experience and respond to events. Various modalities, including facial expressions, subjective psychological tests, and physiological signals, can recognize human emotional states, out of which physiological signals have several advantages over the other modalities, including greater sensitivity to internal feelings and the ability to provide continuous, real-time data for accurate emotional monitoring. In this context, automated Emotion Recognition Systems (ERS) are gaining popularity in predicting human emotions and enhancing health and decision-making. These systems utilize machine learning (ML) and deep learning (DL) algorithms to process wearable biosensor data and classify emotions with high precision and reliability. The automated ERS using traditional ML and DL algorithms directly accesses the user’s raw physiological data to train the model and further classify emotions. It results in a significant loss of privacy protection for the user’s sensitive physiological information. This thesis aims to enhance the automated ERS by improving data privacy concerns and integrity using a novel Federated Learning (FL) paradigm. Unlike traditional machine learning techniques, the FL paradigm creates a decentralized environment (client and server ends), allowing users to transmit only the model weights generated locally on their device rather than the complete raw physiological data to a central server. The server aggregates these weights to create a global model aggregator that updates after each iteration. Apart from privacy, this thesis addresses the research gaps for (1) Lack of multi-modality in FL-based automated ERS with physiological data input; (2) Restricted emotion dimensions in FL-based automated ERS, exploring a smaller range of emotions, (3) Existing FL-based automated ERS fails to address the data heterogeneities occurring in a federated environment. v Firstly, the thesis presents a comprehensive literature review of automated ERS using physiological signals. It includes emotion models, physiological signals, the relation between emotions and physiological signals, technical background including data processing for physiological signals, ML and DL models, and the related works of FL for ERS. Secondly, the thesis proposes a privacy-preserved emotion recognition architecture for multi-modal physiological data combining EEG, ECG, GSR and RESP signal data. For this, the thesis proposes an FL-based Multi-modal Emotion Recognition System (F-MERS) for classifying emotions using Valence, Arousal, and Dominance emotion dimensions. The thesis validates the proposed F-MERS with three different emotion datasets, proving it robust achieving an average testing accuracy of 83.02% with AMIGOS, 86.51% with DEAP, and 75.19% with DREAMER. It assesses its classification performance, scalability with different client distributions, convergence speed, and communication computation (in terms of averaging and training times) and discusses the experimental results. The F-MERS did not address data heterogeneity present in the multi-modal physiological data and the federated environment. Thirdly, the thesis overcomes the challenge of data heterogeneity, lacking in the existing works of FL, by proposing an enhanced Attention-based FederatedLearning for Emotion recognition using Multi-modal Physiological data (AFLEMP) architecture. AFLEMP removes the Variation Data Heterogeneity (VDH) occurring while combining multiple physiological data together by implementing attention mechanisms at the client end. It proposes a novel Scaled-Weighted Federated Averaging (SWFA) algorithm for the server end to reduce the Imbalanced Data Heterogeneity (IDH) occurring due to imbalanced data distribution at the client end within a federated environment. The thesis validates the proposed AFLEMP with two different emotion datasets, achieving the testing accuracy of 90.11% with AMIGOS and 85.12% with DREAMER, proving it to be robust. For assessing the AFLEMP, the thesis evaluates and contrasts it with other FL algorithms for ERS in terms of classification performance, convergence speed and communication computation (in terms of averaging and training times) and discusses its experimental results. Fourthly, the thesis presents that the proposed AFLEMP provides multi-dimensionality in terms of emotion dimensions, which is lacking in the existing works of FL for ERS. vi For this, the thesis proposes AFLEMP to classify a wide spectrum of emotions using a 3D-VAD model of emotions (including Valence-Arousal-Dominance together). The thesis presents the experimental results of the proposed AFLEMP for its classification performance for Valence-Arousal-Dominance together and individually. The research presented in this thesis contributes to the field of emotion recognition based on physiological signals by exploring FL. The FL techniques can assist for providing privacy of emotion recognition systems using multi-modal physiological signals. The proposed F-MERS and AFLEMP are robust, efficient in communication, and scalable.