EXPLAINABLE STAGE-AWARE BHARATANATYAM MUDRA RECOGNITION BY INTEGRATING GEOMETRIC HAND LANDMARK ENCODING AND DOUBLE TRANSFER LEARNING

Abstract

Bharatanatyam mudras constitute an essential component of Indian classical dance, serving as a medium for storytelling, emotional expression, and semantic communication. Automatic recognition of these hand gestures is a challenging task due to subtle inter-class variations, complex finger articulations, viewpoint differences, and limited annotated datasets with exist ing gesture recognition approaches often relyinng solely on visual appearance features and lack robustness, interpretability, and cross-domain adaptability and in addition, limited research has explored the integration of multimodal learning and explainable artificial intelligence for cul turally significant gesture recognition tasks such as Bharatanatyam mudra analysis. This thesis proposes an explainable multimodal dual-stage transfer learning framework for Bharatanatyam mudra recognition by integrating transformer-based visual learning with geo metric hand landmark representations which employs a Swin Transformer Tiny backbone for RGB image feature extraction and a dedicated landmark-processing branch utilizing Medi aPipe hand keypoints to capture structural hand articulation and thus extracted features from both modalities are fused to enhance discriminative representation learning and improve classi fication performance. To address data scarcity and improve transferability, a dual-stage transfer learning strategy is introduced, where the model is initially pretrained on an Indian Sign Lan guage (ISL) gesture dataset and subsequently fine-tuned on Bharatanatyam mudras. To add more, explainable artificial intelligence techniques such as GradCAM-based visualization are incorporated to help with the process of the interpreting model predictions and identification of anatomically significant hand regions influencing recognition decisions. Extensive experiments are conducted on Bharatanatyam mudra datasets along with cross domain evaluation using external gesture datasets to assess robustness and generalization capa bility and the proposed framework is evaluated using multiple performance metrics including accuracy, precision, recall, and F1-score, along with cross-dataset experiments to further ana lyze the effects of domain shift and transferability and the experimental results demonstrate that the proposed multimodal dual-stage transfer learning framework achieves robust and highly ac curate Bharatanatyam mudra recognition while providing improved interpretability and cross domain adaptability thereby contributing towards the development of culturally aware artificial intelligence systems for digital heritage preservation, intelligent dance analysis, and human centered gesture understanding.