A DUAL-STREAM CROSS-ATTENTION FRAMEWORK FOR FINE-GRAINED YOGA POSE CLASSIFICATION USING SPATIOTEMPORAL VISUAL CUES AND 3D SKELETAL GEOMETRY

Abstract

The accurate automated recognition of yoga postures from a single video sequence presents itself as an intricate problem, lying in the intersection between computer vision and biomechanics, and holds enormous potential towards achieving superior fitness training, physical therapy, and man-machine interaction applications. Existing techniques are mainly based on either appearance features, susceptible to changes in clothing and background, or geometric keypoint features that lack contextual information. The main focus of this thesis is the proposition that a combination of the two modalities can outperform each of them independently. We introduce a complete pipeline starting from the pre-processing of untrimmed online videos of the 3DYoga90 dataset and extracting the 3D world-coordinate skeletons with the use of MediaPipe. Two novel backbones are fine-tuned on an extremely strict cross-subject split as the unimodal baselines – the first one is Channel-wise Topology Refinement Graph Convolutional Network (CTR-GCN) on the skeleton modality, and the other is Video Swin Transformer. We propose a logit-level cross-modal fusion model, termed LogitPerceiver, which merges the predictions without requiring full-fledged end to-end training due to its high computational cost. In addition, the model projects the backbone’s logits into a unified space and employs a trainable query to concentrate on two input streams as well as a gated adjustment to the skeleton prediction results. The late-fusion framework enables adaptive and context-dependent fusion of information related to the input samples from both visual and geometric modalities. Experiments carried out using the 3DYoga90 dataset reveal that the skeleton-based CTR GCN can achieve a balanced accuracy of 86.96% compared to 81.64% by the Video Swin Transformer. The proposed fusion framework is able to score 89.64% balanced accuracy, 91.06% overall accuracy, and 88.96% macro-averaged F1-score, beating the performance of all unimodal baselines and establishing a new baseline on the dataset. This thesis provides a reproducible multimodal framework, a light fusion network compatible with frozen experts, and a strong benchmark dataset for yoga position classification.