MULTI MODAL HUMAN ACTIVITY RECOGNITION USING DEEP LEARNING

Abstract

Human activity recognition has been a subject of intense research, as it is at the cross roads of several practical areas such as healthcare monitoring, rehabilitation, fitness analysis, smart environments, surveillance, and human-computer interaction. Pose esti mation methods are used to extract human joint information from visual and sensor data, which are essential for these systems to represent human motions. While significant strides have been made with deep learning, it is not easy to deploy the technology in unstructured environments. Benchmarking performance does not address all the issues that arise from using human postures, occlusion, environmental conditions and from different users. A successful system must achieve recognition accuracy, be computation efficient, account for the temporal aspects of activity sequences and generalize in a consistent manner across different scenarios. The present study explores the design and deployment of deep learning models to realize robust human activity analysis. Pose estimation architectures from convolutional and transformer-based approaches are assessed based on standard metrics like mean Average Precision, Percentage of Correct Keypoints and Mean Per Joint Position Error on the COCO, MPII and the Human3.6M dataset. More complex spatial relationships are easier to capture with more powerful models, but require more computational power. The performance is significantly influenced by viewpoint variation, occlusion, and the possible bias in the datasets, and this reinforces the fact that the selection of the model should be made based on the requirements of a given deployment, and not just the benchmark. A skeleton-based model for both spatial structure and temporal motion is proposed to investigate the effect of pose estimation quality on activity recognition. The key points of the skeleton are extracted and converted from several pose estimation backends to spatial and kinematic representations that include the coordinates of the joints and the distances and the angles between them. They have been processed by dual-branch CNN-LSTM architecture which learns spatial patterns and motion dynamics at the same time. MediaPipe Pose sets a new recognition rate of 94% and offers lightweight recognition alternatives like YOLOv11 Pose with efficient inference times of 125.0ms, 8.0fps, and a model size of 3.68MB. The results indicate that accuracy is not the only criterion that should be used in selecting a model, and that the practical limitations of the deployment environment must be taken into account. v In scenarios where visual data is not sufficient, a Multi-Modal Hierarchical Temporal Transformer framework is developed, which integrates the streams of wearable and ambient sensors. An adaptive fusion mechanism dynamically weights the contribu tion from each sensor to perform the modality specific feature extraction, and the transformer encoder layers with self-attention are used to model long-range temporal dependencies. By using a hierarchical learning strategy, locomotion patterns can be learned concurrently with higher level activities from a shared representation. For Opportunity++, the macro F1 score in Leave-One-Subject-Out cross validation for locomotion and higher-level activity recognition is 0.9290 and 0.8111 respectively, with accuracy of 91.22% and 81.49% respectively. On PAMAP2 and DSADS, accuracy reaches 92.03% and 84.81%. The lightweight version decreases the parameters from 1.16M to 0.009M and the model size from 4.48MB to 0.05MB, enabling deployment on resource-constrained devices. The findings of this research clearly indicate that none of the single modalities or the single model design in the tested scenarios provide reliable activity recognition without any influence in real world conditions. Together, accurate motion representation, well-structured temporal modeling and careful integra tion of complementary data sources are the building blocks of systems that can support real-world applications, such as in the context of healthcare monitoring, assisted living, smart homes, and context-aware computing.