SEMANTIC-GUIDED DEEP LEARNING FRAMEWORKS FOR SCENE RECOGNITION: A COMPARATIVE STUDY OF CNN AND TRANSFORMER MODELS

Abstract

Indoor scene recognition remains a challenging problem in computer vision due to large intra-class variation, strong inter-class similarity, and the complex contextual relationships that exist between objects and spatial layouts within indoor environments. Unlike object recognition, scene understanding requires the model to interpret not only the presence of semantic entities but also their spatial organization and contextual interactions. Conventional visual recognition approaches based solely on appearance features often struggle to capture these higher-level semantic relationships, particularly in scenes where multiple categories share similar visual structures. Semantic guidance has therefore emerged as an effective strategy for improving scene understanding by incorporating object-level contextual information into the recognition process. This thesis investigates how semantic supervision interacts with different deep neural representation architectures for indoor scene recognition. Rather than focusing solely on improving classification accuracy through larger models or architectural complexity, the study examines how the underlying representation structure of a backbone influences the effectiveness of semantic-guided feature learning. The work is structured as a progressive investigation across convolutional and transformer-based architectures under a consistent semantic-aware learning framework. The first phase of the study explores semantic-guided scene recognition using convolutional neural networks. A dual-branch framework consisting of an RGB branch and a semantic branch is employed, where semantic features derived from segmentation maps are integrated with visual representations through attention-based fusion. Within this framework, the effect of backbone architecture is analyzed by comparing ResNet 50 and ResNeXt-50 (32×4d) under identical training and fusion conditions. Experimental observations show that ResNeXt produces stronger scene representations and achieves improved recognition performance on the MIT Indoor-67 dataset. The results suggest that aggregated residual transformations and increased representational diversity enable more effective semantic-guided feature interaction than standard residual learning. Building upon these observations, the second phase extends the investigation to transformer-based architectures in order to analyze how different representation formats respond to semantic supervision. The study evaluates Vision Transformers and hierarchical Swin Transformers within a representation-aligned semantic learning framework. Since transformer architectures organize visual information differently,semantic representations are adapted to match the native structure of each backbone. Semantic maps are converted into token representations for Vision Transformers to enable token-level cross-attention, while hierarchical spatial semantic features are used for Swin Transformers to preserve locality and spatial alignment during fusion. Experimental results indicate that hierarchical transformer representations achieve more effective semantic-guided scene understanding than token-only representations. In particular, Swin-Tiny demonstrates stronger performance and more stable semantic interaction behavior compared to ViT-based models despite lower model complexity. Collectively, the findings of this thesis suggest that the effectiveness of semantic-aware scene recognition depends not only on the availability of semantic information, but also on how naturally the representation structure of the architecture supports semantic integration. Architectures that preserve spatial hierarchy and contextual locality appear to align more effectively with semantic scene cues than architectures relying purely on global token interactions. The study further highlights the importance of representation aware semantic encoding when designing multimodal scene understanding systems. Overall, this thesis presents a structured empirical investigation into semantic-guided representation learning across modern deep neural architectures for indoor scene recognition. The work establishes that semantic supervision becomes more effective when aligned with the native representation structure of the underlying backbone, and it provides insights that may guide future research in semantic-aware visual representation learning, multimodal scene understanding, and architecture-aware fusion design.