COMPONENT-WISE ANALYSIS OF VISUAL ENCODERS AND LANGUAGE DECODERS FOR AUTOMATIC IMAGE CAPTIONING

Abstract

Automatic image captioning combines visual understanding and natural language gen eration to produce textual descriptions of images. Most captioning systems follow the encoder–decoder paradigm, in which a visual encoder extracts a feature representation from the image and a language decoder generates the caption. However, the individual contributions of the visual encoder and the language decoder remain poorly character ized, as the majority of studies evaluate complete end-to-end systems without isolating the role of each component. This lack of component-level interpretability limits the principled design of efficient captioning systems for resource-constrained deployment. This thesis presents a component-wise analysis of visual encoders and language decoders for image captioning. The literature is organized into a unified taxonomy of encoder fami lies, namely CNN-based and transformer-based encoders, and decoder families, namely re current and pretrained language model-based decoders. A controlled experimental frame work is then proposed to evaluate four encoder–decoder combinations — CNN–LSTM, CNN–GPT-2, ViT–LSTM, and ViT–GPT-2 — on the Flickr8k dataset under identical conditions, with both decoder types trained so that the comparison between components is fair. The result here it demonstrate that the choices of language decoder dominates caption quality: GPT-2-based decoders substantially outperform LSTM decoders across the all evaluation metrics, while the visual encoder contributes a smaller but consistent improve ments, with the Vision-Transformer encoder outperforming the CNN based encoder. The best combination, ViT–GPT-2, achieves a BLEU-1 of 0.736, BLEU-4 of 0.629, ROUGE-L of 0.762, and CIDEr of 2.179. The analysis are further extendeds to deployment robust ness, examining how captioning performance degrades under real-world image perturba tions such as blur, sensor noise, compression, low resolution, color jitter, and random crop. Finally, the the work identifies five key research gaps — component interpretability,deployment robustness, lightweight architecture design, hallucination, and metric reliabil ity — and outlines future directions, including the extension of component-wise analysis to modern large multimodal models such as BLIP-2 and LLaVA.