ViBERT: TRANSFORMER-BASEDIMAGE CAPTIONING USING VISION-BERT FUSION — ARCHITECTURE,SURVEYAND IMPLEMENTATION

Abstract

The automatic creation of descriptions for visual stimuli using natural language, known as image captioning, is among the most interesting areas of overlap between computer vision and natural language processing. Although a lot of research has been done in the area, generating fluent and semantically and contextually appropriate de scriptions continues to be a challenging task, especially when dealing with complex scenes involving multiple objects, atypical spatial relationships, and relational reason ing. The present work makes two distinct contributions to the field of image captioning by surveying the architecture of transformer-based image captioning models from the years 2014 to 2025 and proposing a new image captioning model called ViBERT which combines the Vision Transformer and the BERT models. Contribution of the survey, which comprises the first part of this thesis, is to struc ture an enormous number of papers on captioning into nine categories, such as CNN RNN based baseline models, attention-based models, encoder-focused vision trans formers, decoder-focused language model-based methods, cross-modal attention meth ods, contrastive models like CLIP, retrieval based captioning approaches, multimodal transformers, and LLM-based captioning methods. Ten benchmark datasets with spe cific ones on medical images are discussed along with eight evaluation metrics rang ing from simple n-gram similarities at the surface level to semantic graph matching. Nine open problems identified despite advances in benchmarks include caption hal lucinations, medical domain adaptation, accessibility, cross-linguistic generalization, and transformer interpretability. The implementation contribution presents ViBERT, a system utilizing ViT-Base/16 visual encoder with a BERT-base decoder working with a visual prefix injection method. ViBERT was trained on the Flickr8K dataset and tested against two other competi tive baselines such as ResNet50+LSTM+Bahdanau Attention and CLIP+GPT-2. ViB ERT obtained the highest performance metrics in terms of BLEU-4 score with 0.3228 iv points, CIDEr score with 0.9710 points, and SPICE score with 0.3164 points. As for computational performance, ViBERT demonstrated processing speed of 19.5 millisec onds per image. In comparison, it is faster than the LSTM-based baseline almost 20 times due to the benefits from the parallel computing capabilities offered by trans formers. Thus, the research confirmed that the paradigm in image captioning is chang ing and moving from convolution-based methods towards self-attention. Furthermore, the combination of Vision Transformer encoders and bidirectional language models proved to be superior for obtaining better semantic grounding. Overall, the current research provided the analysis of existing models’ performance and their potential for improvement, making the implementation contribution an integral part of the study.