AMULTIMODAL CLINICAL DECISION SUPPORTSYSTEMUSING VISION-LANGUAGE MODELSANDRETRIEVAL-AUGMENTED GENERATION

Abstract

Automated medical image interpretation and clinical report generation have been shown to be very promising areas for the application of recent advances in Vision-Language Models (VLMs). However, existing report generation systems for radiology reports tend to suffer from hallucinated findings, lack of clinical context, and inconsistent structured outputs, which hinders their use for real-world clinical decision support systems. This thesis introduces a multimodal clinical decision support system that integrates Vision-Language Models with Retrieval-Augmented Generation (RAG) to create clinically relevant chest X-ray radiology reports. The proposed system is based on multimodal chest X-ray understanding and re port generation by using the Qwen model series, embedding-based similarity search, and retrieval-augmented contextual grounding using the FAISS similarity search li brary. The Indiana University Chest X-ray dataset was used to build a medical re trieval knowledge base with 3,000 radiology reports. The framework first creates an initial radiology report from the given chest X-ray image, then, it uses a retrieval-aware prompting process to query the knowledge base for similar reports to refine the initial report. Structured JSON-based report generation with Pydantic schemas was used for reliability, parsing consistency, and downstream usage. The Indiana University Chest X-ray dataset was used for experiments with both quantitative and qualitative evaluation techniques. A separate test set (98 chest X-ray cases) was used for evaluation. Lexical similarity metrics, clinical factuality analysis, pathology-level label evaluation, hallucination analysis, parsing robustness, retrieval configuration ablation studies, and runtime analysis were all included in the evalua tion framework. The experimental results showed that the proposed RAG-enhanced framework outperformed the baseline VLM system in terms of the quality of the gen erated report and the clinical relevance of the report. The proposed system improved the ROUGE-1 score from 0.280 to 0.390, ROUGE-L score from 0.220 to 0.310, and clinical label F1 score from 0.631 to 0.762. Moreover, the hallucination rate decreased from 42.0% to 18.0%, and the parsing success rate increased from 85.7% to 98.0%. Further ablation studies showed that the all-MiniLM-L6-v2 embedding encoder with k =3was the best in overall performance. This study’s results clearly illustrate that using multimodal retrieval-guided gener ation significantly enhances the factual consistency, clinical relevance and structural reliability of systems performing automatic radiology report generation. This thesis makes a contribution to the design of trustworthy AI-assisted clinical decision support systems, and demonstrates usefulness of combining VLM and RAG for medical report generation.