MULTIMODAL SURVEILLANCE SYSTEM USING OBJECT DETECTION AND LARGE LANGUAGE MODELS

Abstract

Intelligent surveillance is now an essential component of traffic management, public safety, and institutional security. Conventional CCTV installations mainly preserve visual evidence, while recent deep-learning solutions are able to locate objects in video frames. Even so, many existing systems remain limited to class labels and bounding boxes. They generally do not explain the situational meaning of a detected scene, retrieve relevant safety knowledge, or convert visual observations into practical natural-language alerts for human operators. This work develops an Explainable Intelligent Surveillance System that combines real-time visual perception with language-based event interpretation. The framework uses YOLOv8 to detect surveillance-relevant objects, ByteTrack to preserve object identities across frames, a rule-guided event engine to estimate collision and crowd risks, FAISS to perform vector retrieval, LangChain to coordinate retrieval and prompting, and GPT-4/LLaMA-style models to generate explanations. Each CCTV frame is transformed into a structured event state that records object classes, bounding-box coordinates, confidence values, persistent track IDs, velocity estimates, proximity relations, and risk scores. The event state is embedded and searched against a domain knowledge base containing traffic rules, safety procedures, and surveillance-event descriptions. Retrieved evidence is then combined with the structured visual evidence in a constrained prompt to produce a JSON-style situation report and a concise operator-facing explanation. The evaluation uses public surveillance datasets together with custom traffic CCTV clips. The YOLOv8x configuration provides reliable detection for person and vehicle classes while maintaining a frame rate appropriate for real-time monitoring. The event-analysis module identifies collision risk, bus-stop congestion, unusual stopping, and crowd-density conditions. The retrieval layer improves the factual grounding of generated explanations, while the dashboard brings together live video, event metrics, retrieved evidence, and language-model output in a unified interface. Overall, the study shows that surveillance platforms can progress beyond passive detection toward explainable, auditable, and context-aware decision support. ,