Real-time scene description and interpretation using zero-shot learning and prompt-engineered vision-language models

Abstract

Real-time scene description and interpretation are essential for diverse applications such as surveillance, interactive media, and automated video analysis. However, most existing methods rely heavily on large-scale labeled datasets, thereby limiting their adaptability in dynamic or previously unseen scenarios. In this work, we propose a novel mixed-model framework that integrates Vision-Language Models (VLMs), Large Language Models (LLMs), and lightweight object detection networks (e.g., MobileNet-SSD) through advanced prompt engineering. By leveraging zeroshot learning, our approach generates contextually rich scene descriptions without requiring domain-specific or task-specific retraining. The prompt engineering component reduces sensitivity to subtle linguistic variations, enhancing robustness across diverse input formulations. Furthermore, the lightweight detector ensures real-time performance, making the framework suitable for resource-constrained environments. To address ethical and fairness considerations, we incorporate bias mitigation strategies that limit the propagation of harmful stereotypes from large-scale pretraining data. Experimental evaluations on multiple open-domain scenarios demonstrate that our system offers reliable and efficient scene interpretation, maintaining high accuracy in challenging conditions where traditional supervised techniques often fail. This research paves the way for more flexible, scalable, and responsible visionlanguage systems capable of operating effectively in real-world, zero-shot contexts.