Adaptive navigation for the visually impaired: safe reinforcement learning with real-time computer vision

Abstract

This paper presents a vision-only assistive navigation framework that couples realtime panoptic segmentation with safe reinforcement learning to provide proactive, collision-aware guidance for visually impaired pedestrians in dynamic urban environments using only RGB cameras on resource-constrained devices. The approach integrates a lightweight bottom-up MobileNetV3–FPN panoptic segmentation model for unified scene understanding, a ConvGRU module that predicts short-horizon danger maps from temporal mask sequences, and a multi-input policy that conditions decision-making on both current semantics and anticipated risk. Safety is enforced by a PPO-based controller trained under a constrained formulation (Lagrangian) and supplemented at runtime with an action-shielding safety layer that filters unsafe actions. The system is trained and evaluated in CARLA with domain diversity from Cityscapes and Mapillary Vistas, emphasizing ethical, simulation-first validation and deployment feasibility on edge hardware. Experiments and studies indicate that constrained PPO with action shielding reduces safety violations compared to unconstrained PPO, while ConvGRU-based temporal prediction improves anticipatory avoidance of dynamic obstacles, achieving a favorable speed–accuracy trade-off for wearable use cases with the MobileNetV3 panoptic variant. The work contributes an affordable RGB-only stack, a tightly coupled perception–prediction–control design for proactive safety, and a reproducible benchmark that surfaces limitations in small-object instance quality and sim-to-real transfer, outlining targeted directions for future refinement.