Vision enhancement for autonomous vehicles : a deep learning approach for anticipating and adapting to adverse weather conditions

Abstract

The camera-based perception reliability in autopilot vehicles in poor weather conditions like when it is pouring, there is thick fog, or snow is severely affected. These visual illusions cause missed detections, misclassifications and latent responses hence compromising safety. The current perception systems are usually effective in clear weather but their performance declines drastically during extreme or transitional weather and this poses an operational risk. This thesis presents a framework, which examines the effects of bad weather on visual perception through two complementary tasks, which are weather classification and object detection. An EfficientNet- B3 model with multi-class weather classification was fine-tuned and a pretrained YOLOv8m (COCO weights) tested the strength of object detection under the same weather conditions. This framework measures the impact of environmental degradation on the reliability of perception by correlating classification outputs and detection performance. Moreover, a Custom EfficientNet-B3 classifier was created and had a two-layered classification head (1536 → 256 → 4) with droupout. To balance between feature preservation and domain adaptation, the model uses progressive backbone unfreezing in two training phases. Such a setup was more accurate, and it surpassed the baseline in terms of improved feature extraction and regularization and was also computationally affordable to run on an embedded platform. Our custom driving scenes in Bangladesh with augmented physics based depth-aware fog rendering and CycleGAN generated rain and snow condition form the dataset used in this study. Moreover, for greater detection robustness in adverse weather conditions, the YOLOv8m model was improved with the addition of a Convolutional Block Attention Module (CBAM) subsequent to the last block, C2f, within the model’s backbone. CBAM facilitates the adaptive recalibration of channel-wise as well as spatial attributes, allowing the network to pay attention to appropriate object attributes despite adverse weather conditions. The overall objective is to provide a weather-aware evaluation methodology that enhances understanding of autonomous driving reliability under diverse and challenging environments.