Heritage defect detection under data scarcity: a leakage-aware YOLO–faster R-CNN ensemble with weighted boxes fusion

Abstract

Environmental, biological, and structural reasons are causing cultural heritage sites in Bangladesh to deteriorate more quickly. At the same time, traditional manual inspection procedures are still time-consuming, subjective, and hard to scale. This study introduces an automated deep learning framework for the detection and classification of surface flaws in historical structures, utilizing the Historic Place Dataset including 2,292 photos from Panam City, a UNESCO World Heritage Site in Sonargaon, Bangladesh. We use Weighted Boxes Fusion (WBF) to combine YOLOv8- small for speed and Faster R-CNN for accuracy to find five types of damage: Artistic, Corroded brick, Corroded plaster, Fungus, and Living plant elements. To address the class imbalance and limited data, we use hybrid augmentation methodologies that use both online and offline methods. The experimental results demonstrate that Faster R-CNN has the best test-set performance ([email protected] = 0.8945), and the WBF ensemble ([email protected] = 0.8898) does much better than YOLOv8 alone ([email protected] = 0.864). Per-class analysis shows that features that are easy to tell apart get better results, but damage types that are hard to tell apart are still hard to work with. This study shows that deep learning methods can work even when there isn’t a lot of data available, and it also shows how to improve them through systematic augmentation and architectural optimization.