Road Damage Detection Using YOLOv7 with Cluster Weighted Distance-IoU NMS

Rudy Rachman; Nanik Suciati; Shintami Chusnul Hidayati

doi:10.15575/join.v10i1.1481

Authors

Rudy Rachman Department of Informatic Engineering, Sepuluh Nopember Institute of Technology, Indonesia
Nanik Suciati Department of Informatic Engineering, Sepuluh Nopember Institute of Technology, Indonesia
Shintami Chusnul Hidayati Department of Informatic Engineering, Sepuluh Nopember Institute of Technology, Indonesia

DOI:

https://doi.org/10.15575/join.v10i1.1481

Keywords:

Cluster NMS, NMS, Object detection, Pothole, YoloV7

Abstract

Road damage can occur everywhere. Potholes are one of the most common types of road damage. Previous research that used images as input for pothole detection used the Faster Regional Convolutional Neural Network (R-CNN) method. It has a large inference time because it is a two-stage detection method. The object detection method requires post-processing for its detection results to save only the best prediction from the method, namely, non-maximum suppression (NMS). However, the original NMS could not properly detect small, far, and two objects close to each other. Therefore, this research uses the YoloV7 method as the object detection method because it has better mean Average Precision (mAP) results and a lower inference time than other object detection methods; with an improved NMS method, namely Cluster Weighted Distance Intersection over Union (DIoU) NMS (CWD-NMS), to solve small or close potholes. When training YoloV7, we combined a new, independently collected pothole dataset, with previous public research datasets, where the detection results of the YoloV7 method were better than those of Faster R-CNN. The YoloV7 method was trained using various scenarios. The best scenario during training is using the best checkpoint without using a scheduler. The mAP.5 and mAP.5-.95 value of CWD-NMS was 89.20% and 63.30% with 10.30 millisecond per image for inference time.

References

[1] V. Kaushik and B. S. Kalyan, “Pothole Detection System: A Review of Different Methods Used for Detection,” in 2022 2nd International Conference on Computer Science, Engineering and Applications, ICCSEA 2022, Institute of Electrical and Electronics Engineers Inc., 2022. doi: 10.1109/ICCSEA54677.2022.9936360.

[2] H. Majidifard, Y. Adu-Gyamfi, and W. G. Buttlar, “Deep machine learning approach to develop a new asphalt pavement condition index,” Constr Build Mater, vol. 247, p. 118513, 2020, doi: 10.1016/j.conbuildmat.2020.118513.

[3] F. Meng and A. Li, “Pavement Crack Detection Using Sketch Token,” Procedia Comput Sci, vol. 139, pp. 151–157, 2018, doi: 10.1016/j.procs.2018.10.231.

[4] F. Utaminingrum et al., “Feature selection of gray-level Cooccurrence matrix using genetic algorithm with Extreme learning machine classification for early detection of Pole roads,” Results in Engineering, vol. 20, p. 101437, Dec. 2023, doi: 10.1016/j.rineng.2023.101437.

[5] T. Siriborvornratanakul, “An Automatic Road Distress Visual Inspection System Using an Onboard In-Car Camera,” Advances in Multimedia, vol. 2018, 2018, doi: 10.1155/2018/2561953.

[6] C. Van Geem et al., “Sensors on Vehicles (SENSOVO) - Proof-of-concept for Road Surface Distress Detection with Wheel Accelerations and ToF Camera Data Collected by a Fleet of Ordinary Vehicles,” Transportation Research Procedia, vol. 14, pp. 2966–2975, 2016, doi: 10.1016/j.trpro.2016.05.419.

[7] H. Wu et al., “Road pothole extraction and safety evaluation by integration of point cloud and images derived from mobile mapping sensors,” Advanced Engineering Informatics, vol. 42, no. March, p. 100936, 2019, doi: 10.1016/j.aei.2019.100936.

[8] S. K. Ryu, T. Kim, and Y. R. Kim, “Image-Based Pothole Detection System for ITS Service and Road Management System,” Math Probl Eng, vol. 2015, 2015, doi: 10.1155/2015/968361.

[9] A. Dhiman and R. Klette, “Pothole Detection Using Computer Vision and Learning,” IEEE Transactions on Intelligent Transportation Systems, vol. 21, no. 8, pp. 3536–3550, Aug. 2020, doi: 10.1109/TITS.2019.2931297.

[10] M. H. Yousaf, K. Azhar, F. Murtaza, and F. Hussain, “Visual analysis of asphalt pavement for detection and localization of potholes,” Advanced Engineering Informatics, vol. 38, no. July, pp. 527–537, 2018, doi: 10.1016/j.aei.2018.09.002.

[11] R. Rachman, A. Septiarini, and H. Hamdani, “Detection Of Road Damage Using Faster Regional-Convolutional Neural Network Method,” in 2022 International Conference on Electrical Engineering, Computer and Information Technology (ICEECIT), IEEE, Nov. 2022, pp. 54–59. doi: 10.1109/ICEECIT55908.2022.10030629.

[12] L. Huidrom, L. K. Das, and S. K. Sud, “Method for Automated Assessment of Potholes, Cracks and Patches from Road Surface Video Clips,” Procedia Soc Behav Sci, vol. 104, pp. 312–321, 2013, doi: 10.1016/j.sbspro.2013.11.124.

[13] S. Ren, K. He, R. Girshick, and J. Sun, “Faster R-CNN: Towards Real-Time Object Detection with Region Proposal Networks,” IEEE Trans Pattern Anal Mach Intell, vol. 39, no. 6, pp. 1137–1149, 2017, doi: 10.1109/TPAMI.2016.2577031.

[14] C. Cao et al., “An Improved Faster R-CNN for Small Object Detection,” IEEE Access, vol. 7, pp. 106838–106846, 2019, doi: 10.1109/ACCESS.2019.2932731.

[15] H. Chen et al., “A deep learning CNN architecture applied in smart near-infrared analysis of water pollution for agricultural irrigation resources,” Agric Water Manag, vol. 240, Oct. 2020, doi: 10.1016/j.agwat.2020.106303.

[16] Z. Xue, L. Zhang, and B. Zhai, “Multiscale Object Detection Method for Track Construction Safety Based on Improved YOLOv5,” Math Probl Eng, vol. 2022, 2022, doi: 10.1155/2022/1214644.

[17] C.-Y. Wang, A. Bochkovskiy, and H.-Y. M. Liao, “YOLOv7: Trainable bag-of-freebies sets new state-of-the-art for real-time object detectors,” Jul. 2022.

[18] J. Redmon, S. Divvala, R. Girshick, and A. Farhadi, “You Only Look Once: Unified, Real-Time Object Detection,” Proceedings of the IEEE Computer Society Conference on Computer Vision and Pattern Recognition, vol. 2016-December, pp. 779–788, Jun. 2015, doi: 10.1109/CVPR.2016.91.

[19] T. Gao, M. Wushouer, and G. Tuerhong, “DMS-YOLOv5: A Decoupled Multi-Scale YOLOv5 Method for Small Object Detection,” Applied Sciences (Switzerland), vol. 13, no. 10, May 2023, doi: 10.3390/app13106124.

[20] H. Hongyu, K. Ping, F. Li, and S. Huaxin, “An Improved Multi-Scale Fire Detection Method based on Convolutional Neural Network,” in 2020 17th International Computer Conference on Wavelet Active Media Technology and Information Processing (ICCWAMTIP), IEEE, Dec. 2020, pp. 109–112. doi: 10.1109/ICCWAMTIP51612.2020.9317360.

[21] Y. Song et al., “Intrusion detection of foreign objects in high-voltage lines based on YOLOv4,” in 2021 IEEE 6th International Conference on Intelligent Computing and Signal Processing, ICSP 2021, Institute of Electrical and Electronics Engineers Inc., Apr. 2021, pp. 1295–1300. doi: 10.1109/ICSP51882.2021.9408753.

[22] R. Zhang, M. Jing, Y. Fan, and X. Zeng, “Small Object Detection in Aerial Images,” in Proceedings of International Conference on ASIC, IEEE Computer Society, 2021. doi: 10.1109/ASICON52560.2021.9620245.

[23] Z. Zheng et al., “Enhancing Geometric Factors in Model Learning and Inference for Object Detection and Instance Segmentation,” May 2020.

[24] T. Y. Lin et al., “Microsoft COCO: Common Objects in Context,” Lecture Notes in Computer Science (including subseries Lecture Notes in Artificial Intelligence and Lecture Notes in Bioinformatics), vol. 8693 LNCS, no. PART 5, pp. 740–755, May 2014, doi: 10.1007/978-3-319-10602-1_48.

[25] M. Tkachenko, M. Malyuk, A. Holmanyuk, and N. Liubimov, “Label Studio: Data labeling software.” Accessed: Aug. 14, 2023. [Online]. Available: https://github.com/heartexlabs/label-studio

[26] R. Rachman, M. Alfian, and U. L. Yuhana, “Classification of Illustrated Question for Indonesian National Assessment with Deep Learning,” in 2023 14th International Conference on Information & Communication Technology and System (ICTS), IEEE, Oct. 2023, pp. 77–82. doi: 10.1109/ICTS58770.2023.10330865.

[27] L. Buitinck et al., “API design for machine learning software: experiences from the scikit-learn project,” in ECML PKDD Workshop: Languages for Data Mining and Machine Learning, 2013, pp. 108–122.

[28] Y. Ding, C. Liu, H. Zhu, and Q. Chen, “A supervised data augmentation strategy based on random combinations of key features,” Inf Sci (N Y), vol. 632, pp. 678–697, Jun. 2023, doi: 10.1016/j.ins.2023.03.038.

[29] A. Paszke et al., “PyTorch: An Imperative Style, High-Performance Deep Learning Library,” Dec. 2019.