Research on Asphalt Pavement Surface Distress Detection Technology Coupling Deep Learning and Object Detection Algorithms

Abstract

To address the challenges posed by the vast scale of highway maintenance in China and the high costs associated with traditional inspection vehicles. This study focuses on a routine maintenance project for national and provincial roads in Shanxi Province, with an emphasis on the selection and design of hardware for lightweight, portable pavement inspection devices. A monocular camera was used to capture pavement surface images, resulting in a dataset of 85,511 training samples. Additionally, the YOLOv5 object detection algorithm, combined with convolutional deep learning techniques, was employed to classify and identify pavement surface distresses in the collected images. Through multiple iterations of model tuning and validation, the proposed detection system achieved a false negative rate of 1.13%, a recall rate of 97.35%, and a precision rate of 98.30%. Its high accuracy provides a technical reference for the development and design of portable pavement distress detection devices.

1. Introduction

By the end of 2024, the total highway mileage in China had reached 5.4368 million kilometers, with 99.9% of this network being maintained. Over the past several years, the scope of road maintenance has consistently expanded, demonstrating a steady upward trend. However, large-scale specialized inspection equipment faces significant limitations in terms of detection costs, data acquisition, and processing, creating a major bottleneck in road maintenance management. Additionally, China’s extensive rural road network remains largely underserved by maintenance interventions due to inspection constraints. To address these challenges, this project aims to develop a compact, portable, and cost-effective device for the rapid collection, detection, and assessment of road conditions. The system will also facilitate timely data processing and analysis, ensuring a continuous flow of data to support informed decision-making in road maintenance management.

In the 1970s, the French road management authorities developed the GERPHO system. In the 1980s, a Japanese research team designed the Komatsu system, which was based on analog video technology [1]. By the 1990s, American researchers had introduced the Pavement Condition Evaluation Service (PCES) system, utilizing a line-scan digital camera for pavement distress detection [2,3]. Compared to its predecessors, the PCES system enabled simultaneous image acquisition and processing. However, its functionality was limited, as it was unable to distinguish between different types of pavement distresses [4]. During the same period, rapid advancements in Charge-Coupled Device (CCD) image sensor technology led a Canadian company to develop the Automatic Road Analyzer (ARAN) system [5]. Despite its innovations, the ARAN system faced high hardware costs and lacked synchronization between pavement data collection and distress identification [6,7]. Since the early 21st century, 3D laser scanning technology has rapidly evolved. Researchers have integrated this technology into pavement distress detection systems, such as the DHDV system in the United States, which achieved true automation in detecting pavement distresses. However, its high hardware requirements and maintenance costs have hindered widespread adoption [8]. More recently, studies have utilized 3D point cloud data and YOLOv5 detection models to identify various types of pavement distresses, including longitudinal cracks, transverse cracks, alligator cracks, and potholes. Notable contributions in this field include the work of Ravi Radhika and Ayman Habib from the United States [9], as well as Sami Abdullah from Australia and Sakib Saadman from Bangladesh [10,11,12,13]. Early pavement distress detection methods predominantly relied on traditional texture feature extraction techniques, such as Local Binary Patterns (LBP) and Gabor filters, which often yielded suboptimal performance in practical applications. In contrast, recent advancements in deep learning technologies have enabled the direct learning of feature representations from data, significantly improving the accuracy of pavement distress identification. By applying deep learning-based image processing algorithms to pavement images, researchers can accurately detect and localize distresses. The detected distress regions can then be segmented to extract geometric features such as area, length, and width. Statistical analysis of all detected distresses allows for the calculation of the distress rate (DR) for the evaluated road section, ultimately leading to the derivation of the Pavement Condition Index (PCI). Several other scholars have also made significant contributions to pavement distress detection [13,14,15,16,17,18].

In comparison to foreign countries, China’s development of automatic pavement inspection technology began relatively late. However, since the early 21st century, advancements in hardware capabilities and significant progress in image processing technologies have enabled substantial achievements by domestic research institutes and universities. Several Chinese scholars have conducted extensive research on automatic pavement inspection systems, making significant contributions to image processing techniques for pavement images. Their work has greatly advanced the application of digital image processing technology in road inspection [19,20,21]. In China, pavement inspection technologies are primarily classified into four categories: deep learning-based methods, 3D laser-based techniques, vibration signal-based detection, and pavement texture analysis [21,22,23,24,25,26,27,28].

With the advancement of the 2025 initiative, the digitalization, informatization, and automation of road maintenance have been rapidly progressing. In recent years, a variety of lightweight inspection devices have emerged on the market. Companies such as Baidu, Qianxun, and Shanghai Tonglu Cloud, as well as universities including Tongji University and Southeast University, have actively engaged in research and development. The hardware used in these devices is largely similar, with most relying on monocular cameras, stereo cameras, or industrial cameras to capture road surface data. The collected images are subsequently processed for various recognition tasks to generate pavement inspection data. This study aims to develop a pavement distress detection system by selecting suitable hardware components and utilizing a monocular camera. An enhanced YOLOv5 algorithm is adopted as the target detection model, incorporating convolutional deep learning techniques to classify pavement distress images captured by the monocular camera. By constructing a distress dataset and conducting debugging, validation, and comparative experiments, the proposed model undergoes training and technical validation. Ultimately, the system facilitates the extraction of key physical characteristics of pavement distresses, including length, width, and area, thereby achieving accurate pavement distress detection.

2. Hardware Selection and Algorithm Principles

2.1. Hardware Selection

Based on the demand assessment, the hardware platform must meet the following requirements: (1) The CPU should have a minimum of four cores. (2) The encoding and decoding capabilities must support the processing of two channels of 4K 30 fps video streams. (3) The system should be equipped with at least 8 GB of RAM. (4) A minimum of two USB 3.0 ports should be available. (5) The platform must support hard disk read/write operations and include an HDMI interface for display connectivity.

2.1.1. Hardware Platform

A comparative analysis was performed on several embedded platforms from NVIDIA, including the TX2 and Jetson Xavier, as well as processing platforms such as Raspberry Pi, BeagleBone, and Huawei, as shown in Figure 1. Initially, considering both processor performance and physical size, the TX2 was deemed too large, while platforms such as the Raspberry Pi lacked sufficient computational power and stability. As a result, the Jetson AGX Xavier platform was selected for this study. However, due to its high cost, future iterations of the system may consider adopting the Xavier NX series to optimize cost control.

Considering the installation environment, the camera will be mounted on a data collection vehicle. The vertically installed lens is typically positioned 1 to 3 m above the ground, while the side-mounted lens is generally placed 1 to 10 m from the guardrail. Given these conditions, a short-focus lens with a focal length of 8 mm is selected for the pavement camera.

Currently, positioning systems include GPS, BeiDou, Galileo, and GLONASS. The full deployment of the BeiDou satellite network in China provides robust support for positioning-related equipment and significantly improves location accuracy. In crack detection, the algorithm must accurately determine the location of detected pavement distresses, enabling detailed analysis of distress distribution across entire road segments. These data are crucial for effective road maintenance planning and decision-making. To ensure high-precision positioning, the system will employ a multi-system, multi-frequency positioning board that supports BeiDou navigation to further enhance accuracy.

2.1.2. Camera Selection

Industrial cameras are available with various interface types, including USB, Ethernet (GigE), CameraLink, IEEE 1394, and CoaXPress. When conducting outdoor data collection, industrial cameras offer superior resistance to environmental variations, dust, and vibrations, ensuring greater reliability in complex conditions. To achieve a 1 mm crack resolution over a 4 m wide pavement section, a minimum of 4000 pixels is required. Since pavement inspection typically focuses on a single lane per capture, a 2D area-scan camera is preferred over a line-scan camera.

Industrial camera sensors are typically classified into two types: CMOS and CCD, with CMOS sensors becoming increasingly prevalent. Additionally, industrial cameras use two exposure methods: (1) Global Shutter, which captures the entire image simultaneously; and (2) Rolling Shutter, which exposes different rows at different times, potentially causing distortion when capturing moving objects. Given the vehicle’s motion speed, the image capture response time must be as fast as possible, and the exposure time should be minimized while ensuring image clarity. Therefore, a global shutter CMOS camera is selected to guarantee distortion-free imaging.

FLIR offers both global shutter and rolling shutter cameras, such as the Flea3 USB3 series. However, these cameras have a resolution range limited to 1K to 2K pixels. In contrast, the ORYX 10GigE can capture 4K resolution, 12-bit images at over 60 FPS, but it uses a GigE interface.

In contrast, the Huarui A7A20MU30 from Zhejiang Dahua Technology achieves 4K resolution with a USB 3.0 interface, global shutter, and C-mount compatibility. It supports the USB3 Vision protocol and the GenICam standard, and features a pixel size of 3.45 μm × 3.45 μm. A USB-interface camera was selected for its plug-and-play functionality, which facilitates rapid prototype development and allows for easy future upgrades with similar or higher-performance cameras. Consequently, the Dahua 4K industrial camera was chosen for this study (as shown in Figure 2).

2.1.3. Positioning System

Currently, positioning systems primarily include GPS, GNSS (Global Navigation Satellite System), Galileo, and GLONASS. With the full deployment of the BeiDou satellite network in China, it provides substantial support for positioning-related equipment and significantly improves location accuracy. For crack detection, the algorithm requires precise geolocation of identified distresses, enabling detailed analysis of distress distribution across entire road segments and offering essential data for road maintenance decision-making. To enhance the reliability and accuracy of detection data, this study selects a high-precision, multi-system, multi-frequency positioning board that supports dual-mode BeiDou + GPS positioning. This module features low power consumption and a compact size (as shown in Figure 3), making it easy to integrate into vehicle-mounted and other automated inspection systems.

2.1.4. Movable Central Control System

The data acquisition platform functions as the central control and operation hub for the automated inspection system (as shown in Figure 4). It integrates a positioning data processing module, power module, display unit, and an embedded core processing platform, while incorporating a high-capacity 1 TB solid-state drive (SSD) to support large-scale 4K image data storage.

The device housing has been modified to include connection ports, ensuring efficient interconnectivity among all components. The display unit supports touchscreen operation and is equipped with customized software that includes functionalities such as algorithm parameter configuration, algorithm encapsulation and execution, result visualization, image list display, and processing progress tracking. Additionally, the system supports large-panel display operation for pavement distress detection, thereby significantly enhancing workflow efficiency for maintenance personnel.

2.1.5. Car Triangular Bracket

To facilitate the integration and utilization of various modular components within the pavement distress detection system—including the solid-state storage drive—and to ensure seamless interface connectivity, signal transmission, and external data output, a custom-designed mechanical enclosure was developed to house all system components in a unified structure.

A dedicated hard drive support and fixation bracket was designed, featuring a tripod with suction cups for stability. The industrial camera and high-resolution lens are screw-mounted onto the bracket, which is securely positioned in a triangular configuration at the rear of the vehicle (as shown in Figure 5). A balancing mechanism is incorporated between the camera and the mounting structure, enabling the camera to capture pavement images at a fixed angle relative to the vertical axis, ensuring optimal image acquisition.

2.2. Algorithm Principle and Selection

The computational principles employed in this study primarily include deep learning theory and object detection algorithms.

2.2.1. Data Collection

The core principle of deep learning is to extract hierarchical feature representations through layered networks, which enables the learning and abstraction of complex features. Convolutional Neural Networks (CNNs) are widely used in image classification and object detection tasks. Their architecture is primarily composed of three components: the input layer, hidden layers, and output layer.

In this study, a CNN-based model is employed for training, which consists of two main stages. The first stage is the forward propagation phase, during which data flows from lower-level to higher-level representations. The second stage is backpropagation, in which, if the predicted results deviate from the expected outcomes, the error is propagated backward from higher to lower layers, facilitating model optimization and improved accuracy.

2.2.2. Object Detection Algorithms

Object detection algorithms can be broadly classified into two types. The first type consists of Region Proposal-based methods, such as R-CNN, Fast R-CNN, and Faster R-CNN, which follow a two-stage approach. These methods first generate region proposals using heuristic techniques, such as Selective Search or a CNN-based Region Proposal Network (RPN). Classification and regression are then performed on these proposals. The second type includes one-stage methods, such as YOLO and SSD, which directly predict object categories and locations using a single CNN network. A typical object detection model consists of a feature extraction backbone and a classification-detection network. The design of the backbone network is crucial in determining how features are fused across different network layers. With advancements in deep learning, backbone networks are continuously optimized to enhance feature extraction.

The YOLOv5 network, a representative one-stage detection model, comprises four main components: the input layer, backbone network (Backbone), feature fusion network (Neck), and output layer (Prediction). Compared to other models, YOLOv5 offers superior efficiency and speed, making it particularly well suited for high-frequency pavement distress detection. The architecture of YOLOv5 is shown in Figure 6. This study will perform a comparative analysis to select the optimal YOLOv5 model and integrate deep learning techniques to enhance the accuracy and robustness of pavement distress detection.

YOLOv5 offers several backbone network architectures with varying depths, including YOLOv5s, YOLOv5m, YOLOv5l, and YOLOv5x. These four structures are governed by two key parameters: model depth and the channel width between layers. The number of model parameters increases progressively from YOLOv5s (the smallest) to YOLOv5x (the largest). Specific parameter settings and channel configurations are provided in the

Table 1. Parameters and parameters of backbone networks at each level.

Backbone Network	Model Depth	The Width of the Interstory Passage	Parameter Size/KB
YOLOv5s	0.33	0.5	14,468
YOLOv5m	0.67	0.75	42,367
YOLOv5l	1.0	1.0	93,086
YOLOv5x	1.33	1.25	173,370

.

Using the same Cross Stage Partial (CSP) structure for comparison, the YOLOv5s network includes one residual block, while YOLOv5m utilizes two residual blocks, YOLOv5l employs three residual blocks, and YOLOv5x incorporates four residual blocks at the same locations. This design effectively controls the network’s depth. Additionally, the network width in YOLOv5 is regulated by adjusting the number of convolutional kernels at different stages. Taking the Focus structure as an example, YOLOv5s uses 32 convolutional kernels, YOLOv5m increases this to 48, YOLOv5l to 64, and YOLOv5x to 80.

3. Research on the Construction of Disease Datasets and Scheme Design

3.1. Construction of Road Disease Detection Dataset

In this study, the pavement distress detection dataset was constructed with a strict proportional division into a training set, validation set, and test set. By collecting real-time traffic photos on the road after the equipment is assembled, the dataset can categorize pavement distresses into five types: transverse cracks, longitudinal cracks, alligator cracks, potholes, and patched areas. A total of 85,511 pavement images were collected from actual road surfaces, and 14,641 valid samples were selected to build the dataset. The training and test sets were split into a 9:1 ratio, resulting in 13,388 images for training and 1253 images for testing. Additionally, during training, a validation set was extracted from the training set at a 9:1 ratio to update the model weights, resulting in 1338 validation images. Annotated examples of pavement distresses in the dataset are illustrated in Figure 7.

3.2. Model Training Design

3.2.1. Platform Environment Deployment

In this study, the YOLOv5 object detection algorithm was trained using a self-constructed pavement distress dataset and applied to identify distress types on actual road surfaces. The platform and development environment consist primarily of a Linux-based system and the Python programming language.

3.2.2. Dataset Preparation

The YOLOv5 network does not directly read dataset images and annotations, as this may result in insufficient memory when processing a large number of images. To address this issue, the dataset and annotation files are first stored in a designated directory during training. The image filenames, labels, and bounding box coordinates are then extracted into a single text file, which YOLOv5 reads in batches. This approach prevents memory overflow and enhances data loading efficiency.

During the preprocessing stage, training images are stored in the JPEGImages directory, while annotation files are placed in the Annotations directory. The voc_label.py script is employed to automatically divide the dataset into training and validation sets, generating a labels directory. Within this directory, all annotation data are recorded in text files, including image filenames, label names, and bounding box coordinates (upper-left and lower-right points).

Before training, the dataset parameters must be modified in the test.yaml file. These parameters include the file paths for the training and test datasets, the number of classes, and the class names, which are provided as a string array. The structure of each dataset is illustrated in Figure 8, Figure 9, Figure 10 and Figure 11 the following figure.

3.2.3. Training Parameter Adjustments

Before initiating training, the YOLOv5 algorithm provides configurable network parameters, including the number of training iterations, the batch size for image loading per training step, the configuration file for the backbone network, and the input image size.

3.3. Methods for Analyzing Test Results

In this study, recall, precision, and the false-negative rate are used as evaluation metrics for pavement distress detection. True Positive (TP) refers to a correctly predicted positive instance, True Negative (TN) to a correctly predicted negative instance, False Positive (FP) to a negative instance incorrectly predicted as positive, and False Negative (FN) to a positive instance incorrectly predicted as negative.

(1)

Recall detection rate

Recall is the ratio of correctly identified instances to the total number of instances that should be identified in the dataset.

$$Recall=TP/(TP+FN)$$

(1)

(2)

Precision detection rate

Precision is the ratio of correctly identified instances to the total identified instances.

$$Precision=TP/(TP+FP)$$

(2)

(3)

False-negative rate

The false-negative rate is the ratio of incorrectly unidentified instances to the total number of actual samples.

$$FNR=FN/(TP+FN)$$

(3)

(4)

Test results

Figure 12 shows the detection results of YOLOv5 networks under different backbone networks.

4. Analysis of Asphalt Pavement Surface Damage Detection Technology

4.1. Demonstration and Analysis of the Experimental Results of Object Detection Algorithm

The initial validation experiment in this study aimed to assess the feasibility of the algorithm. This experiment compared the detection performance of the instance segmentation algorithm Mask R-CNN and the object detection algorithm YOLOv5 using the same set of pavement images. Mask R-CNN performs instance segmentation by first locating objects with bounding boxes and then conducting pixel-wise classification within each bounding box to segment the target objects. In contrast, YOLOv5, as an object detection algorithm, only locates objects using bounding boxes without performing pixel-level segmentation.

As shown in Figure 13, the results of the validation experiment indicate that Mask R-CNN exhibits significant deviation in the bounding box localization stage, leading to inaccuracies in the segmentation phase, where it struggles to fully segment the cracks. In contrast, YOLOv5 effectively locates pavement cracks and correctly identifies them individually, demonstrating its suitability for pavement distress detection.

Through four rounds of experimental validation, this study found that the algorithm’s generalization ability is directly related to the size of the dataset. Based on these findings, the final validation phase focused on significantly expanding the pavement distress dataset while ensuring consistent categorization.

During the validation process, pavement distresses were strictly classified into five categories: transverse cracks, longitudinal cracks, alligator cracks, potholes, and patched areas. The dataset was divided into training and validation sets in a 9:1 ratio, resulting in a total of 13,388 image samples for training.

As shown in Figure 14, the results from the four validation experiments demonstrate clear and accurate distress localization and classification. These findings confirm that the YOLOv5 algorithm achieves a high level of model fitting and generalization when applied to a comprehensive pavement distress dataset. Statistical analysis reveals an overall false-negative rate of 1.13%, a recall rate of 97.35%, and a precision rate of 98.30%.

The training results of the YOLOv5 object detection model developed in this study are shown in Figure 15. It can be observed that, during the 200-step iterative process, the GIOU loss and other loss functions exhibit a gradual decrease, while accuracy and recall steadily increase, eventually converging to stable values.

4.2. Analysis of Camera Calibration Algorithms

The calibration images captured for the experiment are shown in Figure 16. Among the 28 images used in this study, the position and angle of the calibration board vary across images, effectively covering the entire field of view of the camera. This diverse positioning strategy significantly reduces calibration errors.

First, a corner detection algorithm was applied to identify the subpixel locations of all corner points in each image. The yellow rectangular box represents the origin corner point, while the green markers indicate the subpixel coordinates of each detected corner. Using the complete set of corner point data from all images, the camera’s intrinsic and extrinsic parameters, as well as distortion coefficients, were optimized using Zhang’s calibration method, the least squares method, and the Levenberg–Marquardt (LM) algorithm. The 3D spatial positions of the monocular camera and all calibration boards are illustrated in Figure 17.

4.3. Measurement Index Table

After obtaining the accurate intrinsic and extrinsic camera parameters, along with the distortion coefficients, an index table can be established. Since this process involves only matrix operations, no additional errors are introduced, and the index table’s error remains consistent with the calibration error of 0.52 pixels.

Given that the camera has a 4K resolution with 3000 × 4096 pixels, the index table contains data for approximately 12 million points, fully representing the physical spatial information of the captured image, as illustrated in Figure 18. Each entry in the index table stores four key pieces of information for each pixel: its corresponding physical length, width, diagonal length, and area, as shown in Figure 19. Ultimately, the index table comprises 48 million precise parameters, which serve as a reference for subsequent pavement distress geometric measurement algorithms.

4.4. Analysis of Broken Geometry Information Measurement Algorithm

Due to the irregular geometric shape of pavement distresses, this study first calculates the length and then derives the average width. However, the accuracy of the distress skeletonization algorithm directly affects the width measurement error. To obtain the actual physical length of the distress, the binarized mask image must be processed into a skeletonized image, which consists of a single-pixel-wide representation of the distress region. The skeletonized image must be continuous and free of discontinuities or noise artifacts to ensure the most accurate length estimation.

The skeletonization algorithm used in this study is a lookup table-based method that iteratively applies logical operations to the outermost white pixels of the distress region. The algorithm determines whether a pixel belongs to the skeleton based on its relationship with neighboring pixels. If the pixel is part of the skeleton, it is retained; otherwise, it is removed. This process ultimately reduces the distress mask to a single-pixel-wide skeletonized representation.

This study categorizes pavement distresses into four major types: simple cracks, alligator cracks, potholes, and patched areas. Among these, potholes and patched areas are excluded from the length and width calculations. The study further classifies simple cracks and alligator cracks into six subtypes, as illustrated in Figure 20. Simple cracks are subdivided into transverse cracks, longitudinal cracks, diagonal cracks, and herringbone cracks, while alligator cracks are divided into simple alligator cracks and complex alligator cracks.

Figure 21 presents the skeletonized images extracted from the six types of pavement distress masks using the lookup table method. As shown in the figure, the width varies across different regions of the distress masks; however, the lookup table method accurately identifies the primary direction of distress propagation. Furthermore, the extracted skeleton images remain continuous, without disruptions or artifacts, ensuring that length calculations are not affected by noise or distortions. With a well-defined skeleton image, the index table can be used to retrieve the physical length corresponding to each pixel position, and these values are summed to obtain the actual physical length and average width of the distress region.

In summary, the distress geometry measurement algorithm enables the precise extraction of physical spatial parameters based solely on the calibration-generated index table and the semantic segmentation-derived distress mask, providing an effective method for measuring pavement distress geometries in real-world conditions.

5. Conclusions

(1)

Through the hardware selection and design process, a lightweight and portable pavement distress detection device was assembled, offering an efficient and practical solution for on-site pavement inspection.

(2)

By integrating the YOLOv5 object detection algorithm with convolutional deep learning techniques, a model was trained using 85,511 pavement sample images. The final statistical results show an overall false-negative rate of 1.13%, a recall rate of 97.35%, and a precision rate of 98.30%, demonstrating the model’s high accuracy and reliability.

(3)

Algorithm validation and analysis confirmed that the distress geometry measurement algorithm can accurately extract physical spatial parameters using only the calibration-generated index table and the semantic segmentation-derived distress mask. The study concludes that the developed pavement distress detection device has significant potential for practical engineering applications.

6. Prospect

(1)

This study utilizes a monocular camera, which effectively identifies two-dimensional pavement distresses, such as transverse and longitudinal cracks, alligator cracking, block cracking, and patched areas. However, it currently lacks the capability to accurately detect three-dimensional distresses, such as potholes and subsidence. In the future, a stereo camera system could be implemented, incorporating existing equipment algorithms and advanced technical approaches to enable comprehensive pavement distress detection.

(2)

This study focuses solely on pavement distress detection for asphalt surfaces. Given the distinct differences between cement and asphalt pavement distresses, future research could explore established distress recognition methodologies to develop an automated detection system for cement pavement distresses.