Neural Network Based Pavement Condition Assessment with Hyperspectral Images

Abstract

Hyperspectral image processing techniques, with their ability to provide information about the chemical compositions of materials, have great potential for pavement condition assessment. This study introduces a novel age-based pavement assessment method, employing an integrated algorithm with artificial neural network (ANN) and spectral angle mapping (SAM) on hyperspectral images. In the proposed method, the resulting ANN prediction outputs are used to make a new prediction along with the results from SAM scores. Tests are performed on hyperspectral images that have 360 spectral bands between 400 and 900 nm, collected by a specifically designed vehicular system for proximal image acquisition. The acquired images have eight classes, including three different pavement classes (good (5-year), medium (10-year), and poor (25-year)), yellow dye, white dye, soil, paving stone, and shadow. Several experiments are performed to evaluate the robustness of the followed methodology with limited learning data that include 5, 10, 25, and 50 samples per class, selected randomly from our independent spectral database. For a fair comparison, the individual ANN, SAM, support vector machine (SVM), and stacked auto-encoders (SAE) algorithms are also evaluated. The classification performances of individual ANN and SAM are significantly increased with their joint use, demonstrating a 1.2% to 21% classification accuracy improvement in relation to the training sample size. The study proves that the proposed approach is quite robust in cases wherein few training data are available, while SAE and standard ANN algorithms are more successful in cases wherein more learning data are present.

1. Introduction

Asphalt is a dark brown to black, cement-like semisolid, solid, or viscous liquid produced by the non-destructive distillation of crude oil during petroleum refining [1]. A flexible pavement profile commonly consists of five different layers, as shown in Figure 1 [2]. After setting the foundation with a subgrade layer, from bottom to top, the subbase layer is placed for foundational support through compacted aggregates, soils or chemical additions. Following that, a base layer is built, frequently above the subbase layer, for drainage purposes. As the top layers, flexible pavement systems have a binder layer—a mixture of aggregate and asphalt—and an asphalt surface layer—the top (crown) of the dense asphalt. This dense asphalt pavement should ideally be structurally resilient for distortion, and skid-resistant under traffic loadings, which is usually evaluated by the automated collection of roughness, rutting, and faulting data across the road networks [2,3].

Pavement management is defined as a set of tools that help decision makers identify optimal plans to provide, evaluate, and maintain pavements in a serviceable condition over a period [4]. The systematic assessment through pavement management systems (PMS) not only enables the identification of current pavement preservation and rehabilitation needs, it also provides a historical angle for better judgments of the road network, especially to evaluate the long-term impacts of material changes, construction implementations or design procedures. In other words, pavement management systems offer help to allocate available resources most effectively by justifying the needs through regularly collected data and dynamic analysis, optimizing the budgetary limitations [3,4]. According to Torres-Machi [5], USD 400 billion is invested each year globally in pavement construction and maintenance, emphasizing the technical, economic, and environmental aspects of the sustainable management of pavement [4,5].

Over time, all pavements’ quality degrades because of structural parameters, such as pavement material or soil characteristics, as well as environmental effects or traffic density [3]. Several conceptual models reveal the same, yet not surprising, trend, which is that the older the pavement gets, the more expensive it is to fix. Based on current pavement conditions and the desired serviceability, several treatment options can be applied on the pavements, as follows [3,4]:

• Preventive maintenance;
• Light to moderate or minor rehabilitation;
• Major rehabilitation;
• Reconstruction.

Figure 2 shows pavement performance change in relation to pavement age. In Figure 2a, you can see the decline in quality within the lifetime of the pavement, associated with the required renovation costs per kilometer. As seen, a poor first strategy will cost about four to five times more in comparison to implementing rehabilitative actions at around the half-life time-point of paved roads [6,7]. Figure 2b illustrates the pavements’ proper treatment type across years, emphasizing that good and fair pavements will also continue to deteriorate unless preventive or rehabilitative actions are taken. To this end, determining and prioritizing the pavement conditions and taking necessary actions at the network level is profoundly significant in terms of economic, technical, and management aspects.

Solving challenging transportation management issues and forming road safety strategies in a timely and cost-efficient manner [8] have specifically driven the focus on remote sensing applications [9,10,11,12,13]. At this juncture, mapping asphalt road quality parameters presents itself as one of the more significant aspects of transportation management issues, which require regular supervision, and does so in a more practical and feasible manner. Pavement surfaces have particularly observable spectral features not only due to their physical–structural characteristics but also in relation to the asphalt layer’s chemical composition [9,10,11,12]. Combining the properties of digital cameras and spectroscopy devices, hyperspectral imaging sensors enable the collection of detailed spectral features of materials in narrow, contagious bands. The acquisition of such information allows identifying target materials in each image pixel through comparisons with existing spectral libraries or via classification algorithms. In this respect, understanding the spectral behavior of the road surfaces and generating accessible spectral libraries can help researchers to perceive road aging and trace deterioration levels in a fully automated manner, as required by authorities. Such a monitoring system would facilitate data collection and analysis, as opposed to the field-intensive demands of traditional systems, which are based on field observations. These field surveys usually require collection of geographical physical parameters of road surfaces (e.g., cracking, rutting and raveling), and analysis of aggregated measures, such as the pavement condition index (PCI) and structure index (SI) [2,14].

The main purpose of this paper is to evaluate the capacity of visible-near infrared (VNIR) hyperspectral image classification and spectral identification algorithms in determining pavement conditions. For this purpose, we propose an age-based classification as a proxy to identify and prioritize the roads that need maintenance in a controlled setting, where we have detailed ground truth information. Several conceptual models elaborate on the inevitable increase in the maintenance costs as the pavement gets older, which might quadruple if past the critical time after mid-age. Given performance change in relation to pavement age, we have acquired well-recorded information of the pavement surfaces in our campus area during our field campaign and determined our testing regions in this controlled setting. Three different predetermined categories, namely good (5-year), medium (10-year) and poor (25-year), are designated during the experimentations based on the ground truth information of the focused study area. Identifying and classifying these pavement categories will enable authorities to simply categorize the road networks via regularly collected and processed image sets, and provide better management of the pavements, as well as budgetary adjustments as to when and which maintenance actions are needed.

In the study, after a comprehensive literature survey on the hyperspectral detection and spectral characteristics of asphalt surfaces (Section 2), a fast and novel methodology for mapping the condition of asphalt surfaces (i.e., good, medium, poor) is presented. For a fair comparison, benchmarking with support vector machine (SVM), artificial neural network (ANN), stacked autoencoder (SAE) classification algorithms are also performed. In Section 3, the algorithms used in the study are detailed, followed by the methodology and data collection in Section 4 and Section 5, respectively. Experimental results are presented with the related performance metrics of the new methodology, along with their comparisons with the benchmarking algorithm outputs in Section 6. General discussion of experiments is given in Section 7 with the conclusions in Section 8.

2. Literature Review

Hyperspectral images provide remarkable capabilities for mapping and identifying urban infrastructures by means of their ability to differentiate the physical and chemical properties of materials. The small coverage of urban surfaces further challenges the algorithms’ performances, demanding fine spectral and spatial resolution data for the analysis. In line with that, the recognition of road networks as well as the generation of detailed road surface maps introduces a greater challenge, requiring especially spectrally high-resolution data to develop automated identification systems [15]. Generations of effective classes, as well as improving algorithm performances at finer scales, are essentially based on the acquisition of state-of-art data, on which new approaches can be applied. In this respect, this section presents a comprehensive study on the applications of road quality assessment with hyperspectral data, and it elaborates on the spectral properties of asphalt pavements.

2.1. Aging of Asphalt Surfaces and Its Spectral Characteristics

Asphalt is a semisolid, solid, or viscous liquid bituminous substance generated by the distillation of crude oil during petroleum refining [1]. It is primarily comprised of hydrocarbons. Although the chemical composition of asphalt is closely related to the chemical composition of the original crude oil, its physical characteristics are dramatically changed by air blowing at high temperatures, depending on the nature of its industrial use (i.e., asphalt pavements, roof asphalts) [1,16]. The elemental analysis reveals that asphalts mainly involve 79–88% carbon, 7–13% hydrogen, traces of 8% sulfur, 2–8% oxygen, and traces of 3% nitrogen by weight [16].

According to the report prepared by Bell [16] on the aging of asphalt, two main phases are responsible for asphalt hardening: volatilization and oxidation. Volatilization mostly occurs in the short-term, such as in construction, while oxidation is a progressive and long-term phenomenon. Both of these factors cause high viscosity in asphalt pavements, which results in the stiffening of the mixture. The surfaces are eventually prone to cracking or raveling due to oxidation products, wear resistance and moisture susceptibility [16]. Petersen [17] lists the three main reasons for asphalt hardening as follows:

(1)

Loss of oily components by volatility or absorption;

(2)

Changes in composition by reaction with atmospheric oxygen;

(3)

Molecular structuring that produces thixotropic effects (steric or structural hardening) [16,17].

Understanding the hardening of asphalt–aggregate mixtures requires extensive laboratory work. The studies cover resilient modulus, indirect tensile, dynamic modulus, micro viscosity, chemical fraction, and ductility tests for the short-term as well as long-term aging assessments [16,17]. These laboratory analyses are time-consuming and economically not feasible with limited impact areas, which calls for effective and generalizable applications.

The spectral behavior of asphalt–aggregate surfaces is a function of the chemical composition along with the physical parameters [9]. The causes of absorption bands in the spectra of minerals are listed as electronic and vibrational processes in general, which are dependent on the discrete energy states of isolated atoms and ions and the bonds in molecules and crystal settings, respectively [12]. Organics also have distinctive absorption features due to C–H stretching around the bands at 1.7 µm, 2.3 µm, and 3.4 µm, enabling their detection with spectroscopic techniques [9,12]. Along with the exposed minerals, these features are also extensively studied in the literature for identifying the deterioration of asphalt pavements. Figure 3a,b illustrate the spectral signatures of asphalt surfaces with different ages [11], and the commonly utilized hydrocarbon index [10] calculation, where λ is wavelength and R is reflectance for each point, representative of asphalt aging, respectively.

According to Herold and Roberts [14], distinguishing the quality of pavements is significantly dependent on the physical and chemical changes the surface has gone through, which are related to the age, quality, and circulation of the roads. The hydrocarbon component of asphalt pavements is usually presented via its distinctive absorption features, specifically in new asphalts where hydrocarbon content is high. As the hydrocarbon amount decreases, these absorption features become vague, even disappearing as the content wears out. The lower amounts of oily components (i.e., hydrocarbon or bitumen content) also signify the increase in hardening and risk of cracking, exposing the pavement’s inherent rocks. In line with that, the features of these exposed rocks and minerals, such as iron or calcite absorptions, become more and more pronounced. The prevalence of these features is also accompanied by comparatively higher overall spectral reflectance (i.e., increase in brightness) within the collected signatures.

In the literature, these characteristics are utilized to categorize asphalt surface conditions, specifically based on calculated ratios around the absorptions. Some of these ratios, and the reasons why they are considered for pavement condition assessment, are given in

Table 1. Some important absorption wavelengths ([11,14,18,19]).

Important Spectral Features for Pavement Condition Assessment	Wavelengths
Organic absorption bands (C–H stretch)	1700 nm (1720 nm,1750 nm), 2200 nm–2500 nm (2300 nm, 2310 nm, 2350 nm) more prominent
Iron oxide absorption features (Electronic absorption processes in the VIS region reflect the dominance of the minerals and result in concave shape with distinct iron oxide absorption features)	520 nm, 670 nm, 870 nm (1700 nm–2300 nm absorption disappears)
Calcite (raveled, gravel exposed pavements dominant)	2320 nm
VIS band difference (this band difference is low for new asphalt surfaces, and increases with age and level of deterioration)	830 nm (a spectral peak between two iron absorption bands) 490 nm (middle of an iron absorption band)
SWIR band difference (there is a significant change in slope in the transition from hydrocarbon to mineral absorption)	For older roads, the slope increases between 2100 nm and 2200 nm and decreases between 2250 nm and 2300 nm

[11,14,18,19].

Given these absorption features, new paved surfaces have minimum values of visible (VIS) differences with maximum short-wave infrared (SWIR) index rates, while the opposite is the case for older pavements. The authors also emphasize the existence of hydrocarbon absorption features in asphalt surfaces that entail structural stresses like cracking, accompanied by a decrease in reflectance [20]. Contrary to structural stresses, it should be noted that aging causes road surfaces to become brighter. Mei et al. [19] also investigates the spectroradiometric properties of asphalt mixtures with several aggregate types, and emphasizes the detectability of bitumen contents higher than 1% [19]. In line with previous studies, these mixtures demonstrate high correlations with the spectral index calculated in the visible region (400–700 nm). The reduction in overall brightness and the slopes in the visible–near infrared spectrum is again recorded in the conducted experiments as the bitumen content increases. The authors’ observations affirm the applicability of spectroscopic analyses to support the pavement management system for road condition classification.

2.2. Hyperspectral Imaging Applications on Pavement Condition Assessment

Road extraction/identification and database construction for road conditions are labor-intensive, time-consuming and costly with the traditional approaches [2,4,14]. Hyperspectral imaging applications’ capability to generate high-accuracy maps renders it an advanced tool for infrastructure mapping, such as extraction and assessment of asphalt pavement characteristics or mapping other impervious surfaces such as rooftops.

A pioneering and comprehensive work on the use of remote sensing data for road condition mapping was sponsored by the United States Department of Transportation in 2001 [8]. The project’s overarching goal is to propose solutions for priority transportation requirements by using hyperspectral sensors. As a partner, the research group of the University of California concentrates on road infrastructure and quality assessment with Airborne Visible/Infrared Imaging Spectrometer (AVIRIS) images (370–2500 nm) to extract pavement-features. The construction of an urban spectral library is an invaluable contribution in terms of providing spectral fingerprints of impervious surfaces/urban land covers and instrumental parameters explaining the absorption features across the spectrum [8,21]. In the project, in addition to the road centerline extraction through spectral classification for different man-made land cover types, indicator wavelengths of asphalt surface condition and age are determined with a detailed investigation of field-collected spectral signatures. These are then utilized, with band ratio index maps, to estimate the asphalt road condition. The analyses provide comprehensive knowledge about urban materials’ spectral characteristics and road surfaces of different types, ages, and conditions [21].

In the following papers [20], the authors focus primarily on pavement surface defects by using AVIRIS (370–2500 nm) and high-resolution HyperSpecTIR (450–2450 nm) data. The findings reveal a significant correlation between the calculated spectral indexes and the measured in-situ pavement condition index (PCI) assigned by the experts. ANOVA calculations between visible–near infrared (VNIR) spectral indexes and management actions, specifically for identifying the required actions of “do nothing” and “maintenance” for the asphalt pavements, confirm the high correlations between the variables [14,20]. Andreou et al. [15] also apply hyperspectral remote sensing techniques for mapping asphalt road conditions. The age, circulation, and material quality of the roads are taken into consideration for determining the site locations for spectral analysis. The asphalt road surfaces are categorized into five different classes, namely very good, good, mediocre, poor condition, and wet asphalt. The CASI 550 hyperspectral sensor (400–1000 nm) and GER1500 radiometer (280–1090 nm) are utilized to classify the categories. The appointed classes are revealed as good indicators for asphalt conditions, pointing out the road areas that need rehabilitation.

Mohammadi [18] similarly conducted research on road surface/material identification in urban areas and road condition assessment. HYMAP (450–2500 nm) hyperspectral data are used for road extraction by spectral angle mapping (SAM) as the first step, followed by the calculation of spectral features based on brightness values. The authors classify the ‘city surfaces’ in their study area, which are concrete, gravel, and asphalt classes, with the help of the mentioned spectral angle and spectral feature metrics. After identifying the road network, the conditions of asphalt surfaces (good, intermediate, and poor) are categorized with spectral functions and ratio images through the well-known spectral features in pavement quality literature [18,20]. Furthermore, Mei et al. [19] focused on the spectro-radiometric properties (350–2500 nm) of frequently used asphalt/aggregate road paving mixtures in relation to their bitumen content to provide a basis for parameterizing physical and chemical characteristics of the asphalt pavement mixtures. Different lithological samples, including two basaltic aggregates and clay granules, are utilized for the composition of the experimental samples on which the spectral signatures are also recorded. The study reveals that radiometric analyses are effective in evaluating the physical features, such as the porosity, water content and lithology, of asphalt pavement aggregates. The experiments indicate the potential use of in-situ radiometric analysis for road engineering investigations, especially for identifying asphalt roads in need of maintenance [19].

As seen in the literature, it is evident that the traditional surveying of road networks to assess their serviceability is unfeasible due to its labor-intensive and costly nature, revealing the need for an economical and timely alternative. To this end, remote sensing methods replace or complement existing traditional methods while serving the many needs of transportation engineers [4,6,7]. Therefore, it is imperative to elaborate on how we evaluate the road networks and prioritize those in need of maintenance quickly and effectively. This study proposes an age-based classification as a proxy to identify and prioritize the roads that need maintenance and to decide the management action properly and promptly. Given the known circulation information and utilized asphalt characteristics in the campus area, we make use of the well-recorded age information of the pavement surfaces in our case in this controlled experimental setting. As stated in previous studies, the cameras working in the 400–2500 nm range may be more suitable for this task. However, as the hyperspectral cameras covering the 400–2500 nm range are quite expensive and not readily available, the present study is conducted with a more affordable 400–1000 nm range hyperspectral camera. To our knowledge, this study is the first to set a custom vehicular system for proximal image acquisition for automated pavement condition assessment in Turkey, which will serve as a pavement management system, optimizing the budget allocation based on the regularly collected and processed image sets.

3. Algorithms

Artificial neural networks (ANNs) have been widely used to classify hyperspectral images [22,23,24]. Plaza et al. [25] use ANNs with small training sets to characterize mixed pixels in hyperspectral images. Similarly, Subramanian et al. [26] implement neural networks with small training samples for classification by utilizing singular value decomposition (SVD) as a pre-processing tool to improve the computation time for the hyperspectral AVIRIS data. With their non-linear capabilities, ANN models do not make any assumptions about data distribution, increasing their classification power. Neurons function as partial simulators through inherent non-linear activation functions, arranged under each layer [27,28].

A simple artificial neuron is the smallest unit of the neural network, on which the weighted sum of its inputs is computed [28]. To overcome the simple linear dependence of this weighted sum on its input features, this score is then passed through an activation function, σ. This non-linear sigma activation function enables the modeling of nontrivial–complex functions, eliminating the flat dependence between the inputs and output scores [27,28]. Figure 4 illustrates the general model used in the study, where X is the input layer connected to H, the hidden layer, and O, the output layer, with weights w_ij and h_ij. In the classification experiments with neural networks, a network with one hidden layer is established. Backpropagation and gradient descent algorithms are used in the ANN experiments for convergence, where parameter n is the number of bands (360), k is the size of the hidden layer (64) and m is the class size (8).

Deep learning methods, widely used in the last decade, are shown to yield high accuracy when extensive learning data are used. To compare the proposed algorithm, we employ a stacked autoencoder (SAE) based classifier and multilayer feed-forward neural networks as deep learning algorithms [29]. A stacked autoencoder is a special form of neural network, designed to learn by stacking additional unsupervised feature learning layers, and can be trained using greedy methods for each additional layer [30]. It consists of multiple layers in which the outputs of each layer are wired to the inputs of the successive layer, and is used for learning unsupervised representations of data through each layer, which are fine-tuned using backpropagation. A deep representation of the input data is provided at the last layer’s output with more than one encoder layer. That is, a layer is trained with the parameters obtained after the training of the previous layer is over. A stacked automatic encoder is created by stacking these hidden input layers. There are different versions of stacked auto-encoder algorithms, such as stacked denoise auto-encoders [31,32] and stacked sparse encoders [33,34].

In our implementation, no dimensionality reduction is applied to high spectral resolution data prior to classification. The network model class outputs eight-element unit vectors, returning binary values to indicate whether that output neuron belongs to the given class. The input layer consists of as many nodes as the number of spectral bands, with three hidden layers fully connected to the output layer. The input is a vector of n × 1, n being the number of bands. A network similar to the one Chen et al. [35] used is illustrated in Figure 5, where X is the input layer connected to H, the hidden layers, and O, the output softmax classifier, with weights w_ij and h_ij.. The system consists of a four-layer structure with three fully connected layers, and the output layer is a softmax classifier layer. In this figure, n is the number of bands, m is the class size, and k and j are the sizes of the hidden layers, which are selected as 128 and 16, respectively.

The spectral angle mapper (SAM) algorithm measures the angular similarity between two spectra by calculating the spectral angle between the target spectrum and image pixels [36]. SAM is used for both classification and target detection purposes [36,37]. In the following expression, s = [s₁ s₂ …. s_n] is the spectral signature of the pixel and d = [d₁ d₂ …. d_n] is the target spectral signature utilized for calculating angular distance:

$$\alpha =co{s}^{-1}\left(\frac{s^{T}d}{\left|\right|s\left|\right| · \left|\right|d\left|\right|}\right)$$

(1)

The support vector machine [38] (SVM) is an effective method, which has been frequently implemented for hyperspectral data classification [29,31,39,40,41]. The SVM training algorithm tries to determine the optimal decision boundary (hyperplane) separating the dataset. Keerthi and Lin [42] studied the different kernel functions for SVM classifiers, such as linear, polynomial, radial basis function (RBF) and sigmoid. RBF kernel, which has been found to be more successful in hyperspectral classification, is selected in this study [39,41]. The equation of gaussian RBF is:

$$\mathrm{K} (X_i,X_j) =exp{(-\gamma \left||X_i-X_j\right||)}^2,\gamma >0$$

(2)

where γ is the kernel parameter [40,42].

4. Proposed Methodology Based on Neural Networks

The proposed methodology integrates the two main algorithms. Firstly, classification via ANN and SAM is performed separately. The acquired output scores of the two methods are integrated at the second step. After obtaining the outcome scores from a neural network for each class, a new score is calculated as the ratio of the NN prediction scores and the spectral angular distance of the pixels. A pseudo-code of the implemented methodology is presented in Figure 6 and the algorithms are made available for convenience (https://github.com/okanbilge/nnRoad). This combined use of ANN and SAM algorithms does not change the results for which ANN makes better decisions, but does direct the classifier output to the one SAM favors when ANN is uncertain. For example, in a two-class problem, the effect of the SAM score is virtually absent when the ANN score is close to one or zero, whereas it might be critical for a non-binary ANN score scenario.

To test the model robustness and observe the effect of training size on overall accuracy, four different experiments are conducted using different numbers of training pixels (5, 10, 20, and 50 pixels per class) on the captured images. The flowchart of the methodology is presented in Figure 7.

The classification performance is calculated based on the prepared ground truth of the asphalt condition categories. The overall classification accuracy is the ratio of correctly classified data points to the total number of classified pixels, which is computed to assess the performance of the proposed algorithm as well as the benchmark methods.

5. Study Area and Data Collection

The study was conducted at the Middle East Technical University campus in Ankara, Turkey. Having a priori knowledge about the infrastructure in the campus region, in the data collection phase, different asphalt types, paving stones, asphalt paints, and soils were captured. In order to avoid complications due to inadequate sunlight, data collection was performed in a cloudless environment in the afternoon. Due to the need for ground truth data, data acquisition was performed in three different low-circulation roads for which the age information is known by the university authorities.

The data collection was performed with a custom-designed system, specifically optimized for proximal image acquisition with a Headwall A-series VNIR camera. The camera has 408 bands, collecting data in the range of 400 nm–1000 nm. It was mounted on an SUV car with a metal cage positioned about two meters in height, as shown in Figure 8.

The size of the collected data in the study is 5000 × 1000 pixels along the track and cross track directions. The spatial resolution of this image is about 5 cm along the track and 1 cm along the cross track direction. The last 68 bands (900–1000 nm) are not used due to the low signal to noise ratio. In the field campaign, six different datasets captured from the designated roads have been utilized for analysis. The constructed spectral library has eight different classes comprising three different asphalt classes (good (5-year), medium (10-year), and poor (25-year)), white and yellow road paint, shadow, paving stone, and soil. The sample pictures from each class are shown in Figure 9.

The experiments were performed to classify pavement classes (good, medium, and poor) with high accuracy, while separating the soil, paving stone, dye, and shadow classes in the acquired images. Four different experiments were conducted using different numbers of training pixels (5, 10, 20, and 50 pixels per class) on the captured images to assess the model’s performance and robustness. Training samples were randomly selected from the existing spectral library. The mean values of the spectral signatures for each class are given in Figure 10a.

As seen in Figure 10a, the three asphalt classes have similar spectral signatures; therefore, each algorithm’s sensitivity to capture the subtle differences will be a key factor in accuracy assessment. The spectral angle mapper algorithm enables us to fine-tune our predictions, with its sensitivity to the characteristic absorption features of spectral signatures that are not visible to the human eye due to scaling or inherent noise. For our asphalt classes, the angular difference ranges between 0.06 and 0.09 radians, pointing out spectral dissimilarities between them. To make it more visually recognizable, we demonstrate our asphalt class signatures in Figure 10b with a closer look, using focused scaling after applying a smoothing filter to remove the noise component. After these operations, the absorption feature around 870 nm was more discernable for medium- and old-age asphalt classes. This absorption characteristic is associated with iron oxide minerals that are exposed within the pavement structure in the literature [11,14].

Furthermore, to explore the between-class variation of the present asphalt classes, we illustrate the class pixels by plotting them across the first three principal components in Figure 11. This plot shows that the three asphalt classes are separated from each other in the 3D space of principal components, although the first visual implication is that they are similar to each other. Contrary to that, these classes are separated from each other in an n-D space provided by the hyperspectral imaging.

6. Experimental Results

In this section, firstly, benchmarking algorithm outputs are presented individually. We then compare them with the classification accuracies of the proposed methodology by also presenting the corresponding output images. To test the model robustness and the effect of training size on the test images, we present the results of four different experiments on each test image.

6.1. The Performance of the Benchmark Algorithms

ANN results in a significantly different classification performance depending on the seed point and the training data. This variation is much more evident, especially when a small set of training data is used. Knowing that the algorithm is sensitive to the size of the training data, several experiments have been carried out to reveal the effect on the collected images.

Among the four performed experiments on the first image set, the average classification accuracy values of the first experiment (5 training samples for each class) are 67.54%, 45.96%, and 49.60% for the ANN implementation. According to the obtained results, ANN can hardly separate different asphalt classes from each other, given the difficulty of distinguishing between shadow, soil, and paving stone classes.

In the fourth experiment (50 training samples per class), SVM results in 75.86%, 70.03%, and 59.76% classification accuracy, while SAM outputs 75.02%, 62.90%, and 57.70% classification performance. The main problem of SVM is the inadequate separation of different pavement classes, as in the case of ANN. At the same time, SAM is found to be more successful in separating asphalt classes to a better extent, which is, however, not sufficient. Unlike the general increase in accuracy rates with higher training samples in SVM, ANN, and SAE, the same tendency is not observed with SAM. The increase in the training data size significantly improves the SAE algorithm’s performance, among other things. In the first and last experiments, the SAE algorithm achieved performance rates of 11.52% and 58.58% for Image 1, 20.58% and 61.87% for Image 2, and 20.58% and 55.63% for Image 3, respectively. The mean performance values for each experiment and the performances of each individual algorithm are given in

Table 2. Average classification accuracies for the first image set.

Train Data	Image 1 (%)					Image 2 (%)					Image 3 (%)
	SAM	SVM	SAE	NN	NN + SAM	SAM	SVM	SAE	NN	NN + SAM	SAM	SVM	SAE	NN	NN + SAM
5	72.49	50.22	11.52	67.54	75.69	49.71	27.10	20.58	45.96	67.77	52.39	29.80	20.58	49.60	68.14
10	73.45	62.90	20.36	82.36	89.28	60.10	49.71	25.61	70.96	77.28	52.72	42.28	25.61	70.52	73.84
20	74.65	73.79	25.60	84.11	92.89	61.17	61.20	44.79	76.10	84.37	55.03	51.86	44.79	73.47	87.10
50	75.02	75.86	58.58	94.06	94.32	62.90	70.03	61.87	82.50	85.92	57.70	59.76	55.63	90.81	91.59

.

Based on the training samples per class, a steady increase in accuracy is observed as the training samples increase, specifically with the feed-forward neural network and our proposed methodology. In order to observe the rate of change of performance in relation to the training sample size, the instances were increased to an adequate level, enabling us to observe the algorithm’s robustness and achieved performances for each method. For all of the implemented algorithms, the performance primarily depends on the discrimination of the three asphalt classes rather than the remaining yellow dye, white dye, soil, paving stone, and shadow categories. The white and yellow paint, shadow, soil, and paving stone classes are most effectively distinguished from each other with the SAM algorithm.

After observing the algorithm’s performance compared to the baseline methods for the first image set (Image 1–3), the proposed methodology was implemented on a second image set, including three more images for testing purposes (Image 4–6). The new experiments validate the previously observed incremental increase with the training data, with an accuracy improvement of 1–17% in comparison to ANN scores. For this image set, the spectral angle inclusion results in a slightly better class separation. The classification accuracies for the second set of images are given in

Table 3. Average classification accuracies for the second image set with ANN and proposed method.

Training Data	Image 4 (%)		Image 5 (%)		Image 6 (%)
	NN	NN+SAM	NN	NN+SAM	NN	NN+SAM
5	69.61	72.32	54.99	55.66	72.22	78.03
10	69.39	86.12	62.10	65.53	83.82	84.77
20	87.70	88.33	74.45	74.98	82.95	83.85
50	91.96	92.33	75.01	75.90	92.00	93.18

.

6.2. Experimental Results for the Proposed Methodology

The proposed methodology combines ANN and SAM scores for the improved differentiation of asphalt classes. In implementing this joint method, the prediction scores of ANN and the calculated angular values of SAM are stored together. For each class, a new estimation value is obtained by taking the ratio of the prediction values by ANN and SAM scores for each class separately. The outputs of the proposed methodology are shown in Figure 12c, along with their corresponding RGB image and ground truths in Figure 12a,b. The algorithm outputs for the first set of images are plotted together in Figure 13, showing the improved performance of the joint methodology over individual benchmark methods.

For the first image set, the overall accuracy is improved from 75.69% to 94.32% for Image 1, from 67.78% to 85.93% for Image 2, and from 68.14% to 91.59% for Image 3, with the proposed method, with an improvement between 1 and 21% over the baseline ANN algorithm. The experiments show that the proposed method performs better for non-binary classification problems, with significant improvements over the baseline ANN method for images with three or more classes. The results also show that the separation between asphalt classes is better with the joint methodology, which indicates its potential for pavement condition assessment.

For the second set of images, which is shown in Figure 14, the overall accuracy is improved from 72.32% to 92.33% for Image 4, from 55.66% to 75.90% for Image 5 and from 78.03% to 92.18% for Image 6 with the proposed method, with an improvement between 1 and 16% over the baseline ANN algorithm. Figure 15 illustrates the change in performance across the experiments for the second data set. The experiments show that the proposed method provides significant improvements over the baseline ANN method for images with three or more classes.

7. Discussion

The experiments with benchmarking algorithms reveal a performance improvement between 1.2–21% across the acquired six images. We observe a gradual increase with SVM algorithm (35–68%), with comparatively higher rates as the training data increase in the first image set. SAM, on the other hand, results in similar scores (58–65%) with less sensitivity to the number of training data, while autoencoder persistently results in poor performance compared to these methods. Through the pixel-wise integration of spectral metrics on neural network prediction scores, proposed methodology outputs consistently higher accuracies to these benchmark methods along with the implemented neural network scores (70–90%).

In order to depict the performance gains through hyperspectral sensor, the algorithm is implemented with also three RGB bands as an input.

Table 4. Average classification accuracies of the proposed methodology with RGB input.

Training Data	Image 1 (%)	Image 2 (%)	Image 3 (%)	Image 4 (%)	Image 5 (%)	Image 6 (%)
5	80.52	60.49	61.82	74.59	50.30	45.31
10	81.09	79.59	71.50	79.61	61.73	81.37
20	96.78	81.70	85.59	81.17	74.26	77.09
50	96.84	83.46	84.49	82.89	74.07	83.97

demonstrates the scores acquired with RGB channels as an input with the proposed method. When the full spectrum is used as an input, the algorithm performances are 5–10% better depending on the training data across the six experiments. As far as the intensity is concerned, our approach outputs better scores when there are three or more classes in the image, as the effect of the SAM score is virtually absent when the ANN score is close to one or zero for a binary scenario. For instance, while the asphalt classes are separated successfully from the paving stone class with the hyperspectral image, it is mixed with the medium age asphalt class with RGB input, resulting in lower classification accuracies.

Across the six experiments, the increase in training samples reduces the dependence on arbitrary seed points. When only five training data for each class are employed, the performance of the algorithm changes in the range of 56–78%, whereas it varies between 76–95% for fifty training samples across the six experiments. With the proposed methodology, the observed variance across experiments is reduced significantly, while the average accuracy is increased consistently regardless of the number of training samples. This improvement is more profound where the training samples are low, revealing the proposed method’s potential.

Furthermore, in order to explore the effect of the presence of secondary classes in the experimental scenes (white dye, yellow dye, shadow, soil) on our overall classification accuracy, we calculate the percentages of these classes by the image size across each experiment. Our calculations reveal that these non-asphalt classes have 1.6 to 7.9% coverage across the captured images. In

Table 5. Confusion matrix for Image 4.

	White Dye	Yellow Dye	Asphalt 1	Shadow	Asphalt 2	Asphalt 3	Paving Stone	Soil
White Dye	74,788	0	623	0	19	1027	6284	14
Yellow Dye	0	44,342	0	0	24	0	228	126
Asphalt 1	1223	0	1,439,545	92	3543	81	6	19
Shadow	0	0	1235	4721	4	0	0	2346
Asphalt 2	2286	11	3045	0	1,044,689	387,177	98,591	1
Asphalt 3	4424	0	98	0	3286	1,365,758	990	0
Paving Stone	94	420	348	0	119,113	22,089	157,785	3011
Soil	0	2020	4511	0	424	1	824	127,035
Overall accuracy between all classes								86.41%
Average accuracy between asphalt classes only								87.57%

, we provide a sample confusion matrix for one of our runs on Image 2, whereby we can investigate all asphalt classes along with all non-asphalt categories. This confusion matrix illustrates the numerical impact of these non-asphalt classes on our reported overall accuracies.

Our results show that the asphalt detection accuracy is, in fact, slightly better when we do not include the non-asphalt classes in our calculations. This is due to the small coverage of these categories across the scenes that could hardly affect the reported overall accuracies.

As a final step, we proceed to calculate the average and standard deviation of the accuracy for each experiment in order to evaluate the impact of the training data across the six experiments. Figure 16 demonstrates the standard deviations and mean values of the classification accuracy for the proposed model and neural networks. The average classification accuracy is improved with the increased training data size, as expected, accompanied by the lower standard deviation of the classification accuracy, emphasizing the profound impact of the proposed methodology when the training sample size is small. The model’s efficiency is proven by the consistent improvement in average performance and the lower variances across experiments in comparison to the benchmarking methods.

8. Conclusions

Pavement sustainability and condition assessment have remained a long-lasting issue for the transportation management authorities [2]. Assessments of road networks and introducing multi-year maintenance plans are essential organizational tasks in terms of strategic asset management by agencies and decision-makers. PMSs integrate technical, economic and environmental factors in these decisions for the sustainable operation of road network conditions. At this point, data collection stands as a crucial and challenging issue due its costly and labor-intensive nature, along with its direct relation to business decisions.

This study presents a new methodology for automated pavement condition assessment based on artificial neural networks and hyperspectral detection methods. To this end, a custom-designed vehicular system was built for data collection, and image acquisition was performed with a Headwall A-series VNIR camera. In the experiments, an age-based classification is implemented using six different images, including three different asphalt classes (good (5-year), medium (10-year) and poor (25-year)) along with the paving stone, white paint, yellow paint, soil and shadow classes, captured on the METU campus area. The spectral detectability of asphalt quality with VNIR brightness indexes has encouraged the research in this controlled setting, offering an alternative pavement management system to identify and prioritize the needed maintenance action. For classification purposes, the ANN, SVM, SAM and SAE algorithms, which are widely used in the literature, are implemented. To overcome the limitations of these widely used algorithms, a novel algorithm combining ANN and SAM scores is presented. The classification performances of individual ANN and SAM are significantly enhanced with their joint use, representing a 1 to 21% classification accuracy improvement across all experiments.

During the experiments, the need for more learning data came forward as a key driver to obtain better classification accuracy in the implemented methods. Especially in the case of the low number of training samples, the classification performance has been found to be low, as the algorithms could not differentiate between asphalt classes. This is because the captured images have different classes of pavement condition with similar spectral signatures. The proposed methodology overcomes this challenge with its combined approach of both neural networks and spectral angle proximity at the same time. Not only would the overall performance of this integrated methodology provide opportunities for identifying the worn pavement surface, but the system could also be useful for the regular monitoring of high-circulation roads. Despite the lack of structural deformities in the controlled test environment of campus roads, the results also suggest their detectability through the proposed methodology. Therefore, the study confirms the applicability of hyperspectral analyses to support strategic asset management and maintenance plans as a component of pavement management systems in agencies, enabling regular and cost-efficient data collection for sustainable pavement management. Future studies will be carried out to identify asphalt surface components using hyperspectral unmixing techniques to capture sub-pixel level information. In addition, new methods of accelerating algorithms will be implemented for the real-time monitoring of asphalt surfaces.