An Adaptive Approach for Multi-National Vehicle License Plate Recognition Using Multi-Level Deep Features and Foreground Polarity Detection Model

Featured Application

We present an adaptive framework for the recognition of multinational vehicles license plates. To make it generalized, this research does not require any prior knowledge of license plate layout and, furthermore, training data is not used from all of the targeted countries. These properties make this approach more suitable in order to get the registration identity of multinational vehicles.

Abstract

License plate recognition system (LPR) plays a vital role in intelligent transport systems to build up smart environments. Numerous country specific methods have been proposed successfully for an LPR system, but there is a need to find a generalized solution that is independent of license plate layout. The proposed architecture is comprised of two important LPR stages: (i) License plate character segmentation (LPCS) and (ii) License plate character recognition (LPCR). A foreground polarity detection model is proposed by using a Red-Green-Blue (RGB) channel-based color map in order to segment and recognize the LP characters effectively at both LPCS and LPCR stages respectively. Further, a multi-channel CNN framework with layer aggregation module is proposed to extract deep features, and support vector machine is used to produce target labels. Multi-channel processing with merged features from different-level convolutional layers makes output feature map more expressive. Experimental results show that the proposed method is capable of achieving high recognition rate for multinational vehicles license plates under various illumination conditions.

1. Introduction

Automatic license plate recognition (ALPR) is a significant topic in the field of intelligent transport systems (ITS) and remains ever challenging in the research era of image processing and computer vision. This is the framework that is used widely to extract the license plate registration number from digital images for vehicle identification. This system also does not require any additional hardware like transmitter or responder to detect vehicles as every vehicle is identified by its license plate. Numerous potential applications have urged researchers to pay more attention to developing an efficient and sophisticated ALPR system. The law enforcement agencies are widely using this system for traffic monitoring, congestion control, borders or restricted area security and also to detect suspicious or theft vehicles. LPR system is also being used in smart parking areas and smart toll stations for flexible entry and exit of vehicle, smooth traffic flow, deduct fee or fine and to enhance the security. Moreover, it can be installed in intelligent vehicles to recognize the identity of neighboring vehicles for communication purposes.

License plate detection (LPD) and license plate recognition (LPR) are two major modules of an ALPR system. LPR module can be further split into license plate character segmentation (LPCS) and license plate character recognition (LPCR). In recent years, particularly focused work has been seen for multinational ALPR systems, but it has been done only for license plate detection and verification [1,2,3,4,5]. Every module has its own importance but LPR is comparatively more important as most required information is extracted at this stage and this is a difficult task, specifically in the case of multi-style LPs for multinational vehicles. The whole system will be useless if we could not get a whole registration number at the LPR stage. In order to complete the system, the proposed work has been done for license plate recognition part that further splits into LPCS and LPCR stages.

Most of the work is seen only for specific types of LPs in existing literature and their effectiveness is limited as it cannot be applied to another region. Moreover, change in license plate regulations would lead to the failure of such specific methods. Various countries do not have standardized LPs and also allow customized license plates that add further complications. In addition, many countries have a lot of diversity in their license plates. Therefore, there is a need to develop a generalized framework for the recognition of multinational vehicle license plates to cater above-mentioned issues. Generally, challenges that are involved in effective character segmentation and recognition of license plates are illumination variance, rotation, shadow and border touched characters due to screws or noise, but some more challenges are involved when we deal with multinational vehicle license plates. They have different background and foreground colors, complex backgrounds containing different shapes and writings, and different shape and size of LPs and characters as shown in Figure 1.

A unique part-based approach is developed that gives very promising results for character segmentation of multinational vehicle license plates despite the above-mentioned challenges. The RGB color-based foreground polarity detection model and character height estimation filter are proposed to accomplish the license plate character segmentation task effectively. To deal with multi-style backgrounds, thresholding under various illumination conditions, skew correction and border touched characters for multinational VLPs are salient features of proposed segmentation method. For the recognition of segmented characters, multi-channel CNN framework with layer aggregation module is proposed to extract deep features, and support vector machine is used as classifier to produce target labels.

Our major contributions in this article are: (i) the proposed generalized method does not require any prior knowledge of LP layout and no training data is required at LPCS stage, furthermore we do not use training data from all targeted countries to extract deep features at LPCR stage, (ii) established color-based foreground polarity detection framework that contributes at both LPCS and LPCR stages. This algorithm significantly classifies the foreground from background based on their colors even under various illumination conditions (iii) multi-channel processing with concatenated output feature vector is introduced in order to get enhanced original image information, (iv) improved CNN is proposed by merging different-level convolutional features to get a more expressive output feature map.

The remainder of the paper is arranged in the following order. Section 2 presents the related work in detail. Section 3 explains the proposed method and Section 4 provides the experimental results and comparison with the latest methods. In Section 5, we present the concluding remarks.

2. Related Work

Generally, connected components with binary thresholding, projection-based analysis, maximally stable extremal regions (MSER), sliding window and many CNN based methods have been extensively used for character extraction from license plates.

The research conducted in [6,7] claims to be implemented for multinational vehicle LPR but was only tested on Israeli and Bulgarian LPs. The two-stage character segmentation approach [8] is proposed for Chinese LPs by using CCA and bank of harrow-shaped filter (HSF). The CCA and kohonen neural network [9] is used to segment characters from Indian LPs. The CCA and Contour detection is used for character segmentation in Bangladesh and Iraq license plates [10,11,12]. CIE-LAB color space, OTSU segmentation and CCA are used to extract characters from a Pakistani vehicle license plate. This method [13] is suitable only for highly contrasting images and very sensitive to shadow. The projection-based methods [14,15] have also been used for China and Iran respectively for LPCS. Usually, this technique is used with one of the binarization methods (OTSU, Adaptive, MET, etc.). The first image is converted into binary then horizontal and vertical projection is used to separate the characters. Maximally stable extremal regions (MSER) is also very famous method for LPCS as used in [16,17] for Chinese license plates. Sliding window technique [18,19] has also been used for America and Iran, but this method requires prior knowledge of window size and it is computationally complex. In recent years many CNN-based approaches are introduced to accomplish segmentation free LPR task as discussed in [20,21]. Recurrent neural network and connectionist temporal classification is used in [20] while in [21], YOLO detector is used to accomplish this task and the target countries are China and Brazil respectively. The author proposed robust license plate recognition by introducing the concept of synthetic images to train a convolutional neural network [22]. A multi-task convolutional neural network [23] is proposed for license plate recognition with better accuracy and lower computational cost.

Once the segmentation is done successfully, the next step is to recognize the characters separately. Generally, four approaches are used to accomplish the recognition task. The simplest and straightforward method is template matching [24,25], which was used a lot in the beginning, but this method is not flexible enough for different fonts, noise, rotation and thickness changes. The artificial neural network [12,26,27,28] is another very popular approach that has been used widely to recognize the license plate characters as well as many other classification tasks. The SVM classifier [13,16,29,30] with various feature extraction methods is also used to perform recognition tasks as SVM is strong and fast classifier for real time applications. The extreme learning machines (ELMs) [31] and hidden Markov model (HMM) [32] are also used as license plate recognition classifiers. The concept of using the conventional low-level features is now considered old fashioned due to the revolution in computer vision by the introduction of deep learning methods. The deep convolutional neural network (CNN) architectures learn a hierarchy of discriminate features automatically that richly describe image content. In recent years, the use of deep learning frameworks [20,21,33,34,35] are also seen in ALPR system because of its powerful recognition capability.

No method mentioned above (CCA, projection, MSER, sliding window) has the capability to handle multinational LPs directly. Some CNN based methods have capability to do this task, but they require a lot of training data which is not feasible in multinational scenario. Our proposed recognition framework is capable enough to recognize the license plate characters of different shapes, styles and having different background/foreground colors under various illumination conditions.

3. Proposed Methodology

The proposed framework is comprised of license plate character segmentation (LPCS) and license plate character recognition (LPCR) modules. The detailed proposed architectures of these modules are discussed in this section.

3.1. License Plate Character Segmentation

Figure 2 shows the complete process of the license plate characters segmentation. As discussed above, we used part-based approach to accomplish this task. The first objective is to segment the region of interest (ROI) in order to reduce the image processing area and to discard a lot of redundant information. In the second part, a proposed character height estimation filter along with connected component analysis (CCA) is applied to extract the required objects. These blocks are discussed below in detail.

LP image detected by [1] is fed to part-I for the foreground and background classification. To distinguish the background from the foreground is one of the most difficult and important steps for object segmentation. The proposed method strongly requires the identification of background/foreground color and polarity, as further segmentation processes based on this step. The term polarity is referred to bright or dark based on the intensity of foreground/background colors. We observed that most of the countries use unique color for background as well as for foreground. Probably, the largest color candidates would belong to background region while foreground would have second highest color candidates, which is the key point that we use for background and foreground classification. We propose a model by using RGB color space which is best suited to identifying any color even under various illumination conditions.

RGB-based color space can be visualized as a cube with corners of black, the three primaries (red, green, blue), the three secondaries (cyan, magenta, yellow), and white. The eight pure color corners and distribution cube is shown in Figure 3 [36]. By considering above mentioned colors and color distribution, a model is proposed that represents all shades of colors into 8 pure colors and has the capability to determine any background and foreground color, which is strongly required in case of multinational VLPs. The total candidates of red, green, blue, yellow, cyan and magenta are computed by (1) as per the color’s definition mentioned in

Table 1. Color definition based on three main groups.

Color Count	$\mathbf{Color}\mathbf{Threshold}\left({\mathit{T}}_{\overline{\mathit{C}}}\right)$	Color Group
$C_{\overline{R}}$	d1 ≥ 0.24 & d3 ≥ 0.24	B < R > G
$C_{\overline{Y}}$	d1 < 0.24 & d3 ≥ 0.24
$C_{\overline{M}}$	d1 ≥ 0.24 & d3 < 0.24
$C_{\overline{BL}}$	d2 ≥ 0.24 & d3 ≥ 0.24	R < B > G
$C_{\overline{CY}}$	d3 ≥ 0.24 & d2 < 0.24
$C_{\overline{M}}$	d1 ≥ 0.24 & d3 < 0.24
$C_{\overline{G}}$	d1 ≥ 0.24 & d2 ≥ 0.24	R < G > B
$C_{\overline{Y}}$	d1 < 0.24 & d2 ≥ 0.24
$C_{\overline{CY}}$	d1 ≥ 0.24 & d2 < 0.24

$C_{\overline{R}}$: Red count, $C_{\overline{Y}}$: Yellow count, $C_{\overline{M}}$: Magenta count, $C_{\overline{BL}}$: Blue count, $C_{\overline{CY}}$: Cyan count, $C_{\overline{G}}$: Green count, $d_1=\left|\left(R-G\right)/R+G)\right|$, $d_2=\left|\left(G-B\right)/G+B)\right|$, $d_3=\left|\left(B-R\right)/B+R)\right|$. The distances between primary colors red-green, green-blue and blue-red are represented by d₁, d₂ and d₃ respectively.

, and the threshold value is determined by using color distribution cube and rigorous experiments on 3718 images in the test dataset.

$$C_{\overline{C}}= {\displaystyle \sum }_{m=0}^{height-1}\left\{{\displaystyle \sum }_{n=0}^{ width-1}T(I\left(x_n , y_m\right))\right\}$$

(1)

$T\left(\zeta \right)=1 when \zeta is T_{\overline{C}}$, $where \zeta =I\left(x_n , y_m\right)$

By using (1) we can get color candidates count ($C_{\overline{C}}$) of red, green, blue, cyan, magenta and yellow, but we cannot set any hard limit to separate the group of black and white pixels as the illumination level is unknown. The values of white and black pixels ($V_{BW})$ are extracted together as one group and the candidate’s vector ($C_{\overline{BW}})$ is obtained as

$$C_{\overline{BW}}=\{\begin{array}{l}{V}_{BW}\hspace{1em}if\hspace{1em}{V}_{BW}\hspace{1em}⋲\hspace{1em}{T}_{\overline{C}}\\ 0\hspace{1em}\hspace{1em} if\hspace{1em}{V}_{BW}\hspace{1em}\notin \hspace{1em}{T}_{\overline{C}}\end{array}$$

(2)

$\mathrm{where} T_{\overline{C}}=\{d_1<0.2 \& d_2<0.2\}$.

Then Otsu’s method based on two intra-class variance is applied to separate them into two classes having black and white pixels separately as shown in Figure 4. Further, larger and smaller class between these two groups is determined as

$$C_{\overline{B}}={\displaystyle \sum }_{i=0}^{SP}{C}_{\overline{BW}\left(i\right)}$$

(3)

$$C_{\overline{W}}={\displaystyle \sum }_{i=SP}^{255}{C}_{\overline{BW}\left(i\right)}$$

(4)

Until this point, we have known the background and foreground colors because the largest candidates group belong to the background while the second largest candidates group belong to the foreground, as shown in Figure 5a. This information is further used to determine the foreground polarity for post processing, as this information is most necessary for adaptive thresholding and for most morphological operations.

Adaptive thresholding technique strongly requires the prior knowledge of foreground polarity to separate background from foreground precisely. This information is also needed for morphological operations to obtain the required results. Algorithm 1 is used to accomplish this task, whereas max1 and max2 belong to $C_{\overline{C}}$ that represent the background and foreground respectively. The background and foreground polarities depend on the sequence of colors as shown in Figure 5b. For example, Ind-1 has brighter polarity as compared to Ind-2. The CG2 (Ind) is determined as

Algorithm 1 Foreground polarity detection process

Input 1: max1   % max1 represents the color pixel count belongs to LP background

Input 2: max2   % max2 represents the color pixel count belongs to LP foreground

If (max1 & max2) $⋲$ CG1

If max1 (Ind) > max2 (Ind)

FP ← bright

Else

FP ← dark

End

Else if (max1$⋲$ CG1& max2$⋲$ CG2) || (max1 $⋲$ CG2 & max2 $⋲$ CG1)

Output:

FP← find {CG2(Ind)}

End

$$CG{1}_{MEAN}=\frac{1}{N}{\displaystyle \sum }_{i=1}^N{V}_{CG1\left(i\right)}$$

(5)

$$CG{2}_{MEAN}=\frac{1}{N}{\displaystyle \sum }_{i=1}^N{V}_{\overline{BW}\left(i\right)}$$

(6)

$$CG2\left(Ind\right)=\{\begin{array}{l}1\hspace{1em}if CG{1}_{MEAN}<CG{2}_{MEAN}\\ 2\hspace{1em}if CG{1}_{MEAN} >CG{2}_{MEAN}\end{array}$$

(7)

The next step is to convert the RGB image into binary image $BI\left(x,y\right)$. For this purpose, first the image is converted into gray image ($GI)$ by using the standard conversion computed by (8) and then we use real-time adaptive thresholding for binary conversion using local mean intensity (first-order statistics) as mentioned in (9) with neighborhood size (Ns). This is most suitable thresholding technique compared to OTSU’s, MET and many others based on two intra-class variance to separate background and foreground particularly in shadow and illumination variance environment. Adaptive thresholding requires prior knowledge of foreground polarity which we have already determined in the above steps.

$$GI=0.2989 \ast R+0.5870 \ast G+0.1140 \ast B$$

(8)

$$BI\left(x,y\right)=\{\begin{array}{l}1\hspace{1em}GI \left(x,y\right)<\frac{{{\displaystyle \sum }}_{i=x-m/2}^{x+m/2}{{\displaystyle \sum }}_{j=y-n/2}^{y+n/2}GI \left(i,j\right)}{mxn}\\ 0\hspace{1em}GI \left(x,y\right)\ge \frac{{{\displaystyle \sum }}_{i=x-m/2}^{x+m/2}{{\displaystyle \sum }}_{j=y-n/2}^{y+n/2}GI \left(i,j\right)}{mxn} \end{array}$$

(9)

$$Ns=2\ast floor\left(size\left(I\right)/16\right)+1$$

(10)

Figure 6 shows the effect of prior knowledge of foreground polarity. In Figure 6a, the LP with dark foreground is well thresholded while in Figure 6b, the LP with bright foreground is well thresholded. So, for multinational vehicle LPs, we must have prior knowledge of foreground polarities in order to separate foreground from background as multinational VLPs have different backgrounds and foregrounds polarities. After getting binary image, we change its background polarity to bright if needed by using (11).

$$B_I\left(x,y\right)=\{\begin{array}{l}{B}_I\left(x,y\right)\hspace{1em}if FP=Dark\\ \overline{B_I\left(x,y\right)}\hspace{1em}if FP=Bright\end{array}$$

(11)

The next step is to extract the ROI which contains only required characters by discarding redundant area of license plate. Let I be a given set of objects of 2 types, then required region $(R_R)$ can be extracted as

$$R_R=\left\{\begin{array}{c}R | R is the maximal connected \\ set of type 1 in I\end{array}\right\}$$

(12)

where type 1 and type 2 represent the white and black objects respectively.

The character height estimation is also one of the most important and difficult tasks to accomplish the character’s segmentation for multinational LPs as different countries LPs have different character size. It also uses further for skew correction and to eliminate the other remaining redundant objects. Until this step, we have already discarded a lot of redundant area of LP by extracting the required background region. To accomplish this task, we propose an adaptive way to estimate the character’s height, which can be estimated as

$$H_{O_{hi}}=\left\{O_{hi} |\hspace{1em}{h}_i=\pm 20\% of h_{max} ; i=1,2,\dots .n\right\}$$

(13)

The $H_{O_{hi}}$ is a set that is used to keep the objects of specific height. Where, $h_{max}$ is the maximum height among all detected bounding boxes in extracted region ($R_R)$ of license plate. The $O_{hi}$ is the targeted object that is selected based on the criteria of height $h_i$. The next bounding box object is compared with $h_{max}$, if its height is $\pm 20\% of h_{max}$ then it will be considered as required object. If it does not fulfil the criteria then the previous bounding box with maximum height will be discarded and a recent one will be considered as maximum height object for further processing. This process will be continued until all required objects are extracted. By following this approach, all larger and smaller objects are eliminated automatically and only required objects, i.e., license plate characters are left.

After finding the heights of required objects, we use this information for skew correction by using (14). The skew detection and correction problem is also addressed in an efficient and accurate way, as it helps to enhance the system performance at segmentation as well as at recognition stage.

$$C{A}_{LP}= {tan}^{-1}\left(\frac{y_{RM\left(O_{hi}\right)}- y_{LM\left(O_{hi}\right)}}{x_{RM\left(O_{hi}\right)}- x_{LM\left(O_{hi}\right)}}\right)$$

(14)

$\mathrm{where}{O}_{hi} ⋲H_{O_{hi}}$, $y_{RM\left(O_{hi}\right)}$ and $y_{LM\left(O_{hi}\right)}$ is y-axis values of right most and left most detected objects while $x_{RM\left(O_{hi}\right)}$ and $x_{LM\left(O_{hi}\right)}$ is x-axis values of right most and left most detected objects of hi.

The border touched characters are separated by using the information of upper and lower boundaries of objects in set $H_{O_{hi}}$ as shown in Figure 8c. The final step is to remove the remaining redundant information and to get bounding boxes on our required objects that are LP characters, so, this task is done by using (15)

$$FI=\{\begin{array}{l}{O}_{hi}\hspace{1em}if O_{hi} ⋲ H_{O_{hi}}\\ 0\hspace{1em}\hspace{1em}if O_{hi} \notin H_{O_{hi}}\end{array}$$

(15)

where, $FI$ represents the final image that have only required LP characters.

The output of above discussed LPCS processes can be seen in Figure 7 and Figure 8. In order to prove the effectiveness of proposed method, Figure 9 presents some sample images of LPs having blur, noise, shadow and also effected by various illumination conditions. The proposed method is not capable of handling the multi-color background/foreground LPs as shown in Figure 10. As discussed above, most of the countries have unique color for background and foreground. Probably, the largest color would belong to background region while foreground would have second largest color, which is the key point that is used to distinguish the background from foreground. In this proposed approach, the rest of the processes are based on this information. That is why this method would give failed segmentation of the license plate characters.

3.2. License Plate Character Recognition

We propose a deep learning (DL) framework to accomplish the license plate character classification task as DL is most recently been introduced in approaches in the area of artificial intelligence. Deep learning is powered by neural networks. Convolutional neural networks (ConvNets or CNNs) are a category of neural networks that have proven very effective in areas such as image recognition and classification. They are made up of neurons that have learnable weights and biases. Each neuron receives some inputs, performs a dot product and optionally follows it with a non-linearity. The whole network still expresses a single differentiable score function: from the raw image pixels on one end to class scores at the other. And they still have a loss function on the last (fully-connected) layer.

Figure 11 represents the proposed structure for character recognition of multinational vehicles LPs. First, the segmented image is decomposed into red, green and blue channels and then it is passed through polarity matching module to process data uniformly in order to eliminate the impact of foreground polarity that varies in multinational VLPs. To hierarchically learn features, these separated channels images are fed to CNNs and get output vectors, which are further concatenated to acquire enhanced image’s feature information. Before sending this output feature vector to the classifier we introduce data normalization module which has a better impact on output scores. Finally, this normalized feature vector is fed to classifier in order to obtain target labels.

The pre-trained network is used as a starting point to learn a new task. Fine-tuning a network with transfer learning is usually much faster than training a network with randomly initialized weights from scratch. Learned features can be transferred quickly to a new task using a smaller number of training images. Some of the well-known pre-trained networks (AlexNet [37], VGG-16 [38], GoogleNet [39], ResNet-18 [40], Inception v3 [41]) have been trained over a million images and can classify images into 1000 object categories and have learned rich feature representations for a wide range of images. The speed is one of the most important parameters as we are working for real-time applications. By keeping this constraint in mind, we choose AlexNet CNN as our base network for transfer learning as it is time efficient among the rest of the four well known CNNs, as shown in Figure 12. The test time is considered for whole test database to test the speed of networks for our particular application. The test database for recognition part contains 21717 license plate characters images that are extracted at segmentation stage from 3718 license plate images of eight different countries.

In deep neural networks, higher-level convolutional layers have rich features representation while the spatial information is deficient. In contrast, in shallow layers, spatial information is reserved at the cost of less expressive features. Generally, features from the last convolutional layer are used by FC layers which are further used for classification. Max-pooling is a down sampling strategy in convolutional neural networks. Therefore, we may have a chance to get spatial information that might be lost during down sampling. Vanishing gradient is another problem that may occur in deep networks. For recognition and detection tasks, performance can be enhanced and information loss can also be reduced by using collective feature information of different convolutional layers, as the authors suggested in [42,43,44,45]. In addition, it is noted that the convergence time is also improved by using the concept of deep layers aggregation. Therefore, we propose an improved CNN at the lines of AlexNet, as shown in Figure 13 and parametric detail is discussed in

Table 2. The parametric detail of improved CNN.

Layers	Configuration
Input	FM: 1	OS: 227 × 227 × 3	KS: -	S: -
Convolution1	FM: 96	OS: 55 × 55 × 96	KS: 11 × 11	S: 4
Max Pooling1	FM: 96	OS: 27 × 27 × 96	KS: 3 × 3	S: 2
Convolution2	FM: 256	OS: 27 × 27 × 256	KS: 5 × 5	S: 1
Max Pooling2	FM: 256	OS: 13 × 13 × 256	KS: 3 × 3	S: 2
Convolution3	FM: 384	OS: 13 × 13 × 384	KS: 3 × 3	S: 1
Convolution4	FM: 384	OS: 13 × 13 × 384	KS: 3 × 3	S: 1
Max Pooling4	FM: 384	OS: 6 × 6 × 384	KS: 3 × 3	S: 2
Convolution5	FM: 256	OS: 13 × 13 × 256	KS: 3 × 3	S: 1
Max Pooling4	FM: 256	OS: 6 × 6 × 256	KS: 3 × 3	S: 2
Concatenation	OS: 6 × 6 × 640	Layer4 © Layer5
FC6	OS: 1 × 1 × 4096
FC7	OS: 1 × 1 × 4096
FC8	OS: 1 × 1 × 37

. In improved CNN, the feature of convolution layer 4 and 5 are merged and one extra max-pooling layer is also introduced to match the features dimensions of both convolution layers.

The next most important step is to choose the classifier that accepts the feature vector from CNN feature learning module and generates output labels. In [46,47,48], the authors claimed that the SVM is a strong and fast classifier for real-time classification applications and great attention has been paid to the fusion of neural networks and SVM [49,50]. That is why the same is used in our proposed system. For the support vector machine algorithm, kernel and hinge loss function is used as described in (16) and (17) [51].

$$G\left(xj , xk\right)=x{j}^{\prime }xk$$

(16)

where G(xj,xk) is element (j,k) of the Gram matrix, where xj and xk are p-dimensional vectors representing observations j and k in X.

$$\ell \left[y,f\left(x\right)\right]=max\left[0,1-yf\left(x\right)\right]$$

(17)

where f(x) = xβ + b, β is a vector of p coefficients, x is an observation from p predictor variables and b is the scalar bias.

4. Experimental Results and Discussion

We have used MATLAB 2019b on an Intel® Core™ i7 CPU having 4.20GHz processing power along with GTX 1080Ti GPU for feature extraction and evaluation. Experiments on 3718 high quality, low quality and blurry images have been performed. Most of the vehicles license plates are from different states of America, Europe along with vehicles from Pakistan, UAE, Canada, Australia, Mexico, UAE, etc.

License plates having different foreground and background colors with a variety of formats are included in the test dataset, as shown in Figure 14. Most of the images are gathered from Media Lab [52] and Olav’s databases [53]. Moreover, the low-resolution images and the images effected by diverse weather and illumination conditions are also included in the test database. In several license plate images, shadow, partial shadow and blur are also included. Some cases in which license plates were under direct sunlight are also included to verify the adaptability of the proposed method.

4.1. LP Characters Segmentation Results Analysis

Regardless of the variety of license plates, the proposed framework effectively segments all the characters of almost 97% of license plates with 98% precision, as shown in

Table 3. Accuracy and precision of segmented characters.

Country	TLPs	NMCLPs	CSc	FSC	Accuracy, %	Precision, %
Australia	247	242	1364	18	97.98	98.70
Canada	329	325	1813	22	98.78	98.80
England	150	144	856	12	96.00	98.62
Mexico	106	103	735	07	97.17	99.06
Pakistan	397	382	2387	37	96.22	98.47
Europe	747	732	4445	75	97.99	98.34
UAE	46	44	237	05	95.65	97.93
USA	1696	1630	9880	191	96.11	98.10
Total	3718	3602	21717	367	96.88	98.33

, which is quite good in the case of multinational VLPs. The major reason for high accuracy for multinational VLPs is that our proposed approach first eliminates the redundant part of license plate by extracting the region of interest (ROI). It consists of the largest area based on unique color that is the salient feature of majority of the country’s LPs. Then, a character height estimation filter is used, which adaptively detects the length of characters in any type of license plate. The accuracy and precision are defined as

$$\begin{array}{c}Accuracy=\frac{NM{C}_{LPs}}{T_{LPs}}\\ \mathit{Precision}=\frac{C{S}_C}{C{S}_C+F{S}_C}\end{array}$$

and

$$T_{LPs}=NM{C}_{LPs}+M{C}_{LPs}$$

where $NM{C}_{LPs}$ (no-missing character license plates) are license plates of all detected characters and ${MC}_{LPs}$ (missing-characters license plates) are LPs having partially detected characters. $C{S}_C$ represents the correctly segmented characters and $F{S}_C$ denotes the falsely segmented characters of no-missing character license plates ($NM{C}_{LPs}$) and only these LP characters are further used in recognition stage, as the LPs with missing characters ($M{C}_{LPs})$ are useless for further processes.

4.2. LP Characters Recognition Results Analysis

In this part, each character extracted from the license plate will be separately recognized. In total, 21717 characters of 36 (0–9, A–Z) classes having different length of each class are extracted in the previous part. The test dataset consists of different style characters with different background and foreground polarities are extracted from various VLP formats to evaluate the performance of proposed feature model under different conditions. As we have used SVM for classification, which is a supervised learning model, training images are required to get a decision boundary for performance assessment. Therefore, a database of 84058 characters of 37 (0–9, A–Z, EC) classes and almost 2250 characters of each class are used for training. One extra-class (EC) is also introduced to eliminate the falsely segmented characters (FS_C). In order to support the adaptability constraint for recognition of multinational VLPs and to prove the effectiveness of the proposed framework, license plate characters from one country are gathered for training and made it suitable for the test data to have any background and foreground color and polarity. All training data consist of gray scale images to eliminate the effect of color and all images have dark foreground polarity to make it uniform. This approach will make this system more generic in nature to support multinational framework, as it is not feasible to gather training data from all targeted countries.

Our proposed model contributes well at three different levels and achieved outstanding performance collectively for such a diversified data. The individual and collective performance of these levels is discussed in the next sections.

4.2.1. Layers Aggregation Module

The CNN module is the backbone of the proposed method as it plays a vital role in extracting the features from license plate images, which greatly affects the recognition ability of the proposed framework. This is the reason behind proposing and evaluating four different suggested modified CNN structures by using layer aggregation models to fuse low- and high-level features. These four models are referred to as CNN_C15, CNN_C25, CNN_C35 andCNN_C45 and concatenate the feature map of convolution layers 1, 2, 3 and 4 with convolution layer 5 respectively. CNN_B denotes the base structure and outputs only convolution layer 5 feature map as traditional structure works. The proposed layers aggregation models for CNN structure can be seen in Figure 15, where C represents the convolution and P represents the pooling layer.

Table 4. Performance comparison of base CNN with modified CNN structures.

	Recognition Accuracy, %
Country	CNNB	CNNC15	CNNC25	CNNC35	CNNC45
Australia	89.67	88.49	74.10	91.14	91.55
Canada	78.76	82.47	63.72	82.41	84.62
England	85.12	84.9	72.14	84.89	88.43
Mexico	93.47	93.47	85.71	95.78	96.19
Pakistan	83.48	84.12	67.77	85.75	90.89
Europe	90.78	91.44	85.47	91.98	93.76
UAE	88.11	88.11	87.67	89.87	91.25
USA	80.64	81.97	63.03	82.00	85.69
Total	86.25	86.87	74.95	87.98	90.30

demonstrates the comparative study of four different layer aggregation and traditional models for 8 different countries. It is observed that network with CNN_C45 model gives better character recognition as compared to CNN_B, CNN_C15, CNN_C25 and CNN_C35. Probably, CNN_C15, CNN_C25 and CNN_C35 still contain excessive background noise as compared to CNN_C45, which would lead to reducing the performance of the system. Table 4 shows that CNN_C45 got 90.30% recognition accuracy, which is highest among all other structures even it beats the state-of-the-art network (CNN_B).

4.2.2. Multi-Channel CNN Architecture Module

Multiple image fields are the essential requirement for efficient object classification as image features have direct impact on performance of object recognition framework. Here we propose multi-channels based multi-CNN architecture along with concatenated output feature vector, that is capable to deal with comprehensive and more appropriate features to enhance the original image’s feature information. First, red, green and blue channel images are obtained by decomposing the original RGB image, then it is fed to separately trained neural networks for each channel image and finally the fully connected layer is obtained by concatenating one-dimensional eigenvectors of three single image domains, as shown in Figure 11.

Experimental results proved that the proposed framework with multi-channel CNN module (M_MC-CNN) can learn image features more effectively as compared to traditional convolutional neural network (CNN-LA), and achieved 92.13% overall recognition accuracy which is 2% more than traditional CNN structure, as shown in

Table 5. Performance comparison of different modules of proposed recognition network.

		Recognition Accuracy, %
Country	CNNB	CNN-LA	MMC-CNN-LA	MMC-CNN-LA © MPM
Australia	89.67	91.55	92.64	97.23
Canada	78.76	84.62	87.99	92.77
England	85.12	88.43	90.47	93.39
Mexico	93.47	96.19	96.53	98.86
Pakistan	83.48	90.89	93.92	98.71
Europe	90.78	93.76	94.16	97.86
UAE	88.11	91.25	92.53	93.04
USA	80.64	85.69	88.76	96.44
Total	86.25	90.30	92.13	96.04

.

4.2.3. Impact of FG-Polarity Matching Module

We explored the proposition that, in terms of foreground polarity, data uniformity enhances the network recognition performance. Our focused work is for multinational VLPs, and we know that different country LPs have different foreground polarities, even one country LPs may have different FP. For this purpose, we propose and integrate polarity matching module before feeding the image to multichannel convolutional neural network, that checks and inverts the foreground polarity of an image if needed, as we already have FP information that is extracted by using foreground/background classification framework at segmentation stage. The foreground polarity matching module works as follows

$$I_{FP}\left(x,y\right) = \{\begin{array}{l}{I}_{FP}\left(x,y\right)\hspace{1em}if FP=Dark\\ \overline{I_{FP}\left(x,y\right)}\hspace{1em}if FP=Bright\end{array}$$

The FP matching module is integrated between multi-channel input and multi-channel CNN network module to rectify the mismatch of foreground polarity and gives more accurate recognition results as compared to the previous stage. A data normalization module is also used to make an output feature vector in standardized form. The total 96% recognition accuracy is achieved after the integration of these modules in the proposed framework.

For better understanding, we graphically present the performance comparison of every module for all targeted countries separately, as shown in Figure 16 and Figure 17. Figure 16 shows the cumulative accuracy of every integrated module while Figure 17 only demonstrates the percentage increase in accuracy that every module contributes in the proposed system. It is observed that cumulative recognition performance enhances from 86.25% to 96% for our proposed network with the combination of different modules at different levels, as shown in Figure 18. Base represents the result achieved by the base network while Level 1, Level 2 and Level 3 represent the output of layer aggregation, multi-channel CNN and FG polarity matching modules respectively.

A comparison of the proposed method with already existing works is also provided. Five existing state-of-the-art methods have been implemented in MATLAB and their performance is recorded on test dataset. These existing methods are compared with the proposed method in terms of recognition accuracy and computational time and results are presented in

Table 6. Comparison with existing methods.

Method	Year	RA, %	Time, ms
Ref. [31]	2018	82.2	5.6
Ref. [13]	2018	93.1	4.0
Ref. [34]	2018	92.5	7.0
Ref. [21]	2018	83.5	1.9
Ref. [30]	2018	88.3	6.8
Proposed	-	96.04	3.5

. It has been observed that the proposed method outperforms all the existing methods by achieving 96.04% recognition accuracy. Furthermore, the proposed method is faster, in terms of computational time, than all tested methods except the method in [21]. Although the method in [21] has little better computational time than the proposed method, its accuracy is almost 12.5% less than the proposed method.

5. Conclusions

In this work, we propose an adaptive framework that is well suited for the recognition of multinational vehicle license plates. To make it generalized, this research does not require any prior knowledge of LP layout and furthermore we do not use training data from all targeted countries. This is the prominent feature of the proposed methodology. A foreground polarity detection model is developed for the classification of background and foreground based on their colors, which is not only the backbone of the LPCS stage but contributes well at the LPCR stage. This model also works well in the presence of shadow and under various illumination conditions. The multi-channel processing with improved CNN structure is proposed to extract deep features, and the real-time SVM classifier is used to get output labels. Features from convolution layer 4 and 5 are merged in a layer aggregation module in order to get a more expressive output feature map. The recognition performance is enhanced at three different levels and 96.04% recognition accuracy is achieved, which is a significant improvement in the case of multi-style license plates.