Face Recognition in an Unconstrained and Real-Time Environment Using Novel BMC-LBPH Methods Incorporates with DJI Vision Sensor

Abstract

Face recognition (FR) in an unconstrained environment, such as low light, illumination variations, and bad weather is very challenging and still needs intensive further study. Previously, numerous experiments on FR in an unconstrained environment have been assessed using Eigenface, Fisherface, and Local binary pattern histogram (LBPH) algorithms. The result indicates that LBPH FR is the optimal one compared to others due to its robustness in various lighting conditions. However, no specific experiment has been conducted to identify the best setting of four parameters of LBPH, radius, neighbors, grid, and the threshold value, for FR techniques in terms of accuracy and computation time. Additionally, the overall performance of LBPH in the unconstrained environments are usually underestimated. Therefore, in this work, an in-depth experiment is carried out to evaluate the four LBPH parameters using two face datasets: Lamar University data base (LUDB) and 5_celebrity dataset, and a novel Bilateral Median Convolution-Local binary pattern histogram (BMC-LBPH) method was proposed and examined in real-time in rainy weather using an unmanned aerial vehicle (UAV) incorporates with 4 vision sensors. The experimental results showed that the proposed BMC-LBPH FR techniques outperformed the traditional LBPH methods by achieving the accuracy of $65\%$, $98\%$, and $78\%$ in 5_celebrity dataset, LU dataset, and rainy weather, respectively. Ultimately, the proposed method provides a promising solution for facial recognition using UAV.

1. Introduction

Facial recognition (FR) has drawn more and more attention among researchers and practitioners over the last few decades due to its extensive use in many security fields. Nowadays, FR techniques are extensively used in public places such as airports, shopping malls, and stadiums due to security concerns. For example, FR systems have been used to identify the valid ticket holder in 2008 Olympic games in China [1] and; police identified unknown bodies by facial recognition techniques in California [2]. The existing FR techniques show promising results in favorable conditions (i.e., proper lighting, illumination) using the web camera and small data. However, the performance of FR techniques is not satisfactory in many cases due to the following limitations: large dataset led to high computation cost [3]; low light/illuminations caused low FR accuracy [1]; inadequate features of the face produced unstable FR models [4].

With the advent of deep learning techniques, deep learning-based FR has been widely used in many areas. However, the application of deep learning requires a massive amount of data to develop stable FR techniques. Thereby, general algorithms such as Eigenfaces, Fisherfaces, and Local binary pattern histogram (LBPH) are still extensively used to develop FR applications due to the overall suitable and simple performance [1,5,6,7]. Among them, LBPH showed robustness in unconstrained conditions like low light and illumination variations, so it has been more widely used than the other two algorithms (Fisherface and Eigenfaces) [8,9]. Will LBPH FR strategies perform appropriately in adverse environments (i.e., foggy, cloudy, rainy), as it performed in indoor environments? It is difficult to answer this question, since none of the previous research considered FR in bad climate. Therefore, this research aims to understand LBPH FR techniques’ performance comprehensively, focusing on its parameters (i.e., radius, neighborhood, cell) in an unconstrained environment. The overall experiment is carried out on two datasets–5_celebrity (obtained from Kaggle database repository) [10] and the Lamar University data base (LUDB) (contains the facial image of Lamar University students) [1].

Recently, drones are used extensively in agriculture, security, and entertainment industries. Most of the recent drones produce high-resolution images; such large resolutions images with deep learning approaches usually results in a complex and computationally expensive model. Considering those factors, we developed and tested novel algorithms named Bilateral Median Convolution-Local binary pattern histogram (BMC-LBPH) in this work to develop a fast FR system with limited data. The experiment was conducted under foggy, cloudy, and rainy weather. Based on the authors’ knowledge, this is the first research to demonstrate FR techniques incorporating with the drone techniques in unconstrained weather. The novelties of this research are summarized as follows:

• A systematic analysis of LBPH parameter;
• A novel BMC-LBPH methods; and
• Finally, a pilot test for real-time FR in rainy weather using DJI Phantom 4 UAV (DJI Phantom 4 is chosen due to its capability of sensing obstacle and highly capable camera sensor (i.e., effective pixels: 20 M, forward, backward and downward vision system) [11]) based on LBPH and BMC-LBPH FR technique.

Figure 1 represents some of the face images from the dataset used during the experiments.

The overall datasets used in this study is detailed in

Table 1. Dataset used during this study.

Dataset	Number of Participants	Number of Face/Subject	Total Images
5_celebrity	20	5	100
LUDB	10	10	100

.

1.1. Radius and Neighbor of Lbph

Radius and neighbor value are the two most important parameters in the LBPH algorithm. The Radius ranges from 1 to 5, with 1 as the default value [12]. Increasing the radius may help to capture the details on a large scale. On the other hand, the neighbor value should be a positive integer varying from 4 to 24 and has 8 as default value [12]. Ojala et al. (1994) [13] stated that image textures could be described as a $2^p$ binary, regarded as uniform patterns. More uniform patterns mean that the image contains more texture information. According to Ojala, Pietikäinen, and Harwood (1996) [14], increasing the radius and neighbor size could reduce the number of uniform patterns in the images, which later postulated by another study conducted by Ahonen, Hadid, and Pietikainen (2004) [15]. However, the effect of neighbor value and radius on time complexity regarding face recognition was not justified boldly in previous research.

Table 2. Different LBP radius and neighbor values uniform pattern in the image.

${\mathit{L}\mathit{B}\mathit{P}}_{\mathit{P}}{,}_{\mathit{R}}$	% of Uniform Pattern in the Image	Reference
$8,1$	$90\%$	Ojala et al. (2002) [16]
$8,1$	$90.6\%$	Ahonen et al. (2004) [15]
$8,2$	$85.2\%$	Ahonen et al. (2004) [17]
$16,2$	$70\%$	Ojala et al. (2002) [16]

presents the percent of uniform patterns in the image with different neighbor values and radius. The neighbor value and the radius increased while the rate of uniform pattern abated.

The previous studies did not show how neighborhood and radius affect FR accuracy and computational. Therefore, in this work, the effect of radius and neighborhood in unconstrained weather is statistically measured with computational time and accuracy.

1.2. Cell/Grid

Grid/Cell is another parameter of LBPH, in the face recognition method provided by OpenCV. An image contains several cells in horizontal and vertical directions, which can be denoted as grid x in the horizontal direction and grid y in the vertical direction. In Figure 2), the image was divided into multiple grids, horizontally and vertically. Each cell value was used to create the histogram of each region. The cell size was used to divide and normalize the histogram of the image. The histogram determines the binary distribution clustered over the image to produce the output features.

Figure 3 shows a 4-neighborhood LBP operator on a single image. The face region was divided into nine overlapping areas, and a 16-bin histogram was calculated for each area. Then the result was concatenated into a single 144-bin histogram. On the other hand, an $LB{P}_{8,2}^{u2}$ also applied to the whole $19\times 19$ size images to produce a 59-bin histogram. In total $203(59+144)$, bin histogram was used as a descriptor of the face region. Note that there was no specific boundary about horizontal and vertical cells on the images.

The number of the grid has no specific range, and none of the previous research accounted for these issues. As a result, it was noticed that different studies used unique grid values (i.e., $8\times 8$, $18\times 21$, and $24\times 24$) in their work [16,17]. For example, Ahonen et al. (2004) [17] used a length of $18\times 21$ grid with $LB{P}_{8,2}^{u2}$ and attained higher accuracy; Reference [16] applied a set of grid value such as $8\times 8$, $16\times 16$ and $24\times 24$ for texture classification.

However, some of the literature have optimized window length, radius, and neighbor value incorporate $8\times 8$ grid values to develop an optimized LBPH model. For instance, Urdal (2016) [19] proposed an LBPH based image processing techniques considering different radius and neighbor values; Mujdat et al. (2015) [20] experimented with varying sizes of the window of $3\times 3$, $5\times 5$, and $7\times 7$, and obtained $86.7\%$, $91.4\%$, and $94.09\%$ accuracy respectively; however, grid size was disregarded during the study, like it was absent in Reference [19]. Therefore, distinctive cell sizes, including $8\times 8$, $16\times 16$, and $24\times 24$, are tested (inspired by Reference [20]) along with various radius and neighbor value with the focus on LBPH FR techniques in this work.

1.3. Mathematical Formulation

Consider an image $I(x,y)$ and let $g_c=I(x,y)$ denote the gray level of the pixel $(x,y)$. Let $g_p$ denotes the gray value of a sample point of the evenly spaced circular neighborhood of P sampling points and R radius around $(x,y)$: $g_p=I(x_p,y_p)$ where $p=0$,...., $p-1$ and

$$x_p=x+Rcos\left(2\pi p/p\right)$$

(1)

$$y_p=y-Rsin\left(2\pi p/p\right).$$

(2)

Let the local texture of the image $I(x,y)$ characterized by the joint distribution of the gray value of $p+1(p>0)$ pixels:

$$T=t(g_c,g_0,g_1,....,g_{p-1}).$$

(3)

Without losing information, it is possible to subtract the central pixel value from the neighborhood:

$$T=t(g_c,g_0-g_c,g_1-g_c,\dots ,g_{p-1}-g_c).$$

(4)

In the next step, the joint distribution is approximated, and the central pixel is assumed to be independent, which allows the factorization of the distribution:

$$T=t\left(g_c\right)t(g_0-g_c,g_1-g_c,\dots ,g_{p-1}-g_c).$$

(5)

Here the first factorization $t\left(g_c\right)$ does not contain useful information since the distribution is over $I(x,y)$. Thus, we consider joint distribution differences: $t(g_0-g_c,g_1-g_c,....,g_{p-1}-g_c)$. However, collecting texture information from the multidimensional data is still difficult. Ojala, Pietikäinen, and Harwood (1996) [14] proposed a method that uses vector quantization. They used the codebook of 384 codewords to reduce the dimensionality [14]. Vector quantization is also difficult because the codebook must be trained similarly to the other texton-based methods. To get rid of this problem, the only sign of the difference is considered: $T(s(g_0-g_c),s(g_1-g_c),....,s(g_{p-1}-g_c))$, where $s\left(z\right)$ is the threshold function.

$$S\left(z\right)=\left\{\begin{array}{cc}1\hfill & \mathrm{if}z\ge 0\hfill \\ 0\hfill & \mathrm{if}z\le 0\hfill \end{array}\right.$$

(6)

The generic LBP can be obtained by summing the threshold differences weighted by powers of two. The $LB{P}_{P,R}$ operator, can be defined as

$$LB{P}_{P,R}(x_c,y_c)=\sum _{p=0}^{p-1}s(g_p-g_c)2^p.$$

(7)

From that equation, the texture can be described as:

$$T\backsim t\left(LB{P}_{P,R}(x_c,y_c)\right).$$

(8)

2. Methodology

Figure 4 summarizes the framework to evaluate different LBPH parameters in FR methods. There is no specific experiment that considers the four basic parameters of LBPH (radius, neighbor, grid x, and grid y) in FR methods regarding accuracy and execution time. This paper is the first detailed attempt considering the unconstrained situations.

Many of the earlier studies used LBPH parameters between the following range: radius = (1,5), neighbor = (4,24) [12,21]. However, other parameters grid_x and grid_y have no specific parameter range and most of the previous research used the grid value as 8 [12,21]. This study found that increasing the grid value did not increase the accuracy or reduce the execution time, nevertheless, grid value ranges from 8 to 24 was also considered to support the previous disclosure. Hence, experimenting with each of the four parameters of LBPH, will create a overall combination of 400,000 experiment. Thus, as an alternative approach, only following parameters was considered for the whole experiment: radius = (1 to 5), neighbors = (4,8), grid_x, grid_y = (8,8), (16,16), (24,24).

3. Experimental Results

Effect of LBPH Parameters

Table 3. Result comparison in Lamar University data base (LUDB) and 5_celebrity dataset.

Database	LBPH Parameters
	Radius ($\mathit{R}$)	Neighbor ($\mathit{N}$)	Cell $\mathit{x}$	Cell $\mathit{y}$	Accuracy		Execution Time (sec)
LUDB	1	4	24	24	Max	90%	–	0.36
	2	4	16	16	Max	90%	–	0.20
	1	8	24	24	Max	90%	Max	0.53
	1	8	16	16	Max	90%	–	0.29
	2	8	16	16	–	90%	–	0.27
	5	4	24	24	Min	70%	–	0.31
	1	8	24	24	–	–	Max	0.53
5_celebrity	2	8	24	24	–	32%	Max	4.17
	3	4	8	8	–	33%	Min	0.38
	5	4	8	8	–	34%	Min	0.68
	5	8	8	8	Max	42%	–	0.99
	2	4	16	16	Min	22%	–	0.98

summarizes the overall result for the maximum and minimum accuracy and computation time in both LU and 5_celebrity databases. In 5_celebrity, the maximum accuracy found for the following parameters: r = 5, n = 8, x = 8, y = 8, and the minimum execution time yielded for the following parameters, r = 3,5; n =4; x = 8; y = 8. In contrast, in the LU database, several parameters yielded the maximum accuracy; and the low computation time eventuated for parameters such as r = 5, n = 4, x = 8, and y = 8.

From the above experiment, it was found that there are no specific standard parameters with maximum accuracy with low computation in both databases (see Table 3). Moreover, it can be concluded that execution time will not be an issue if a user uses a fast GPU.

4. Proposed FR Framework

A novel LBPH based method, bilateral median convolution-Local binary pattern histogram (BMC-LBPH), was proposed to improve the FR accuracy in an unconstrained environment. The schematic representation was illustrated in Figure 5. Initially, the image is acquired and converted into a gray-scale scale image, which is time efficient and allows multiple-image manipulation process on the image. Any unnecessary object is removed from the image by selecting the face region and detecting the face using a cascade classifier. Then the simple slice method is used to crop the face for feature extraction. Once the face is cropped, the following filter is used for further image processing-firstly, a bilateral filter is applied to reduce the noise and illumination variation of face image [22,23,24]; secondly, the median blur is used to smoothen the overall image; finally, a convolution filter is used to produce robust feature vectors [25].

The overall experiment was conducted using all the images (100) from 5_celebrity dataset (contains 100 images), and 40 images (20 cloudy and 20 foggy) from LUDB (Other face images captured in rainy and sunny conditions were ignored since those are less challenging, and the existing LBPH and proposed BMC-LBPH showed almost $100\%$ accuracy on those scenarios.) dataset with 5-fold cross-validation; and the performance of proposed BMC-LBPH and existing LBPH were statistically calculated and compared using the following formulas [1]:

$$Accuracy=\frac{t_p+t_n}{t_p+t_n+f_p+f_n}$$

(9)

$$Precision=\frac{t_p}{t_p+f_p}$$

(10)

$$Recall=\frac{t_p}{t_n+f_p}.$$

(11)

$$F1=2\times \frac{Precision\times Recall}{Precision+Recall}$$

(12)

where,

• True Positive ($t_p$) = Face detected and matched with the true labels.
• False Positive ($f_p$) = Face detected but matched with the false labels.
• True Negative ($t_n$) = Not face and not matched with the labels.
• False Negative ($f_n$) = Face detected but not matched with the labels.

Experiment with BMC-LBPH Methods

Table 4. Result comparison between LBPH and bilateral median convolution-Local binary pattern histogram (BMC-LBPH) methods.

Dataset		Methods	Evaluation Matrix
			Precision	Recall	F1 Score
5_celebrity		LBPH	0.36	0.4	0.358
		BMC-LBPH	0.67	0.65	0.619
LUDB	Cloudy image	LBPH	0.898	0.85	0.838
		BMC-LBPH	1.0	1.0	1.0
	Foggy image	LBPH	0.978	0.966	0.964
		BMC-LBPH	0.966	0.95	0.946

presents the performance of existing LBPH and the proposed BMC-LBPH methods in 5_celebrity and LUDB dataset. The proposed methods (BMC-LBPH) outperformed existing LBPH methods in terms of precision and f1 scores in every scenario. However, the recall score for existing methods (LBPH) is slightly better in the LUDB dataset, especially on foggy images.

Figure 6a shows, the proposed BMC-LBPH outperformed existing LBPH FR techniques with higher accuracy on both datasets. However, on a foggy image, existing LBPH’s accuracy ($96.6\%$) is slightly higher than the proposed BMC-LBPH ($95\%$ accuracy). Similarly, Figure 6b revealed that the computation time of BMC-LBPH is efficient compared to traditional LBPH FR techniques both on 5_celebrity and LUDB.

5. Real-Time Face Recognition Using UAV

During the literature review, it was noticed that real-time FR systems in bad weather (i.e., foggy, cloudy, rainy) using UAV are either missing or ignored. Therefore, a pilot test was carried out with the initially available drone, DJI Phantom 4. A detailed specification of all necessary parts about the drone can be found in Reference [11]. Drone specifications are the following-DJI Phantom drone with 4 vision sensor; Intelligent Flight Battery—5350 mAh; Remote Controller; Operating Frequency—2.400–2.483 GHz; Gimbal Control range from −900 to +300; Camera—12.4 M pixel.

Figure 7 shows a flowchart of the proposed methodology for real-time FR using a DJI Phantom 4 drone in bad weather.

Initially, using GPS and DJI GO mobile app, the Phone was connected to the controller (Figure 8a); with the drone live streaming was induced by the YouTube platform(Figure 8b)-DJI GO app has an option for live streaming using either Youtube/Facebook. If the battery strip on the drone becomes green (Figure 8c), then it is expected that the drone is ready to fly and for the pilot test. Figure 8d, shows recognized frontal face in rainy weather by DJI phantom 4.

Test Analysis

The test was implemented in rainy weather with two subjects.

Table 5. LBPH and BMC-LBPH FR techniques performance in rainy weather.

Subject	Methods	Precision	Recall	F1 Score
1	Existing LBPH	0.61	1.0	0.75
	Proposed BMC-LBPH	0.71	1.0	0.83
2	Existing LBPH	0.61	1.0	0.756
	Proposed BMC-LBPH	0.77	1.0	0.83

presents the overall performance of existing LBPH and proposed BMC-LBPH in real-time FR under rainy weather. The overall precision and f1 score of subjects 1 and 2 for BMC-LBPH are higher than traditional LBPH FR techniques, yet the recall score is 1 for both algorithms.

Besides, the accuracy achieved by the proposed BMC-LBPH is higher than that of the existing LBPH FR techniques in real-time for both subjects (Figure 9). However, in real-time, the computation time for the proposed FR technique is equal to existing LBPH for subject one but slightly higher for subject two.

6. Discussion

In this work, a schematic examination was performed to evaluate the four LBPH parameters, including radius(r), neighborhood(n), grid x, and grid y. The experiment revealed that default parameters of LBPH (r = 1, n = 8, x = 8, y = 8) performed well both on 5_celebrity and LUDB dataset. Increasing radius and neighbor values resulted in higher calculation time and lowering those values, can not improve FR accuracy. Similarly, changing parameters of grid x and grid y can not improve FR accuracy or computation time at a satisfactory level. Thus, a novel LBPH based FR strategy, Bilateral Median Convolution-Local binary pattern histogram (BMC-LBPH), was proposed and experimented. The proposed method outperformed existing LBPH techniques on both datasets in terms of accuracy. However, in a foggy image, it performed slightly lower than traditional LBPH.

Moreover, a pilot test of a real-time FR in rainy weather was performed to assess the performance of the existing and proposed LBPH methods utilizing a DJI Phantom 4. The proposed LBPH outperformed the current LBPH strategies in real-time FR in unconstrained weather (rainy). The proposed methods increased the accuracy of FR up to $16\%$. Then again, the execution time for existing and recommended LBPH strategies are practically comparative. However, it is more likely to achieve higher accuracy using other strategies such as using convolutional neural network (CNN), deep CNN, and recurrent neural network (RNN) techniques, which are computationally expensive and require big data. Therefore, it could be inferred that the performance of the proposed BMC-LBPH technique is acceptable in real-time FR in bad weather concerning accuracy and execution time.

However, only two subjects were tested due to the time constraints, and the pilot test was conducted only in rainy weather. More subjects and varying weathers should be considered in the future study.

7. Conclusions

Real-time face recognition in rainy weather using DJI Phantom 4 drone was successfully implemented with the novel Bilateral Median Convolution-Local binary pattern histogram (BMC-LBPH) methods. The BMC-LBPH methods exceed the existing LBPH algorithm with increased accuracy up to $10\%$ by taking the low computation time. Four LBPH parameters (radius, neighbor, grid x, grid y) were also tested in an unconstrained environment. The results confirmed that the default parameters (radius = 1, neighbors = 8, grid x = 8, grid y = 8) are ideal for accuracy and execution time. The overall experiment was conducted using small datasets (5_celebirty consists of 100 images, and LUDB comprises 40 images); therefore, further improvement of FR’s accuracy is feasible using large scale data. Future works include but not limited to: face recognition in different unconstrained situations such as low light, face angle, and motions; testing the proposed BMC-LBPH methods on large scale data incorporates with various weather conditions such as foggy, cloudy, and sunny.