Robust Hand Gesture Recognition Using HOG-9ULBP Features and SVM Model

Abstract

Hand gesture recognition is an area of study that attempts to identify human gestures through mathematical algorithms, and can be used in several fields, such as communication between deaf-mute people, human–computer interaction, intelligent driving, and virtual reality. However, changes in scale and angle, as well as complex skin-like backgrounds, make gesture recognition quite challenging. In this paper, we propose a robust recognition approach for multi-scale as well as multi-angle hand gestures against complex backgrounds. First, hand gestures are segmented from complex backgrounds using the single Gaussian model and K-means algorithm. Then, the HOG feature and an improved 9ULBP feature are fused into the HOG-9ULBP feature, which is invariant in scale and rotation and enables accurate feature extraction. Finally, SVM is adopted to complete the hand gesture classification. Experimental results show that the proposed method achieves the highest accuracy of 99.01%, 97.50%, and 98.72% on the self-collected dataset, the NUS dataset, and the MU HandImages ASL dataset, respectively.

1. Introduction

Hand gesture-based human–computer interaction technologies are widely used in driving assistance, sports activity assistance, underwater rescue, firefighting assistance, etc. [1,2]. According to the World Health Organization, millions of people around the world suffer from hearing impairment [3]. Most of them can only communicate with others through sign language or gestures [4]. Gesture-based human–computer interaction can greatly improve the living quality of people suffering from hearing impairment [5,6]. Although many achievements have been made in gesture recognition, changes in scale, angle, and position, as well as complex backgrounds still adversely affect the recognition system [7,8,9].

The general gesture recognition procedure consists of pre-processing, feature extraction, and classification. Skin color segmentation is a critical step in the image pre-processing and greatly affects the feature extraction and gesture recognition. However, it is relatively difficult to segment gestures from complex backgrounds, especially under skin-like interference [10,11,12]. As to the feature extraction of multi-scale multi-angle (MSMA) gestures, there are two challenges which need to be addressed: (I) how to extract the most distinguished features from the gesture images; (II) how to increase robustness to the changes in the scale and rotation.

The widely used features in gesture recognition are Hu moments [13,14], Krawtchouk moments [15,16], histogram of oriented gradients (HOG) [17] and local binary patterns (LBP) [18,19]. Hu Moments are an image descriptor used to characterize the shape of an object in an image. Krawtchouk moments are a set of moments formed using Krawtchouk polynomials as the basis function set, and found to be robust to viewpoint changes and capable of achieving good recognition with a small number of training samples. HOG can fully capture the edge and contour information [20,21]. LBP can effectively describe the texture characteristics [22,23] and its improved feature can possess the property of rotation invariance [24]. Because many images have rich contour and texture information, some researchers combined HOG and LBP to increase the recognition precision, such as the detection of building appearance [25], brain tumor detection [26], the gesture recognition in Android devices [27] and pedestrian detection [28]. To increase the scale and/or rotation invariance of the recognition systems, ref. [15] used rotation normalization to align the gestures. Ref. [29] put forward a contour direction-based normalization method to increase the recognition rate of multi-scale gestures. Ref. [30] proposed a radiation projection transformation model to identify the MSMA static gestures. However, the proposed approach was based on the contour information and may cause misclassification when the boundary contours were similar. Ref. [31] proposed a descriptor incorporating three types of parameters of multi-scales to make a discriminative representation of the hand shape. Ref. [32] put forward an algorithm to detect the state of the fingers by the parallel feature of the finger contours. The proposed approach has rotation invariance, but it is vulnerable to the modeling accuracy.

Hence, there is an immediate need for the development of a robust hand gesture recognition method. In this paper, we propose an improved 9 uniform LBP feature and combine it with the HOG feature to realize the feature extraction of MSMA hand gestures, and then use support vector machine (SVM) to complete the feature classification. To verify the effectiveness of the proposed method, we construct a dataset via Kinect with 20,000 MSMA hand gesture images against different backgrounds. Experiments are then carried out on the self-made dataset, the NUS dataset and the MU HandImages ASL dataset. Experimental results on these datasets verify the effectiveness and superiority of our method. The main contributions of this paper can be summarized as follows:

• We use single Gaussian models (SGM) and the K-means algorithm to segment hand gestures against complex backgrounds. The SGM method can segment hand gestures from a general non-skin background. However, for the skin-like background, the skin color pixels and the skin color-like pixels cannot be correctly segmented. After noticing that there are three types of pixels in the images after SGM segmentation, i.e., the black background pixels, the skin color-like pixels and the skin color pixels, we further use the K-means algorithm to cluster these three kinds of pixels, to separate the skin color-like pixels and skin color pixels and achieve segmentation of hand gestures against the skin-like interference.
• We propose an improved 9ULBP descriptor and combine it with the HOG feature for feature extraction of MSMA hand gestures. The HOG feature is used to capture the contour information and maintain the scale invariance. The proposed 9ULBP descriptor can capture the texture information effectively and is rotationally invariant. So the combined HOG-9ULBP feature has not only the scale and rotation invariance, but also rich contour and texture information. SVM is then used to complete the feature classification.

2. Methods

The proposed hand gesture recognition method includes the hand gesture segmentation against complex backgrounds, the feature extraction by combing the HOG feature and an improved 9ULBP descriptor and the gesture classification via SVM. The framework of the proposed recognition method is shown in Figure 1.

2.1. Hand Gesture Segmentation

In this subsection, we classify the background in two different types: the skin-like background and the non-skin background. We use the SGM to segment hand gestures against the non-skin background and SGM-Kmeans method to segment hand gestures against the skin-like background.

Skin color is one of the most prominent surface features of the human body and can be applied to gesture segmentation by detecting skin color [33,34]. As shown in Figure 2, SGM can effectively segment gestures from non-skin backgrounds, but cannot eliminate the interference of skin-like backgrounds. By observing the images after SMG segmentation in Figure 2b, we find that there are three kinds of pixels, i.e., the skin color, the skin-like background and the black background. To segment the skin color hand gesture from the skin-like background, we need to cluster the three different kinds of pixels and eliminate the skin-like interferences. Experiments show that the K-means algorithm can effectively separate the three different kinds of pixels and the skin-like interference can be eliminated effectively, where K is chosen as 3, as shown in Figure 2c. The segmented gesture images are converted to grayscale images and scaled to $64\times 64$ pixels to reduce the data processing burden in feature extraction and gesture classification.

2.2. Feature Extraction

MSMA hand gestures are naturally used for communications. So the feature not only needs to capture the characteristics of the gestures, but also has the property of scale and rotation invariance. In this subsection, we combine the HOG feature and an improved 9ULBP descriptor to realize the feature extraction.

2.2.1. HOG Feature

The fundamental principle of the HOG feature is that the shape or contour of an object in an image can be described by the distribution of its intensity gradients and edge directions. Because of its robustness to the appearance and contour information of the human body, the HOG feature can effectively characterize the edges and contours of the human body and has achieved great success in gesture detection and recognition. Moreover, the HOG feature also possesses good geometric invariance and can be used for the feature description of multi-scale images.

The main steps to obtain the HOG feature descriptor are as follows:

• Calculate the horizontal gradient $g_x$ and vertical gradient $g_y$ of each pixel $(x,y)$, respectively:

  $$\begin{array}{c}\hfill g_x(x,y)=I(x+1,y)-I(x-1,y),\end{array}$$

  (1)

  $$\begin{array}{c}\hfill g_y(x,y)=I(x,y+1)-I(x,y-1),\end{array}$$

  (2)

  where $I(x,y)$ is the pixel value.
• Calculate the gradient magnitude $m(x,y)$ and orientation $\theta (x,y)$ as follows:

  $$\begin{array}{c}\hfill m(x,y)=\sqrt{g_x{(x,y)}^2+g_y{(x,y)}^2},\end{array}$$

  (3)

  $$\begin{array}{c}\hfill \theta (x,y)=\arctan \frac{g_y(x,y)}{g_x(x,y)}.\end{array}$$

  (4)
• Calculate the orientation histogram vector of each cell. We take the calculation of 9-dimensional gradient orientation histogram vector for instance. First, the image is divided into some specified blocks and cells. Then, the range $[0,\pi ]$ is equally divided into 9 bins. Let ${\theta }_0,{\theta }_1,{\theta }_2\dots ,{\theta }_9$ represent $0,\pi /9,2\pi /9,\dots ,\pi$, respectively. For each pixel $(x,y)$ in a cell, if ${\theta }_i\le \theta (x,y)<{\theta }_{i+1}(i\in 0,1,2,\dots ,8)$, let $B_i=B_i+m(x,y)$. The obtained $B_i$ is then $L_2$-norm normalized as follows:

  $$\begin{array}{c}\hfill B_i^{\prime }=\frac{B_i}{\sqrt{{∥B_i∥}_2^2+\epsilon }}\end{array}$$

  (5)

  where $\epsilon$ is a very small normalization constant. Finally, the 9-dimensional gradient orientation histogram vector for each cell can be formulated as

  $$\begin{array}{c}\hfill vecto{r}_{cell}=\{B_0^{\prime },B_1^{\prime },B_2^{\prime },\dots ,B_8^{\prime }\}.\end{array}$$

  (6)
• HOG feature calculation. The calculation procedure of the HOG feature is illustrated in Figure 3. First, the HOG feature of each block is obtained by sequentially connecting the gradient orientation histogram of all the cells in the block. Then, the HOG feature $F=(x_1,x_2,\cdots ,x_n)$ of the entire image is obtained using the 4-pixel stride overlap strategy.

To verify the scale invariance of the feature, we extract the HOG feature of 4 hand gestures with different scales, as shown in Figure 4. It can be seen that the histograms of different scale gestures possess high similarity, which demonstrates the scale invariance of the HOG feature.

2.2.2. 9ULBP Feature

It is known that in addition to the contour and shape information, the hand gestures have rich texture information. LBP can be used to describe local texture feature and is widely used in texture segmentation [22,35,36]. In the following, we first introduce the calculation procedure of the conventional LBP descriptor and then propose an improved 9ULBP descriptor to capture the texture information and improve the rotation invariance of the conventional LBP feature.

Figure 5 shows the calculation procedure of the LBP feature. First, the thresholds of the pixel values in the $3\times 3$ neighborhood are set according to the value of the center pixel. Next, the thresholds are concatenated into an 8-bit binary number to obtain the LBP code of the center pixel. Then, the LBP code is converted to its decimal format. Finally, the decimal values of the entire image are used to generate the histogram (LBP descriptor).

The LBP descriptor has 256 individual patterns. It is obvious that for hand gestures after preprocessing, the substantial amount of patterns are detrimental to the texture extraction and the description of texture features, because too many patterns will not only make the feature vector dimension too long, but also make the histogram too sparse. The sparse property of the LBP histogram distribution of hand gestures can be seen in Figure 5b. Therefore, it is of great importance to seek a texture analysis scheme suitable to the texture characteristics of segmented hand gestures. By observing that the uniform patterns histogram can provide better texture analysis than the histogram of individual patterns [37], we use the uniform pattern LBP descriptors in texture analysis of hand gestures.

The uniformity measure U needs to be introduced to define the uniform patterns, which corresponds to the number of the spatial transitions (bitwise 0/1 changes) in the pattern. A uniform pattern refers to a pattern with a limited transition $(U\le 2)$ in a cyclic binary and a uniform $LB{P}_P^R$ operator which has $P(P-1)+2$ different uniform LBP patterns, where R refers to the radius and P refers to the number of sampling points. When the sampling point P is 8 and the radius R is 1, the corresponding 58 uniform pattern (58ULBP) can gain the highest proportion in texture analysis [38]. So we take 58ULBP for further discussion. Figure 6a shows the 58ULBP codes, and Figure 7a shows the histogram of the 58ULBP image. It can be seen that for hand gesture recognition, the histogram distribution is sparse and nearly 20 patterns are not present, which indicates that the pattern is still excessive.

To further reduce the patterns, we introduce an improved 9ULBP based on the 58ULBP by limiting the uniformity measure $(U\le 1)$ in binary rather than in cyclic binary. The 9ULBP descriptor can be obtained as follows:

• Count the number of 1s in the 58ULBP code.
• Divide the LBP codes into 9 categories based on the number of 1s.
• Calculate decimal values of LBP codes in each class to find the smallest LBP code.
• Choose the smallest LBP code as the 9ULBP code.

By this way, we can transform the 58ULBP codes to 9ULBP codes, as shown in Figure 6b. The corresponding 9ULBP image and the histogram are shown in Figure 7b.

It can be seen that the histogram and pattern of 9ULBP can also capture the texture characteristics of the gesture image. Moreover, because the procedure of calculating the 9ULBP code can be viewed as rotating the 58ULBP code clockwise until it reaches the corresponding smallest decimal number in the same class, the improved 9ULBP descriptor has the property of rotation invariance. To verify it, we rotate a hand gesture image in five directions and then extract their 9ULBP histograms, respectively. The experimental results are shown in Figure 8. It can be seen that the 9ULBP histogram is not affected by the image rotation. Therefore, the proposed 9ULBP code can not only reduce the dimension of LBP descriptor, but also enhance its rotation invariance, which is of great importance for the feature extraction of multi-angle hand gestures.

2.2.3. Feature Fusion

The HOG feature has good scale invariance and can describe the contour of gestures well. The proposed 9ULBP feature has excellent rotaional invariance and can capture the texture of gestures. It is reasonable to fuse the HOG and 9ULBP features to obtain a new HOG-9ULBP feature for the gesture recognition.

HOG and 9ULBP with different sizes perform differently in capturing the outline and texture of hand gestures. If the block size and cell size are too large, the information of descriptors may be insufficient. Conversely, if the sizes are too small, the redundant information in the extracted descriptors may be excessive, or even lead to a dimensionality disaster. To obtain an optimal size, we compare the HOG and 9ULBP features with different block sizes and cell sizes as shown in Figure 9. In our experiments, we compare the HOG-9ULBP feature with different sizes to obtain the best fusion characteristics. The procedure for the feature fusion of HOG and 9ULBP is shown in Figure 10.

2.3. Gesture Classification

We use SVM to complete the gesture classification. SVM is a type of classifier with a supervisor, whose operation is based on finding optimal separation between different classes. Compared to other classifiers, SVM is based on the principle of structural risk minimization (SRM) to ensure that machine learning has a good generalization performance [39]. The principle of SVM is to transform the classification problem to a convex optimization problem [40]. Moreover, SVM uses the statistical theory to find the optimal solution and needs, relatively, a much smaller number of samples. During the experiments, we use the SVM based on radial basis function (RBF) due to its good nonlinear mapping capability. Ten cross-validation experiments are carried out to determine the SVM parameters and to obtain the best classifier model.

3. Results

In this section, we carry out comparative experiments to verify the effectiveness of the proposed MSMA hand gesture recognition algorithm. Experiments are performed on a Windows 10 system with a 3.40 GHz 8-core Intel Core i7-6700 processor and 8 GB of main memory.

3.1. Dataset Creation

Because currently the MSMA hand gesture datasets are few, we first use deep camera Kinect to create a dataset containing 20,000 MSMA gestures with complex backgrounds. The dataset contains 10 most widely used symbols of 20 people in different backgrounds, which are shown in Figure 11a. The procedure of the production of the homemade dataset is as follows:

• Use Kinect’s human skeleton recognition technology to track the center point of the right hand palm.
• Intercept the right hand gesture pictures at four different scales.
• Rotate the obtained gesture images in five directions (−90${}^{\circ }$, −45${}^{\circ }$, 0${}^{\circ }$, 45${}^{\circ }$, 90${}^{\circ }$).
• Organize these multi-scale and multi-angle gesture images into the dataset.

Figure 11b shows the scale and angle characteristic in the dataset. All the gestures are characterized by different scales and angles, which will increase the difficulty for accurate gesture recognition.

3.2. Experiment on Self-Collected Dataset

We first determine the parameters of HOG and 9ULBP descriptors via experiments. The performance of HOG and 9ULBP descriptors with different sizes are shown in

Table 1. Performance of HOG with different sizes.

Block	Cell	Dimension	Recognition Rate
(64, 64)	(32, 32)	36	36.39%
(64, 64)	(16, 16)	144	77.87%
(64, 64)	(8, 8)	576	90.02%
(32, 32)	(16, 16)	2916	89.41%
(32, 32)	(8, 8)	11,664	89.03%

and

Table 2. Performance of 9ULBP with different sizes.

Cell	9ULBP Dimension	Recognition Rate
(32, 32)	36	20.97%
(16, 16)	144	40.49%
(8, 8)	576	85.47%
(4, 4)	2304	85.22%

, respectively. We can see that using either the HOG feature or 9ULBP feature to characterize MSMA gestures can only achieve relatively low recognition rates, not exceeding $90.02\%$ and $85.47\%$, respectively. The recognition rate does not monotonously increase with respect to the descriptor dimension. The reason for this is that there is more redundant information in the high-dimensional descriptors and the redundant information can lead to performance degradation. In [41], it is shown that the performance of the fusion feature is much better than the single feature, but the parameters will affect the feature extraction performance. Next, we carry out experiments to seek the optimal feature fusion scheme.

Now we fuse the HOG and 9ULBP descriptors with different sizes into new HOG-9ULBP descriptors, which are fed to the SVM, and the experimental results are shown in

Table 3. Performance of HOG-9ULBP descriptors with different sizes.

HOG	9ULBP	HOG-9ULBP	Recognition Rate
2916	576	3492	97.46%
2916	144	3060	97.43%
2916	36	2952	97.40%
576	576	1152	99.01%
576	144	720	98.69%
576	36	612	94.50%
144	576	720	91.07%
144	144	288	89.91%
144	36	180	88.54%

. We can see that compared with HOG or 9ULBP features, the proposed HOG-9ULBP fusion feature can significantly improve the recognition rate. The reason is that the combined HOG and LBP feature can present both the contour and texture information, and possess the characteristic of scale invariance and rotation invariance. We can also see that under different dimensions, the recognition rate is different. The system can achieve optimal classification performance when combining the 576-dimensional HOG feature and 576-dimensional LBP descriptor.

Finally, we compare the proposed method with the optimal parameters with the other gesture recognition methods, i.e., LBP feature and SVM [8], Krawtchouk polynomials and minimum distance (KMD) classifier [15], HOG feature and SVM [17], contour feature and SVM [29], fast region-based convolutional neural networks (R-CNN) [42] and Hu moment feature and SVM [43]. The experimental results on the self-collected dataset are shown in

Table 4. Comparison results on self-collected dataset.

Recognition Method	Recognition Rate
The proposed method	99.01%
Fast R-CNN [42]	97.15%
KMD Classifier [15]	96.94%
Contour feature and SVM [29]	95.00%
Hu moment feature and SVM [43]	91.80%
HOG feature and SVM [17]	90.02%
LBP feature and SVM [8]	85.47%

. We can see that the proposed method has the highest recognition precision of 99.01%.

3.3. Experiment on the NUS Dataset

The NUS dataset [44] consists of 10 classes of gestures with 24 sample images per class as shown in Figure 12a, which are constructed by varying the position and size of the hand within the image frame. These gestures have less inter-class variations in appearance, which will undoubtedly increase the difficulty of recognition.

To further verify the effectiveness of the proposed MSMA gesture recognition approach on the NUS dataset, we rotate each gesture image in five directions, as shown in Figure 12b. During the experiments, we divide the dataset into two parts: a training set with 600 images and a test set with 600 images. The 10 cross-validation method was used to obtain the optimal SVM parameters. The comparative experimental results on the NUS dataset are shown in

Table 5. Comparison results on the NUS dataset.

Recognition Method	Recognition Rate
The proposed method	97.50%
Fast R-CNN [42]	95.32%
Color-texture feature and neural network [45]	95.00%
HOG feature and SVM [8]	94.50%
Shape, texture and color feature [46]	94.36%

. It can be seen that the proposed method has the highest recognition rate of 97.5%.

3.4. Experiment on the MU HandImages ASL Dataset

The MU HandImages ASL dataset [47] contains 2425 images of five individuals for each of the 36 American sign language gestures. The 36 gestures consist of 26 letters and 10 digits, which are shown in Figure 13a. Moreover, as shown in Figure 13b, within the dataset some gestures are highly similar and will definitely increase the recognition difficulties.

To further demonstrate the performance of the proposed MSMA gestures recognition algorithm, we rotate each image in five directions, as shown in Figure 13c and obtain an extended dataset with 12,125 images. We divide the dataset into two parts, i.e., a training set with 6062 images and a test set with 6063 images. The 10 cross-validation method was used to obtain the optimal SVM model. We can see from the experimental results (

Table 6. Comparison results on the MU HandImages ASL dataset.

Recognition Method	Recognition Rate
The proposed method	98.72%
NMF and CS [48]	98.13%
Fast R-CNN [42]	97.78%
Zernike and nearest neighbor classifier [49]	97.00%
DSAE 5 layers and Softmax [50]	83.36%

) that compared to other methods, the proposed method has the highest recognition rate of 98.72%.

4. Conclusions

We propose a novel hand gesture recognition method that can effectively recognize MSMA gestures against complex backgrounds. First, the SGM-Kmeans method is put forward to segment gestures from complex backgrounds. Then, the HOG feature and an improved 9ULBP feature are combined to obtain a novel gesture feature. The proposed HOG-9ULBP feature not only has the rich contour information and texture information, but also the scale invariance and rotation invariance. Comparative experimental results show that the proposed gesture recognition method can achieve a higher recognition rate. In the future, we will add in-depth information to tackle MSMA gesture recognition problems in space and will investigate the dynamic gesture recognition based on depth and RGB information.