Dynamic Hand Gesture Recognition for Smart Lifecare Routines via K-Ary Tree Hashing Classifier

Featured Application

The proposed system is an image processing module that monitors, tracks, and recognizes hand gestures and has been evaluated over publicly available benchmark datasets. However, this technique can be used over automated home appliances as well as security systems to control surrounding environments and classify their events.

Abstract

In the past few years, home appliances have been influenced by the latest technologies and changes in consumer trends. One of the most desired gadgets of this time is a universal remote control for gestures. Hand gestures are the best way to control home appliances. This paper presents a novel method of recognizing hand gestures for smart home appliances using imaging sensors. The proposed model is divided into six steps. First, preprocessing is done to de-noise the video frames and resize each frame to a specific dimension. Second, the hand is detected using a single shot detector-based convolution neural network (SSD-CNN) model. Third, landmarks are localized on the hand using the skeleton method. Fourth, features are extracted based on point-based trajectories, frame differencing, orientation histograms, and 3D point clouds. Fifth, features are optimized using fuzzy logic, and last, the H-Hash classifier is used for the classification of hand gestures. The system is tested on two benchmark datasets, namely, the IPN hand dataset and Jester dataset. The recognition accuracy on the IPN hand dataset is 88.46% and on Jester datasets is 87.69%. Users can control their smart home appliances, such as television, radio, air conditioner, and vacuum cleaner, using the proposed system.

1. Introduction

Over the past few years, intelligent human-computer interaction recognition in the smart home environment is getting more attention in many fields, including architecture, robotics, biomedical, and smart home appliances [1,2,3]. The easiest way of controlling smart home appliances is through hand gestures [4,5,6]. The appliances that you use every day are an important part of your home. Consumers today are very careful about their comfort and safety and are more interesting in smart appliances [7,8,9]. They will facilitate their daily life by controlling the lights, speakers, air conditioners, and similar robots through hand gestures [10,11,12]. By making gestures, you can control all the devices in your home. A motion sensor is one of the most important tools in your smart home [13,14,15].

Different models proposed for controlling smart home appliances using hand gesture recognition can be divided into two main streams. The first approach is based on the recognition of hand gestures using motion sensors embedded in smart home appliances [16,17,18]. In motion-based sensors, one inertial sensor or an array of sensors is used. These sensors are responsible for tracking the acceleration, velocity, and position of the hand. Such type of motion features help to control smart appliances such as television, radio, and lighting of rooms [19,20,21], however, the drawback of using motion-based sensors in smart home appliances is the high sensitivity. The second approach is the use of image sensors [22,23,24] or cameras to obtain the commands from hand gestures [25]; the sensors are trained on image features which include color, shape, texture, position contours, and motion of the hands. Our proposed model is based on the second approach and recognizes the hand gestures using imagery sensors or cameras [26].

In this research article, we propose a robust method for recognizing hand gestures for controlling smart home appliances. For this, we use the IPN hand dataset and Jester dataset. Initially, preprocessing of the video samples for frame conversion, motion blur noise reduction, and resizing is performed. The next step is hand detection via SSD-CNN. After that, the hand skeleton is extracted to process these data sources by various algorithms for features extraction, optimization of the extracted features, and recognition of the hand gestures. The main contributions of this paper are:

• For hand motion and position analysis, we propose a method for extracting hand skeletons;
• For the recognition of image-based hand gestures, we have extracted novel features based on point-based trajectories, frame differencing, orientation histogram, and 3D point clouds.

The article is subdivided as follows: we start with the related works section, which is followed by our system methodology. Then, the detailed experimental setup is discussed, and finally, an overview of the paper is presented in the Conclusions Section.

2. Related Work

Hand gesture recognition can help computers translate and interpret specific motions to control smart home appliances. With the advancement of technology, various hand gesture recognition systems have been developed via smart tools and various classification approaches. In this section, we will discuss the detailed description of various HGR models developed in the past few years.

Table 1. A comprehensive review of relevant research.

Hand Gestures Recognition for Controlling Smart Home Appliances
Methods	Main Contributions
H. Khanh et al. (2019) [24]	The system was developed for controlling smart home appliances using two deep learning models fused with mobile sensors to recognize hand gestures. The mobile sensors were instrumented on smartwatches, smartphones, and smart appliances. The deep learning models helped in the learning and representation of the mobile sensors’ data.
Ransalu et al. (2012) [25]	The HGR model was developed to automate the home appliances using hand gestures. First, the hand was detected using the Viola-Jones object detection algorithm. To segment the hand from the image, the YCbCr skin color segmentation technique was used. The filtered hand was refined using dilation and eroding. At last, a multilayer perceptron was used to classify the four-hand gestures, i.e., ready, swing, on/speedup & off.
P. N. et al. (2017) [26]	The model consists of a few steps. The hand gestures were captured using the web camera and were then preprocessed to detect the hands. Corner point detection was used to de-noise the images using a MATLAB simulation tool. Based on the gestures, different threshold values were used for controlling the home appliances. The threshold values were generated using the Fast Fourier transform algorithm and the appliances were controlled by the micro-controllers.
V. Utpal et al. (2011) [27]	For controlling home appliances and electronic gadgets using hand gestures, the authors detected hands using the YCbCr skin color segmentation model and traced edges. For gesture recognition, the number of fingers was counted, and its orientation was analyzed. The reference background was stored from each frame which was compared with the next frame for reliable hand gesture recognition.
Santhana et al. (2020) [28]	They developed a hand gesture recognition system using Leap motion sensors. The system was customized to recognize multiple motion-based hand gestures for controlling smart home appliances. The system was trained using a customized dataset containing various hand gestures to control daily household devices using a deep neural network.
Qi et al. (2013) [29]	They developed a hand recognition system for controlling television. The system was categorized into three sub-categories. (1) For static hand gesture recognition, hand features were extracted using a histogram-oriented gradient (HOG), and for recognition, Adaboost training was used. (2) For dynamic hand gesture recognition, first the hand trajectory was recognized and passed through the HMM model for recognition. (3) For finger click recognition, a specific depth threshold was fixed to detect the fingers. The distance between the palm and the fingertip was calculated. The accumulated variance was calculated for each fingertip to recognize the finger click gesture.
Yueh et al. (2018) [30]	The authors developed a system to control a TV using hand gestures. First, the hand was detected through skin segmentation and the hand contour was extracted. After that, the system was trained using CNN to recognize hand gestures that were categorized into five branches; (1) menu, (2) direction, (3) go back, (4) mute/unmute, and (5) nothing. After that, CNN also helped in tracking the hand joints to detect commands: (1) increase/ decrease the speed, (2) clicking, and (3) cursor movement.

includes a comprehensive review of recent research in this area.

3. Material and Methods

In this section, a detailed description of the proposed model is given. First, the video input is converted into RGB frames, then the frames are resized to a fixed dimension. Noise is removed and the quality of images is enhanced and sharpened. The second step is hand detection by first removing the background and extracting the foreground. A hand skeleton is extracted for localizing the points on the entire hand. Then the point-based and texture-based features are extracted. These features are optimized by using an optimization algorithm. At last, a classification algorithm is used for the classification of hand gestures for controlling smart home appliances. Figure 1 illustrates the proposed HGR system model structural design.

3.1. Preprocessing of the Input Videos

Before the localization of the hand points, some preprocessing techniques are applied to save computational cost and time. Initially, video data is converted into frames (images). These frames are set to a fixed dimension of 452 × 357. After that, the frames are denoised using the median filtering algorithm. The median filtering is used for detecting the distorted pixels in the images and replacing the corrupted pixels values with the median values. A 5 × 5 window is used to de-noise the image [31,32,33,34,35,36,37]. The median filter is defined in Equations (1)–(3);

$$Med \left(I\right)=Med\{I_p)$$

(1)

$$=\frac{I_p\left(k+1\right)}{2};k is odd$$

(2)

$$=\frac{1}{2}\left[I_p\left(\frac{k}{2}\right)+I_p\left(\frac{k}{2}\right)+1\right],$$

(3)

where I₁, I₂, I₃,…, I_k is the sequence of neighboring pixels. Before applying the filter, all pixels of the images should be arranged in ascending or descending order. After sorting the pixels, the sequence of the pixels will be $I_{p1}<I_{p2}<I_{p3}<I_{pk}$, where k is usually odd. Figure 2 shows the results of preprocessing on video frames.

3.2. Hand Detection Using Single Shot MultiBox Detector

The ridge detection of the human silhouette comprises two steps that are binary edge extraction and ridge data generation [38,39,40]. In the binary edge extraction step, the binary edges are extracted from the RGB and depth silhouettes are obtained in the described preprocessing stage. The distance maps are produced on the edges by using distance transform (See Figure 3). While, in the ridge data generation step, the local maximal is obtained from the pre-computed maps, which produces ridge data within the binary edges [41,42]. A further description of binary edge detection and generation of ridge data is described below in Algorithm 1:

Algorithm 1: Hand detection using a single shot multibox detector.

Input:        Optimized feature vectors. Output:     Hand gesture classification. Step 1:       Check the length of the hash table (say n).           Check the number of entries in the hash table by setting a fixed threshold           (say T = 40). If (n > T) then Find the correlation matching or minimum distance between the vectors by the following equation: $${\gamma }_K={\displaystyle \sum }_{x=1}^n\left|\left|{\alpha }_x\right|-\left|\beta \right|\right| where K=1,2,3,4$$ where $\alpha$ represents the centroid of the vectors stored in the hash table, β represents the new vectors of the test image, $\gamma$ represents the distance between the stored values of the hash table and the new vectors.  Now, finding the sum of the vectors $$\theta ={\gamma }_1+{\gamma }_2+{\gamma }_3+{\gamma }_4$$ End Step 2: /*Check the correlation ${\gamma }_K$ of the new entry*/    If $({\gamma }_K\ge 0.98$)    Match exists    Else    Match does not exist End

3.3. Hand Landmarks Localization Using Skeleton Method

To localize the hand landmarks points, the first step is to localize the palm region. For doing this, we selected the palm area via a single shot multibox detector and removed the fingers; with this help, a bounding box appears on the palm. Then the extreme top-left, top-right, bottom-left and bottom-right points are calculated and marked with the 4 points [43,44,45,46].

The next step is to localize the finger points. For this, the palm region is removed and only the fingers are left. The extreme points are identified with the help of a scanning window that moves from top to bottom, identifying all the extreme points. As a result, the extreme top points of all fingers are marked with the 5 points. Similarly, the extreme bottom points are identified using the scanning window that moves from bottom to top marking 5 points on the bottom of the fingers [47]. Figure 4 shows the results of hand point localization.

3.4. Features Extraction

For features extraction, we have extracted both points-based and appearance-based features for better classification of the hand gestures. For point-based features, we have used Bezier curves and frame differencing. For appearance-based features, 3D point clouds are mapped on the hands.

3.4.1. Bezier Curves

The landmark points localized on the entire hand are utilized for Bezier curves fitting for analyzing the trajectories of the hand in different gestures. For this, we have taken three control points to represent a curve using Equation (4);

$$Curve\left(x\right)={{\displaystyle \sum }}_{i=0}^n{Q}_{i,n}\left(x\right)Y_i$$

(4)

The points along the curve are determined by x, where 0 $\le x\le 1$. The degree of the curve is denoted by n, which is one less than the control points. Y_i is the i-th control point where Y(0) = Curve(0) and Y(n) = Curve(1). $Q_{i,n}$ is the Bernstein polynomial and is calculated in Equation (5).

$$Q_{i,n}=\frac{n!}{i!\left(n-i\right)!}{n}^i{\left(1-x\right)}^{n-i}$$

(5)

We have used Bezier curves with three control points describing the quadratic curve. Therefore, the Bernstein polynomial with n = 2 is calculated as in Equations (6)–(8) [48,49,50,51].

$$Q_{0,2}={\left(1-x\right)}^2$$

(6)

$$Q_{1,2}=2x\left(1-x\right)$$

(7)

$$Q_{2,2}=x^2$$

(8)

Therefore, the equation of a Bezier curve with three control points is simplified as in Equation (9). Figure 5 shows the results of the Bezier curves fitting on the hand.

$$Curve\left(x\right)={\left(1-x\right)}^2+2x\left(1-x\right)+x^2$$

(9)

3.4.2. Frame Differencing

Keyframes are the representation of the elements in the image sequences. In this model, the keyframes have been extracted that exhibit the dynamic hand gestures. Each hand shows a gesture that is localized by a set of points. To find the difference in the positions of the landmarks, the first and the pause frame sequences are taken as the keyframes. Figure 6 illustrates the change in the position of the pixel between different frames. The first frame is well established. To find the motion of the hand gestures, we have adopted the following method of frame difference as defined in Equation (10) [52,53,54,55,56].

$$Dif{f}_k\left(x,y\right)=\left|Fram{e}_k\left(x,y\right)-Fram{e}_{k-1}\left(x,y\right)\right|$$

(10)

where $Fram{e}_k\left(x,y\right)$ and $Fram{e}_{k-1}\left(x,y\right)$ are the two consecutive frames in which the hand is not moving and $Dif{f}_k$ is defined as in Equation (11).

$$Dif{f}_k\left(x,y\right)=\left|Fram{e}_k\left(x,y\right)-Fram{e}_{k-1}\left(x,y\right)\right|\approx 0$$

(11)

where $Fram{e}_k\left(x,y\right)$ is the pause keyframe. The two continuous frames are impossible to be the keyframes. Thus, if $Fram{e}_{k-1}\left(x,y\right)$ is the keyframe, then $Fram{e}_k\left(x,y\right)$ is not the keyframe. Therefore, for each frame sequence, $Fram{e}_k\left(x,y\right)$ with N number of frames: The following approach should be outlined as:

• Initialize the keyframe number with n = 1. So, the keyframes are marked as $M_1\left(x\right)=1, M_k\left(x\right)=0, k=\left(2,\dots , N\right).$ After that, compute the difference frame $Dif{f}_k$ between $Fram{e}_k\left(x,y\right)$ and $Fram{e}_{k-1}\left(x,y\right)$, $k=\left(2,\dots , N\right)$. Compute the valid pixels of N.
• If $N>Thres{h}_1$ and $M_{k-1}\left(x\right)=0,$ set n = n + 1, set $M_k\left(x\right)=1$.
• Set k = k + 1 if the value of k is less than N. Then repeat the steps, otherwise end the procedure.
• After calculating the frame difference, each key point L in the first keyframe and L^′ points in the other keyframes positions are calculated using the distance formula defined in Equation (12).

  $$Distance=\sqrt{{\left(L{1}^{\prime }-L1\right)}^2+\left(L{2}^{\prime }-L2\right)}$$

  (12)

D Point Clouds

For appearance-based features, extracted the 3D point clouds. The following are the main steps of extracting the appearance-based features [57,58,59].

• First, a central point on the palm is taken and the maximum distance d between the central point and the edges E_i point of the gesture region is calculated. After that, ten different lengths of the radius are defined as $X=n\times \frac{d}{10} where n=1,2,3,\dots 10$. Next, the center of the rhombus is defined as C and the radiuses as r_n. We have drawn 10 rhombuses (innermost is the first rhombus and outmost is the tenth rhombus) as shown in Figure 7. To highlight the effect of changing hand gestures, the color of the rhombus changes on the hand.
• In Figure 7, it is visible that every rhombus has a different number of intersections with hand gesture regions. For finding the number of stretched fingers S, we have taken the sixth rhombus (according to the thumb rule). In the sixth rhombus, we extracted those points whose colors vary from green to yellow and yellow to green. We define G_i as the point whose color changes from green to yellow and Y_i as the point whose values changes from yellow to green.
• Now, for midpoint identification, we define it as M_i which is the midpoint of G_i and Y_i. Then, each midpoint M_i and the central point C can be connected through a line and the angles between the adjacent lines are calculated. The angles are represented as An_j (j = 1, 2, 3…I − 1).
• Using the thumb rule, the fifth rhombus is taken as a boundary line to divide the hand gesture into two parts. For instance, we have taken the first part as P1 and the second part as P2, where P1 lies inside the rhombus and P2 is the outside area of the rhombus. Then the ratio R of P1 and P2 is calculated. The R is the gesture region area distribution feature as shown in Algorithm 2. Figure 7 shows the appearance features using 3D point clouds.

Algorithm 2: Feature extraction.

Input: Hands Point based and texture-based data (x, y, z). Output: Feature Vectors (v1, v2, …, vn).     featureVectors $\leftarrow [ ]$     window_size $\leftarrow \mathrm{Get} \mathrm{windowSize}( )$     $\mathrm{Overlap}\leftarrow \mathrm{Get} \mathrm{Overlapping} \mathrm{Time}( )$ For HandComponent in [x,y,z] do     Hand$\leftarrow$window$\leftarrow \mathrm{Getwindow}\left(\mathrm{HandComponent}\right)$   /* Extracting features */   BezierCurves $\leftarrow$ ExtractBezierCurvesFeatures (Hand_window)   Frame Differencing $\leftarrow$ ExtractFrameDifferencingFeatures (Hand_window)   3D Point Clouds $\leftarrow$ Extract3DPointCloudsFeatures (Hand_window)   featureVectors $\leftarrow$ GetFeatureVectors [BezierCurves, Frame Differencing, 3D Point Clouds]   featureVectors.append (featureVectors)   End for featureVectors $\leftarrow$ Normalize (featureVectors)   return featureVectors

3.5. Feature FOptimization Using Fuzzy Logic

The objective of using fuzzy logic optimizer is to recognize the hand gestures based on the information obtained through different feature descriptors. Each feature descriptor value is labeled with a specific variable and is mapped to their respective fuzzy sets. For instance, we have five fingers labeled as; F1 (thumb), F2 (index finger), F3 (middle finger), F4 (ring finger), and F5 (little finger). The joints of the fingers are labeled as J1, J2, J3, and J4. Similarly, the distance between the fingers is denoted by Di,j showing the distance between the fingers Fi and Fj.

Since any movement of the hand shows the variation of the position of the hand in a sequence of images (frames), to simulate the data transfer, a hand configuration is generated by a tuple of angles. For each tuple of angles, the data is represented using a set of linguistic variables such as curve, straight, and bent. The separation of the fingers is represented as open, closed, crossed, and semi-open. By these notations, the set of features is optimized, helping to reduce the overall computational time and complexity [60,61,62,63]. Figure 8 shows the result of the fuzzy logic optimization.

3.6. Hand Gestures Recognition

For hand gesture recognition, K-Ary Tree Hashing (KATH) classifier is used for the first time in our proposed model [64,65,66,67]. The KATH classifier takes the feature descriptors values of each image corresponding to the hand gesture and projects in a common space without the subtree pattern prior knowledge. Then similar pattern feature descriptors are kept in the traversal table. The unique patterns are specified by passing through recursive indexing N numbers to generate (n − 1). After that, the hand gesture is classified by the sub-patterns created by MinHash. The experimental results show that KATH classified different hand gestures more accurately than many other state-of-the-art methods, i.e., ANN and decision tree, as shown in Figure 9. In our proposed model, the graph g = (v, ɛ, l) is given input with the number of iterations I and F representing the feature space. To assign a new label l, the nodes v are relabeled considering the neighboring nodes WV. The traversal table T is generated and stored. After the traversal table, MinHash classifies data. For dimensional reduction, PCA is used to plot results in 3D feature space.

4. Experimental Setting and Results

4.1. Datasets Descriptions

The IPN hand dataset [68] is a large-scale hand gesture video dataset. It contains 13 gestures, including pointing with one finger, pointing with two fingers, click with one finger, click with two fingers, throw up, throw down, throw left, throw right, open twice, double click with one finger, click with two fingers, zoom in and zoom out. The IPN dataset contains RGB videos with a resolution of 640 × 480 at 30 fps. Figure 10 shows the example images of the IPN hand dataset.

The Jester dataset [69] contains a large collection of labeled hand gestures video clips collected by webcam. The dataset contains 148,092 videos, and each video frame is converted into a jpg image at the rate of 12 frames per second. There are 27 classes of hand gestures named: swiping down, swiping left, swiping right, swiping up, thumb down, thumb up, zooming in with full hand, zooming out with full hand, stop, and so on. Figure 11 shows the example images of the Jester dataset.

4.2. Performance Parameters and Evaluations

4.2.1. Experiment I: The Hand Detection Accuracies

In this experiment, the hand detection accuracies on different hand gestures over IPN Hand dataset and Jester dataset are shown in

Table 2. Hand detection accuracies over IPN Hand dataset.

Hand Gestures	Number of Samples	Plain Background	Accuracy (%)	Cluttered Background	Accuracy (%)
POF	30	30	100	25	83.3
PTF	30	30	100	26	86.6
COF	30	30	100	26	86.6
CTF	30	28	93.3	26	86.6
TU	30	29	96.6	29	96.6
TD	30	29	96.6	30	100
TL	30	27	90	30	100
TR	30	28	93.3	30	100
OT	30	29	96.6	30	100
DCOF	30	30	100	29	96.6
DCTF	30	30	100	29	96.6
ZI	30	30	100	28	93.3
ZO	30	29	96.6	30	100
Mean Accuracy Rate			97.1%		94.3%

POF = pointing with one finger, PTF = pointing with two fingers, COF = click with one finger, CTF = click with two fingers, TU = throw up, TD = throw down, TL = throw left, TR = throw right, OT = open twice, DCOF = double click with one finger, DCTF = double click with two fingers, ZI = zoom in, ZO = zoom out.

and

Table 3. Hand detection accuracies over Jester Hand dataset.

Hand Gestures	Number of Samples	Plain Background	Accuracy (%)	Cluttered Background	Accuracy (%)
SD	30	30	100	25	83.3
SL	30	29	96.6	27	90
SR	30	28	93.3	24	80
SU	30	28	93.3	26	86.6
TD	30	29	96.6	24	80
TU	30	28	93.3	30	100
ZIF	30	29	96.6	25	83.3
ZOF	30	27	90	30	100
S	30	29	96.6	25	83.3
RF	30	30	100	29	96.6
RB	30	30	100	23	76.6
PI	30	28	93.3	28	93.3
SH	30	29	96.6	30	100
Mean Accuracy Rate			95.6%		88.6%

SD = swiping down, SL = swiping left, SR = swiping right, SU = swiping up, TD = thumb down, TU = thumb up, ZIF = zooming in with full hand, ZOF = zooming out with full hand, S = stop, RF=rolling hand forward, RB = rolling hand backward, PI = pulling hand in, SH = shaking hand.

, respectively. Table 2 represents the results of the hand gestures of the IPN Hand dataset on both plain and complex backgrounds. We took 30 samples of each hand gesture in the plain and complex background and obtained 97.1% accuracy on plain background samples and 94.3% accuracy on complex background samples.

Table 3 represents the results of the 13 hand gestures of the Jester dataset on both plain and complex backgrounds. We took 30 samples of each hand gesture in the plain and complex background and obtained 95.6% accuracy on plain background samples and 88.6% accuracy on complex background samples.

4.2.2. Experiment II: Hand Gestures Classification Accuracies

For hand gestures classification, we used a KATH classifier. The design method was evaluated using the leave one subject out (LOSO) cross-validation method. In

Table 4. Confusion Matrix results over IPN Hand dataset.

Class	POF	PTF	COF	CTF	TU	TD	TL	TR	OT	DCOF	DCTF	ZI	ZO
POF	8	1	0	1	0	0	0	0	0	0	0	0	0
PTF	0	9	0	0	0	0	1	0	0	0	0	0	0
COF	0	0	10	0	0	0	0	0	0	0	0	0	0
CTF	0	1	0	9	0	0	0	0	0	0	0	0	0
TU	1	0	0	0	9	0	0	0	0	0	0	0	0
TD	0	0	1	0	0	8	0	0	0	1	0	0	0
TL	0	2	0	0	0	0	7	0	0	0	0	1	0
TR	1	0	0	0	0	0	0	8	0	0	0	1	0
OT	0	0	0	0	0	0	0	0	9	0	1	0	0
DCOF	0	0	0	0	0	0	0	0	0	10	0	0	0
DCTF	0	0	0	0	0	0	0	0	0	0	9	0	1
ZI	0	0	0	0	0	0	0	0	0	0	0	10	0
ZO	0	0	0	0	0	0	0	0	0	0	0	1	9
Hand Gestures classification mean accuracy = 88.46%

POF = pointing with one finger, PTF = pointing with two fingers, COF = click with one finger, CTF = click with two fingers, TU = throw up, TD = throw down, TL = throw left, TR = throw right, OT = open twice, DCOF = double click with one finger, DCTF = double click with two fingers, ZI = zoom in, ZO= zoom out.

, the results over the IPN hand video dataset show 88.46% hand gestures classification accuracy.

Table 5. Confusion Matrix results over Jester dataset.

Gestures	SD	SL	SR	SU	TD	TU	ZIF	ZOF	S	RF	RB	PI	SH
SD	9	0	0	0	1	0	0	0	0	0	0	0	0
SL	0	9	0	0	0	0	0	0	1	0	0	0	0
SR	0	0	9	0	1	0	0	0	0	0	0	0	0
SU	0	0	1	8	1	0	0	0	0	0	0	0	0
TD	0	0	0	0	10	0	0	0	0	0	0	0	0
TU	0	0	0	0	1	8	0	0	1	0	0	0	0
ZIF	0	0	0	0	0	0	9	0	0	1	0	0	0
ZOF	0	0	0	0	1	0	0	9	0	0	0	0	0
S	0	1	0	0	0	0	0	0	8	0	0	1	0
RF	0	0	0	0	1	0	0	0	0	9	0	0	0
RB	0	0	0	0	0	0	0	0	0	0	10	0	0
PI	0	0	0	0	0	0	0	0	0	0	1	9	0
SH	0	0	0	0	0	0	1	0	0	0	1	1	7
Hand Gestures classification mean accuracy = 87.69%

SD = swiping down, SL = swiping left, SR = swiping right, SU = swiping up, TD = thumb down, TU = thumb up, ZIF = zooming in with full hand, ZOF = zooming out with full hand, S = stop, RF = rolling hand forward, RB = rolling hand backward, PI = pulling hand in, SH = shaking hand.

represents the confusion matrix for the Jester dataset with 87.69% mean accuracy for hand gestures classification.

4.2.3. Experiment III: Comparison with Other Classification Algorithms

In this segment, we compared the recall, precision, and F1-measure over the IPN hand dataset and the Jester dataset. For the classification of hand gestures, we used a decision tree, an artificial neural network, and we associated the consequences with the KATH classifier. Figure 12 shows the results over the IPN hand dataset and Figure 13 shows the results over the Jester dataset.

4.2.4. Experiment IV: Comparison of our Proposed System with State-of-the-Art Techniques

In this section, we have compared the proposed model with other well-known techniques using the same datasets.

Table 6. Hand gestures recognition results from the proposed model with other state-of-the-art techniques.

Authors	IPN Hand Dataset (%)	Authors	Jester Dataset (%)
Yamaguchi et al. (2022) [70]	60.00	Zhou et al. (2018) [71]	82.02
Gammulle et al. (2021) [72]	80.03	Shi et al. (2019) [73]	82.34
Garcia et al. (2020) [68]	82.36	Kopuklu et al. (2018) [74]	84.70
TSN [75]	68.01	MFFs [76]	84.70
Proposed method	88.46	Proposed method	87.69

shows the comparative results between the proposed model and other state-of-the-art techniques.

5. Conclusions

This article is based on a hand gestures recognition system for controlling smart home appliances. Two benchmark datasets were selected for experiments: the IPN hand dataset and Jester dataset. Initially, images are acquired, in which hands are detected and landmarks are localized on the palm and the fingers. After that, the textures-based and point-based features are extracted. The hand skeletons are used for extracting the point-based features, whereas the full hand is used for extracting the texture-based features. For feature reduction and optimization, the fuzzy logic is adopted, and finally, the K-ary classification algorithm is used for classifying the hand gestures for operating smart home appliances. For the IPN hand dataset, we achieved the mean accuracy of 88.46% and for the Jester dataset, a mean accuracy of 87.69% was achieved. The proposed system’s performance shows a significant improvement compared to existing state-of-the-art frameworks. The limitation of the proposed framework is due to the complexity in the videos, such as cluttered backgrounds and various illumination conditions, which make it difficult to achieve more accurate results.