A Comparative Study of Local Descriptors and Classifiers for Facial Expression Recognition

Abstract

Facial Expression Recognition (FER) is a growing area of research due to its numerous applications in market research, video gaming, healthcare, security, e-learning, and robotics. One of the most common frameworks for recognizing facial expressions is by extracting facial features from an image and classifying them as one of several prototypic expressions. Despite the recent advances, it is still a challenging task to develop robust facial expression descriptors. This study aimed to analyze the performances of various local descriptors and classifiers in the FER problem. Several experiments were conducted under different settings, such as varied extraction parameters, different numbers of expressions, and two datasets, to discover the best combinations of local descriptors and classifiers. Of all the considered descriptors, HOG (Histogram of Oriented Gradients) and ALDP (Angled Local Directional Patterns) were some of the most promising, while SVM (Support Vector Machines) and MLP (Multi-Layer Perceptron) were the best among the considered classifiers. The results obtained signify that conventional FER approaches are still comparable to state-of-the-art methods based on deep learning.

1. Introduction

Facial Expression Recognition (FER) has a number of different applications, including service robots [1], driver fatigue monitoring [2], mood prediction [3], sentiment analysis in customer reviews [4], and many others. For humans, recognizing and interpreting facial expressions comes naturally and is a basic form of communication. In some cases, facial expressions are the only means of communication, such as in newborns and unconscious patients in the ICU [5]. However, facial expression recognition is still challenging for computer systems due to factors such as camera view angles, occlusions, image noise, and changes in scene illumination. All these issues must be accounted for when developing a robust system for a given application. In the past decade, much work has been done to recognize a group of expressions, which are often related to prototypical facial expressions associated with basic emotions  [6]. A FER system usually operates in three stages: in the first stage, a face region is located, then facial features are extracted in the second stage, and finally, an expression is classified in the third stage.

FER systems can either be categorized as frame or sequence-based depending on whether a static image or a video sequence is used to extract facial features for classification [7]. In frame-based approaches, only spatial information, such as the appearance and geometry of the facial image, can be used to describe the expression. In contrast, the sequence-based approach can extract spatial and temporal features to describe the evolution of expressions during a video sequence. Frame-based approaches are preferred due to their simplicity, and no assumption is made about how the expression evolves (an assumption for many sequence-based approaches). Therefore, much attention has been given to developing FER systems based on static images. In this category, we can identify two main trends in the development of FER systems. On the one hand, there are FER systems based on the extraction and classification of handcrafted features, and on the other, we have approaches based on deep learning. Handcrafted techniques rely on domain-specific knowledge about the human face, such as facial muscle deformations and the movement of facial components, such as raising the eyes and the opening/closing of the mouth during emotion elicitation.

Within the handcrafted techniques, the geometrical features were developed to describe the shape and location of facial components [8]. Some work was also undertaken to reconstruct Action Units and interpret their combinations as facial expressions [9]. Geometric-based FER systems were difficult to develop because they required the accurate labeling of Action Units [6] and landmarks, which is time-consuming and tedious. During this time, appearance-based FER systems were being developed, giving rise to local descriptors for facial expression representation. Local descriptors were used to extract texture information such as edges, corners, and spots that make up facial expressions and achieved the same or better results while being less tedious to develop than the geometric features.

Recently, deep neural networks have been studied for facial expression recognition [10,11]. Unlike traditional techniques, deep learning includes little knowledge about facial structure and appearance. Deep learning systems automatically detect and extract facial features based on a series of layers, such as convolution, pooling, and fully-connected layers. This field is attracting more and more attention due to the improved recognition capacity offered by deep neural networks for static and sequence-based recognition. Mainly, CNN (Convolution Neural Networks) is the basic model for extracting more discriminative spatial features from face images, while RNN (Recurrent Neural Networks) and LSTM (Long Short-Term Memory) networks are suitable for characterizing the spatio-temporal features of sequences [12].

In the literature, there have been a few attempts to compare the different methods based on local descriptors and traditional classifiers for facial expression recognition. In 2018, Turan and Lam [13] studied 27 local descriptors under different conditions, such as varying image resolutions and the number of sub-regions. Two classifiers were used to recognize a different number of expressions on four facial expression databases. Their results were comparable to the state-of-the-art deep learning approaches on well-known datasets. Slimani et al. [14] studied the independent performance of 46 LBP variants for facial expression recognition. A single classification technique (i.e., SVM) was used to classify seven expressions. The study showed that several local descriptors, initially proposed for other classification problems, outperformed the state-of-the-art four databases.

This study investigates the performances of six local descriptors and four machine-learning techniques for automatic facial expression recognition. The novelty of this approach is in using a variable sub-region size and a variable number of histogram bins to optimize the performance of histogram-based local descriptors. Face registration and feature vector normalization also contributed significantly to the results obtained in this work compared to previous works. Many experiments were conducted under different settings, such as varied extraction parameters, different numbers of expressions, and two datasets, to discover the best combinations of local descriptors and classifiers for facial expression recognition. The rest of this paper is organized as follows: Section 2 reviews local descriptors and classifiers used in facial expression recognition, Section 3 is the materials and methods, Section 5 presents the experimental results and discussion, and Section 6 concludes the paper.

2. Related Works

The following lines describe the various local descriptors and classifiers used in our experiments in more detail.

2.1. Local Descriptors

2.1.1. Local Binary Patterns

The Local Binary Patterns (LBP) descriptor describes the spatial structure of a local patch containing a center pixel $x_c$ surrounded by p equally-spaced neighbors [15]. Given a texture image, I, the LBP patterns of each pixel can be computed by assigning a binary code to the center pixel, $x_c$ of a $3\times 3$ patch with eight neighbors and computing an LBP histogram used to characterize the image. In [16], a complete version of LBP was developed to allow for a variable number of neighbors p located on a circular radius r. The LBP pattern is defined as

$$LB{P}_{r,p}=\sum _{j=0}^{p-1}S(x_j-x_c)2^j$$

(1)

where $S\left(A\right)$ is defined as

$$S\left(A\right)=\left\{\begin{array}{cc}1\hfill & \mathrm{if}A\ge 0\hfill \\ 0\hfill & \mathrm{else}\hfill \end{array}\right.$$

(2)

Since its introduction as a textured-based descriptor in 2002, LBP has undergone several improvements to produce a more robust descriptor. For example, Shan et al. [17] improved LBP by dividing the face image into sub-regions of various sizes and positions, and the AdaBoost algorithm was used to learn the most discriminative LBP histograms. The so-called Boosted-LBP features were then classified as expressions using machine learning techniques such as template matching, Support Vector Machines, Linear Discriminant Analysis, and linear programming. LBPTOP extends LBP to encode temporal changes and spatial changes [18]. This method extracts LBP patterns from three orthogonal planes: the spatial plane, the vertical spatio-temporal plane, and the horizontal spatio-temporal plane. In [19], a new kernel-based manifold learning method called kernel discriminant isometric mapping is proposed to reduce the dimension of LBP features. The features are then classified using the nearest neighbor classifier. Recently, Guo et al. [20] proposed the Extended Local Binary Patterns on three Orthogonal Planes (ELBPTOP) for spontaneous micro-expressions recognition, and in [21], a smaller LBP feature vector that is also resistant to noise is presented. The novel descriptor considers four neighbors and diagonal neighbors separately and an adaptive window and averaging in radial directions to improve the feature extraction.

2.1.2. Compound Local Binary Patterns

Compound Local Binary Patterns (CLBP) extend the LBP descriptor by encoding both the magnitude and the sign of the differences between a center (or threshold) pixel and its P neighbors [22]. Unlike LBP, which replaces each gray value which a P-bit code, CLBP assigns a $2P$-bit code. Considering $i_c$ is the gray value of the center pixel, $i_p$ is the gray value of the p-th neighbor, and $M_{avg}$ is the average magnitude of the difference between $i_p$ and $i_c$ in a local neighborhood, the CLBP code is defined as

$$f_c(x,y)=\sum _{p=0}^{P-1}s(i_p,i_c)2^{2p}$$

(3)

where $s(i_p,i_c)$ is defined as

$$s(i_p,i_c)=\left\{\begin{array}{cc}00\hfill & \mathrm{if}{i}_p-i_c<0\mathrm{and}|i_p-i_c|\le M_{avg}\hfill \\ 01\hfill & \mathrm{if}{i}_p-i_c<0\mathrm{and}|i_p-i_c|>M_{avg}\hfill \\ 10\hfill & \mathrm{if}{i}_p-i_c\ge 0\mathrm{and}|i_p-i_c|\le M_{avg}\hfill \\ 11\hfill & \mathrm{otherwise}\hfill \end{array}\right.$$

The code is then split into two sub-CLBP patterns by concatenating the bit values corresponding to the following sequence $(1,2,5,6,\dots ,2P-3,2P-2)$ and $(3,4,7,8,\dots ,2P-1,2P)$, respectively, of the original CLBP code. The two sub-CLBP codes are treated as separate P-bit codes, and a histogram is computed for each. Finally, the two histograms are combined to form a feature vector.

2.1.3. Local Directional Patterns

Local Directional Patterns (LDP) is based on the edge responses in a local neighborhood [23]. When applied to a texture image, the descriptor computes eight directional edge responses at each pixel and encodes the responses as an 8-bit binary code using the relative strengths of the edge responses. Specifically, the edge responses are computed using Kirsch edge masks (see Figure 1). To form a binary code, the k most significant responses are set to 1 while the remaining $8-k$ bits are set to 0. In summary, the LDP code is defined as

$$LD{P}_k=\sum _{j=0}^{7}B(m_j-m_k)2^j$$

(4)

where $m_j$ is the j-th directional response, $m_k$ is the k-th most significant directional response and $B\left(a\right)$ is defined as

$$B\left(a\right)=\left\{\begin{array}{cc}1\hfill & \mathrm{if}a\ge 0\hfill \\ 0\hfill & \mathrm{otherwise}\hfill \end{array}\right.$$

(5)

2.1.4. Angled Local Directional Patterns

Angled Local Directional Patterns (ALDP) were introduced by Shabat and Tapamo [24] to improve upon the LDP descriptor by addressing two drawbacks of LDP: (1) the static choice of the most significant bits and (2) the value of the center pixel was ignored. To address these issues, the ALDP descriptor is extracted by first generating Kirsch mask responses $(m_0,\dots ,m_7)$ just like LDP, then computes angular vector components $(p_0,\dots ,p_7)$ in four angles (0, 45, 90, 135). One of the drawbacks of ALDP is the large size of the feature vector. Considering all eight angular vector components, the basic histogram size is 256, unlike LDP, which has only 56 possible codes. More details about the ALDP descriptor can be found in [24].

2.1.5. Weber’s Local Descriptor

Weber’s Local Descriptor (WLD) was inspired by Weber’s law, which states that the human perception of a pattern depends not only on the change in stimulus (such as illumination variation) but also on the original intensity of the stimulus [25]. WLD consists of a differential excitation ($\xi$) and a gradient orientation ($\theta$). A feature vector is obtained by rearranging the differential excitations several times into subgroups and then creating histograms of those subgroups. Finally, the histograms are reordered and concatenated into a single feature vector for classification. Given the texture image G, a gray-level $x_c$, and p neighbors, the differential and excitation components of WLD are defined by Equations (6) and (7). More details about the implementation can be found in [25].

$$\xi \left(x_c\right)={\tan }^{-1}\left(\frac{V_s^{00}}{V_s^{01}}\right)={\tan }^{-1}\left[\sum _{i=0}^{p-1}\left(\frac{x_i-x_c}{x_c}\right)\right]$$

(6)

The second component of WLD is the gradient orientation $\theta \left(x_c\right)$, which is defined as

$$\theta \left(x_c\right)={\tan }^{-1}\left[\frac{V_s^{11}}{V_s^{10}}\right]={\tan }^{-1}\left[\frac{x_5-x_1}{x_7-x_3}\right]$$

(7)

2.1.6. Histogram of Oriented Gradients

Dalal and Triggs [26] originally proposed the Histogram of Oriented Gradients (HOG) descriptor to tackle objection detection. The HOG descriptor counts the occurrences of gradient orientation in a local sub-region of an image. The HOG descriptor is applied by computing the gradient directions over the pixels of small sub-regions called cells. Subsequently, the histogram of the gradient directions is used as a feature. HOG is especially powerful because it applies block normalization schemes (e.g., L2-norm and L2-Hys) to the histogram. In block normalization, a window is moved over the input image, and the HOG of each cell within the window is normalized as a group. The concatenation of normalized histograms is treated as the feature vector. Given a texture image L, a sub-region size of N × N pixels, then the gradient orientation ${\theta }_{x,y}$ of a pixel located at $(x,y)$ can be computed by

$${\theta }_{x,y}={\tan }^{-1}\left(\frac{L(x,y+1)-L(x,y-1)}{L(x+1,y)-L(x-1,y)}\right)$$

(8)

2.2. Classifiers

2.2.1. Support Vector Machines

Support Vector Machines (SVM) have become the standard for many FER approaches due to the advantages such as fast training and no direct probability estimation [8,27,28]. SVM is a machine learning algorithm that classifies input features by defining a separating hyperplane. Given l training data vectors $x_i=(x_1,x_2,\dots ,x_l)$ and the corresponding labels $y_i\in \{-1,1\}$, the following primal optimization problem is defined as

$$\underset{w,b,\xi }{\min }\left\{1/2w^{\top }w+C\sum _{i=1}^l{\xi }_i\right\}$$

(9)

$$\begin{array}{c}\mathrm{Subject}\mathrm{to}{y}_i(w^{\top }\varphi \left(x_i\right)+b)\ge 1+{\xi }_i\\ {\xi }_i>0,i=1,\dots ,l\end{array}$$

where ${\xi }_i$ represents the misclassification error for the i-th training vector, $\xi$ is the total misclassification error; w is the normal vector to the hyperplane; $b/\parallel w\parallel$ represents the offset of the hyperplane from the origin along the normal vector w ($\parallel ·\parallel$ being the norm operator); $\varphi \left(x_i\right)$ is the kernel function, mapping $x_i$ into a higher dimensional space and C is the regularization parameter. Scikit-learn’s implementation of the LIBSVM library [29] is used in our experiments.

2.2.2. Naïve Bayes Classifier

The Naïve Bayes (NB) classifier assigns a label $y\in (y_1,y_2,\dots ,y_M)$ to a feature vector $X(x_1,x_2,\dots ,x_N)$ based on the maximum likelihood framework defined by

$$\widehat{y}=\underset{y}{argmax}P\left(X\right|y)$$

(10)

By assuming that the features in X are mutually independent given a class label y, Equation (10) reduces to:

$$\widehat{y}=\underset{y}{argmax}\prod _{i=1}^{N}P\left(x_i\right|y)$$

(11)

The different Naive Bayes classifiers differ based on how they model the probability distribution of features given a class label $\left(P\left(x_i\right|y)\right)$. The Gaussian Naïve Bayes classifier is the most common and assumes a Gaussian distribution as defined in Equation (11). The Cauchy distribution has also been investigated for FER [30].

$$P\left(x_i\right|y)=\frac{1}{\sqrt{2\pi {\sigma }_y^2}}exp\left(-\frac{{(x_i-{\mu }_y)}^2}{2{\sigma }_y^2}\right)$$

(12)

The NB classifier is a standard due to its simplicity and good performance in many classification problems. In this study, scikit-learn’s [29] implementation of the Gaussian Naïve Bayes algorithm was used.

2.2.3. K-Nearest Neighbors

K-Nearest Neighbors (KNN) is a non-parametric method that classifies a new sample based on a majority vote system: a new data point is assigned to the class with the most representatives within the k nearest neighbors of that point. The neighbors are taken from the training set consisting of sample points for which the class is known.

Despite being the simplest machine learning algorithm, KNN has achieved encouraging results in the FER problem. Sohail and Bhattacharya [31] developed an approach based on eleven feature points representing the principal muscle actions and the KNN classifier to recognize six basic facial expressions, achieving an average accuracy of 90.76%. Panchal and Pushpalatha [32] proposed a new feature descriptor based on LBP and Asymmetric Region LBP and classification was done using KNN. When classifying seven expressions, an overall recognition accuracy of 95.1% was achieved on the JAFFE database.

It should be noted that the KNN algorithm does not construct a general internal model but simply stores training data instances. Hence, no explicit training is required. Hence, there are zero training costs. However, it becomes computationally expensive to find the k neighbors when classifying high-dimensionality features. In this study, scikit-learn’s implementation of the KNN classifier was used [29].

2.2.4. Multi-Layer Perceptron

Multi-Layer Perceptron (MLP) is the traditional basic neural network that learns a non-linear model to map training features X $(x_1,x_2,\dots ,x_m)$ to output targets y [29]. The model comprises an input layer, one or more hidden layers, and an output layer. Each layer is made up of neurons that transform the value of the previous layer with a weighted linear summation $w_1{x}_1+w_2{x}_2+\dots +w_m{x}_m$ followed by a non-linear activation function $g(·):R\to R$.

The MLP neural network has shown encouraging performances in recognizing facial expressions of emotion. Dino and Abdulrazzaq [33] proposed a system for classifying facial expressions using the Viola-Jones algorithm for face detection, the HOG descriptor for feature extraction, PCA for dimensionality reduction, and MLP for classification. They reported an average accuracy of 82.97% on the CK+ database when classifying eight basic expressions. Boughrara et al. [34] developed a new constructive training algorithm for the Multi-Layer Perceptron (MLP) applied to FER. Unlike most traditional algorithms, which fix the structure of the neural network before training, the proposed constructive training algorithm learns the architecture of the network and performs the learning process simultaneously. The approach uses the Perceived Facial Images (PFI) in eight directions to extract features from a facial image, and the predicted expression is obtained by a fusion of the neural networks corresponding to the eight directions. The main drawback of MLP is that it is often time-consuming and tedious to determine a proper structure than can guarantee convergence and avoid over-fitting.

3. An Approach for Optimized Facial Expression Recognition

A simple and effective method for facial expression recognition based on local descriptors and conventional classifiers was defined by Slimani et al. [14]. The method involves dividing the face region into several non-overlapping regions and extracting local descriptors from each sub-region. The histogram of each sub-region is calculated separately and concatenated to form a single feature vector for classification. The current work extends Slimani et al.’s [14] approach by optimizing the feature extraction process. Figure 2 summarizes the proposed approach for optimized facial expression recognition, consisting of two phases: a training phase and a testing phase. In the training phase, images are pre-processed using face detection and registration. Next, a local descriptor is applied to the registered faces to extract facial features. Subsequently, feature vectors are generated by dividing a face region into several sub-regions and calculating the histograms of each sub-region. The feature vectors are normalized based on the type of local descriptor used (see Section 3.2). To optimize the process of feature vector calculation, the extraction parameters (i.e., the sub-region size and the number of histogram bins) are varied, producing several feature sets. Then, the best extraction parameter settings are found by comparing the classification performances produced by each feature set. Finally, the best feature set and corresponding expressions are used to train a machine-learning model.

In the testing phase, images are processed, and the feature vectors are generated based on the best extraction parameter values obtained from the training phase. After that, the feature vectors are classified as facial expressions using the trained classifier. Additionally, the classifier’s performance is evaluated by 10-fold cross-validation. The novelty of this approach is in using a variable sub-region size and a variable number of histogram bins to optimize the performance of each local descriptor.

3.1. Face Detection and Registration

Faces were detected using a frontal face detector implemented by the OpenCV library for the Python programming language (OpenCV is available at https://pypi.org/project/opencv-contrib-python/ accessed on 24 July 2021). The faces were then registered, giving them a predefined pose, shape, and size. First, an input image was converted to gray-scale, and then Contrast Limited Histogram Equalization (CLAHE) was applied using Equation (13). Then, a face blob was obtained by performing a thresholding operation followed by filtration techniques. Then, an ellipse-fitting algorithm (the ellipse fitting algorithm was implemented by the Python OpenCV library.) estimates the shape and angle of inclination of the face blob. The ellipse’s angle is used to rotate the face to a vertical position. This process is not always perfect. Hence, the eye locations are used to rotate the image a second time. Finally, an elliptical crop followed by a rectangular crop is employed to produce an image where the eyes are in a predefined position relative to the sides of the image. Figure 3 illustrates face detection and registration.

$$g_{i,j}=⌊(L-1)\sum _{n=0}^{I_{i,j}}{p}_n⌋$$

(13)

where $p_n$ is defined as

$$\begin{array}{c}{p}_n=\frac{{\sum }_{i,j}A(I_{i,j}=n)}{M\times N};n=0,1,\dots ,L-1;0\le i<M;0\le j<N;\end{array}$$

and $A\left(x\right)$ is defined as

$$\begin{array}{c}A\left(x\right)=\left\{\begin{array}{cc}1,\hfill & \mathrm{if}x\mathrm{is}\mathrm{true}\hfill \\ 0,\hfill & \mathrm{otherwise}\hfill \end{array}\right.\end{array}$$

3.2. Feature Extraction and Feature Vector Calculation

In this study, six local descriptors were investigated: Local Binary Patterns (LBP), Compound Local Binary Patterns (CLBP), Local Directional Patterns (LDP), Angled Local Directional Patterns (ALDP), Weber’s Local Descriptor (WLD), and Histogram of Oriented Gradients (HOG). Feature extraction and feature vector calculations are key stages of this FER method. Although the two processes were portrayed separately in the preamble of this section, their workings are very interlinked. The method used in this study divides the face image into several equally sized non-overlapping sub-regions, and a local descriptor is applied to each sub-region, producing several histograms. The final feature vector is generated by normalizing the histograms and concatenating them into a single 1D vector. It must be noted that the sub-region histograms are calculated separately and normalized such that the frequencies sum to one (this rule is applied to all the considered descriptors except the HOG descriptor, where the blocks are normalized in groups using L2-Hys [26]). The normalized histograms are then concatenated to form the final feature vector. Suppose $I_j$ is a $W_S\times W_S$ matrix of intensities representing the j-th sub-region in the image, and the function $D(x,y)$ computes the descriptor at a position $(x,y)$ in $I_j$, and then the histogram associated with this sub-region is defined as

$$H_j={\left\{h_i\right\}}_{i=0,1,\dots ,N_b-1}$$

(14)

$$h_i=\sum _{x=0}^{W_S-1}\sum _{y=0}^{W_S-1}\delta (D(x,y),i)$$

$$\delta (x,i)=\left\{\begin{array}{cc}1,\hfill & \mathrm{if}L\left(i\right)\le x<U\left(i\right)\hfill \\ 0,\hfill & \mathrm{otherwise}\hfill \end{array}\right.,L\left(i\right)=i\times \frac{n}{N_b},U\left(i\right)=(i+1)\times \frac{n}{N_b},$$

where n is the number of possible descriptor values, and $N_b$ is the number of histogram bins.

The final feature vector is given by

$$H=\left\{\frac{H_0}{{\left(W_S\right)}^2},\frac{H_1}{{\left(W_S\right)}^2},\dots ,\frac{H_{m-1}}{{\left(W_S\right)}^2}\right\}$$

(15)

where m is the number of sub-regions.

3.3. Feature Set Evaluation

In this step, a model is trained with different feature sets, and the best feature set is selected based on its classification performance. The performances were ranked using the average recall during 10-fold cross-validation. Then, the best feature set and the corresponding best parameter setting are identified. Algorithm 1 defines the process of feature set evaluation in more detail.

Algorithm 1 Feature Set Evaluation

Inputs:    $(F_1,\dots ,F_m):\mathrm{feature}\mathrm{sets}$   $(p_1,\dots ,p_m):\mathrm{extraction}\mathrm{parameters}\mathrm{for}\mathrm{each}\mathrm{feature}\mathrm{set}$ Outputs:    $F_{best}:\mathrm{best}\mathrm{feature}\mathrm{set}$   $p_{best}:\mathrm{best}\mathrm{extraction}\mathrm{parameters}$ 1:    $F_{best}\leftarrow F_1$         // Initialize the best feature set 2:    $p_{best}\leftarrow p_1$         // Initialize the best parameters 3:    $model\leftarrow NewModel\left(\right)$         // Construct a new model e.g., SVM, KNN etc. 4:    $maxScore\leftarrow CrossValidation(model,F_1$)         // Obtain a cross-validation score 5:    for i in $(2,3,\dots ,m)$ do         // Train new models with the remaining feature sets 6:        $model\leftarrow NewModel\left(\right)$ 7:        $score\leftarrow CrossValidation(model,F_i)$ 8:        if $score>maxScore$ then         // Find the maximum score 9:            $F_{best}\leftarrow F_i$ 10:            $p_{best}\leftarrow p_i$ 11:            $maxScore\leftarrow score$ 12:        end if 13:  end for 14:  return $F_{best},p_{best}$

3.4. Facial Expression Classification

This research is focused on the recognition of prototypic facial expressions of emotion from facial images. The classification stage aims at training a machine learning model using labeled feature vectors extracted from the facial images. Once trained, the model can make predictions on new data. The current work considers four classification methods, including Support Vector Machines (SVM), K-Nearest Neighbors (KNN), Naive Bayes (NB) (this work used scikit-learn’s [29] implementation of the Gaussian Naive Bayes classifier), and Multi-layer Perceptron (MLP).

Table 1. Classifier hyper-parameters.

Classifier	Parameters
SVM	kernel = Radial Basis Function, C = 1000, gamma = 0.05
NB	Automatically selected
KNN	k = 50
MLP	solver = adam, number of passes over the training data = 200, learning rate = 0.001

gives the hyper-parameter settings used to train each classifier.

4. Datasets

This study conducted experiments using two well-known FER datasets: the Extended Cohn–Kanade (CK+) and Radboud Faces Dataset (RFD).

CK+ gathers facial expression images of subjects from various ethnicities, ages, and genders. This database consists of 593 image sequences from 123 subjects. The database contains posed and non-posed images taken under lab-controlled conditions. To form the first dataset for our study, we followed the guidelines in [35] by selecting our samples as follows: the last images in the sequences for anger, disgust, and happiness; the last and fourth images for the first 68 sequences related to surprise; the last and fourth from last images for the images of fear and sadness. A second dataset is created by adding images for the expression related to neutral. The number of images collected amounted to 347 and 407 for the first and the second datasets, respectively.

The Radboud Faces Database contains pictures of 67 subjects (adults and children) displaying eight facial expressions of emotion. The subjects are of Caucasian ethnicity, and the images are taken in different gaze directions and head orientations [36]. Three subsets were created: the first subset consisted of 402 images labeled with six expressions, including anger, disgust, fear, happiness, sadness, and surprise; the second subset was formed by augmenting the first set with 67 images labeled as contempt, and the third subset was formed by adding 67 images labeled as neutral to the second set. The total number of image samples was 536.

Table 2. Summary of the datasets.

Dataset	Database Name	Number of Images	Number of Classes
DS1	CK+	347	6
DS2	CK+	407	7
DS3	RFD	402	6
DS4	RFD	469	7
DS5	RFD	536	8

summarizes the number of samples and expressions in each dataset.

5. Experimental Results and Discussion

5.1. Performance Analysis for Varying the Number of Histogram Bins

In this experiment, the face regions were first detected and aligned using a face registration algorithm that automatically positions the eyes and crops on the face image to discard background pixels. Then, the aligned face images were resized to a resolution of 100 × 100 pixels. By setting the size of the sub-regions to 13 × 13 pixels (i.e., 49 sub-regions in total), feature vectors were extracted from each image. The number of histogram bins was varied to reduce the feature vector length while achieving a high recognition rate. Note that for WLD, the histogram bins were set as $M=2$, $S=2$, and T was varied from 2 to 24. Figure 4 gives the results (results are given as the average accuracy during 10-fold cross-validation) obtained when the number of histogram bins is varied for different descriptor and classification combinations. We observe different trends depending on the type of classifier used. When SVM and MLP classifiers are used, the classification performance either improves or stays the same as the number of bins increases. A similar result is seen in the KNN classifier, although the large size of the features hurts the recognition rates from some descriptors (e.g., with the LDP descriptor, the performance decreases as the number of bins increases). On the other hand, when using the NB classifier, the recognition rates decrease as the number of bins rises, regardless of the type of features used.

5.2. Performance Analysis for Varying the Sub-Region Size

A sub-region can be represented by a square of size $l\times l$, where l varies from 7 to 25, taking steps of 3. Figure 5 gathers the results for varying the sub-regions size on the RFD dataset. As observed in the tables, the larger the sub-regions size (or, the smaller the number of sub-regions), the better the recognition rates. This trend is observed for all other classifier and descriptor combinations. It is also worth noting that no single sub-region size works best for all descriptor and classifier combinations. However, our results suggest that sub-regions of 10 × 10 and 13 × 13 pixels give some of the highest recognition rates. Based on the results from the previous two experiments (see Section 5.1 and Section 5.2), the optimum parameter values were selected.

Table 3. Best parameters for each combination of classifier and descriptor (number of bins, block size).

Descriptor	SVM	NB	KNN	MLP
WLD	(88, 19)	(8, 19)	(96, 10)	(88, 13)
LDP	(48, 7)	(12, 10)	(8, 10)	(48, 13)
ALDP	(112, 13)	(16, 16)	(112, 13)	(128, 7)
LBP	(96, 19)	(32, 19)	(128, 13)	(128, 10)
CLBP	(248, 10)	(24, 10)	(248, 13)	(216, 10)
HOG	(50, 13)	(6, 7)	(9, 10)	(50, 13)

gives the parameter values used in the next series of experiments.

5.3. Performance Analysis of the Classifiers

Figure 6 summarizes the best results obtained when varying the sub-region sizes on the CK+ dataset with six expressions. We observe that the best recognition rates (95–98%) were achieved using SVM and MLP classifiers, while the lowest accuracies were produced by the NB and KNN classifiers (73–91%). As reported in Figure 7, a similar trend is observed on the RFD dataset with six expressions. To further analyze the performances of the SVM and MLP classifiers, the previous experiments were repeated on other datasets with varying numbers of expressions.

Table 4. Comparison of recognition rates on the CK+ and RFD datasets with a varying number of expressions.

	CK+		RFD
Descriptor + Classifier	6-Class	7-Class	6-Class	7-Class	8-Class
WLD + SVM	96.0	92.4	96.7	95.9	91.5
LDP + SVM	96.1	93.3	97.0	96.8	94.0
ALDP + SVM	97.5	91.2	97.0	96.8	94.1
LBP + SVM	96.4	93.0	96.6	97.0	93.0
CLBP + SVM	95.3	93.0	97.5	97.0	93.5
HOG + SVM	97.4	93.4	96.6	97.0	93.8
WLD + MLP	96.3	94.2	96.9	95.9	91.9
LDP + MLP	95.9	93.2	96.9	96.2	93.4
ALDP + MLP	96.7	92.7	95.4	95.3	93.0
LBP + MLP	97.2	93.1	96.7	96.4	94.0
CLBP + MLP	97.5	93.5	96.8	97.4	93.1
HOG + MLP	98.3	95.1	97.4	96.4	94.0

gathers the results obtained when the number of expressions was varied. For the CK+ dataset, the expressions were anger, disgust, surprise, sadness, happiness, and disgust in the six-class problems, and the seven-class problem added the neutral expression to the previous list. For the RFD dataset, the expressions were anger, disgust, surprise, sadness, happiness, and disgust in the six-class problem, the seven-class problem added the expression of contempt to the previous list, and the eight-class problem added the neutral expression. Considering the six-class problem, SVM and MLP achieve similar classification rates (above 95%) on the two datasets. Although their performances are very similar, the MLP classifier achieved slightly better recognition rates than SVM when classifying WLD, ALDP, LBP, CLBP, and HOG descriptors. Both classifiers are negatively affected by the introduction of the neutral. The largest performance drop is seen in the ALDP + SVM combination, where the accuracy drops by almost 7%. The recognition rates of both classifiers do not change much when the Contempt expression is introduced in the RFD dataset with seven expressions.

5.4. Analyzing the Computational Costs

This section examines the computational costs of using different combinations of descriptors and classifiers. The extraction times of each local descriptor were measured by extracting a feature vector with a sub-region size of 13 × 13 pixels and 56-bin histograms. The same parameter setting was used in all descriptors to make the comparison fair. On the other hand, when measuring the prediction time, the optimum parameters (see Table 3) were used to extract feature vectors before making predictions on those feature vectors.

Table 5. Extraction times (in seconds).

WLD	LDP	ALDP	LBP	CLBP	HOG
0.54	3.79	3.9	5.69	12.49	1.72

and

Table 6. Prediction times of various classifiers (times are given in milliseconds).

Descriptor	WLD	LDP	ALDP	LBP	CLBP	HOG	Average
SVM	1300	530	1240	1050	2520	1970	1435
NB	150	240	180	200	200	210	197
KNN	3010	500	3260	3560	6360	1160	2975
MLP	310	140	460	460	680	580	438

give the extraction and prediction times, respectively, where each value is the average time over 500 iterations. Results show that the best extraction time was achieved by WLD (0.54 s), followed by HOG (1.72 s). The longest times are seen in LBP and CLBP. NB was the fastest among the considered classifiers, with an average prediction time of 197 ms, followed by MLP, which took an average of 438 ms. SVM came in the third position, and KNN was the last.

The time costs for training each classifier using different descriptors were also measured. We used the same extraction parameters for all the considered classifiers to ensure a fair comparison. We set the sub-region size to 19 × 19 pixels and the number of bins to 48. The features were extracted from the RFD dataset with six expressions. As seen in

Table 7. Training times of various classifiers (times are given in milliseconds).

Descriptor	WLD	LDP	ALDP	LBP	CLBP	HOG	Average
SVM	41	30	22	26	39	14	29
NB	9	6	6	8	6	4	6
KNN	1	1	1	1	1	1	1
MLP	6520	5680	7489	5991	6234	2836	5792

, KNN and NB had the fastest training times (a few milliseconds), followed by SVM, which took 29 ms. MLP was the slowest, taking up to 6 seconds during training. All our experiments were performed on a Windows 10 PC with an AMD Ryzen 7 (3700U) processor and 12 GB of RAM.

5.5. Comparison with the State-of-the-Art

Several local descriptors have been proposed in recent years to tackle the FER problem [33,37,38].

Table 8. Comparison with state-of-the-art methods.

Database	Ref	Year	Features	Samples	Classifier	No. Classes	Accuracy (Measure)
CK+	[40]	2017	LTP + HOG	610	SVM	7	96% (10-fold)
CK+	[38]	2018	MRDTP + MRDNP	1281	ELM-RBF	7	98.4% (10-fold)
CK+	[41]	2017	Deep	927	FRR-CNN	6	92.06% (10-fold)
CK+	[24]	2017	ALDP	-	SVM	7	97% (80–20%)
CK+	[10]	2017	Deep	927	FN2EN	6	98.6% (10-fold)
CK+	[33]	2019	HOG	634	MLP	8	82.97% (10-fold)
RaFD	[39]	2014	PHOG	630	KNN	7	100% (67–33%)
RaFD	[11]	2012	PZM	-	KNN	6	94.51% (70–30%)
RaFD	[37]	2020	HOG + LLE	469	SVM	6	93.54% (10-fold)
CK+	Proposed	2022	HOG	347	MLP	6	98.4% (10-fold)
CK+	Proposed	2022	ALDP	347	SVM	6	97.5% (10-fold)
RaFD	Proposed	2022	CLBP	402	SVM	6	97.5% (10-fold)

gives the performances of various existing FER systems compared to our proposed best combinations of local descriptors and classifiers. The table shows that several local descriptors work as well as deep neural networks. Furthermore, Shokrani et al. [39] proposed the Pyramid Histogram of Oriented Gradients (PHOG) descriptor and KNN classifier, achieving an average accuracy of 100% on the CK+ dataset. Results such as these are difficult to compare with other approaches because they depend on the selection of the train and testing images. The more recent studies prefer 10-fold cross-validation as a more reliable measure of the accuracy of FER classifiers. This is because each sample is used for training and testing once. Hence, a high accuracy during a 67–33% split, as in [39], is not always a good indicator of performance.

It is clear from the table that our approach using HOG and CLBP either performs better or as efficiently as other approaches. Although some existing approaches have produced higher accuracies, they generally produce larger feature vectors. The proposed method finds the best balance between a small feature vector and high accuracy by exploring several feature sets. Our findings reveal that a small feature vector can achieve the same accuracy as a large feature vector if the extraction parameters are appropriately set, as seen in Figure 4. This finding could greatly benefit existing methods with high dimensionality and large memory footprints. It should be noted that, unlike many recent approaches, this study did not use any dimensionality reduction or subspace-learning methods and still achieved competitive results.

6. Conclusions

This study evaluated different combinations of local descriptors and classifiers for facial expression recognition. The experiments comprehensively evaluated six descriptors and four classifiers on two famous datasets, such as CK+ and RFD. Notably, the appropriate choice of extraction parameters improved several descriptors’ performances. We can single out HOG as the descriptor that achieved the most consistent performance across all the datasets. The computational costs of each classifier and descriptor were also studied, and we found that the NB and MLP classifiers provided the fastest predictions. However, MLP can be time-consuming during training compared to the other classifiers. When evaluating the classifiers’ recognition efficiencies, we found that SVM and MLP achieved the best results.

Out of the six considered descriptors, HOG and ALDP are some of the most promising, while SVM and MLP rank at the top of the considered classifiers. We also found that SVM was faster to train and required little model tuning compared to MLP. On the other hand, MLP has a notably faster prediction, which could be an advantage in real-time applications.