Research on a Lip Reading Algorithm Based on Efficient-GhostNet

Abstract

Lip reading technology refers to the analysis of the visual information of the speaker’s mouth movements to recognize the content of the speaker’s speech. As one of the important aspects of human–computer interaction, the technology of lip reading has gradually become popular with the development of deep learning in recent years. At present, most the lip reading networks are very complex, with very large numbers of parameters and computation, and the model generated by training needs to occupy large memory, which brings difficulties for devices with limited storage capacity and computation power, such as mobile terminals. Based on the above problems, this paper optimizes and improves GhostNet, a lightweight network, and improves on it by proposing a more efficient Efficient-GhostNet, which achieves performance improvement while reducing the number of parameters through a local cross-channel interaction strategy, without dimensionality reduction. The improved Efficient-GhostNet is used to perform lip spatial feature extraction, and then the extracted features are inputted to the GRU network to obtain the temporal features of the lip sequences, and finally for prediction. We used Asian volunteers for the recording of the dataset in this paper, while also adopting data enhancement for the dataset, using the angle transformation of the dataset to deflect the recording process of the recorder by 15 degrees each to the left and right, in order to be able to enhance the robustness of the network and better reduce the influence of other factors, as well as to improve the generalization ability of the model so that the model can be more consistent with recognition scenarios in real life. Experiments prove that the improved Efficient-GhostNet + GRU model can achieve the purpose of reducing the number of parameters with comparable accuracy.

1. Introduction

Lip reading [1] is a human–computer interaction technology that has been widely used in various fields in recent years. Computer vision technology is responsible for continuously recognizing the speaker’s face information from images and videos, and further extracting the continuous lip change features when the speaker speaks, and then inputting the recognized mouth features into the lip-motion recognition model to recognize what the speaker says. Currently, lip reading technology is used in the fields of assisted automatic driving, assisted communication for hearing impaired people, and public safety [2,3]. The earliest research on lip reading originated from the 1950s, when Sumby, W.H. [1] first proposed that the series of lip movements during human speech could be used as a means of information acquisition, and thus the concept of lip reading was introduced, opening a new chapter of research in the field of lip reading. At present, lip reading systems are mainly divided into traditional lip reading systems and lip reading systems based on deep learning. Traditional lip reading is mainly composed of two parts: mouth feature extraction and lip movement recognition. The usual practice in mouth feature extraction is a pixel-based approach, which is to take the pixel values of the mouth region as the input visual information and extract the image features by discrete cosine transform (DCT) [4] and principal component analysis (PCA) [5]. For lip movement recognition, the Hidden Markov Model (HMM) [6] is most typically used, which considers lip movement sequences to be linear in a very short period of time and able to be represented by linear parametric models, and by connecting these linear models in series in time a Markov chain can be formed, and after obtaining the lip movement sequences and combining the known model parameters, the maximum probability recognition results. In recent years, with the continuous development of deep learning, it has started to gradually enter the field of lip movement. Oxford University, Google DeepMind, and the Canadian Institute for Advanced Research (CIFAR) jointly designed and studied the end-to-end lip reading model LipNet, which achieved 93.4% accuracy on the GRID corpus [7]. Jake Burton, David Frank, and others found that a CNN + LSTM based fusion network is the most featured input powerful recognition network [8]. Chen X., Du J. and others proposed a deep learning network DenseNet + resBiLSTM model for Mandarin sentence-level lips with good lip recognition, with a 13.91% improvement compared to LipNet [9].

In recent years, the recognition rate of lip reading models has been increasing and the network structure is growing, and as the network deepens and the amount of data increases, it poses a great challenge to the computing power as well as the storage capacity of the devices. Lightweight designs for models started to gradually become the focus of research today. In 2020, the Huawei team proposed GhostNet [10], which is inspired by the convolution process. Inspired by the similarity of generated feature maps, the Ghost module was introduced to generate the same number of feature maps using fewer parameters. Based on this, this paper adopts GhostNet as the backbone network for spatial feature extraction and improves on the network by introducing the Efficient Channel Attention module [11], which replaces the original Squeeze-and-Excitation module [12] to achieve performance improvement through a local cross-channel interaction strategy without dimensionality reduction. The purpose of reducing the number of parameters is also achieved. Since lip reading requires learning the back-and-forth association of lip-movement sequences, the improved Efficient-GhostNet is combined with the GRU [13] network to extract the temporal features of lip reading. At the same time, we carried out the work of creating our own dataset. Asian volunteers were selected for the dataset production for the recording of the dataset, and data enhancement was performed [14,15] by recording the dataset in three angles, as well as changing the brightness of the data and adding Gaussian noise to expand our dataset. The rest of the paper is as follows: in the second part, we present the network structure of the lip-recognition model. The third section contains the data of our proposed method, as well as the analysis and results. The fourth part contains the conclusion and future research directions.

2. Framework

In this section, we will introduce the research framework and the main research steps of this paper; the framework of the research is shown in Figure 1. First, we needed to perform video extraction with a fixed number of frames for the lip video, and then we used the Dilib model to obtain the face features, determine the location of the lip, and crop the obtained lip-motion sequence with a fixed size. Then, the obtained lip-movement sequences were inputted to the improved Efficient-GhostNet to obtain the feature vectors of the sequences, which would be outputted, and finally, we used the output of the previous part as the input of GRU to extract the temporal features of the lip-movement sequences using GRU, and then obtained the final prediction results by a SoftMax layer [16].

2.1. Image Feature Extraction-GhostNet

2.1.1. Ghost Module

With the continuous development of deep learning there are more and more various network architectures, but due to the increase in data volume, people started to focus on limited memory and computational resources to develop lightweight network models; therefore, there are ways of model compression such as network pruning [17,18,19], quantization [18] and knowledge distillation [20]. In addition to these methods, there are some effective network architectures that can also achieve lightweightedness: MobileNet [21,22], ShuffleNet [23] and DenseNet [24]. MobileNet uses depthwise and pointwise convolution to build a unit, and uses larger convolutional kernels to convolve the original layers. ShuffleNet achieves model lightness by changing the channel structure. In this paper, we use GhostNet, a lightweight model, to perform feature extraction for lip reading. The first step is to build the Ghost module, whose structure is shown in Figure 2, which uses some low-cost linear operations to generate many feature maps and is also able to obtain the information under the eigenfeatures map. Specifically, first use the ordinary 2D convolution to generate part of the feature map, and then use the feature map for depth-separable convolution to obtain another part of the feature map, and then the two are stacked output.

Assuming that the input data are $X\in R^\left(c\times h\times w\right)$, where c is the number of channels of the input data and h and w are the height and width of the input data, respectively, the algorithms of n feature maps generated by any convolutional layer can be expressed as follows:

$$Y=X*f+b$$

(1)

where * indicates the convolution operation, b is the bias term, $Y\in R^{h^{\prime }\times w^{\prime }\times n}$ is the output feature map of n channels, and $\in R^{c\times k\times k\times n}$ is the convolution filter of the layer. $h^{\prime }$ and $w^{\prime }$ are the height and width of the output data, respectively, and k × k is the kernel size of the convolution filter. In the convolution process, the expression of FLOPs is $n*h^’*w^’*c*k*k$. Since the number of filters n and the number of channels c are large, the calculation result of FLOPs will also be large.

In the base of Equation (1), it is obvious that the number of parameters to be optimized (in f and b) is determined by the dimensionality of the input and output feature maps. As shown in Figure 3, the number of output feature maps of the convolutional layers often contain a large number of redundancies, some of which are very similar. We believe that it is not necessary to generate all redundant feature maps one by one, because this consumes a large number of FLOPs and parameters, which is a waste of resources. Suppose that the output eigenmaps generated by some inexpensive transformations are “ghosts” of the eigenmaps, which are usually small in size and generated by the original convolution kernel. In particular, the m eigenmaps $Y^{\prime }\in R^{h^{\prime }\times w^{’}\times m}$ are generated using a basic convolution operation:

$$Y^{\prime }=X*f^{\prime }$$

(2)

where $f^{\prime }\in R^{c\times k\times k\times m}$ is the filter used, and to further obtain the desired n feature maps some ordinary convolution and depth-separable convolution operations are used to generate s ghost features on the present evidence feature map $Y^{\prime }$ according to the following function, which is as follows.

$$y_{ij}={\mathsf{\Phi }}_{i,j}\left(y_i^{’}\right), \forall i=1,\dots ,m, j=1,\dots ,s,$$

(3)

where $y_i^{’}$ denotes the i-th intrinsic feature mapping in $Y^{\prime }$, and ${\Phi }_{i,j}$ in the above equation is the j-th current operation used to generate the j-th ghost feature mapping $y_{ij}$, which means that $y_i^{’}$ can have one or even more ghost feature mappings ${\left\{y_{ij}\right\}}_j^s=1$, and the last ${\Phi }_{i,s}$ is used to save the identity mapping shown in the above Figure 2b. The identity mapping of the intrinsic feature mapping is shown in Figure 2b above. With the above equation, $n=m*s$ feature mappings, and $Y=y_{11},y_{12},\dots ,y_{ms}$ can be obtained as the output of the Ghost model in Figure 2. Since the linear operations in each channel are composed of both depth-separable convolution and ordinary convolution, the computational cost of this structure is much smaller than that of the channel computation composed of ordinary convolutions.

2.1.2. Ghost Bottleneck

The structure of Bottleneck with two step sizes is shown in Figure 4, which shows that it is made by stacking two Ghost modules and finally connecting a residual structure to form a Bottleneck with step size 1, which is intended to deepen the network structure. The Bottleneck with step size 2 also consists of two Ghost modules, with the difference being that a deep separable convolution operation with step size 2 is added in the middle, and the same residual structure as ResNet [25] is added to the output, and the output is $H\left(x\right)=F\left(x\right)+x$. The purpose is to effectively avoid the problem of gradient disappearance and performance degradation. After each layer, a batch normalization [26] process is performed and the ReLU function [27,28] is activated after the first bottleneck structure. The purpose of this structure is to deepen the network structure while changing the size of the feature layer.

2.1.3. Efficient-GhostNet

The improvement structure of GhostNet is shown in Figure 5 by building a network based on Ghost Bottleneck and improving GhostNet. The improved GhostNet mainly consists of a stack of Ghost bottleneck; firstly, the first layer is a feature layer with 16 channels composed of 3 × 3 convolution kernels in 2D, and after that it is stacked by Ghost bottleneck with different steps to make it deepen the number of channels and also change the size of the feature map. Finally, a global average pooling [29] and 2D convolution [30] are used to convert the features into 1280-dimensional feature vectors. Compared with the original GhostNet, the more efficient channel attention module is introduced into the model instead of squeeze-and-excitation, which can reduce the number of parameters while obtaining comparable accuracy. * in the figure indicates multiplication.

2.1.4. Efficient Channel Attention

Nowadays, channel attention mechanisms are widely used in convolutional neural networks; however, most current methods are designed to be overly complex in order to achieve better performance, which leads to an increase in the number of parameters and computational effort. In this paper, we introduce a more efficient ECA module, which can avoid the effect of dimensionality reduction and has the advantage of achieving performance improvement and reducing the number of parameters without dimensionality reduction through a local cross-channel interaction strategy; we also introduce adaptive selection of the convolutional kernel size, i.e., the size of the convolutional kernel will be changed flexibly according to the size of the number of channels, which aims to better achieve the inter-channel information exchange between channels. The structure is shown in Figure 6. The input feature map is first subjected to a global average pooling operation, and then a fast one-dimensional convolution operation of size K, where the size of K indicates the coverage of local cross-channel interactions, i.e., how many domains are involved in the attention prediction of a channel, and finally the obtained attention weights are multiplied with the corresponding elements of the original feature map to obtain the final output feature map.

Since the module needs to find the appropriate cross-channel interaction, it needs to determine the coverage of the interaction, that is, the value of k, which is the size of the sliding window. If the value of k needs to be manually adjusted for different structures, it will consume a relatively large amount of computational resources; in fact, the cross-channel coverage is positively correlated with the channel in a more reasonable way, that is, there is a mapping ϕ between the channel C and k:

$$C=\varphi \left(k\right)$$

(4)

The simplest mapping is a linear function, which is $\varphi =\gamma *k-b$. Since linear functions have limitations on the characterization of eigenrelations, and also the number of channels is usually set to the power of 2, the linear function is extended to a nonlinear function to introduce a possible solution:

$$C=\varphi \left(k\right)=2^{\gamma *k-b}$$

(5)

Then, given the channel dimension C, the convolution kernel size k can be determined as:

$$k=\Psi \left(C\right)={\left|\frac{lo{g}_{2}C}{\gamma }+\frac{b}{\gamma }\right|}_{odd}$$

(6)

where ${\left|t\right|}_{odd}$ denotes the odd number closest to t, γ and b are set to 2 and 1, respectively.

2.2. Gated Recurrent Unit (GRU)

For lip reading like this, the whole process of lip reading needs to be analyzed and it is necessary to obtain a time-dependent lip reading sequence. In deep learning, RNN networks are the most commonly used models to deal with a temporal correlation problem, but RNN models all suffer from a long-term dependency, as well as gradient disappearance and gradient explosion problems. In 1997, the LSTM [31] proposed by Hochreiter and Schmid-Huber solved the above problem very well by introducing the concept of gate structure to achieve feature retention and discard. The Gated Recurrent Unit (GRU) is a variant of the one mentioned above, which has the advantage of a simpler network structure, as shown in Figure 7.

To solve the problem of gradient disappearance and gradient explosion in RNN, an update gate and reset gate are used in GRU to replace the structure of input gate, output gate, and forget gate in LSTM, three gates which can judge and decide whether the information is passed as output or not. The update gate can control the information of the previous state to be brought to the current state, and the reset gate controls the information of the previous state to be written to the current candidate state, so that many pieces of sequential information can be retained and the irrelevant information can be deleted. The formula of each part of GRU network is as follows:

$$r_t=\sigma \left(W_{xr} x_t+W_{hr}{h}_{t-1}+b_r\right)$$

(7)

$$z_t=\sigma \left(W_{xz}{x}_t+W_{hz} h_{t-1}+b_z\right)$$

(8)

$$\widetilde{h_t}=tanh\left(W_{x\tilde{h}} x_t+W_{h\tilde{h}}(r_t\odot h_{t-1}\right)+b_{\tilde{h}})$$

(9)

$$h_t=z_t\odot \widetilde{h_t}+(1-z_t)\odot h_{t-1}$$

(10)

$$y_t=\sigma \left(W_0 · h_t\right)$$

(11)

where $W_{xz}$ and $W_{hz}$ are the respective corresponding weight vectors, $b_z$ is the bias vector, and the update gate is passed through the sigmoid function to obtain the gating threshold, $W_{x\tilde{h}}$, $W_{h\tilde{h}}$ denote the weight vectors corresponding to the lattice, $b_{\tilde{h}}$ is the bias vector, and the size of $r_t$ will determine how much of the past information is retained by the model. $z_t$ is the update gate signal mentioned above, indicating that the output signal is controlled by the update gate signal.

3. Results and Discussion

3.1. Dataset

At present, most of the lip reading datasets [32,33,34,35] are recorded by European and American people, and there are few datasets made for Asian people. Therefore, in this paper, we produced an Asian dataset for training, which is more consistent with the lip reading characteristics of Asian people, meaning the model has better recognition effect for Asian people.

Meanwhile, since the ultimate purpose of the system of lip reading is for it to be applied in, for example, the recognition of public places, the lip reading of disabled people, etc., this paper proposes data enhancement of the lip dataset based on different angles of the lip dataset, i.e., recording the lip dataset with frontal, 15 degrees to the left, and 15 degrees to the right perspectives. The purpose of doing so is as follows. Firstly, to be able to improve the robustness of the network and better reduce the influence of other factors, and secondly because such data enhancement is also a better reflection of real life, in which the videos in real life are usually unlikely to be completely positive videos.

Based on this, the dataset in this paper is our self-made lip-language video database, recorded by 10 speakers, containing five boys and five girls, the data contain ten numbers 0–9, and each number is repeated 20 times. In the process of recording, the camera and the head of the person are maintain relative stillness, respectively, three cameras from different angles are used at the same time to record the dataset, and each video is independently edited. Therefore, it is possible to obtain the front, left 15 degrees, right 15 degree video dataset, totaling 6000 samples, while adding Gaussian noise to these samples and changing the brightness to expand the samples to 12,000, with a resolution of 1920 × 1080 and a frame rate of 25 frames per second, about 1–2 s per video. After obtaining the dataset, we marked and intercepted all video mouth parts, and drew frames by ten frames for each video, and processed the processed dataset into 224 × 224 pixels. We divided the dataset into 80% as the training set and 20% as the test set, and part of the dataset is shown in Figure 8.

3.2. Results

To validate the effectiveness of Efficient-GhostNet, the lightweight network used in this paper, we compared its computational effort with Top-1 accuracy in the table. We used MobileNet, GhostNet, and our Efficient-GhostNet, and divided these models into three complexity levels. As can be seen from the

Table 1. Performance comparison of lightweight models under homemade datasets.

Models	FLOPs (M)	Top-1 Acc. (%)
MobileNetV2 0.3× GhostNet 0.5× Efficient-GhostNet 0.5×	41 42 40	54.7 55.6 56.2
MobileNetV2 0.6× GhostNet 1.0× Efficient-GhostNet 1.0×	141 141 138	65.6 66.4 67.8
MobileNetV2 1.0× GhostNet 1.3× Efficient-GhostNet 1.3×	299 266 251	72.4 74.7 76.3

, the Efficient-GhostNet network improves compared to the pre-improvement one for the limits of 40, 150, and 300 for MFLPs, respectively, and the larger FLOPs highlight the effectiveness of their models better.

The self-made dataset was inputted into the Efficient-GhostNet network to obtain spatial features, and the feature vectors were inputted into the GRU network to obtain temporal features, and the loss function and accuracy of each cycle were recorded, and the results are shown in Figure 9. The convergence of the model during training and the feasibility of the model were evaluated.

In Figure 9, when the period reaches 70, the convergence rate starts to slow down and gradually stabilizes and the loss function and accuracy rate both tend to stabilize, indicating that the model has reached the optimum and the accuracy rate of the proposed network is 88.8%.

In this paper, in order to evaluate the performance of the lip-reading system, we conducted an experimental comparison of some current mainstream models and the results are shown in

Table 2. Performance comparison of each model with the homemade dataset.

Models	Params./M	Time/s	Acc./%
VGG16 + LSTM	139	16.3	91.4
Resnet50 + LSTM	25	5.7	87.8
MobileNet + GRU	5.4	3.1	88.6
GhostNet + GRU	5.1	2.9	88.7
Efficient-GhostNet + GRU	3.8	2.2	88.8

. It can be seen that the model proposed can perform the prediction task of lip-reading well, while the model we used can reduce the number of parameters of the network with comparable accuracy and also reduce the prediction time of lip-reading, so our proposed model is effective.

We tested the accuracy of the two models individually for ten words both before and after the improvement, and the results are shown in Figure 10. The experimental results can show that the improved model has better recognition accuracy. In addition, the model has better recognition for “two” and “four”, which is due to the relatively simple pronunciation of “two” and “four” and the more obvious lip movements; the recognition of “zero” is worse, which is due to the relatively complex pronunciation of “zero”, the large variation of mouth pronunciation and individual differences, and also its reliance on the tongue to control.

In order to test the recognition ability of our proposed model for phrases, we used the existing dataset GRID corpus for our experiments and also conducted experimental comparisons with some current models; the results are shown in the following

Table 3. Performance comparison of each model with the GRID dataset.

MODEL	Recognition Task	Rec. Rate/%
HOG + SVM [36]	Phrases	71.2
CNN [37]	Phrases	64.8
Feed-forward + LSTM [38]	Phrases	84.7
3D-CNN + highway + Bi-GRU + attention [39]	Phrases	97.1
STCNN [40]	Phrases	95.5
MobileNet + GRU	Phrases	92.8
Efficient-GhostNet + GRU	Phrases	94.1

.

Through our experiments, we can see that our proposed model can accurately identify the results and perform better than some of the previously proposed models in the process of phrase recognition.

4. Conclusions

In this paper, we propose an optimized lightweight network based on the improved GhostNet network by replacing the original SE module with the ECA module to achieve better performance and reduce the number of parameters through a local cross-channel interaction strategy without dimensionality reduction, and since lip reading also needs to consider the temporal features of picture frames, the GRU network with fewer parameters is used. The GRU network with fewer parameters is used for the extraction of temporal features. The lip-synthesis dataset in this paper uses 10 speakers, including five boys and five girls, and the data contain ten numbers from 0 to 9. The experiments show that the proposed model reduces the prediction time by 24.1% and the amount of model parameters by 25.4% under the condition that the accuracy is guaranteed to be comparable, which validates the effectiveness of the proposed model in this paper. In order to demonstrate the recognition ability of the proposed model for phrases, we conducted a set of comparison experiments and the results show that our model also has a better recognition ability for phrases. In future research, we will expand our dataset to enhance the generality of the lip reading model. Since our recognition study was conducted in chronological order rather than for specific types of lip movements at specific times, the real-time performance is not good and future research will address how to improve the real-time performance of the model for the purpose of deployment to mobile devices.