Facial Skincare Products’ Recommendation with Computer Vision Technologies

Abstract

Acne is a skin issue that plagues many young people and adults. Even if it is cured, it leaves acne spots or acne scars, which drives many individuals to use skincare products or undertake medical treatment. On the contrary, the use of inappropriate skincare products can exacerbate the condition of the skin. In view of this, this work proposes the use of computer vision (CV) technology to realize a new business model of facial skincare products. The overall framework is composed of a finger vein identification system, skincare products’ recommendation system, and electronic payment system. A finger vein identification system is used as identity verification and personalized service. A skincare products’ recommendation system provides consumers with professional skin analysis through skin type classification and acne detection to recommend skincare products that finally improve skin issues of consumers. An electronic payment system provides a variety of checkout methods, and the system will check out by finger-vein connections according to membership information. Experimental results showed that the equal error rate (EER) comparison of the FV-USM public database on the finger-vein system was the lowest and the response time was the shortest. Additionally, the comparison of the skin type classification accuracy was the highest.

1. Introduction

Since most people have not received professional medical knowledge training, if they use inappropriate products at will, they will easily become self-defeating and make the skin condition more serious, which will cost much money and time to remedy, and even lead to repeated skin issues such as acne. Acne is also called pimples, and the main cause of acne is excessive sebum secretion, which tends to occur in places with more sebaceous glands. According to the global acne market report for 2016–2026 [1], acne is one of the most common diseases for which dermatologists provide treatment assistance and more than 90% of the world’s population suffer from acne symptoms.

The sales’ channels of skincare products are mainly physical stores, which can be further divided into open drugstores and skincare products’ counters. Open drugstores have a leisure and free shopping environment, but the advice obtained by consulting the store staff may not be rigorous, and skincare products’ counters are prone to frequent sales’ promotion by the clerks, which reduces the clients’ willingness to purchase. Online shopping products often contain defects and counterfeit products, which are prone to the risk of harm. The recent prevalence of unmanned stores has many similarities with the above factors. The unmanned automated business model can effectively reduce labor costs, cash exchanges, and contact between cashiers and customers to maintain hygiene and safety and provide better personalized services. According to the power of artificial intelligence (AI) for cosmetics brands [2], a personalized recommendation system has become a future trend, which not only stimulates more consumption power but also strengthens the relationship with consumers. In view of this, this work proposes to realize a new business model of facial skincare products with computer vision (CV) technology. Based on the concept of unmanned stores, a fast and contactless finger-vein identification system was designed to allow consumers to instantly verify their membership. This identification system can also be used as a checkout feature to save time for taking out wallets and credit cards, and thereby improves shopping efficiency by saving time in queuing. At the same time, it can also record the purchasing habits of consumers to provide more consumer-friendly commodities of preferences and needs.

The skincare products’ recommendation system in this work adopts machine learning (ML) and deep learning (DL) methods. Compared with traditional CV technologies, it improves the issue of environmental constraints and low recognition rate. In recent years, many studies have been adopting DL methods to discuss skin quality as well. As the symptoms of various skin diseases are similar, it is difficult for doctors to distinguish with naked eyes. Junayed et al. [3] used a convolutional neural network (CNN) to develop a deep residual neural network (DRNN), which can identify five types of acne to assist doctors in diagnosis. As skin cancer is steadily increasing around the world, preventive medicine is also becoming increasingly important. Vesal et al. [4] used U-Net [5] to identify skin lesions before skin cancer, and the experimental results reached 93% on the sensitivity (SE) target. Hameed et al. [6] used CNN and support vector machine (SVM) [7] to identify skin diseases with an accuracy rate of 90%. The above studies showed that the features obtained by CNN can enhance the classification effect of a variety of skin diseases. Due to the increasing types of skin diseases, there have been more and more studies discussing skin pigmented lesions in recent years. Goyal et al. [8] identified skin melanoma by a region-based convolutional neural network (R-CNN) [9], and the accuracy rate reached 98%. Adegun et al. [10] used a deep convolutional neural network (DCNN) and the sub-network in the encoder/decoder architecture to identify pigmented tumors on the skin surface, and the accuracy rate reached 96%. As mentioned above, using DL methods to identify the state of skin quality is not only less restricted by a specific environment, but also easier for us to recognize ambiguous skin features, and the accuracy is also sufficient for the practical stage, as shown in

Table 1. Comparison of various methods of skincare products’ recommendation system.

Method	Environmental Impact	Accuracy
Traditional CV	High	Low
ML & DL	Low	High

. Therefore, we proposed to use multi-feature classification by the ML method and pixel-wise segmentation by the DL method.

One of the risks of traditional identity identification methods is that the clients may forget their own passwords. Therefore, a variety of biometric methods [11] have been developed to use physiological features that do not need to be carried to reduce unnecessary risks. The comparison of identification methods is shown in

Table 2. The comparison of biometrics’ identification method.

Method	Defect
Password	Risk of forgetting, loss, and theft
Face recognition [12,13]	Susceptible to masks, face movement, and light sources for recognition
Finger-print recognition [14]	Susceptible to oil or sweat for recognition
Iris recognition [16]	Uncomfortable for long-term use
Finger-vein recognition [17]	None

. In the method of biometric recognition, face recognition [12,13] is susceptible to masks, face movement, and light sources in the process of capturing images, which leads to illegible results. Finger-print [14] and palmprint recognition [15] uses touch input to capture texture, so oil or sweat on hands may affect its recognition. Besides, fingerprints are human external feature and they are easy to be copied and obtained by people with bad intentions. Although iris recognition [16] has a high accuracy, long-term use will cause discomfort. The biological features of the above recognition methods are all outside of the body surface, and there are still some unreliable external factors in use. Finger-vein identification [17] is based on the biological features of the body surface, and it requires a living body to be recognized. Therefore, it is not easy to be copied or abrased. At the same time, its contactless nature avoids safety of health issues, and it is not easy to interfere with recognition in face of dirt and oil stains. As a result of the above reasons, we decided to use finger-vein identification as a security system in this work.

The remainder of this paper is organized as follows. We describe the proposed method in Section 2, present experimental results in Section 3, and draw conclusions in Section 4.

2. Proposed Method

The main architecture of this work is composed of a finger-vein identification system, a skincare products’ recommendation system, and an electronic payment system, as shown in Figure 1. The first subsystem is the finger-vein recognition system, which is used as identity verification and a personalized service. The second subsystem is the skincare products’ recommendation system, which provides consumers with professional skin analysis through skin type classification and acne detection to improve skin issues of consumers and recommends skincare products finally. The third subsystem is the electronic payment system, which provides a variety of checkout methods, and the system will check out by finger-vein connections according to membership information.

2.1. Finger-Vein Identification System

When the finger is irradiated with near-infrared light (NIR), the hemoglobin in the venous red blood cells absorbs the near-infrared light and presents a shadow. Based on this principle, the near-infrared vein image can be obtained. Finger-vein identification method has the characteristics of being contactless, fast, and stable as well as satisfying the conditions of hygiene, real time, and correctness simultaneously, making users have a better experience. The flowchart of the finger-vein recognition system is shown in Figure 2. The system first uses 13/7 discrete wavelet transform (DWT) [18] to reduce the dimensionality of the infrared image captured by the raspberry pi camera, retaining the features while shrinking the image, thereby improving computation speed. Then, parametric-oriented histogram equalization (POHE) [19] is used for image enhancement to solve the issue of difficult extraction of features due to light sources, making subsequent features easier to extract. Finally, it uses Gabor filters [20,21] to extract vein features to input the data into the SVM model for training. The system can use a pre-trained model to efficiently and quickly complete membership registration or identification.

2.1.1. Image Pre-Processing

This system uses two-dimensional masked 13/7 DWT in the image pre-processing step to extract the converted Low-Low (LL) band coefficient information [22,23] for subsequent use. DWT uses a double-layer, low-pass filter and has the characteristic of energy concentration. It uses adjacent pixels to perform a two-dimensional convolution operation. The image input from an infrared camera can preserve more features to a low-dimension image after convolution operation, and dimensionality-reduced images also greatly improve the effectiveness of subsequent feature extraction work.

2.1.2. Contrast Enhancement

Histogram equalization (HE) is a method to enhance the contrast of local images. HE also includes local histogram equalization (LHE) and global histogram equalization (GHE). In LHE, each pixel is computed separately with a different conversion function; so, it has a better image enhancement effect. However, the computational performance is not suitable for real-time recognition systems. Although GHE has higher computational performance, its enhancement effect is not ideal.

In view of this, POHE used in this system not only achieves the effect of LHE contrast enhancement, but also consumes a smaller amount of computing power. The method adopts the concept of integral image and simplifies the transformation function of LHE. First, calculate integral image I_i,j¹ and I_i,j^2. The calculation method is shown in Equation (1). Then, each pixel is independently enhanced, and finally the enhancement method is shown in Equations (2) and (3) [19].

$$I_{i,j}^{\left(k\right)}={\displaystyle \sum }_{m=0}^i{\displaystyle \sum }_{n=0}^j{x}_{m,n}^k=x_{i,j}^k+I_{i-1,j}^{\left(k\right)}+I_{i,j-1}^{\left(k\right)}-I_{i-1,j-1}^{\left(k\right)}$$

(1)

where $I_{i,j}^{\left(k\right)}$ can be denoted as integral image.

$$f\left(x_{i,j}\right)=L\times c\left(x_{i,j}\right)$$

(2)

where $c\left(x_{i,j}\right)$ denotes the cumulative distribution function, L is the distribution of original image, and $f\left(x_{i,j}\right)$ is the resulting pixel after equalization.

$$c_{Gaussian}\left(x_{i,j}\right)=\frac{1}{2}\left[1+erf\left(\frac{x_{i,j}-{\mu }_{i,j}^{est}}{\sqrt{2}\sigma }\right)\right]$$

(3)

where ${\mu }_{i,j}^{est}$ denotes the mean in a kernel and $\sigma$ is sigma in a kernel for the transformation function of POHE.

2.1.3. Feature Extraction and SVM Training

The Gabor wavelet filter method is very similar to the visual stimulus response in the human visual system. It has the characteristics of achieving the best localization in the spatial domain and the frequency domain, and it is also sensitive to the edges of the image. It can be used at different scales and directions of the frequency domain to extract features and, thus, is suitable for extracting vein pattern features. The method first divides the signal into many small time intervals, and analyzes the time intervals by Fourier transform [24]. The definition of the two-dimensional Gabor filter [25] is shown in Equation (4), and the results of image pre-processing and feature extraction are shown in Figure 3.

$$\mathrm{G}\left(x, y, \theta , \mu , \sigma \right)=\frac{1}{{2\pi \sigma }^2}exp\left\{-\frac{x^2{+y}^2}{2\sigma }\right\}exp\left\{2\pi i\mu \right(xcos\theta +ysin\theta \left)\right\}$$

(4)

where $\theta$ represents the orientation of the normal to the parallel stripes of a Gabor function, $\mu$ is mean, and$\sigma$ is the standard deviation of the Gaussian envelope.

After the step of feature extraction, we used the Otsu method [26] for image binarization, and then input vein features into SVM model for training. The system may operate a pre-trained model efficiently and inference for identification to complete membership registration or electronic payment quickly.

2.1.4. Identification

SVM is a well-developed, supervised learning method. After SVM prediction is performed, we can obtain experimental data results of a false acceptance rate (FAR) and false rejection rate (FRR) with different thresholds. The data points are plotted as the linear graph is called the receiver operating characteristic curve (ROC). When the curves of FAR and FRR are equal at a certain point, the data at that point are the equal error rate (EER). EER is a target used to evaluate the effect of finger-vein identification. The feature score of LINEAR core function, which is adopted by this system, is higher than that of other methods. We can, therefore, infer that the identification result operated by LINEAR core function is better.

2.2. Skincare Products Recommendation System

The skincare products’ recommendation system is divided into two parts: skin type classification and acne detection. It is implemented by combining multi-feature processing with the ML classification technology and the DL semantic segmentation [27] technology. Finally, according to the analysis of skin type and acne status, facial skincare products are recommended to consumers. The flowchart of the skincare products’ recommendation system is shown in Figure 4.

2.2.1. Image Capture and Extract Region of Interest (ROI)

This system uses Logitech’s C310 camera to capture facial images, and then uses the cross-platform CV library (Open Source Computer Visual Library, OpenCV) facial recognition model to detect facial regions in the image. Facemark [28] is then used to identify 68 features of the face, and, at the same time, marks the output on the display screen for the user to confirm whether the facial contour has been captured correctly. Then, it uses the four areas including left and right cheeks, forehead, and chin as ROI image, which is prone to acne and oil, and finally outputs the ROI captured image to skin type classification and acne detection.

2.2.2. Skin Type Classification

In the skin type classification part, the input ROI image is converted into a grayscale image first, and then a two-level Haar [29] DWT is performed on the grayscale image to obtain seven sub-bands. The reason for choosing Haar as the wavelet transform is that Haar wavelet is orthogonal and double-orthogonal simultaneously. Besides, the spatial resolution is low and the quality is high [30]. According to the seven sub-bands after Haar wavelet transformation, the system uses the LL band coefficients in the two-level transformation [31] to calculate them in a histogram, and then uses the cumulative number of the four selected intervals [32] in the histogram as parameters. The selected interval includes [110, 120), [120, 130), [180, 190), and [190, 200), and these four parameters are used in the classification of the first stage of skin type classification.

The second stage of classification is to use the ROI image transformed by Haar wavelet to calculate texture features. According to the distance and angle between two pixels in the image, the cumulative number with the same gray level is calculated and implemented by a co-occurrence matrix. Then, based on the co-occurrence matrix [32], contrast is calculated, as shown in Equation (5). Inverse difference moment (IDM) is calculated, as shown in Equation (6). Entropy is calculated, as shown in Equation (7). They are finally used for the second stage of classification.

$$Constrast={\displaystyle \sum _{n=0}^{Eg-1}{n}^2}\times \left\{{\displaystyle \sum _{i=1}^{Eg}{\displaystyle \sum _{j=1}^{Eg}p(i,j|d,0°)}}\right\}\left|i+j=1\right|$$

(5)

$$\mathrm{IDM}={\displaystyle \sum _i{\displaystyle \sum _j\frac{1}{1+{(i-j)}^2}\times p(i,j | d,0°)}}$$

(6)

$$Entropy=-{\displaystyle \sum _i{\displaystyle \sum _{j}p(i,j | d,0°)\times \log }}\{p(i,j | d,0°)\}$$

(7)

where d is the distance of two pixel, and Eg is the value of maximum grayscale value minus minimum grayscale value.

In the skin type classification part, the linear SVM is used to classify the skin type. According to the four sub-band parameters after statistics as the first type of feature, the oily skin and non-oily skin are first distinguished, and then the texture feature is used as the second type of feature to distinguish neutral skin and dry skin from non-oily skin. The advantage of SVM is that it can classify from explicit features and only takes a small number of samples to train the classification model; so, this model is used as the basis for classification.

2.2.3. Acne Detection

In the part of acne detection, the captured ROI image is input to the DeepLab-v3+ [33] model for pixel-level prediction. First, the encoder adopts the atrous spatial pyramid pooling (ASPP) architecture and calculates the multi-dimensional features with different convolution rates. Then, it performs bilinear upsampling on the acquired features and concatenates them with bottom layer features, which have the same spatial resolution. After concatenation, convolution is used to refine the features, and then a bilinear upsampling is performed. In order to reduce the computational complexity, this system separates the standard convolution into a depth separable convolution and performs spatial convolution independently for each layer of input channel; pointwise convolution is used for combining the output of convolution.

In the part of acne statistics, we analyze the segmented images and calculate the proportion of the acne area occupying the ROI image area. Finally, the user’s acne severity is determined according to the calculated acne ratio value and the resulting image is divided into areas of mild, moderate, and serious acne. The higher the ratio, the more serious the issue is. We classify the severity of acne according to the proportion of acne occupying the ROI image: 25% is mild, 50% is moderate, and more than 75% is severe. Because there is no clinically correct acne detection label, we self-label the acne data on the Internet and test it to get an 80% segmentation accuracy.

2.2.4. User Interface

In this work, facial care products were used as the products of the vending machine, as shown in Figure 5. When the user operates the vending machine, the lens of the vending machine will detect whether there is a user, and then the voice system will inform the user of the operation steps. The user stands at a specified distance and aligns the recognition display screen through the voice system, and the system can perform skin type classification and acne detection. Finally, the system analyzes the user’s facial status based on the classification and detection results, and provides recommendations for skincare products. At this time, the user can choose whether to buy skincare products. If the user would like to purchase products, the user only needs to click on the screen directly to purchase. This facial skincare product vending machine provides electronic payment and checkout features. After the checkout, the skincare products will be placed in the pick-up port for the purchaser to pick up. This composite system not only avoids the cost of store rent, but also make it relatively more convenient for management and replenishment. It can also cooperate with major skincare product brands to put on vending machines. Its future development is worth looking forward to.

The user interface of skincare products’ recommendation system is shown in Figure 6. The user can determine whether the facial contour is accurately captured according to the screen, then press the recommendation button, as shown in Figure 6a, and view the analytical results and recommended results, as shown in Figure 6b.

The establishment of a skincare products’ database is an indispensable part of this work. The database of this work contains 110 skincare products, including mild-oily, mild-neutral, mild-dry, moderate-oily, moderate-neutral, moderate-dry, serious-oily, serious-neutral, and serious-dry of nine types of skincare products. Due to the large number of items, only one skincare product is listed, as shown in

Table 3. List of skin care products.

Item	Mild	Moderate	Serious
Oily	NRK Rose Astringent	PAULA’s CHOICE	PAULA’s CHOICE
			Salicylic Acid
	Lotion	Pore-Reducing Toner	Blemish-prone Skin
			Extra Strength
Neutral	SHISEIDO Revital	SHISEIDO Vital-Perfection White Revitalizing Emulsion	PAULA’s CHOICE
			Salicylic Acid
	Lotion Ex		Blemish-prone Skin
Dry	ORBIS Clear Moisture	PAULA’s CHOICE CALM Soothing Toner	PAULA’s CHOICE 2% BHA Gel Exfoliant

.

2.3. Electronic Payment System

The electronic payment system in this work links the finger-vein identification system and the checkout page through the cloud database serialization technology. The relationship graph of the electronic payment system is shown in Figure 7.

2.3.1. Cloud Database Construction

Setting up a cloud database with XAMPP, this database stores the identity data registered by each user, and corresponds to the user’s first scan of the vein record as a mechanism to confirm the identity. The database verification method uses encryption and comparison, and the database also records the purchase data of users in recent times, providing an analysis of the willingness to purchase various products and the user’s purchasing habits.

2.3.2. Identity Verification and Electronic Payment Mechanism

In the part of the identity verification mechanism, the system uses the developed finger-vein identification system for verification, compares the user’s identity according to the cloud database, and, after confirming the identity, the skincare product system can be used. It can also be used according to the cloud database, providing recommendations for items that have been purchased. In the part of the electronic payment mechanism, we use cross-border e-commerce payment API to redirect to the payment page, on which consumers can finally pay for purchase.

This system only needs to perform identity verification through finger-vein identification, and then consumers can use credit cards to make purchases. It does not require personal mobile devices or chip cards, such as barcodes, near-field communication (NFC), etc.

2.3.3. User Interface

As shown in Figure 8a,b, the user must fill in the payment method and other information and complete the operation of photographing finger-veins. After the first registration, if the payment method inquiry is not turned on in future purchases, payment can be made through vein authentication without filling in any information. The payment method can be changed through the membership system.

3. Experimental Results

3.1. Accuracy of Skin Type Classification

The accuracy of skin type classification in this work is shown in

Table 4. Accuracy of skin type classification.

Method	Oily	Neutral	Dry
This work	83%	94%	77%
J. Lee et al. [32]	82%	85%	83%

Through cooperation with dermatologists, systematic tests were carried out from the perspective of the medical profession, and the doctors’ professional advice on various types of skin was carried out. The skin type classification accuracy rate is the comparison statistics between the results of this work and the results diagnosed by the doctors. In this work, we used 20 people’s skin type data diagnosed by professional dermatologists for training and 15 people’s skin type data for testing. They were all between 18 and 23 years old.

3.2. The Security of Finger-Vein Identification System

EER is an important indicator of security. There are two possible errors in the identification system, FRR and FAR. FRR represents the rate at which the system should pass the result but not pass the recognition, while FAR represents the rate at which the system should not pass the result but pass the recognition. The curve formed by FAR and FRR will intersect at a point, which is called EER, which is the point at which the two recognition error rates are the same. At this point of time, the sum of the two recognition error values is at the minimum value. When the similarity is set to EER, there will be the most balanced performance; so, in general, the value of EER will be used as an index to identify the performance of the system. The smaller the value of EER, the better the performance of the identification system. This work is optimized by adjusting the parameters of SVM.

Table 5. Comparison of EER.

Method	EER
	Our Database	FV-USM
N. Miura et al. [35]	2.5%	7.2%
E-C. Lee et al. [36]	3.13%	10.2%
Z. Liu et al. [37]	1.51%	5.2%
This work	1.8%	2.8%

shows the EER comparison of each method. Public database FV-USM [34] and the self-created dataset were used in this work. Table 5 shows that the FV-USM database had high security, and the self-created dataset had the second highest security in this work.

3.3. The Response Time of Finger-Vein Identification System

Table 6. Comparison of response time.

Method	Image	Extraction	Recognition	Total
N. Miura et al. [35]	320 × 240	6.74 s	1.64 s	8.38 s
Wang et al. [38]		3.52 s	1.43 s	4.95 s
Z. Liu et al. [37]		1.63 s	1.27 s	2.90 s
This work		0.59 s	0.201 s	0.79 s

is a comparison between the response time of this work and that of other methods in identification. According to the results of the comparison, in the same development environment, it can be seen from Table 6 that the response time of this work was reduced by more than 4 times than other methods.

4. Conclusions

This work proposes the use of CV technology to realize a new business model of facial skincare products. The overall framework is composed of a finger-vein identification system, a skincare products’ recommendation system, and an electronic payment system. The difference between this work and the previous literature is that this work proposes a composite system architecture, based on the concept of unmanned stores, to design a fast and contactless finger-vein recognition system. In addition to facial analysis of skin type classification and acne detection, recommending for skincare products is also provided. The experimental results include accuracy of skin type classification, EER comparison of each method of finger-vein identification system, and the comparison of response time. The EER comparison of the FV-USM public database on the vein system is the lowest and the response time is the shortest. Additionally, the comparison of the skin type classification accuracy is the highest.