Face Detection Based on Skin Color Segmentation Using Fuzzy Entropy

Abstract

Face detection is the first step of any automated face recognition system. One of the most popular approaches to detect faces in color images is using a skin color segmentation scheme, which in many cases needs a proper representation of color spaces to interpret image information. In this paper, we propose a fuzzy system for detecting skin in color images, so that each color tone is assumed to be a fuzzy set. The Red, Green, and Blue (RGB), the Hue, Saturation and Value (HSV), and the YCbCr (where Y is the luminance and Cb,Cr are the chroma components) color systems are used for the development of our fuzzy design. Thus, a fuzzy three-partition entropy approach is used to calculate all of the parameters needed for the fuzzy systems, and then, a face detection method is also developed to validate the segmentation results. The results of the experiments show a correct skin detection rate between 94% and 96% for our fuzzy segmentation methods, with a false positive rate of about 0.5% in all cases. Furthermore, the average correct face detection rate is above 93%, and even when working with heterogeneous backgrounds and different light conditions, it achieves almost 88% correct detections. Thus, our method leads to accurate face detection results with low false positive and false negative rates.

1. Introduction

In recent years, there has been intensive research to develop complex security systems involving biometric features. In general, a biometric system is a pattern recognition system that makes a personal identification by determining the authenticity of a specific physiological or behavioral characteristic possessed by the user. One of the most popular biometric techniques to identify users is face recognition. In general, an automatic face recognition system is based on extracting a set of features from the user’s face, either geometric characteristics or some information of textures and shapes of the different elements that constitute a human face. Before a recognition algorithm is applied, faces must have been detected in an image using some detection method. Therefore, detecting a face is the first step for the recognition process to be efficiently completed.

Early face detection algorithms, which used heuristic and anthropometric techniques, appeared in the 1970s [1]. These methods often failed, since they were very sensitive to changes in lighting conditions or image scale; moreover, they only detected a face in well-defined environments, such as high-contrast images with a white background. In order to detect faces in complex environments, the algorithm had to be totally re-designed.

New security requirements and some technology advances, such as the massive popularization of multimedia applications in telecommunication systems, allowed face detection algorithms to be re-visited, and newer and more efficient systems have been developed in the last few years, including low-level analysis, active shape models, feature analysis, etc. [2,3].

There are many approaches to face detection, such as knowledge- and appearance-based methods, feature-invariant algorithms or template-matching techniques; for further details, see [2,4,5]. In this work, we shall focus on a pre-processing stage for detecting faces in color images. To do this, we assume that the problem consists of detecting skin regions in the image, and then, the segmented skin clusters will be further processed to validate that they belong to face regions.

Recently, skin color detection has been intensively researched in computer vision, becoming very popular as a powerful cue for detecting, segmenting and tracking faces and hands [6,7,8]. One of the main requirements for a face detection system is that it can handle the variations that occur in natural image sequences. These variations can be in the image conditions (size, location, lighting, orientation, point of view), in the face appearance (glasses, beard, mustache, makeup, expression) or in the image contents (complexity of the background, occlusion by other objects, number of faces). Human skin-color is one of the most robust face features regarding the variations mentioned above and is used as the core for the color segmentation model developed in this paper.

The main goal of skin color modeling is to extract a set of decision rules that efficiently find the pixels in an image that belong to human skin and eliminate those that represent any other elements in a scene. There are several possibilities to classify these methods; for instance, in [9], two categories for modeling skin color distributions were defined: physical-based approaches and statistical approaches. The first one describe the formation of images; e.g., evaluating how the appearance of skin color varies according to light changes in a scene [10]. Focusing on the statistical approaches, in [2], three categories for modeling skin color distributions were defined: explicitly-defined skin region, non-parametric skin distribution and parametric skin distribution models, either adaptive or non-adaptive. We will use this classification as a basis for our review.

In general, the first method to develop a skin classifier consists of defining the bounding limits of the regions corresponding to a color belonging to the skin, by means of some numerical (and often empirical) rules. In other terms, we define a set of threshold values in order to classify each pixel as “skin” or “non-skin” by evaluating the color for every pixel in a new input image. It is clear that the efficiency of the color segmentation stage highly depends on the selected color space. Thus, RGB is the most commonly-used basis for color descriptions, though it has the negative aspect that each of the coordinates (red, violet and blue) is subject to luminance effects from the lighting intensity of the environment, which may result in a misclassification of skin and non-skin pixels. That is to say, color spaces with separated luminance and chrominance components, like HSV or YCbCr, seem to be more appropriate for face detection; thus, the HSV color space is widely used for skin color detection, due to its relation to how humans perceive colors. On the other hand, YCbCr is useful in compression applications, but the specification of colors is somewhat unintuitive. Many other proposals using different color spaces can be found in the literature [11,12,13]. The main advantage of using these techniques is the simplicity of the skin detection rules, which permit a fast classification of pixels. The major drawback is the difficulty in finding both robust color spaces and reliable decision rules for a convenient skin detection. Nevertheless, as shown in [4], if an optimum skin detector is designed for every color space, their performance will be almost the same.

On the other hand, there is a set of techniques that estimate the distribution of skin color by means of a training phase, without deriving an explicit model. These methods are often referred to as non-parametric skin models. Generally, they create a skin probability map (SPM) [14,15], which consists of assigning a probability to each point in different color spaces. The illumination conditions are often unknown in an arbitrary image, so the variation in skin colors is much less constrained than in the controlled setup. Again, the selection of an appropriate color space is essential to achieve optimal results. Many non-parametric methods have been proposed for skin color modeling: histogram lookup tables (LUT) [16,17], SPMs using Bayesian theory [18,19] or neural networks [20,21], among others.

Non-parametric methods achieve low computation times for both training and classifying skin pixels, and they do not depend on the skin distribution shape or on the selection of the color space, either. However, they require a huge storage capacity, and the training must be performed with a representative dataset.

Finally, the searching for a more compact skin model representation for certain applications, as well as the need of having a generalization of the training data have resulted in the development of the so-called parametric skin distribution models. These algorithms consist of building a decision rule where an explicit skin distribution model is obtained, using different features, such as mean values, covariance matrices, mixtures of Gaussians or elliptic boundary models [22,23]. Parametric methods can also be fast, as they make it possible to interpolate data and to generalize the training phase for incomplete data. In spite of this, their performance highly depends on the skin distribution shape, and many parametric models ignore the static non-skin color. Subsequently, this fact leads to high false positive rates compared with non-parametric methods.

Many recent works [2,4,24] are based on the underlying idea of representing the skin color in an optimal color space (such as RGB, HSV or YCbCr) by means of the so-called skin cluster. Color information is an efficient tool for identifying facial areas if the skin color model can be properly adapted for different lighting environments. As a consequence, our proposal consists of taking the main advantages of explicitly-defined skin models, i.e., the simplicity of their decision rules, and combining them with the main benefits of non-parametric skin models, that is to say, low computation times for training and classifying. To do this, our method includes a fuzzy model for detecting skin in color images. The completed experiments (see

Table 1. A summary of the face detection results with some color skin segmentation algorithms. SPM, skin probability map (Bold text represents the system with better results).

Algorithm	Incorrect (%)	Correct (%)
Explicit RGB rules	12.03	87.97
SPM HSV	22.29	77.71
Our proposal (in the RGB system)	6.47	93.53

, where bold text represents the system with better results) show that our method gives accurate results and that using a fuzzy approach to skin color detection allows the system to obtain a correctly-detected face even when working with uncontrolled environments and different light conditions. This model is explained in the following section.

This paper is organized as follows: In Section 2, we will define some terms first, and afterwards, the process of designing the fuzzy systems for detecting skin in RGB, HSV and YCbCr color spaces will be widely described. The face detection algorithm is also outlined at the end of Section 2. Then, the experiments on (i) segmenting skin and (ii) detecting faces are explained and discussed in Section 3, where a comparison with some explicitly-defined skin segmentation algorithms and some SPM methods is also included. Finally, we shall conclude with a discussion of the results in Section 4, and some remarks on our work are summarized in Section 5.

2. Development of a Fuzzy Face Detection System

As described before, automated face detection mainly consists of identifying the regions of the image containing a face or a set of faces. In this section, we will first develop a fuzzy system to extract skin from color images, taking into account different color spaces. Clearly, not all of the detected skin regions would contain faces: some of them will belong to other skin areas, such as the neck, hands, etc., while others will correspond to objects with similar colors to the skin. Thus, a method to detect where faces are in the segmented images must be developed, as well. Figure 1 shows an overview of our face detection algorithm.

Our approach involves two phases: in the first one, the system is trained. Thus, all of the parameters needed for the definition of the fuzzy rules for skin color detection are estimated here. Then, the second step implies an extensive set of experiments with different databases for validating the proposed model. In order to achieve our goals, let us define some terms first.

2.1. Definitions

Definition 1.

A region R in an image I is said to be connexif there is a neighborhood relation between any pixel $p\in R$.

Definition 2.

A connex region $R_F$ in an image I contains a face F when the set of the main features of F, $\varphi$, is included in $R_F$.

The extracted features $\varphi$ may include information about geometrical features (position of the eyes, mouth, etc.), texture information, color spaces, etc. In our case, $\varphi$ will consider the face area, shape and skin color detection.

It is clear that any image I may contain several faces and that a connex region $R_i$ may not contain any face. Let us assume that a face $F_i$ is defined as the i-th rectangular connex region $R_{F_i}$ containing the distinctive features of a face, $\varphi$, and that is limited by the pixels $p_{1_i},p_{2_i}$, located at the top and bottom vertexes of the region.

In this work, we shall focus on face detection in color images. A color space is a method by which color can be specified, created and visualized.

Definition 3.

A color vector $c\in C$, where C is a color space, of a pixel p is defined as an l-tuple of color components $c\left(p\right)=\{c_1\left(p\right),c_2\left(p\right),\dots ,c_l\left(p\right)\}$, where $c_i\left(p\right)$, for $i=1,2,\dots ,l$, may have N different values.

For instance, for the RGB color space, l = 3 and $c\left(p\right)=\{r_p,g_p,b_p\}$, where $\{r_p,g_p,b_p\}\in [0,255]$.

After the face detection process is completed, the system creates an output binary image O where every pixel is labeled as foreground, i.e., face pixel, or background, i.e., non-face pixel. Therefore, a face detection problem is a segmentation problem, in which the main goal is to find image regions containing a face. This can be stated as follows:

Definition 4.

Let I be an image of size $W=n\times m$ pixels. Let ${\mathsf{\Gamma }}_I$ be a set of faces in I, formed by a set of k non-overlapped regions, $R_{F_i}$, each one containing a face: ${\mathsf{\Gamma }}_I=\{R_{F_1},R_{F_2},\dots ,R_{F_k}\}$. A face detector is a function that converts pixels $p_j$, for $j=1,2,\dots ,W$, in the original image I into a binary output image O as follows:

$$O\left(p_j\right)=\left\{\begin{array}{c}1\mathit{if} p_j \mathit{belongs} \mathit{to} a \mathit{face}\hfill \\ 0\mathit{if} p_j \mathit{does} \mathit{not} \mathit{belong} \mathit{to} a \mathit{face}\hfill \end{array}\right.$$

(1)

This definition can be used for any face detection scheme, such as knowledge-based, template matching or feature-based methods [5]. Due to the fact there are some situations (such as lighting conditions, face orientation or similar colors between faces and background) that may not achieve an optimum segmentation when applying a face detection scheme, some post-processing filter must be applied to obtain the set ${\mathsf{\Gamma }}_I$. For our proposals, face detection will be performed using a skin color segmentation algorithm in a convenient color space C and a subsequent computation of the areas of the segmented connex regions $R_i\in {\mathsf{\Gamma }}_I$.

2.2. Fuzzy Sets and Skin Detection

In Section 1, we have outlined the main approaches to skin color detection. As we know, in any image segmentation scheme, there is a high level of uncertainty for a classifier to automatically obtain an optimum segmentation [25]. This fact can be also extended to face detection and, in particular, to skin color segmentation. Thus, we find that applying fuzzy theory can be a convenient way to obtain good detection rates, since a fuzzy set-theoretic model provides a mechanism to represent and manipulate uncertainty within an image.

Color image segmentation using fuzzy classification is a pixel-based segmentation method. This method assigns a color class to each pixel of an input image by applying a set of fuzzy rules on it. We can use this approach to achieve our goal: a pixel can be classified as “skin” or “non-skin” according to a set of fuzzy rules extracted from a training stage using different color spaces. To do this, each color plane will be considered as a fuzzy set, so that the skin detection is performed through fuzzy functions representing the membership degree of each pixel to the different classes. This is accomplished as follows:

Definition 5.

Given a color image I of size $W=n\times m$ pixels, where each pixel is defined using a color vector c in a color space C, so that $c\left(p\right)=\{c_1\left(p\right),c_2\left(p\right),\dots ,c_l\left(p\right)\}$, $\forall p\in I$, the histogram of $C,\mathsf{\Psi }\left(C\right)$, is defined as a $b\times l$ array $\mathsf{\Psi }\left(C\right)=\{f_1,f_2,\dots ,f_l\}$, such that each $f_i$ is the frequency vector of the color component $c_i$, for $i=1,2,\dots ,l$, using b bins, on the image I.

As a result, the value of each bin is the number of pixels in image I having the color $c_i$. If $\mathsf{\Psi }\left(C\right)$ is normalized by W, then $\mathsf{\Psi }\left(C\right)$ takes the color space C into the interval [0, 1]; that is, $\mathsf{\Psi }\left(C\right)$ represents the probability distribution of each color $c_i$ to be present in image I. According to Zadeh’s theory [26], a fuzzy set is a pair $(A,m)$ where A is a set and m: $A\to [0,1]$. This can be applied to the color histogram, where the fuzzy set can be defined as the pair $(C,\mathsf{\Psi })$, where C is the color space and Ψ: $C\to [0,1]$ is the normalized histogram. For each $c\in C,\mathsf{\Psi }\left(c\right)$ is the grade of membership of c, so that $c\in (C,\mathsf{\Psi })\iff c\in C$ and $\mathsf{\Psi }\left(c\right)\ne 0$.

The research on skin color face detection is mostly based on the popular RGB, YCbCr and HSV color space models [27]. In addition, some previous works have modeled these color spaces by means of fuzzy sets and fuzzy relations. Thus, the RGB system was used to represent skin colors [28], where a fuzzy model is extracted after asking people to classify human faces according to their skin color; then, a modified RGB fuzzy skin detector was presented in [29], based on the work of [30], trying to improve the skin detection performance. For the HSV space and its variants, the hue component can be defined by means of a fuzzy representation in order to take into account the non-uniformity of the colors’ distribution. Consequently, the authors in [31] proposed to represent colors with trapezoidal or triangular fuzzy subsets, associating colors with fuzzy sets; fuzzy linguistic hierarchies with different numbers of labels, depending on the desired granularity, were used in [32]. Then, a fuzzy classifier to detect the presence of faces in small windows, with an HSV color model to detect skin, was presented in [33]. Finally, fuzzy representations of YCbCr for modeling skin can be found in [34,35].

As shown, since RGB, YCbCr and HSV are the most common color spaces models for segmenting skin, we shall use these color systems in this paper, as well. In order to use our fuzzy approach, we must calculate first the normalized histogram for the considered color spaces, RGB, HSV and YCbCr:

• ${\mathsf{\Psi }}_{RGB}\left(C\right)=\{\mathsf{\Psi }\left(R\right),\mathsf{\Psi }\left(G\right),\mathsf{\Psi }\left(B\right)\}$;
• ${\mathsf{\Psi }}_{HSV}\left(C\right)=\{\mathsf{\Psi }\left(H\right),\mathsf{\Psi }\left(S\right),\mathsf{\Psi }\left(V\right)\}$;
• ${\mathsf{\Psi }}_{YCbCr}\left(C\right)=\{\mathsf{\Psi }\left(Y\right),\mathsf{\Psi }\left(Cb\right),\mathsf{\Psi }\left(Cr\right)\}$.

Since the skin detection will be used as a pre-processing task for detecting a face in an image, the training has been performed using a set of 200 images from the XM2VTS database [36], extracting only the skin information, using different ethnic groups and changing lighting conditions. The XM2VTS face database contains eight recordings of 295 subjects each, acquired over a period of four months. The background in this set of images is homogeneous, with a good contrast for detecting skin. A group of images from the training set is shown in Figure 2.

The results after obtaining ${\mathsf{\Psi }}_{RGB}\left(C\right)$ are shown in Figure 3.

From Figure 3a, the membership functions ${\mu }_{{\mathit{SKIN}}_i}\left(c_i\right)$ for the skin color in each plane can be modeled using a bell-shaped function, such that:

$${\mu }_{{\mathit{SKIN}}_i}\left(c_i\right)={\beta }_i{e}^{-\frac{{(c_i-{\alpha }_i)}^2}{2{\sigma }_i^2}}$$

(2)

where $i=\{R,G,B\};\{c_R,c_G,c_B\}\in [0,255];$ ${\beta }_i=\max {\mathsf{\Psi }}_{RGB}\left(c_i\right),{\sigma }_i^2$ is the variance of each fuzzy set $c_i$ and ${\alpha }_i=\arg \underset{c_i}{\max }{\mathsf{\Psi }}_{RGB}\left(c_i\right)$. The results of the model for the skin pixels are shown in Figure 3b.

Finally, for the background pixels, i.e., the non-skin pixels in the image, let us consider a variation of the model introduced in [37], which identifies the fuzziness in the transition region between the object (in this case, the skin) and the background classes. Thus, the membership value of a point to the object is determined by applying an S-function and a Z-function to the each color plane, so that:

$${\mu }_{{\mathit{NSKIN}}_i}^S\left(c_i\right)=\left\{\begin{array}{c}0\mathrm{for}{c}_i\le a_{S_i}\hfill \\ \\ 2{\left(\frac{c_i-a_{S_i}}{{\gamma }_{S_i}-a_{S_i}}\right)}^2\mathrm{for}{a}_{S_i}\le c_i\le b_{S_i}\hfill \\ \\ 1-2{\left(\frac{c_i-{\gamma }_{S_i}}{{\gamma }_{S_i}-a_{S_i}}\right)}^2\mathrm{for}{b}_{S_i}\le c_i\le {\gamma }_{S_i}\hfill \\ \\ 1\mathrm{for}{\gamma }_{S_i}\le c_i\end{array}\right.$$

(3)

$${\mu }_{{\mathit{NSKIN}}_i}^Z\left(c_i\right)=\left\{\begin{array}{c}1\mathrm{for}{c}_i\le a_{Z_i}\hfill \\ \\ 1-2{\left(\frac{c_i-a_{Z_i}}{{\gamma }_{Z_i}-a_{Z_i}}\right)}^2\mathrm{for}{a}_{Z_i}\le c_i\le b_{Z_i}\hfill \\ \\ 2{\left(\frac{c_i-a_{Z_i}}{{\gamma }_{Z_i}-a_{Z_i}}\right)}^2\mathrm{for}{b}_{Z_i}\le c_i\le {\gamma }_{Z_i}\\ \\ 0\mathrm{for}{\gamma }_{Z_i}\le c_i\end{array}\right.$$

(4)

where $i=\{R,G,B\}$; $\{c_R,c_G,c_B\}\in [0,255]$ and $\{a_{S_i},b_{S_i},{\gamma }_{S_i},a_{Z_i},b_{Z_i},{\gamma }_{Z_i}\}$ are the cross-over points that determine the shape of the membership functions for the non-skin pixels.

Accordingly, the same process has been adopted for the HSV and YCbCr color spaces. Although many previous works claim that color information in the HSV system is mainly contained either only in the hue component [38] or in the hue and saturation components [39], from our experiments, it became clear that the recognition of skin is also influenced by the intensity (value) of the image. This fact makes us consider all three color planes for achieving an accurate color skin detection. The results of calculating the histogram for each color plane in HSV are shown in Figure 4a. Note that intensity values are between zero and 255.

In the same way, for the YCbCr space, the two chrominance components are usually used to extract skin clusters, whereas the luminance component Y is discarded. However, practically speaking, the skin-tone color is non-linearly dependent on luminance [35], so for our proposals, we have also taken into account all three components, whose values are between zero and 255. The histograms are shown in Figure 5a, and their Gaussian approximations can be seen in Figure 5b.

As in the RGB case, each plane ($H,S$ and V) and ($Y,Cb$ and $Cr$) can be modeled, respectively, using a bell-shaped function (see Figure 4b and Figure 5b) defined as stated in Equation (2), where $i=\{R,G,B,H,S,V,Y,Cb,Cr\};{\beta }_i=\max {\mathsf{\Psi }}_k\left(c_i\right),$ ${\alpha }_i=\arg \underset{c_i}{\max }{\mathsf{\Psi }}_k\left(c_i\right);$ $\{c_R,c_G,c_B,c_H,c_S,c_V,c_Y,c_{Cb},c_{Cr}\}\in [0,255]$; k refers to RGB, HSV or YCbCr color systems, and ${\sigma }_i^2$ is the variance of each fuzzy set $c_i$.

Finally, for the non-skin pixels in the image, we will apply the S- and Z-functions defined by Equations (3) and (4), where $i=\{R,G,B,H,S,V,Y,Cb,Cr\}$.

Now, given an input image I, for any pixel $p\in I$, its color components are fuzzified, according to the parameters defined in Equations (2)–(4) and using some of the three color spaces considered above. In order to determine these parameters, a maximum entropy criterion will be used.

Shannon entropy [40] has been widely used in information theory. It is a measure of the uncertainty associated with a random variable. Specifically, Shannon entropy quantifies the expected value of the information contained in a message [41]. As a result, Shannon entropy is an extremely powerful approach in image segmentation [42,43,44]. In general terms, the selection of an appropriate threshold using Shannon entropy consists of finding an optimum threshold by maximizing the entropy function. In [45], Cheng et al. used the concept of fuzzy c-partition and the maximum entropy principle to select threshold values for gray-level images. When it is used for multilevel thresholding segmentation, the fuzzy c-partition entropy measures the quantity of extracted information in the image segmentation [46]. A larger value of entropy indicates that more information is extracted in this process. Taking this methodology into account, the skin color detection process is a segmentation process, as images are divided into a suitable number of fuzzy sets to represent skin and non-skin pixels. From the study developed in this section, it becomes clear that a fuzzy three-partition entropy approach for each color plane will segment images, giving as a result skin and non-skin regions.

Let $p_{{\mathit{SKIN}}_i},p_{{\mathit{NSKIN}}_i}^S,p_{{\mathit{NSKIN}}_i}^Z$ be the probabilities of the three fuzzy sets resulting from Equations (2)–(4), defined as follows:

$$\begin{array}{c}{p}_{{\mathit{SKIN}}_i}={\sum }_{c_i=0}^{255}{\mathsf{\Psi }}_k\left(c_i\right){\mu }_{{\mathit{SKIN}}_i}\left(c_i\right),\hfill \\ \\ p_{{\mathit{NSKIN}}_i}^S={\sum }_{c_i=0}^{255}{\mathsf{\Psi }}_k\left(c_i\right){\mu }_{{\mathit{NSKIN}}_i}^S\left(c_i\right),\hfill \\ \\ p_{{\mathit{NSKIN}}_i}^Z={\sum }_{c_i=0}^{255}{\mathsf{\Psi }}_k\left(c_i\right){\mu }_{{\mathit{NSKIN}}_i}^Z\left(c_i\right)\hfill \end{array}$$

(5)

Then, the entropy of the fuzzy three-partition can be calculated as:

$$H(a_{Z_i},b_{Z_i},{\gamma }_{Z_i},a_{S_i},b_{S_i},{\gamma }_{S_i})=-p_{{\mathit{SKIN}}_i}\log \left(p_{{\mathit{SKIN}}_i}\right)-p_{{\mathit{NSKIN}}_i}^S\log (p_{{\mathit{NSKIN}}_i}^S)-p_{{\mathit{NSKIN}}_i}^Z\log (p_{{\mathit{NSKIN}}_i}^Z)$$

(6)

The optimal combination of the parameters in Equation (6) can be computed by the maximum entropy criterion, that is the entropy $H(a_{Z_i},b_{Z_i},{\gamma }_{Z_i},a_{S_i},b_{S_i},{\gamma }_{S_i})$ will be maximized when selecting parameters $(a_{Z_i},b_{Z_i},{\gamma }_{Z_i},a_{S_i},b_{S_i},{\gamma }_{S_i})$ in a proper way. This process must be repeated for each plane and each color system used in the system.

After the fuzzy inference system has been defined, skin-candidate regions are detected. That is, we must remark that the segmented regions have the same color as that of the skin. The following process is to assure that these regions actually belong to a face or not. This fact will be discussed in the following subsection.

2.3. Face Detection

After the segmentation process, the system creates an output binary image where pixels are segmented as skin or non-skin pixels. The skin region localization process is achieved by applying an eight-connected blob algorithm [47]. Then, after identifying skin regions, the algorithm only accepts those having an appropriate size over the total image area. Let $I,O$ be the input and output images, both of size $W=n\times m$ pixels; let $R_k$ be a region segmented as skin in O. The system will consider $R_k$ as a face region $R_{F_k}$ if the following relationship is accomplished:

$$\frac{A_k}{W}\ge p$$

(7)

where the area of region $R_k$, $A_k$, represents the number of skin pixels in $R_k$. Hence, regions above threshold p are assumed to contain human faces.

3. Results

Let us show now the results of some experiments completed for our model and compare it with some other existing methods.

First, the values of the parameters in order to maximize the entropy $H(a_{Z_i},b_{Z_i},{\gamma }_{Z_i},a_{S_i},b_{S_i})$ must be found. After performing the experiments with the training set defined in Figure 2, we found that the optimum values are: $a_{S_i}={\gamma }_{Z_i}=\arg \underset{c_i}{\max }{\mathsf{\Psi }}_{RGB}\left(c_i\right)$; $a_{Z_i}=0.5{\gamma }_{Z_i}$; ${\gamma }_{S_i}=1.5a_{S_i};$ and $b_{S_i},b_{Z_i}$ correspond to the membership value of $c_i=b_{S_i}$ (or $b_{Z_i}$) in ${\mu }_{{\mathit{NSKIN}}_i}^S\left(c_i\right)$, ${\mu }_{{\mathit{NSKIN}}_i}^Z\left(c_i\right)$ that is equal to 0.5. The results for the models of the fuzzy skin and non-skin classes for the RGB color space are shown in Figure 6.

The results for the models of the fuzzy skin and non-skin classes for the HSV and YCbCr color spaces are shown in Figure 7.

3.1. Results for the Proposed Fuzzy Systems

The tests have been performed with three different databases:

• The XM2VTS face database [36], described before in Section 2.1;
• The VALID database [48], which consists of five recording sessions of 106 subjects over a period of one month. One session is recorded in a studio with controlled lighting and no background noise, and the other four sessions are recorded in office-type scenarios;
• The color FERET database [49], which is composed of 11,338 face images of 994 different persons, taken from different angles of view, over the course of 15 sessions between 1993 and 1996. For our experiments, we used the sets of images fa (images with frontal view) and fb (frontal images from the same individuals but with different facial expression), corresponding to 843 individuals with frontal images only.

For all of the experiments in this section, we have used a set of 200 images from each database. Our set was taken from persons of different ethnicities: Caucasian, Asian and African.

According to the fuzzy skin detector defined before, Figure 8, Figure 9 and Figure 10 show the results for different images in all three databases. The images include different genders and ethnic groups and, for the VALID database, complex backgrounds and changing lighting conditions.

From these results, we must point out that the proposed method leads to very accurate results for the XM2VTS database, which has both good illumination conditions and high contrast between the foreground and the background. In the case of the VALID database, the results of the skin detector are also quite precise, in spite of the unfavorable conditions for achieving good quality results. Some errors appear mainly due to background colors that are very similar (or even identical) to skin colors; in addition, the lighting conditions will affect the final result. This effect is also noticeable for the images in the color FERET database, where there is a great variability in the illumination conditions of the images. However, the general results are good, and after applying a post-processing stage for detecting a face from the skin segmentation results, most of these errors can be corrected, as we will discuss in Section 3.3.

3.2. Comparison with Other Methods

Let us now compare our system with some other existing algorithms for skin color detection. First, an explicitly-defined skin region algorithm for the RGB color space is used; in [2], a pixel $(R_0,G_0,B_0)$ is classified as skin if:

$$\begin{array}{c}\left(\mathrm{a}\right){\mathrm{R}}_0>95\wedge {\mathrm{G}}_0>40\wedge {\mathrm{B}}_0>20,\mathrm{and}\hfill \\ \left(\mathrm{b}\right)\max ({\mathrm{R}}_0,{\mathrm{G}}_0,{\mathrm{B}}_0)-\min ({\mathrm{R}}_0,{\mathrm{G}}_0,{\mathrm{B}}_0)>15,\mathrm{and}\hfill \\ \left(\mathrm{c}\right)\mid {\mathrm{R}}_0-{\mathrm{G}}_0\mid >15\wedge {\mathrm{R}}_0>{\mathrm{G}}_0\wedge {\mathrm{R}}_0>{\mathrm{B}}_0\hfill \end{array}$$

(8)

We also found similar proposals for HSV and YCbCr color systems. In [50], it was found that, as we stated before, the recognition of skin color is influenced by the intensity (value) and saturation of the image; consequently, the distribution of intensity, saturation and hue should be defined for accurate skin detection. After training 200 pictures, including different ethnic groups (Asian, Caucasian and African), they obtained the following rules for a pixel $(H_0,S_0,V_0)$ to be classified as skin, where the three components are normalized:

$${\displaystyle \begin{array}{l}\left(\mathrm{a}\right)0.1<{\mathrm{S}}_0<0.68,\mathrm{and}\\ \\ \left(\mathrm{b}\right)\frac{0.4-{\mathrm{V}}_0}{0.014}\le {\mathrm{H}}_0\le \frac{{\mathrm{V}}_0+0.062}{0.01},0.13\le {\mathrm{V}}_0\le 0.25,\mathrm{or}\\ \\ \left(\mathrm{c}\right)\frac{0.4-{\mathrm{V}}_0}{0.014}\le {\mathrm{H}}_0\le \frac{0.67-{\mathrm{V}}_0}{0.014},0.25\le {\mathrm{V}}_0\le 0.38,\mathrm{or}\\ \\ \left(\mathrm{d}\right)\frac{{\mathrm{V}}_0-0.34}{0.03}\le {\mathrm{H}}_0\le \frac{0.67-{\mathrm{V}}_0}{0.014},0.38\le {\mathrm{V}}_0\le 0.46,\mathrm{or}\end{array}}$$

$${\displaystyle \begin{array}{l}\left(\mathrm{e}\right)\frac{{\mathrm{V}}_0-0.34}{0.03}\le {\mathrm{H}}_0\le \frac{{\mathrm{V}}_0-0.31}{0.009},0.46\le {\mathrm{V}}_0\le 0.6,\mathrm{or}\\ \\ \left(\mathrm{f}\right)\frac{0.91-{\mathrm{V}}_0}{0.14}\le {\mathrm{H}}_0\le \frac{{\mathrm{V}}_0-0.31}{0.009},0.6\le {\mathrm{V}}_0\le 0.76,\mathrm{or}\\ \\ \left(\mathrm{g}\right)\frac{0.91-{\mathrm{V}}_0}{0.14}\le {\mathrm{H}}_0\le \frac{1.17-{\mathrm{V}}_0}{0.0082},0.76\le {\mathrm{V}}_0\le 0.91,\mathrm{or}\\ \\ \left(\mathrm{h}\right)\frac{{\mathrm{V}}_0-0.91}{0.0041}\le {\mathrm{H}}_0\le \frac{1.17-{\mathrm{V}}_0}{0.0082},0.91\le {\mathrm{V}}_0\le 1\end{array}}$$

(9)

Finally, in [51], a pixel $(Y_0,C{b}_0,C{r}_0)$ is classified as skin if:

$$\begin{array}{c}\left(\mathrm{a}\right)60<{\mathrm{Y}}_0<255,\mathrm{and}\hfill \\ \left(\mathrm{b}\right)100<{\mathrm{Cb}}_0<125,\mathrm{and}\hfill \\ \left(\mathrm{c}\right)135<{\mathrm{Cr}}_0<170\hfill \end{array}$$

(10)

On the other hand, a skin probability map (SPM) approach to each one of the color spaces is also taken into account. For this method, given skin and non-skin histogram models, a skin pixel classifier is defined using the Bayes’ theorem. Then, the inequality $P\left(skin\right|c)\ge T$ can be used as a skin detection rule, where T is a selected threshold value and $P\left(skin\right|c)$ is the probability of observing skin, given a concrete c color value (see [14,23] for further details). We performed a training with the same images used in the development of the fuzzy systems, and obtained, as a result, the probability distribution for all three considered color systems.

Figure 11, Figure 12 and Figure 13 show the results for some images for our proposal and the existing skin segmentation methods named before. In each case, for the SPM models, we show the threshold T that minimizes the number of false positives and false negatives.

Finally, we have manually extracted the ideal skin segmentation for all of the images and compared it with the segmented results of all of the methods implemented. Therefore, in

Table 2. Comparison between algorithms.

Algorithm	XM2VTS			VALID
	FP (%)	FN (%)	Correct (%)	FP (%)	FN (%)	Correct (%)
Explicit RGB rules	0.12	5.75	94.14	0.40	14.47	85.13
Explicit HSV rules	0.47	5.12	94.41	2.19	12.91	84.91
Explicit YCbCr rules	0.55	4.12	95.33	0.80	15.14	84.06
SPM RGB	1.65	3.43	94.92	6.40	17.53	76.06
SPM HSV	0.81	6.39	92.80	0.98	13.97	85.04
SPM YCbCr	0.23	6.06	93.71	0.59	9.39	90.02
Fuzzy RGB	0.22	1.66	98.12	0.64	5.70	93.66
Fuzzy HSV	0.18	2.12	97.70	0.91	5.62	93.47
Fuzzy YCbCr	0.18	3.31	96.52	0.60	6.38	93.02
Algorithm	FERET			Global
	FP (%)	FN (%)	Correct (%)	FP (%)	FN (%)	Correct (%)
Explicit RGB rules	3.14	2.45	94.41	1.22	7.56	91.22
Explicit HSV rules	6.38	4.82	88.80	3.01	7.61	89.38
Explicit YCbCr rules	1.69	7.93	90.39	1.01	9.06	89.93
SPM RGB	2.65	5.44	91.91	3.57	8.80	87.63
SPM HSV	6.30	0.41	93.29	2.70	6.92	90.38
SPM YCbCr	7.10	6.16	86.74	2.64	7.20	90.16
Fuzzy RGB	0.75	2.56	96.69	0.54	3.31	96.16
Fuzzy HSV	0.60	3.13	96.27	0.56	3.62	95.82
Fuzzy YCbCr	0.67	4.27	95.05	0.48	4.65	94.87

, we show the results of the false positive (FP) and false negative (FN) rates, as well as the correct skin detection rate for the images in the experiments, for all of the algorithms considered. Our method obtains the better results for each color system, as shown in bold text in Table 2.

3.3. Results on Face Detection

As mentioned in Section 2.3, after defining a skin color detection system, we will need to determine if the resulting pixels belong to a face or not. To do this, we proposed a threshold p relating the area of the segmented skin regions and the number of pixels in the input image. After a training process, we found $p=0.3$ as the best threshold in order to identify if a skin region belongs to a face. Let us consider first well-illuminated images with a homogeneous background, such as the ones in the XM2VTS database; we can see an example in Figure 14. Here, the results after the skin segmentation process and the subsequent face detector are shown; in order to compare different methods, we have used the ones that obtained the best results in Table 2 for each kind of algorithm. Therefore, the explicit rules and our fuzzy proposal for the RGB color system are taken into account, as well as the skin probability map for the HSV system.

As expected, the results show that when working with well-illuminated images and homogeneous backgrounds, faces are detected properly for all of the algorithms considered. There are some false positive skin pixels belonging to the shirt region, but they are isolated in small regions whose areas are below threshold p, so these regions are not considered as face areas, and consequently, the face is detected correctly.

Let us explain now a more complex example, where the Valid database is utilized for the tests. As mentioned before, most images in the Valid database were acquired in a noisy “real-world” office environment with no control of illumination, so face detection algorithms must deal with tougher conditions than in both the XM2VTS and FERET databases. Again, the results after the skin segmentation process and the subsequent face detector are shown in Figure 15.

The results of the Valid database are highly dependent on the conditions where images were taken, since a great amount of images in this database are recorded in office-type scenarios. For the example shown, we can clearly appreciate the differences between the results of all three algorithms, and what is more, it becomes clear how the face is not well detected for the first two cases. Thus, for the explicit RGB system, although the algorithm extracts in a proper way the facial skin pixels, some pixels in the shirt are also considered as belonging to skin and connected to the face region (i.e., they are false positives); this way, they cannot be separated from the face, and as a consequence, this results in a bad face detection. Accordingly, an incorrect face detection is also obtained when using the SPM for the HSV color system. In this case, even a bigger portion of the shirt is considered as skin, leading again to a bad detection.

On the contrary, for our fuzzy RGB detector, the pixels of the face are correctly segmented and classified as belonging to the same region. The area of this region is above the threshold ratio p, so this region is considered as a face $R_F$. Moreover, false positive skin pixels, such as the ones in the shirt, are isolated from the face, and the areas of these regions are below threshold p, so these regions are not considered as belonging to a face, which results in a very accurate face detection.

To sum up,

Table 3. Comparison of face detection with different color skin segmentation algorithms.

Algorithm	XM2VTS		VALID
	Incorrect (%)	Correct (%)	Incorrect (%)	Correct (%)
Explicit RGB rules	5.82	94.16	21.06	78.94
SPM HSV	10.10	89.90	37.43	62.57
Fuzzy RGB	1.98	98.02	12.56	87.44
Algorithm	FERET		Global
	Incorrect (%)	Correct (%)	Incorrect (%)	Correct (%)
Explicit RGB rules	9.21	90.79	12.03	87.97
SPM HSV	19.33	80.67	22.29	77.71
Fuzzy RGB	4.87	95.13	6.47	93.53

shows the results on face detection for all of the images used for testing the system from the three databases. In addition, a comparison between our detector with other related works is shown in

Table 4. Comparison of other face detection methods.

Algorithm	Incorrect (%)	Correct (%)
Viola and Jones [52]	7.63	92.37
Li et al. [53]	6.87	93.13
Fuzzy RGB	6.47	93.53

. Thus, the seminal work of Viola and Jones [52] is selected first. Moreover, convolutional neural networks (CNN) are becoming extremely popular in pattern recognition in the last few years. Recently, Li et al. [53] built a CNN cascade for fast face detection, and the results obtained from this method are considered in Table 4, as well.

The best results on face detection using the mentioned dataset are obtained for our fuzzy RGB detector (as shown in bold text in Table 4), with an average of 93.53% correct detections; even for the Valid database, it achieves almost 88% correct face detections, whereas the explicit RGB algorithm has about 79%, and the SPM for the HSV color system only gets about 62% correct detections. Finally, our system improves the results of the Viola and Jones face detector, which is the de facto standard in many applications of face detection and achieves similar results to the CNN cascade proposal. Consequently, we believe that the developed fuzzy skin detector is a reliable face detector.

4. Discussion

According to the results of the experiments, it seems clear that our fuzzy proposal for segmenting skin achieves good detection rates in all three color systems considered, whilst false positives and false negatives are kept at low levels. Moreover, our approach can classify faces from different ethnic groups, giving an accurate skin segmentation result. Thus, the best global results for all of the algorithms implemented in terms of correct classified pixels (96.16%) are for the fuzzy RGB detector, followed by the other two fuzzy proposals (the fuzzy HSV detector and the fuzzy YCbCr detector achieve 95.82% and 94.87% correct segmented pixels, respectively). This gives an improvement of about 5% compared with the explicit and SPM detection algorithms.

Regarding the false positive errors, we must recall that this rate measures the proportion of actual negative pixels that are incorrectly identified as skin pixels; this rate is critical, because a face can be misdetected due to a high number of false positive pixels, such as hair, clothes and background colors that are similar to skin. In this case, the fuzzy YCbCr detector gives the lowest FP rate (only 0.48%), followed by the other two fuzzy detectors, which are slightly above 0.5%. Again, these rates are better than the other implemented algorithms. Moreover, considering the false negative errors, where skin pixels are not identified properly, the fuzzy RGB detector has the lowest rate (3.31%), followed another time by the other two fuzzy proposals (3.62% for the fuzzy HSV detector and 4.65% for the fuzzy YCbCr method). These results improve the ones from the other algorithms about 4%–5% in most cases.

When working with good illumination conditions and homogeneous backgrounds, such as the case of the images in the XM2VTS database, the results obtained for all of the algorithms are good, with correct detection rates reaching nearly 95%. For this database, the fuzzy RGB detector is the best one, with 98.12% correct classification of pixels. Besides, images in the color FERET database have generally homogeneous backgrounds and uniform lighting conditions, but sometimes the colors in the background are similar to the ones in human skin, which leads to the highest false positive rates for all three databases (see Figure 8, Figure 9 and Figure 10 and Figure 13). Nevertheless, the correct detection results for our fuzzy proposals are higher than 95%, with FP rates below 1% in all of the cases.

On the other hand, working in unconstrained environments, such as the ones we can find in the VALID database, gives, as expected, the worst segmentation results. Thus, the explicit and SPM algorithms hardly arrive at 85% correct results. However, all of the three fuzzy methods proposed have detection rates above 93% for this database, and as a consequence, they behave reasonably well for complex backgrounds and changing illumination environments, giving very accurate results.

To sum up, we have verified that our proposal of using fuzzy expert systems for skin color face detection gets precise results even when dealing with uncontrolled scenarios and changing lighting conditions, making it a suitable technique for segmenting skin in different environment conditions.

5. Conclusions

Face recognition systems have become a popular tool for identifying and verifying the identity of users in recent years. In order to complete successfully the recognition process, an initial, robust face detection algorithm must be implemented. Among the methods for face detection, color skin segmentation algorithms have become important in the last few years, providing a robust tool for detecting faces.

This article has presented a review of different skin detection algorithms on color images, having studied the main features of the most common color spaces: RGB, HSV and YCbCr. A proposal of fuzzy skin color detectors for these color spaces has been discussed throughout this work. Thus, each color plane has been modeled using different fuzzy sets for the skin and non-skin classes and the inference system and defuzzifying process have been explained. The results of the experiments show that our fuzzy algorithms give the better results regarding the classification of pixels as skin (between 94%–96%), improving about 5% both the explicit rules and the SPM methods. Considering face detection, the fuzzy RGB system achieves almost 94% of correct detections, being robust even when dealing with uncontrolled environments and changing light conditions. Therefore, our method leads to very accurate face detection results with low false positive and false negative rates.

Future works aim at extending the fuzzy system to other color spaces, such as TSL or YIQ. An adaptive training of the skin model can be used to improve the classification of skin regions for our fuzzy method. We are also focusing on working with images where more than one face appears, adapting the face detection algorithm accordingly.