Combining Fuzzy C-Means Clustering with Fuzzy Rough Feature Selection

Abstract

With the rapid development of the network, data fusion becomes an important research hotspot. Large amounts of data need to be preprocessed in data fusion; in practice, the features of datasets can be filtered to reduce the amount of data. The feature selection based on fuzzy rough sets can process a large number of continuous and discrete data to reduce the data dimension, making the selected feature subset highly correlated with the classification but less dependent on other features. In this paper, a new method of fuzzy rough feature selection is proposed which combines the membership function determination method of fuzzy c-means clustering and fuzzy equivalence to the original selection. Different from the existing research, our method takes full advantage of knowledge about the dataset itself and the differences between datasets, which makes the features selected have a higher correlation with the classification, improves the classification accuracy, and reduces the data dimension. Experimental results on the UCI machine learning repository datasets confirmed the performance and effectiveness of our method. Compared to the existing method, smaller subsets of features and an average of 1% higher classification accuracies were achieved.

1. Introduction

We live in an age of data explosion—dealing with the growth of datasets usually requires much time and expense if we use the existing computers and algorithms. We want the dataset to contain more and more features to increase the likelihood of distinguishing different categories. Unfortunately, it may not be right. A higher-dimensional dataset increases the possibility of discovering incompletely valid false patterns. An effective way of resolving this problem is to select some of the most relevant and informative features from the dataset and eliminate redundant or irrelevant features. Unlike other dimensional reduction methods, feature selection retains the original meaning of features. This method can effectively reduce the size of a dataset without influencing the information expressed by the data, thus reducing cost and saving time.

Researchers have carried out different definitions of feature selection. Ideally, feature selection is to find the minimum feature subset which is necessary and sufficient to identify the target [1]. The definition of feature selection from the angle of improving the prediction accuracy is a process which can increase the classification accuracy or reduce the characteristic dimension without lowering the classification accuracy [2]. The basic method of feature selection is to generate a feature subset (search algorithm) and then evaluate the subset (evaluation criteria). The selection algorithm and evaluation criteria are two important parts of feature selection; an excellent search algorithm can speed up the search of features to find the optimal solution. The normal search algorithms contain global optimization, random search, and heuristic search. Evaluation criteria are defined as to evaluate the feature subset which is selected with some evaluation criterions. The evaluation decides directly the output of algorithm and performance of a classification model. An excellent evaluation criterion ensures the chosen subset contains a large amount of information and tiny redundancy. Evaluation functions can be divided into filter (evaluation function is independent of classifier), wrapper (the error probability of classifier is used as evaluation function), and embedding (a mixture of the first two). Common feature selection methods include Relief (relevant features), LVW (Las Vegas wrapper), LARS (least angle regression), and attribute reduction of rough set.

Fuzzy rough set is the extension of rough set. Two important applications of rough set are rule induction and attribute reduction (the meaning is the same as feature selection, but in rough set theory it is often called attribute reduction). However, rough set feature selection can only reduce the dataset of discrete data, and to overcome the drawback, people combined fuzzy set to rough set. Fuzzy rough feature selection (FRFS) provides a method which can effectively reduce the datasets of real value or discrete noise data (or both) without users’ information. In addition, the technology can be applied to data with a continuous or nominal decision attribute, thus it can be used in the regression and classification dataset, and only requires additional information for each feature of fuzzy partition which can be automatically derived from the data. An important problem of FRFS is the determination of the similarity relations between objects. The existing methods are not flexible nor pertinent. In this paper, a method combines the membership function generation method based on fuzzy c-means clustering and fuzzy equivalence is proposed. The improving FRFS can automatically generate a membership function based on the knowledge of the dataset itself and complete feature selection.

This paper is structured as follows: Section 2 is literature review. The theoretical background is given in Section 3, introducing the rough set attribute reduction, membership function, and fuzzy rough feature selection. In Section 4, the improved fuzzy rough feature selection is presented, and an experiment is provided in Section 5. The paper is concluded in Section 6.

2. Literature Review

The rough set theory is a frame represented by Zdzislaw Pawlak in 1989, which can construct concept approximation with incomplete information. The available information contains a set of examples of concept and the relationship to each other, such as indiscernibility, set approximation, reducts, and dependency [3,4]. Rough set as a method of soft computing receives more and more attention. Nowadays, rough set is still a research hotpot in the field of artificial intelligence. Hu and Yao proposed structured rough set approximation in complete and incomplete information systems to serve as a basis of three-way decisions with rough set [5]. To deal with an incomplete information system, a more generalized approach that considered potential candidates was presented [6].

Rule induction and feature selection are two important applications of rough set. Every component of the model of rule induction is introduced in detail in [7]. In the literature [8,9], rule induction is carried out for the absence of feature values in the information system. In the literature [10,11], the researchers used the result of attribute reduction to classify datasets with neural networks; the testing result indicated that with less study time the misclassification does not increase significantly, and they declared that the attribute reduction of rough set has the possibility of practical application. Because the attribute reduction of rough set is an NP-hard problem, many pieces of research pay attention to the acceleration algorithm [12,13,14]. Recently, two quick feature selection algorithms based on the neighbor inconsistent pair were presented which can reduce the time consumed in finding a reduct [15].

Fuzzy sets were introduced independently by Lotfi A. Zadeh and Dieter Klaua in 1965 as an extension of the classical notion of set [16]. Because both rough set and fuzzy set are used to deal with uncertain data, so many scholars compared the two methods and make great contributions [17,18]. Dubois and Prade first combined fuzzy set and rough set together [19], hereafter the research centering on fuzzy rough set appear one after another [20,21,22,23,24,25], and in the meantime, the accelerating algorithm came out, such as feature selection based on ant colony optimization [26] and information entropy [27]. In recent years, the feature selection algorithm based on a new definition of fuzzy rough set approximations based on the divergence measured of fuzzy sets is proposed and its properties were explored [28]. Another interest is the accelerator of fuzzy rough feature selection, a method based on sample reduction and dimensionality reduction was proposed [29].

3. Theoretical Background

Rough set theory is presented by Zdzislaw Pawlak in 1981 [30,31]. A basic notion of rough set is the concept of lower and upper approximations. A vague concept is an approximation of precise concepts. Objects belonging to the same category have the same attributes that are indistinguishable.

3.1. Rough Set Attribute Reduction

The central notion of RSAR (rough set attribute reduction) is indiscernibility. Assume there is an information system $I=\left(U,A\right)$, where $U$ is a non-empty finite subset of objects (the universe), $A$ is a non-empty finite set of attributes such as for each $a\in A,U\to V_a$, where $V_a$ is the value set for attribute $a$. $A=\left\{C\cup D\right\}$ where $C$ is the set of condition attributes and $D$ is the set of decision attributes.

Definition 1 (Indiscernibility).

For any $P\subseteq A$, there is an associated equivalence relation $IND\left(P\right): IND\left(P\right)=\{\left(x.y\right)\in U^2|\forall a\in P a\left(x\right)=a\left(y\right)\}$, if $\left(x,y\right)\in IND\left(P\right)$, then $x$ and $y$ are indiscernible from attributes of $P$. The equivalence classes of the $P$-indiscernibility relation are denoted as ${\left[x\right]}_P$.

Definition 2 (Lower Approximation).

The lower approximation of $P$ is defined as $\underset{\_}{P}X=\left\{x|{\left[x\right]}_P⊑X\right\},X⊑U$.

Definition 3 (Positive Region).

$P$ and $Q$ are equivalence relations over $U$, the positive region is defined as $PO{S}_P\left(Q\right)={\bigcup }_{X\in U/Q}\underset{\_}{P}X$. The positive region contains the objects of $U$ which can be divided into $U/Q$ class, using the knowledge of attribute $P$.

Definition 4 (Dependency).

An important work of data analysis is finding the dependency of attributes. If all values of a set of attributes $Q$ depend uniquely on another set of attributes $P$, or there is a functional dependency between values of $Q$ and $P$, then $Q$ depends totally on $P$, denoted $P\Rightarrow Q$. Dependency can be defined as: for $P,Q\subset A$, $Q$ depends on $P$ with degree $k$ $\left(0\le k\le 1\right)$, defined $P{\Rightarrow }_{k}Q$, $k=\gamma P\left(Q\right)=\frac{\left|PO{S}_P\left(Q\right)\right|}{\left|U\right|}$, if $k=1$, $Q$ depends totally on $P$. If $k<1$, $Q$ depends partially on $P$ with degree $k$. If $k=0$, $Q$ does not depend on $P$.

A basic idea is to calculate the dependencies of all the possible subsets of $C$, any subset with $\gamma \left(D\right)=1$ is a reduct. The smallest subset is the minimum reduct. However, this idea is impossible for large datasets. Algorithm 1 is called QUICKREDUCT Algorithm [20]; we do not need to calculate all the possible subsets. Start with an empty set and add attributes to the set in proper order to obtain the maximum increase of dependency until we get the maximum possible number (usually equal to 1). We need to pay attention to that the algorithm may not get the minimum set of reduct necessarily every time. In the worst situation, the complexity reaches $n!$ for attribute dimensionality of $n$.

Algorithm 1: RSAR QUICKREDUCT

Input:$\mathbf{R}\leftarrow${empty set}, $\mathbf{C}\leftarrow${set of all features},

Output:$\mathbf{R}$

1: do

2:  $\mathbf{T}\leftarrow \mathbf{R}$

3:  $\forall \mathbf{x}\in \left(\mathbf{C}-\mathbf{R}\right)$

4:  $\mathbf{if} {\mathbf{\gamma }}_{\mathit{R}\cup \left\{\mathit{x}\right\}}\left(\mathit{D}\right)>{\mathbf{\gamma }}_{\mathit{T}}\left(\mathit{D}\right)$

5:   $\mathbf{T}\leftarrow \mathbf{R}\cup \left\{\mathit{x}\right\}$

6:   $\mathbf{R}\leftarrow \mathbf{T}$

7: until ${\mathbf{\gamma }}_{\mathit{R}}\left(\mathit{D}\right)={\mathbf{\gamma }}_{\mathit{C}}\left(\mathit{D}\right)$

8: return R

3.2. Membership Function

American cybernetician L.A. Zadeh created fuzzy sets theory with a groundbreaking paper in 1965 [16]. Fuzzy set is the extension of classical set, which contains more general and various mathematical concepts, forming a new fuzzy mathematics discipline.

Assuming $X$ is the universe, $A$ is a subset of $X$ and can be represented by characteristic functions such as map $A:X\to \left\{0,1\right\},\forall x\in X,A\left(x\right)=\{\begin{array}{c}0,x\notin A\\ 1,x\in A\end{array}$. For fuzzy subset $A$ of $X$, any $x\in X$, $x$ is not absolutely belonging to $A$ nor absolutely not belonging to $A$. The degree to which $x$ belongs to $A$ can be represented by the value in $\left[0,1\right]$.

Definition 5 (Membership Function).

Assuming $X$ is the universe, the map $A:X\to \left[0,1\right]$ is called a fuzzy subset of $X,$ F set for short. Map $A$ is called membership function of F set about $A$, $A\left(x\right)$ is called degree of membership of $x$ about $A$.

The generation of the fuzzy membership function is fundamentally important. It is important to find a proper membership function. A basic method of constructing membership functions is the reference function; there are some commonly used membership functions, choosing proper parameters to get the membership functions which we need, such as triangular membership functions:

$$f\left(x,a,b,c\right)=\{\begin{array}{c}\begin{array}{c}0, x<a\\ \frac{x-a}{b-a},a\le x\le b\end{array}\\ \begin{array}{c}\frac{c-x}{c-b},b<x\le c\\ 0, x>c\end{array}\end{array}.$$

(1)

Trapezoidal membership function:

$$f\left(x,a,b.c.d\right)=\{\begin{array}{c}\begin{array}{c}0, x<a\\ \frac{x-a}{b-a},a\le x<b\end{array}\\ 1,b\le x<c\\ \begin{array}{c}\frac{d-x}{d-c},c\le x\le d\\ 0, x>d\end{array}\end{array}.$$

(2)

Gaussian membership function:

$$f\left(x,\sigma ,c\right)=\exp (-\frac{{\left(x-c\right)}^2}{2{\sigma }^2}).$$

(3)

Membership function can be generated from available data. Many methods can be used, such as the histogram method, the transformation of the probability distribution to the probability distribution, clustering, and neural network [32,33,34,35,36]. We need an effective membership function generation mechanism to make full use of fuzzy theory and it must have the following advantages [37]:

• Accuracy. Membership function should reflect the knowledge contained in the data accurately.
• Flexibility. This method can provide a broad family of membership function.
• Computability. The method should be feasible in calculation so that it has practical application value. The literature [38] emphasizes the importance of easy optimization and adjustment for membership functions.
• Ease of use. Once the membership function is determined, for any given $\mathrm{x}$, the corresponding $\mathrm{A}\left(\mathrm{x}\right)$ can be found easily.

In this paper, we use clustering to find membership functions. The fuzzy c-mean clustering (FCM) method is used to generate the fuzzy membership function during clustering. During the clustering process of FCM, the fuzzy membership function is generated.

3.3. Fuzzy Rough Feature Selection

The RSAR we mentioned only applies on the discrete datasets, but in real life, datasets usually contain real values and noises. Using fuzzy set theory, we can deal with this complex situation. Fuzzy mathematics is a mathematical theory in which studies utilize fuzzy phenomena. The core of fuzzy mathematics is the fuzzy set, which is different from the classical set. It has no definite elements and can only be mastered by membership function.

The intersection, union, and complement operations of fuzzy sets are similar to those of classical sets, but in some cases, the general $\mathrm{and}$ operator may fail. The selection of the $\mathrm{and}$ operator should be analyzed in detail. T norm and s norm can be thought of as generalized operations, but they are not actually. This question remains uncertain.

Definition 6 (t-norm).

A triangular norm or shortly t-norm better reflects the nature of the logic operator $\mathrm{and}$. T-norm is a binary function $T$ on $\left[0,1\right]$, which satisfies the exchange law, associative law, monotonicity, and the boundary condition. That is to say $T:\left[0,1\right]\times \left[0,1\right]\to \left[0,1\right],\forall x,y,z\in \left[0,1\right]$:

$$T\left(x,y\right)=T\left(y,x\right)$$

$$T\left(x,T\left(y,z\right)\right)=T\left(T\left(x,y\right),z\right)$$

$$y\le z,T\left(x,y\right)\le T\left(x,z\right)$$

$T\left(x,1\right)=x$.

There are some frequently used t-norms $\otimes$:

The standard min operator $x\otimes y=min\left\{x,y\right\}$

The algebraic product $x\otimes y=xy$

Lukasiewicz t-norm $x\otimes y=max\left\{x+y-1,0\right\}$.

Definition 7 (s-norm).

S-norm is also called a triangular conorm and shortly t-conorm. S-norm is a binary function $S$ on $\left[0,1\right]$, which satisfies the exchange law, associative law, monotonicity, and the boundary condition. That is to say $S: \left[0,1\right]\times \left[0,1\right]\to \left[0,1\right],\forall x,y,z\in \left[0,1\right]$:

$$S\left(x,y\right)=S\left(y,x\right)$$

$$S\left(x,S\left(y,z\right)\right)=S\left(S\left(x,y\right),z\right)$$

$$y\le z,S\left(x,y\right)\le S\left(x,z\right)$$

$S\left(x,0\right)=x$.

Three well known s-norms $\oplus$ are:

The standard max operator $x\oplus y=max\left\{x,y\right\}$

The probabilistic sum $x\oplus y=x+y-xy$

The bounded sum $x\oplus y=min\left\{x+y,1\right\}$.

The fuzzy rule can be expressed as “if $A$ then $B$”, shortly $A\to B$. $A$ and $B$ are fuzzy sets, the true degrees are expressed as $A\left(x\right)$, $B\left(y\right)$. The true degree of $A\to B$ is expressed as $A\left(x\right)\to B\left(y\right)$; the degree of the proposition depends on the true degree of the former and the latter.

Definition 8 (Fuzzy Implicator).

Find a binary function $R:\left[0,1\right]\times \left[0,1\right]\to \left[0,1\right]$ to represent the fuzzy rules properly. Meet left monotone (or right monotone)

$R\left(1,0\right)=0,R\left(1,1\right)=R\left(0,1\right)=R\left(0,0\right)=1$.

The frequently used implicators are:

Mamdani implicator: $x\to y=x\wedge y$

Zadeh implicator: $x\to y=\left(1-x\right)\vee \left(x\wedge y\right)$

Kleene–Dienes implicator: $x\to y=\left(1-x\right)\vee \left(x\wedge y\right)$

Lukasiewicz implicator: $x\to y=\min \left\{1,1-x+y\right\}$.

Fuzzy upper approximation and lower approximation are defined with the fuzzy division of input [39] or with the following definition:

$${\mu }_{\underset{\_}{R_P}X}\left(x\right)=in{f}_{y\in U}I\left({\mu }_{R_P}\left(x,y\right),{\mu }_X\left(y\right)\right),$$

(4)

where $I$ is the fuzzy implicator and $R_P$ represents the fuzzy similar relation in the subset of feature set $P$.

$${\mu }_{R_P}\left(x,y\right)={\cap }_{a\in P}\left\{{\mu }_{R_a}\left(x,y\right)\right\},$$

(5)

where ${\mu }_{R_a}\left(x,y\right)$ represents the degree of similarity of $x$ and $y$ about feature $a$.

$${\mu }_{R_a}\left(x,y\right)=max(\mathit{min}\left(\frac{a\left(y\right)-\left(a\left(x\right)-{\sigma }_a\right)}{a\left(x\right)-\left(a\left(x\right)-{\sigma }_a\right)},\frac{a\left(x\right)+{\sigma }_a-a\left(y\right)}{a\left(x\right)+{\sigma }_a-a\left(x\right)}\right),0)$$

(6)

The fuzzy positive region and dependency are defined as in rough set [40].

$${\mu }_{PO{S}_{R_P}\left(Q\right)}\left(x\right)=su{p}_{x\in U/Q} {\mu }_{\underset{\_}{R_P}X}\left(x\right)$$

(7)

$${{\mathsf{\gamma }}^{\prime }}_P\left(Q\right)=\frac{{\sum }_{x\in U}{\mu }_{PO{S}_{R_P}\left(Q\right)}\left(x\right)}{\left|\mathbb{U}\right|}$$

(8)

An example with the dataset in

Table 1. Dataset with continuous data.

Object	a	b	c	q
1	−0.4	−0.3	−0.5	1
2	−0.4	0.2	−0.1	2
3	−0.3	−0.4	−0.3	1
4	0.3	−0.3	0	2
5	0.2	−0.3	0	2
6	0.2	0	0	1

follows. There are six objects, features (condition attribute) $a,b$ and $c$, label (decision attribute) $q$. Assuming $C=\left\{a,b,c\right\},D=\left\{q\right\},IND\left(q\right)=\left\{\left\{2,4,5\right\},\left\{1,3,6\right\}\right\}$, using Equation (6), we can obtain the fuzzy similarity matrixes as follows:

$$U_{R_a}\left(x,y\right)=\left(\begin{array}{cccccc}1.0& 1.0& 0.987& 0.013& 0.006& 0.006\\ 1.0& 1.0& 0.987& 0.013& 0.006& 0.006\\ 0.987& 0.987& 1.0& 0.026& 0.019& 0.019\\ 0.013& 0.013& 0.026& 1.0& 0.993& 0.993\\ 0.006& 0.006& 0.019& 0.993& 1.0& 1.0\\ 0.006& 0.006& 0.019& 0.993& 1.0& 1.0\end{array}\right)$$

$$R_b\left(x,y\right)=\left(\begin{array}{cccccc}1.0& 0.0& 0.568& 1.0& 1.0& 0.0\\ 0.0& 1.0& 0.0& 0.0& 0.0& 0.137\\ 0.568& 0.0& 1.0& 0.568& 0.568& 0.0\\ 1.0& 0.0& 0.568& 1.0& 1.0& 0.0\\ 1.0& 0.0& 0.568& 1.0& 1.0& 0.0\\ 0.0& 0.137& 0.0& 0.0& 0.0& 1.0\end{array}\right)$$

$$R_c\left(x,y\right)=\left(\begin{array}{cccccc}1.0& 0.0& 0.036& 0.0& 0.0& 0.0\\ 0.0& 1.0& 0.036& 0.518& 0.518& 0.518\\ 0.036& 0.036& 1.0& 0.0& 0.0& 0.0\\ 0.0& 0.518& 0.0& 1.0& 1.0& 1.0\\ 0.0& 0.518& 0.0& 1.0& 1.0& 1.0\\ 0.0& 0.518& 0.0& 1.0& 1.0& 1.0\end{array}\right)$$

$${\mu }_{\underset{\_}{R_a}\left\{1,3,6\right\}}\left(x\right)=in{f}_{y\in U}I\left({\mu }_{R_a}\left(x,y\right),{\mu }_{\left\{1,3,6\right\}}\left(y\right)\right)$$

$$\begin{gathered} {\mu }_{\underset{\_}{R_a}\left\{1,3,6\right\}}\left(3\right)={\inf }_{y\in U}I\left({\mu }_{R_a}\left(3,y\right),{\mu }_{\left\{1,3,6\right\}}\left(y\right)\right)= \\ \inf \left\{I\left(0.699,1\right),I\left(0.699,0\right),I\left(1,1\right),I\left(0,0\right),I\left(0,0\right),I\left(0,0\right)\right\}=0.301. \end{gathered}$$

Others are the same as object 3.

$${\mu }_{\underset{\_}{R_a}\left\{1,3,6\right\}}\left(1\right)=0.0$$

$${\mu }_{\underset{\_}{R_a}\left\{1,3,6\right\}}\left(2\right)=0.0$$

$${\mu }_{\underset{\_}{R_a}\left\{1,3,6\right\}}\left(4\right)=0.0$$

$${\mu }_{\underset{\_}{R_a}\left\{1,3,6\right\}}\left(5\right)=0.0$$

$${\mu }_{\underset{\_}{R_a}\left\{1,3,6\right\}}\left(6\right)=0.0$$

For $\left\{2,4,5\right\}$

$${\mu }_{\underset{\_}{R_a}\left\{2,4,5\right\}}\left(1\right)=0.0$$

$${\mu }_{\underset{\_}{R_a}\left\{2,4,5\right\}}\left(2\right)=0.0$$

$${\mu }_{\underset{\_}{R_a}\left\{2,4,5\right\}}\left(3\right)=0.0$$

$${\mu }_{\underset{\_}{R_a}\left\{2,4,5\right\}}\left(4\right)=0.301$$

$${\mu }_{\underset{\_}{R_a}\left\{2,4,5\right\}}\left(5\right)=0.0$$

$${\mu }_{\underset{\_}{R_a}\left\{2,4,5\right\}}\left(6\right)=0.0.$$

Thus, the positive regions of every object are:

$${\mu }_{PO{S}_{R_a}\left(Q\right)}\left(1\right)=0.0$$

$${\mu }_{PO{S}_{R_a}\left(Q\right)}\left(2\right)=0.0$$

$${\mu }_{PO{S}_{R_a}\left(Q\right)}\left(3\right)=0.301$$

$${\mu }_{PO{S}_{R_a}\left(Q\right)}\left(4\right)=0.301$$

$${\mu }_{PO{S}_{R_a}\left(Q\right)}\left(5\right)=0.0$$

$${\mu }_{PO{S}_{R_a}\left(Q\right)}\left(6\right)=0.0.$$

The dependency of feature $a$:

$${{\mathsf{\gamma }}^{\prime }}_{\left\{a\right\}}\left(Q\right)=\frac{{\sum }_{x\in U}{\mu }_{PO{S}_{R_P}\left(Q\right)}\left(x\right)}{\left|\mathbb{U}\right|}=\frac{0.602}{6}=0.1003.$$

Using the same algorithm, we can calculate the ${{\mathsf{\gamma }}^{\prime }}_{\left\{b\right\}}\left(Q\right)=0.3597$, ${{\mathsf{\gamma }}^{\prime }}_{\left\{c\right\}}\left(Q\right)=0.4078$, in the first circulation we choose feature $c$. Then, in the second circulation we obtain ${{\mathsf{\gamma }}^{\prime }}_{\left\{a,c\right\}}\left(Q\right)=0.5501,{{\mathsf{\gamma }}^{\prime }}_{\left\{b,c\right\}}\left(Q\right)=1.0$, and in the end, we choose the feature subset $\left\{b,c\right\}$.

4. New Method

In the existing fuzzy rough feature selection algorithm, there are two methods for choosing the fuzzy set, one is given a fuzzy set while inputting data [39]. The other is definite with fuzzy similarity relations and a fuzzy implicator [40,41]. Both methods have their own drawbacks. The first one complicates the algorithm, so we need to add some knowledge to the feature selection, which is departing from our original intension. The second one has some problems in the definition of fuzzy similarity relations. The common definitions of relation at present are:

$${\mathsf{\mu }}_{R_a}\left(x,y\right)=1-\frac{\left|a\left(x\right)-a\left(y\right)\right|}{\left|a_{max}-a_{min}\right|},$$

(9)

$${\mathsf{\mu }}_{R_a}\left(x,y\right)=\exp \left(-\frac{{\left(a\left(x\right)-a\left(y\right)\right)}^2}{2{\sigma }_a^2}\right),$$

(10)

$${\mathsf{\mu }}_{R_a}\left(x,y\right)=max\left(min\left(\frac{\left(a\left(y\right)-\left(a\left(x\right)-{\sigma }_a\right)\right)}{\left(a\left(x\right)-\left(a\left(x\right)-{\sigma }_a\right)\right)},\frac{\left(\left(a\left(x\right)+{\sigma }_a\right)-a\left(y\right)\right)}{\left(\left(a\left(x\right)+{\sigma }_a\right)-a\left(x\right)\right)}\right),0\right),$$

(11)

where ${\sigma }_a^2$ is the variance of feature $a$. As we can see, what is said above defines the fuzzy similarity relations of all the datasets with a single equation but ignores the difference between the datasets. Generating a fuzzy set automatically is extremely urgent. A dataset is the universe of fuzzy sets that contains many fuzzy sets. We can abstract a fuzzy set and fuzzy similarity relations from a dataset and make it different between datasets, so the algorithm model has better generalization ability.

4.1. Reduction

The fuzzy c-means clustering algorithm (FCM) has a wide range application and is more successful in numerous fuzzy clustering algorithms. It obtains the membership degree of every sample point to the class center through the optimization of objective function [42].

Objective function is represented by the Euclidean distance of clustering center and sample point. Solving every clustering center to the minimum of the value function of the non-similarity index. The vague generalization is:

$$\mathrm{J}\left(\mathrm{U},c_1,c_2,\dots ,c_c\right)={\displaystyle \sum }_{i=1}^c{J}_i={\displaystyle \sum }_{i=1}^c{\displaystyle \sum }_j^n{u}_{ij}^m{d}_{ij}^2$$

(12)

where $u_{ij}$ is between 0 and 1, $c_i$ is the clustering center of the fuzzy set, $d_{ij}=\Vert c_i-x_j\Vert$ is the Euclidean distance of $i$th clustering center $j$th sample point. $m$ is the weighted index number. To construct the Lagrangian multiplier of the constraint formula, derivate all the input parameters to make Equation (12) reach the minimum:

$$c_i=\frac{{\sum }_{j=1}^n{u}_{ij}^m{x}_j}{{\sum }_{j=1}^n{u}_{ij}^m}$$

(13)

$${\mathsf{\mu }}_{ij}=\frac{1}{{\sum }_{k=1}^c{(\frac{d_{ij}}{d_{kj}})}^{2/\left(m-1\right)}}.$$

(14)

The whole processors of the clustering algorithm are as in Algorithm 2. The output of FCM are centers $c$ and membership matrix $U$. $U$ contains the degree of every object belonging to each center.

Algorithm 2: Fuzzy C-Means (FCM)

Input: Matrix of membership degree $\mathit{U}$ initialized with random numbers between 0 and 1, and satisfied the normalization, accuracy of convergence $\mathbf{\epsilon }$, maximum number of iterations $\mathit{T}$ and weighted index $\mathit{m}$.

Output:$\mathbf{U}$

1: t = 1

2: do

3:  ${\mathit{c}}_{\mathit{i}},\mathit{i}=\mathbf{1},\mathbf{2},\dots ,\mathit{c}$ with Equation (13)

4:  $\mathit{U}=[{\mathit{u}}_{\mathit{ij}}]$ with Equation (14).

5:  Update ${\mathit{c}}_{\mathit{i}}$ and $\mathit{U}$

6:  t←t + 1

7: Until $\mathit{t}=\mathit{T} \mathit{or} \Vert {\mathit{U}}_{\mathit{t}}-{\mathit{U}}_{\mathit{t}-\mathbf{1}}\Vert \le \mathbf{\epsilon }$

8: Return U

The definition of the lower approximation of fuzzy rough set is:

$${\mu }_{\underset{\_}{R_P}X}\left(x\right)=in{f}_{y\in U}I\left({\mu }_{R_P}\left(x,y\right),{\mu }_X\left(y\right)\right)$$

(15)

where $I$ is the fuzzy implicator. ${\mu }_{R_P}\left(x,y\right)$ represents the similarity relation between $x$ and $y$ in the whole subsets of feature set $P$. In order to contain only one similarity relation, we take the intersection of relations in all subsets of $P$, where the intersection is calculated with the t-norm.

$${\mu }_{R_P}\left(x,y\right)={\displaystyle \bigcap }_{a\in P}\left\{{\mu }_{R_a}\left(x,y\right)\right\}$$

(16)

where ${\mu }_{R_a}\left(x,y\right)$ represents the similarity degree of $x$ and $y$ about feature $a$.

Fuzzy clustering on every feature with Algorithm 2 to get the membership degree of every object to the feature:

$${\mu }_{R_a}\left(x\right)=U_{FCM\left(a\right)}\left[x\right].$$

(17)

Because equivalence relations are used to model equality, fuzzy equivalence relations are commonly considered to represent approximate equality or similarity [41,43]. We use the fuzzy equivalence relation $\mathrm{R}$ in the literature [44]:

$$R\left(x,y\right)=\frac{a-a\left|x-y\right|+bmin\left(x,y\right)}{a-\left(a-1\right)\left|x-y\right|+bmin\left(x,y\right)}.$$

(18)

Figure 1 shows that with different values of $a$ and $b$, the equation has a different form. The value of $a$ defines the basic shape of the function. Additionally, the value of $b$ defines the size of the opening on one side. As we use the function to describe the similarity of two objects, so we choose $b$ equal to 0 to make sure that the function is in balance of the two objects even if they are great values or small values. When $a$ goes to 0, the function performs a crisp relation that when two objects are equal the function output is 1, otherwise it is 0. Thus, we chose a small value on $a$, because after the FCM algorithm, the difference between objects was small.

Algorithm 3: Fuzzy C-Means Fuzzy Rough Feature Select (C-FRFS)

Input: the set of all conditional attributes C, the set of all decision attributes D and the set of all objects U. R $\leftarrow$ {empty set}

Output: R

1: For each x, y in U:

2:  ${\mathbf{\mu }}_{\mathit{x}}=\mathit{FCM}\left(\mathit{x}\right)$, ${\mathbf{\mu }}_{\mathit{y}}=\mathit{FCM}\left(\mathit{y}\right)$

3:  Calculate $\mathit{E}\left(\mathit{x},\mathit{y}\right)$

4:  ${\mathbf{\gamma }}_{\mathit{prev}}^{\prime }=\mathbf{0},{\mathbf{\gamma }}_{\mathit{best}}^{\prime }=\mathbf{0}$

5: do

6:  $\mathit{T}\leftarrow \mathit{R}$

7:  ${\mathbf{\gamma }}_{\mathit{prev}}^{\prime }={\mathbf{\gamma }}_{\mathit{best}}^{\prime }$

8:  $\mathit{if} {\mathbf{\gamma }}_{\mathit{R}\cup \left\{\mathit{a}\right\}}\left(\mathit{D}\right)>{\mathbf{\gamma }}_{\mathit{T}}\left(\mathit{D}\right)$, $\forall \mathit{a}\in \left(\mathit{C}-\mathit{R}\right)$,

9:   $\mathit{T}\leftarrow \mathit{R}\cup \left\{\mathit{a}\right\}$

10:  $\mathit{R}\leftarrow \mathit{T}$

11:  ${\mathbf{\gamma }}_{\mathit{best}}^{\prime }\leftarrow {\mathbf{\gamma }}_R\left(\mathit{D}\right)$

12: until $\left({\mathbf{\gamma }}_{\mathit{best}}^{\prime }-{\mathbf{\gamma }}_{\mathit{prev}}^{\prime }\right)\times |\mathit{U}|<\mathbf{1}$

13: return R

According to the clustering membership and Equation (18), we can get the fuzzy similarity relation of two objects.

$${\mu }_{R_a}\left(x,y\right)=\frac{a-a\left|U_{FCM\left(a\right)}\left[x\right]-U_{FCM\left(a\right)}\left[y\right]\right|}{a-\left(a-1\right)\left|U_{FCM\left(a\right)}\left[x\right]-U_{FCM\left(a\right)}\left[y\right]\right|}$$

(19)

The definitions of positive region and dependency are the same as we mentioned above [40].

$${\mu }_{PO{S}_{R_P}\left(Q\right)}\left(x\right)=su{p}_{x\in U/Q} {\mu }_{\underset{\_}{R_P}X}\left(x\right)$$

(20)

$${{\mathsf{\gamma }}^{\prime }}_P\left(Q\right)=\frac{{\sum }_{x\in U}{\mu }_{PO{S}_{R_P}\left(Q\right)}\left(x\right)}{\left|\mathbb{U}\right|}$$

(21)

The steps of the algorithm are as in Algorithm 3, and we simply called it C-FRFS, which means fuzzy rough feature selection based on clustering. We apply the fuzzy c-means clustering on every object in the universe $C$. For every two objects $x$ and $y$, use the fuzzy equivalence relation Equation (19) to describe the fuzzy similarity relation. Then, according to Equations (15), (20), and (21), we can obtain the dependency degree $\gamma$ of every feature in $C$. Start with an empty set $R$, and each time select a feature which has the greatest increase in the dependency degree. The algorithm stops when adding a feature cannot result in classifying at least one object.

4.2. Example

We still use the example in Table 1, and with the FCM we calculate the membership matrix as in

Table 2. Membership matrix of dataset in Table 1.

Object	a	b	c
1	0.9973	0.9967	0.9657
2	0.9843	0.9777	0.8617
3	0.9966	0.9014	0.9965
4	0.9973	0.9693	0.9441
5	0.9900	0.9967	0.9965
6	0.9966	0.9967	0.9965

. Assuming $C=\left\{a,b,c\right\},D=\left\{q\right\},IND\left(q\right)=\left\{\left\{2,4,5\right\},\left\{1,3,6\right\}\right\}$, with Equation (19), we can get the fuzzy similarity matrix:

$$U_{R_a}\left(x,y\right)=\left(\begin{array}{cc}\begin{array}{ccc}1& 0.0707& 0.5893\\ 0.0707& 1& 0.0744\\ 0.5893& 0.0744& 1\end{array}& \begin{array}{ccc}1& 0.1196& 0.5893\\ 0.0707& 0.1492& 0.0744\\ 0.5893& 0.1306& 1\end{array}\\ \begin{array}{ccc}1& 0.0707& 0.5893\\ 0.1196& 0.1492& 0.1306\\ 0.5893& 0.0744& 1\end{array}& \begin{array}{ccc}1& 0.1196& 0.5893\\ 0.1196& 1& 0.1306\\ 0.5893& 0.1306& 1\end{array}\end{array}\right)$$

$$U_{R_b}\left(x,y\right)=\left(\begin{array}{cc}\begin{array}{ccc}1& 0.0490& 0.0094\\ 0.0490& 1& 0.0120\\ 0.0094& 0.0120& 1\end{array}& \begin{array}{ccc}0.0343& 1& 1\\ 0.1066& 0.0490& 0.0490\\ 0.0135& 0.0094& 0.0094\end{array}\\ \begin{array}{ccc}0.0343& 0.1066& 0.0135\\ 1& 0.0490& 0.0094\\ 1& 0.0490& 0.0094\end{array}& \begin{array}{ccc}1& 0.0343& 0.0343\\ 0.0343& 1& 1\\ 0.0343& 1& 1\end{array}\end{array}\right)$$

$$U_{R_c}\left(x,y\right)=\left(\begin{array}{cc}\begin{array}{ccc}1& 0.0085& 0.0306\\ 0.0085& 1& 0.0064\\ 0.0306& 0.0064& 1\end{array}& \begin{array}{ccc}0.0433& 0.0306& 0.0306\\ 0.0110& 0.0064& 0.0064\\ 0.0178& 1& 1\end{array}\\ \begin{array}{ccc}0.0433& 0.0110& 0.0178\\ 0.0306& 0.0064& 1\\ 0.0306& 0.0064& 1\end{array}& \begin{array}{ccc}1& 0.0178& 0.0178\\ 0.0178& 1& 1\\ 0.0178& 1& 1\end{array}\end{array}\right)$$

For every element in $IND\left(q\right)$, calculate the lower approximation of every object. We give an example with $\left\{1,3,6\right\}$ in $IND\left(q\right)$ and object 2:

$${\mu }_{\underset{\_}{R_a}\left\{1,3,6\right\}}\left(x\right)=in{f}_{y\in U}I\left({\mu }_{R_a}\left(x,y\right),{\mu }_{\left\{1,3,6\right\}}\left(y\right)\right).$$

$$\begin{array}{ll}\hfill {\mu }_{\underset{\_}{R_a}\left\{1,3,6\right\}}\left(2\right)& ={\inf }_{y\in U}I\left({\mu }_{R_a}\left(2,y\right),{\mu }_{\left\{1,3,6\right\}}\left(y\right)\right)\\ & =\inf \left\{I\left(0.0707,1\right),I\left(1,0\right),I\left(0.0744,1\right),I\left(0.0707,0\right),I\left(0.1492,0\right),I\left(0.0744,1\right)\right\}\\ & =0.8508\end{array}$$

Others in the same way:

$${\mu }_{\underset{\_}{R_a}\left\{1,3,6\right\}}\left(1\right)=0.0$$

$${\mu }_{\underset{\_}{R_a}\left\{1,3,6\right\}}\left(3\right)=0.0$$

$${\mu }_{\underset{\_}{R_a}\left\{1,3,6\right\}}\left(4\right)=0.0$$

$${\mu }_{\underset{\_}{R_a}\left\{1,3,6\right\}}\left(5\right)=0.0$$

$${\mu }_{\underset{\_}{R_a}\left\{1,3,6\right\}}\left(6\right)=0.0.$$

For $\left\{2,4,5\right\}$

$${\mu }_{\underset{\_}{R_a}\left\{2,4,5\right\}}\left(1\right)=0.0$$

$${\mu }_{\underset{\_}{R_a}\left\{2,4,5\right\}}\left(2\right)=0.0$$

$${\mu }_{\underset{\_}{R_a}\left\{2,4,5\right\}}\left(3\right)=0.0$$

$${\mu }_{\underset{\_}{R_a}\left\{2,4,5\right\}}\left(4\right)=0.0$$

$${\mu }_{\underset{\_}{R_a}\left\{2,4,5\right\}}\left(5\right)=0.8508$$

$${\mu }_{\underset{\_}{R_a}\left\{2,4,5\right\}}\left(6\right)=0.0.$$

Thus, the positive region of every object:

$${\mu }_{PO{S}_{R_a}\left(Q\right)}\left(1\right)=0.0$$

$${\mu }_{PO{S}_{R_a}\left(Q\right)}\left(2\right)=0.8508$$

$${\mu }_{PO{S}_{R_a}\left(Q\right)}\left(3\right)=0.0$$

$${\mu }_{PO{S}_{R_a}\left(Q\right)}\left(4\right)=0.0$$

$${\mu }_{PO{S}_{R_a}\left(Q\right)}\left(5\right)=0.8508$$

$${\mu }_{PO{S}_{R_a}\left(Q\right)}\left(6\right)=0.0.$$

The dependency of feature $\mathrm{a}$:

$${{\mathsf{\gamma }}^{\prime }}_{\left\{a\right\}}\left(Q\right)=\frac{{\sum }_{x\in U}{\mu }_{PO{S}_{R_P}\left(Q\right)}\left(x\right)}{\left|\mathbb{U}\right|}=\frac{1.7016}{6}=0.2836.$$

In the same way, we can obtain ${{\mathsf{\gamma }}^{\prime }}_{\left\{b\right\}}\left(Q\right)=0.4624$, ${{\mathsf{\gamma }}^{\prime }}_{\left\{c\right\}}\left(Q\right)=0.4837$.

Thus, in the first circulation, we choose feature $c$. Then, in the second circulation, we obtain ${{\mathsf{\gamma }}^{\prime }}_{\left\{a,c\right\}}\left(Q\right)=0.6305,{{\mathsf{\gamma }}^{\prime }}_{\left\{b,c\right\}}\left(Q\right)=0.9831$, and in the end we choose feature $b$ and $c$ as our result of feature selection.

5. Experiments

In this part, we used nine classifiers to classify nine datasets from UCI machine learning repository [45], to verify and evaluate our feature selection algorithm. The algorithm uses fuzzy clustering to generate fuzzy sets, which are defined by Equation (17). After feature selection, the datasets are reduced. These reduced datasets are classified using related classifiers (unreduced datasets do not use feature selection steps).

The T-FRFS is the short of threshold fuzzy rough feature select which was represented in [46] and FRFS is mentioned in [40]. C-FRFS is our proposed method mentioned in this paper. All the three methods were written in MATLAB 2017b which runs on a computer with following characteristics:

• OS: Microsoft Windows 10
• CPU: Intel Core i5-8400 CPU 2.80GHz
• RAM: 8GB.

We use these two proposed successors to compare with our new method and this indicted that we have made some progress on fuzzy rough feature selection. Nine datasets as depicted in

Table 3. Information of nine datasets and reduct size of three feature selection methods. C-FRFS: c-means fuzzy rough feature select. T-FRFS: threshold fuzzy rough feature select.

No.	Dataset	Objects	Features	Classes	Reduct Size
					C-FRFS	T-FRFS	FRFS
1	Glass	214	9	7	4	8	9
2	Iris	150	4	3	3	3	4
3	Cleveland	303	13	5	4	6	8
4	Wine	178	13	3	4	4	5
5	User	431	5	4	3	4	5
6	Vertebral	310	6	4	4	5	6
7	Sonar	208	60	2	2	4	5
8	Yeast	1484	8	10	6	7	8
9	Hillvalley	606	100	2	24	96	100

, were employed to evaluate each methods’ performance. The results of all methods, as well as unreduced datasets in terms of the number of selected features, are also demonstrated in Table 3. As shown, our new method C-FRFS always gets the smallest reduced subset and then the T-FRFS. The FRFS does not perform well and sometimes cannot select features.

Figure 2 shows the running time of three methods. In Figure 2a, our new method is always slower than the other two methods. However, in the Hillvalley dataset, our new method performs better, which means it is more suitable for high-dimensional data than the other two methods. The reason is that our method obtains a smaller reduct, so the program ends earlier and running time is less.

To test the performance of the new method, we applied three methods on different sizes of datasets. With the number of objects from 200 to 5000, the time consumption increased exponentially. However, as shown in Figure 3, the three methods have the same tendency and are almost coincident, which demonstrates that clustering nearly not influences the performance of our new method. In the same case, our new method can get the best consequence but has no impact on efficiency.

Nine classifiers of different categories such as Bayes Net, Naïve Bayes, RBFNetwork, Jrip, PART, BFTree, FT, J48, and NBTree in Weka were selected to classify resulting subsets of features by each FS method [47]. To obtain more accurate results, we used 10-fold cross validation in the modeling of each classifier method. The results are presented in

Table 4. Resulting classification accuracies. Unred.: unreduced.

Datasets	Glass	Iris	Cleveland	Wine	User	Vertebral	Sonar	Yeast	Hillvalley	Mean
C-FRFS	63.66	94.52	55.01	95.51	91.01	80.90	72.44	56.36	50.59	73.33
T-FRFS	64.85	94.52	54.46	94.82	90.99	71.11	72.49	56.55	50.84	72.29
FRFS	66.46	94.82	54.61	95.38	90.65	81.11	70.04	56.99	51.03	73.45
Unred.	66.46	94.82	54.42	95.13	90.65	81.11	76.07	56.99	51.03	74.08

where the average classification accuracies of nine classifiers for each dataset worked with three selection methods along with unreduced datasets shown in each column, and the last column indicated the mean of nine classification accuracies. The highest mean of classification accuracies was gained in expense of employing all features, which means that feature selection methods are not always successful in increasing classification accuracy, but in decreasing model complexity by sacrificing inconsiderable accuracy. As Table 4 shows, for four datasets, the original model gained the highest classification accuracy, three for both FRFS and our new method.

Since both the classification accuracies and the number of selected features are important, divisions of classification accuracies, as shown in Table 4, by the number of selected features, as shown in Table 3, were considered as a measure to compare the results [48]. Therefore, a method which has a higher classification accuracy and smaller set of features is regarded as a better method. The results are shown in

Table 5. Division of accuracies and number of selected features.

Datasets	Glass	Iris	Cleveland	Wine	User	Vertebral	Sonar	Yeast	Hillvalley	Mean
C-FRFS	15.92	31.51	13.75	23.88	30.34	20.23	36.22	9.39	2.11	24.55
T-FRFS	8.11	31.51	9.08	23.71	22.75	14.22	18.12	8.08	0.53	18.21
FRFS	7.38	23.71	6.83	19.08	18.13	13.52	14.01	7.12	0.51	14.66
Unred.	7.38	23.71	4.19	7.32	18.13	13.52	1.27	7.12	0.51	10.79

. The lowest value was from the original datasets, and our new method reaches the highest value in each dataset. Sometimes, the T-FRFS also reaches the highest value.

A statistical test with the Friedman test was used to compare the methods.

Table 6. Average ranks of methods.

Method	Rank
C-FRFS	1.0556
T-FRFS	1.9444
FRFS	3.3333
Unred.	3.6667

shows the average ranks of methods which correspond to Table 5. The chi-square with 3 freedom was 26.1 and P-value was 0, which means there are statistical difference between methods.

6. Discussion and Conclusions

A new method named C-FRFS based on fuzzy rough feature selection has been presented in this paper. The first development based on the fuzzy c-means clustering uses a new kind of membership function which automatically forms the membership matrix only with the knowledge contained in the dataset itself. The second development employs the setting up of the fuzzy relation—different to methods used in many articles which use a single equation—we have combined fuzzy equivalence mentioned in other research and the membership function generated through the above way to set up the fuzzy relation. An example is given to illustrate how reduction may be achieved. Note that no user-defined thresholds are required for the method, although a choice must be made regarding fuzzy connections and the priori coefficient of fuzzy equivalence. Experimental results over nine datasets taken from UCI showed the applicability and effectiveness of our proposed new method. In order to measure the performance of every method, we considered the number of selected features, classification accuracies, and the division of classification accuracies by the number of selected features. We compared the results operated on different methods, such as T-FRFS, FRFS, unreduced, and our new method C-FRFS. The comparisons indicated that our method was outperformed. We also compared the time taking of three methods on different sizes of datasets. The results demonstrated that our new method did not influence the efficiency remarkably. That was to say, we improved the accuracy and validity of feature selection with nearly no time loss.

Further research in this area will include a more in-depth experimental investigation of the prior coefficient with its evaluation and a more efficient fuzzy clustering method to raise up the accuracies of our feature selection method.