Adaptive Two-Index Fusion Attribute-Weighted Naive Bayes

Abstract

Naive Bayes (NB) is one of the essential algorithms in data mining. However, it is rarely used in reality because of the attribute independence assumption. Researchers have proposed many improved NB methods to alleviate this assumption. Among these methods, due to its high efficiency and easy implementation, the filter-attribute-weighted NB methods have received great attentions. However, there still exist several challenges, such as the poor representation ability for a single index and the fusion problem of two indexes. To overcome the above challenges, we propose a general framework of an adaptive two-index fusion attribute-weighted NB (ATFNB). Two types of data description category are used to represent the correlation between classes and attributes, the intercorrelation between attributes and attributes, respectively. ATFNB can select any one index from each category. Then, we introduce a regulatory factor $\beta$ to fuse two indexes, which can adaptively adjust the optimal ratio of any two indexes on various datasets. Furthermore, a range query method is proposed to infer the optimal interval of regulatory factor $\beta$. Finally, the weight of each attribute is calculated using the optimal value $\beta$ and is integrated into an NB classifier to improve the accuracy. The experimental results on 50 benchmark datasets and a Flavia dataset show that ATFNB outperforms the basic NB and state-of-the-art filter-weighted NB models. In addition, the ATFNB framework can improve the existing two-index NB model by introducing the adaptive regulatory factor $\beta$. Auxiliary experimental results demonstrate the improved model significantly increases the accuracy compared to the original model without the adaptive regulatory factor $\beta$.

1. Introduction

The naive Bayes (NB) classifier is a classical classification algorithm. Due to its simplicity and efficiency, it is widely used in many fields such as data mining and pattern recognition.

Assume that a dataset $D=\{x_1,x_2,\dots ,x_m\}$ contains m training instances, an instance $x_i$ can be represented by an n-dimensional attribute value vector $<x_{i1},x_{i2},\dots ,x_{in}>$. The NB classifier uses Equation (1) to predict the class label of the instance $x_i$.

$$c\left(x_i\right)=\underset{c\in C}{argmax}P\left(c\right)\prod _{j=1}^{n}P\left(x_{ij}\right|c)$$

(1)

where C is the set of all possible class labels c, n is the number of attributes, and $x_{ij}$ represents the value of the jth attribute of the ith instance. $P\left(c\right)$ is the prior probability of class c, and $P\left(x_{ij}\right|c)$ is the conditional probability of the attribute value $x_{ij}$ given the class c, which can be calculated by Equations (2) and (3), respectively.

$$P\left(c\right)=\frac{{\sum }_{i=1}^m\delta (c_i,c)+1}{m+\vartheta \left(C\right)}$$

(2)

$$P\left(x_{ij}\right|c)=\frac{{\sum }_{i=1}^m\delta (c_i,c)\delta (x_{ij},A_j)+1}{{\sum }_{i=1}^m\delta (c_i,c)+\vartheta \left(A_j\right)}$$

(3)

where $A_j$ represents all the values of the jth attribute in training instances. $\vartheta$(·) is a custom function to calculate the number of unique data in C or $A_j$. $c_i$ denotes the correct class label for the ith instance. $\delta$(·) is a binary function, which takes the value one if $c_i$ and c are identical and zero otherwise [1].

Duo to the attribute independence assumption, the NB classifier is a simple, stable, easy to implement, and better classification algorithm for various applications. However, real data are complicated and diverse, which make it difficult to satisfy this assumption. Thus, researchers have proposed many methods to reduce the influence of the attribute independence assumption. These methods can be divided into six categories: Structure extension models directed arcs to represent the dependence relationship between attributes [2,3,4,5,6]. Fine-tuning adjusts the probability value to find a good estimation of the desired probability term [7,8]. The purpose of instance selection is to construct NB model on a subset of training set instead of the whole training set [9,10]. Instance weighting consists of assigning instances different weights by different strategies [11,12,13]. Attribute selection is the process of removing redundant attributes [14,15,16,17,18,19]. Distinguished from attribute selection, attribute weighting assigns a weight to each attribute in order to relax the independence assumption and make the NB model more flexible [20,21,22,23,24,25,26,27].

In this paper, we focus our attention on attribute weighting, which is further divided into wrapper methods and filter methods. The wrapper methods optimize the weighted matrix by using gradient descent to improve classification performance. Wu et al. proposed a weighted NB algorithm based on differential evolution, which gradually adjusted the weights of attributes through evolutionary algorithms to improve the prediction results [28]. Zhang et al. proposed two attribute-value-weighting models based on the conditional log-likelihood and mean square error [1]. However, these methods are often less efficient due to the time-consuming optimization process. Another category obtained the weights by analyzing the correlation of attributes [20,21,24,26,27]. Since correlation can be easily and efficiently obtained by various measurement indexes, the computational efficiency of filter methods obviously increases. Related filter methods are detailed introduced in Section 2. Although filter methods have some advantages such as being flexible and computational efficient, there are still two problems. Most of the methods utilize a single index, which expresses the data characteristic, to determine the attribute weight. However, a single index cannot comprehensively discover information about the dataset. In order to fully dig up the information from the dataset, a two-index fusion method was proposed, which could achieve better performance [29]. However, the ratio of the two indexes became the second problem. The method assumed that the contributions of the two indexes were equivalent and ignored the difference in contribution between the two indexes.

To overcome the above problems, we propose an adaptive two-index fusion attribute-weighted naive Bayes (ATFNB) method. ATFNB can select any index from two categories of data description, respectively. The first category describes the correlation between attributes and classes, and the second category describes the intercorrelation between attributes and attributes. Once two indexes have been selected, ATFNB fuses the two indexes by introducing a regulatory factor $\beta$. Due to the diversity of datasets, the regulatory factor $\beta$ can be adaptively adjusted to get the optimal ratio between the two indexes. Moreover, a range query method is proposed to obtain the optimal value of the regulatory factor $\beta$. To verify the effectiveness of ATFNB, we conduct extensive experiments on 50 UCI datasets and a Flavia dataset. Experimental results show that ATFNB has a better performance compared to NB and state-of-the-art filter NB models.

To sum up, the main contributions of our work include the following:

• We propose a general framework, ATFNB, which fuses any two indexes from two categories of data description. Our framework can derive all existing filter-attribute-weighted NB models by selecting difference indexes.
• We introduce a regulatory factor $\beta$, which can adaptively adjust the optimal ratio of two indexes, and shows which index is more important on various datasets.
• A quick range query method is proposed to obtain the optimal value of the regulatory factor $\beta$. Compared to the traditional method, step-length searching, our method obviously speeds up the optimization.
• The existing two-index NB methods’ performance are significantly improved by introducing the regulatory factor $\beta$.

The rest of the paper consists of the following parts. Section 2 comprehensively reviews the filter-attribute-weighted methods. Section 3 proposes an adaptive two-index fusion attribute-weighted naive Bayes method. Section 4 presents the experimental datasets, setting and results. Section 5 further discusses the experimental results. Finally, Section 6 summarizes the research and gives some future work.

2. Related Work

Given a dataset D with n attributes and K classes. The naive Bayes weight matrix is shown in

Table 1. The naive Bayes weight matrix.

Class	${\mathit{A}}_1$	${\mathit{A}}_2$	……	${\mathit{A}}_{\mathit{n}-2}$	${\mathit{A}}_{\mathit{n}-1}$	${\mathit{A}}_{\mathit{n}}$
$c_1$	$w_1$	$w_2$	……	$w_{n-2}$	$w_{n-1}$	$w_n$
$c_2$	$w_1$	$w_2$	……	$w_{n-2}$	$w_{n-1}$	$w_n$
$...$	$...$	$...$	……	$...$	$...$	$...$
$c_{K-1}$	$w_1$	$w_2$	……	$w_{n-2}$	$w_{n-1}$	$w_n$
$c_K$	$w_1$	$w_2$	……	$w_{n-2}$	$w_{n-1}$	$w_n$

.

The naive Bayes method incorporates the attribute weight into the formula as follows:

$$\widehat{c}\left(x_i\right)=\underset{c\in C}{argmax}P\left(c\right)\prod _{j=1}^{n}P{\left(x_{ij}\right|c)}^{w_j}$$

(4)

where $w_j$ is the weight of the jth attribute $A_j$. The most critical issue of filter-weighted NB methods is how to determine the weight $w_j$ of each attribute, which has attracted more great attention. Many weighted NB methods have been proposed based on various measurements of attribute weighted. Here, we introduce several state-of-the-art filter-weighted NB methods.

Ferreira et al. firstly proposed a weighted naive Bayes method to alleviate the independence assumption, which assigned weights to different attributes [30]. Based on this idea, Zhang et al. presented an attribute-weighted model based on a gain ratio (WNB) [31]. Attributes with a higher gain ratio deserved higher weights in WNB. Therefore, the weight of each attribute can be defined by Equation (5).

$$w_j=\frac{GainRatio(D,A_j)}{\frac{1}{n}{\sum }_{j=1}^{n}GainRatio(D,A_j)}$$

(5)

where $GainRatio(D,A_j)$ is the gain ratio of attribute $A_j$ [32].

Then, Lee et al. proposed a novel model that used the Kullback–Leibler metric to calculate the weight of each attribute [33]. This model was certain information between each attribute and the corresponding class label c, which was obtained by Kullback–Leibler [34] measuring the difference between the prior distribution and the posterior distribution of the target attributes. The weight value of the jth attribute is shown in Equation (6).

$$w_j=\frac{1}{Z}\frac{{\sum }_{j|i}P\left(x_{ij}\right)KL\left(c\right|x_{ij})}{-{\sum }_{j|i}P\left(x_{ij}\right)log\left(P\left(x_{ij}\right)\right)}$$

(6)

where $P\left(x_{ij}\right)$ means the probability of the value $x_{ij}$, and $Z=\frac{{\sum }_i^n{w}_i}{n}$ is a normalization constant. $KL\left(c\right|x_{ij})$ is the average mutual information between the class label c and the attribute value of $x_{ij}$.

Next, Jiang’s team proposed a series of filter-attribute-weighted methods, which included deep feature weighting (DFW) [26] and correlation-based feature weighting (CFW) [29]. DFW assumed that more independent features should be assigned higher weights. The correlation-based feature selection was used to evaluate the degree of dependence between attributes [35]. According to this selection, the best subset was selected from the attribute space. The weight value assigned to the selected attribute was two, and the weight value assigned to other attributes was one, as shown in Equation (7).

$$w_j=\left\{\begin{array}{cc}2,& if{A}_{j}isselected.\\ 1,& otherwise.\end{array}\right.$$

(7)

Compared with the above methods with a single index, CFW was the first two-index weighted NB method, which used the attribute-class correlation and the average attribute–attribute intercorrelation to constitute the weight of each attribute. The mutual information was measured from the attribute–class correlation and the attribute–attribute intercorrelation, defined as Equations (8) and (9), respectively.

$$I(A_j;C)=\sum _{a_j}\sum _{c}P(a_j,c)log\frac{P(a_j,c)}{P\left(a_j\right)P\left(c\right)}$$

(8)

$$I(A_i;A_j)=\sum _{a_i}\sum _{a_j}P(a_i,a_j)log\frac{P(a_i,a_j)}{P\left(a_i\right)P\left(a_j\right)}$$

(9)

where $a_i$ and $a_j$ represent the values of attributes $A_i$ and $A_j$, respectively. $I(A_j;C)$ is the correlation between the attribute $A_j$ and class C. $I(A_i;A_j)$ is the redundancy between two different attributes $A_i$ and $A_j$. Finally, the weight of the attribute $w_j$ is defined as Equation (10).

$$w_j=\frac{1}{1+e^{-\left(NI(A_j;C)-\frac{1}{n-1}{\sum }_{j=1\cap j\ne i}^{n}NI(A_i;A_j)\right)}}$$

(10)

where $NI(A_j;C)$ and $NI(A_i;A_j)$ are the normalized values, which, respectively, represent the maximum correlation and the maximum redundancy.

3. ATFNB

3.1. The General Framework of ATFNB

The filter-weighted NB methods assign a specific weight for each attribute to alleviate the independence assumption. However, there are still some challenges, such as the poor representation ability for a single index and the fusion problem of two indexes. Therefore, we propose an adaptive two-index fusion attribute-weighted NB. The framework of ATFNB is shown in Figure 1. Given a dataset, two indexes are selected from a class–attribute category and an attribute–attribute category, respectively. Then, the regulatory factor $\beta$ is utilized to fuse the two indexes, and adaptively generate the optimal ratio value. Next, the weight of each attribute is calculated via the optimal regulatory factor $\beta$. Finally, the attribute weights are incorporated into the NB classifier to predict the class labels.

3.2. Index Selection

As shown in Figure 1, the ATFNB framework contains two widely used types of the attribute correlation: class–attribute and attribute–attribute. The class–attribute category is to measure the correlation between attributes and classes. The stronger the correlation between an attribute and a class, the more significant the attribute’s contribution to the classification. Thus, the index value is positively correlated with the weight. Common indexes in this category include the mutual information, Pearson correlation coefficient, information gain, gain ratio, etc. The attribute–attribute category is to measure the redundancy between attributes. In order to satisfy the independence assumption of the naive Bayes classifier as much as possible, attributes with high redundancy are assigned small weights. Thus, the weight is inversely correlated to the index value. Common measures of redundancy between attributes include the mutual information and Pearson correlation coefficient, etc. Except for the above six indexes in Figure 1, researchers can add any other index or design new indexes to represent data characteristics.

By selecting different indexes, the ATFNB framework can become any weighted NB model, including the existing weighted NB models. If only the gain ratio is selected from the class–attribute category, ATFNB degenerates into the single-index WNB model. If the class–attribute and attribute–attribute category both choose the mutual information, ATFNB becomes the two-index CFW model. Thus, the index selection is a critical step in the ATFNB framework. Any two indexes selected from the two categories can generate various models, which may achieve different results. In Section 5.3, the classification performance of different index selections is discussed in detail.

3.3. Range Query Method for the Regulatory Factor $\beta$

Since the significantly discriminative attribute should be highly correlated with the class and has low redundancy with other attributes, its weights should be positively associated with the difference between the class–attribute correlation and attribute–attribute intercorrelation [29]. The mathematics formula of the weight $w_j$ can be defined by Equation (11).

$$w_j=\underset{C{A}_j}{\underbrace{class\_attribute}}-\underset{A{A}_j}{\underbrace{attribute\_attribute}}$$

(11)

where $C{A}_j$ and $A{A}_j$ represents the values of the selected class–attribute index and attribute–attribute index, respectively. The existing two-index methods, CFW, consider that the contributions of two indexes are equivalent [29]. However, various indexes contain different characteristics, and an equivalent contribution of the two indexes is unreasonable. Thus, we introduce a regulatory factor $\beta$ to adaptively control the ratio of two indexes. After incorporating the regulatory factor $\beta$, Equation (11) can be rewritten as Equation (12).

$$w_j=\beta \times C{A}_j-(1-\beta )\times A{A}_j$$

(12)

where the regulatory factor $\beta \in [0,1]$.

Conventionally, the step-length searching strategy can be applied to search the optimal interval of $\beta$. However, the accuracy and computational efficiency are affected by the step size. When the step size gets smaller, the optimal interval of $\beta$ is more accurate but the search gets much slower. Thus, we propose a range query method to calculate the optimal interval of the regulatory factor $\beta$. Firstly, the basic weighted NB model (Equation (4)) is logarithmically transformed, and the detailed transformation process is shown as follows.

$$\begin{array}{cc}\hfill T(x_i,c)& =log\left(P\left(c\right)\right)+\sum _{j=1}^{n}log{\left(P\left(x_{ij}\right|c)\right)}^{w_j}\hfill \\ \hfill & =log\left(P\left(c\right)\right)+\left[\begin{array}{c}{w}_1,...,w_j\end{array}\right]\left[\begin{array}{c}log\left(P\left(x_{i1}\right|c)\right)\\ ...\\ log\left(P\left(x_{ij}\right|c)\right)\end{array}\right]\hfill \\ \hfill & =\beta \underset{K_c}{\underbrace{\left[\begin{array}{c}(C{A}_1+A{A}_1),...,(C{A}_j+A{A}_j)\end{array}\right]\left[\begin{array}{c}log\left(P\left(x_{i1}\right|c)\right)\\ ...\\ log\left(P\left(x_{ij}\right|c)\right)\end{array}\right]}}\hfill \\ \hfill & +\underset{M_c}{\underbrace{log\left(P\left(c\right)\right)-\left[\begin{array}{c}A{A}_1,...,A{A}_j\end{array}\right]\left[\begin{array}{c}log\left(P\left(x_{i1}\right|c)\right)\\ ...\\ log\left(P\left(x_{ij}\right|c)\right)\end{array}\right]}}\hfill \\ \hfill & ={\beta }_i\times K_c+M_c\hfill \end{array}$$

(13)

where $T(x_i,c)$ is the probability value that the instance $x_i$ belongs to class c. ${\beta }_i$ is the interval when instance $x_i$ is correctly classified.

Based on Equation (13), a probability set $S_i$ can be constructed to store the probability values of instances $x_i$ belonging to different classes. The probability set of $x_i$ is $S_i=\{T(x_i,c_1),T(x_i,c_2),\dots ,T(x_i,c_K)\}$. If the correct label of $x_i$ is $c_k$, $T(x_i,c_k)$ should be greater than the other probability values in $S_i$. This can be defined as follows:

$$T(x_i,c_k)>\{S_i-T(x_i,c_k)\}$$

(14)

When the instance $x_i$ is correctly classified, the interval of ${\beta }_i$ can be obtained. For m instances, a set $G=\{{\beta }_1,{\beta }_2,\dots ,{\beta }_m\}$ contains the m intervals corresponding to each instance. To calculate the optimal interval ${\beta }^{*}$ from G, so that any value in the interval can obtain the same classification accuracy on the training set, the upper and lower bounds of all intervals in G are sorted in ascending order $Q=\{valu{e}_1,valu{e}_2,\dots ,valu{e}_q\}$. Any two adjacent values in Q are regarded as the lower and upper bounds of a subinterval. Thus, Q can generate $q-1$ subintervals. The subintervals in $R=\{{\gamma }_1,{\gamma }_2,...,{\gamma }_{q-1}\}$ satisfying Equation (15) are taken as ${\beta }^{*}$.

$$max\{\sum _{i=1}^m\tau ({\gamma }_1,{\beta }_i),\sum _{i=1}^m\tau ({\gamma }_2,{\beta }_i),...,\sum _{i=1}^m\tau ({\gamma }_{q-1},{\beta }_i)\}$$

(15)

where $\tau$(·) is a binary function, which takes the value 1 if ${\gamma }_{q-1}$ is a subset of ${\beta }_i$ and 0 otherwise, as shown in Equation (16).

$$\tau =\left\{\begin{array}{cc}1,& if{\gamma }_{q-1}\subseteq {\beta }_i.\\ 0,& otherwise.\end{array}\right.$$

(16)

According to the above derivation processes, the range query method for the regulatory factor (RQRF) is described in Algorithm 1.

Algorithm 1: RQRF

Input: class–attribute ($C{A}_j$), attribute–attribute ($A{A}_j$), dataset D For each instance $x_i$ in D:       For each class c in C:             Calculate $K_c$ and $M_c$ in Equation (13).             According $K_c$ and $M_c$, get $T(x_i,c)$.       End       $S_i=\{T(x_i,c_1),T(x_i,c_2),\dots ,T(x_i,c_K)\}$       If instance $x_i$ label is $c_k$:             Find the value that satisfies Equation (14); it is recorded as ${\beta }_i$,             otherwise ${\beta }_i$ = ∅.       End       $G=\{{\beta }_1,{\beta }_2,\dots ,{\beta }_m\}$ End For each ${\gamma }_{q-1}$       Find the subinterval ${\beta }^{*}$ that conforms to Equation (15). End Output:${\beta }^{*}$

Any value in ${\beta }^{*}$ can achieve a consistent classification accuracy in dataset D, so we choose any value from the optimal interval ${\beta }^{*}$. After obtaining the value of the regulatory factor, the weight $w_i$ can be calculated by Equation  (12).

3.4. The Implementation of ATFNB

The general framework of ATFNB is briefly described in Algorithm 2. According to Algorithm 2, we can see that how to select two indexes $A{A}_j$ and $C{A}_j$, and how to learn the regulatory factor $\beta$ are two crucial problems. To select $A{A}_j$ and $C{A}_j$, several indexes are listed in Section 3.2. To learn the value of the regulatory factor $\beta$, we single out an RQRF algorithm in Section 3.3. Once the value of the regulatory factor $\beta$ is obtained, we can use Equation (12) to calculate the weights of each attribute. Finally, these weights are applied to construct an attribute-weighted NB classifier.

Algorithm 2: ATFNB Framework

Input: Training set D, test set X       (1)   For each attribute $A_j$ in D                   Calculate (attribute–attribute) index $A{A}_j$                   Calculate (class–attribute) index $C{A}_j$.       (2)   According to RQRF, the value of the regulatory factor $\beta$ is solved.       (3)   According to Equation (12), the weight matrix is obtained.       (4)   According to Equation (4), the class label of each instance in X is predicted. Output: The class label of instances in X

4. Experiments and Results

4.1. Experimental Data

To verify the effectiveness of ATFNB, a collection of 50 benchmark datasets and 15 groups of the leaf dataset were conducted.

The 50 benchmark classification datasets were chosen from the University of California at Irvine (UCI) repository [36], representing various fields and data characteristics, as listed in

Table 2. Descriptions of 50 UCI datasets used in the experiments.

Dataset	Instance Number	Attribute Number	Class Number
abalone	4177	8	3
acute	120	6	2
aggregation	788	2	7
balance-scale	625	4	3
bank	4521	16	2
banknote	1372	4	2
blood	748	4	2
breast-cancer	286	9	2
breast-tissue	106	9	6
bupa	345	6	2
car	1728	6	4
chart_Input	600	60	6
climate-simulation	540	18	2
congressional-voting	435	16	2
connectionist	208	60	2
dermatology	366	34	6
diabetes	768	8	2
ecoli	336	7	8
energy-y1	768	8	3
fertility	100	9	2
glass	214	9	6
haberman-survival	306	3	2
iris	150	4	3
jain	373	2	2
knowledge	172	5	4
libras	360	90	15
low-res-spect	531	100	9
lymphography	148	18	4
magic	19,020	10	2
mammographic	961	5	2
promoters	106	57	2
splice	3190	60	3
nursery	12,960	8	5
page-blocks	5473	10	5
pima	768	8	2
planning	182	12	2
post-operative	90	8	3
robotnavigation	5456	24	4
seeds	210	7	3
sonar	208	60	2
soybean	683	35	18
spect	265	22	2
synthetic-control	600	60	6
tic-tac-toe	958	9	2
titanic	2201	3	2
twonorm	7400	20	2
wall-following	5456	24	4
waveform	5000	21	3
wilt	4839	5	2
wine	178	13	3

. We used the mean of the corresponding attribute to replace missing data values in each dataset, then applied chi-square-based algorithm to discretize the numerical attribute values [37]. The amount of discretization of each attribute was consistent with the number of types of class labels.

The Flavia dataset contains 32 types of leaf and each leaf has 55–77 pieces. Four texture and ten shape features of each leaf were extracted based on grayscale and binary images [38]. We constructed 15 groups for comparative experiments, and in each group we randomly selected 15 kinds of leaves from the whole Flavia dataset. The detailed characteristics of the 15 groups are listed in

Table 3. Descriptions of 15 groups from Flavia dataset used in the experiments.

Dataset	Instance Number	Attribute Number	Class Number
$G\_1$	869	14	15
$G\_2$	888	14	15
$G\_3$	865	14	15
$G\_4$	887	14	15
$G\_5$	884	14	15
$G\_6$	919	14	15
$G\_7$	892	14	15
$G\_8$	881	14	15
$G\_9$	864	14	15
$G\_10$	879	14	15
$G\_11$	895	14	15
$G\_12$	888	14	15
$G\_13$	927	14	15
$G\_14$	924	14	15
$G\_15$	904	14	15

. Then, the same preprocessing pipeline as for the UCI dataset was applied to discretize the continuous attributes.

4.2. Experimental Setting

ATFNB is a general framework of attribute-weighted naive Bayes, which can adaptively fuse any two indexes. According to Figure 1, we selected two simple and popular indexes from two categories: the information gain from the class–attribute category and the Pearson correlation coefficient from the attribute–attribute category. Notably, ATFNB refers to a specific NB model fused from the above two indexes in the following experiments, and no longer represents a general framework.

For the class–attribute category, the information gain describes the information content provided by the attribute for the classification. The formula of the information gain is shown in Equation (17).

$$Gain(D;A_j)=Ent\left(D\right)-\sum _{v=1}^V\frac{|D^v|}{\left|D\right|}Ent\left(D^v\right)$$

(17)

where $Ent\left(D\right)$ is the information entropy. The discrete attribute $A_j$ has V values $\{a^1,a^2,\dots ,a^V\}$. $D^v$ indicates that the vth branch node contains all the instances in dataset D, whose value is $a^V$ on the attribute $A_j$.

For the attribute–attribute category, the Pearson correlation coefficient was used to calculate the correlation between attributes $A_i$ and $A_j$, and the formula can be written as Equation (18).

$$\rho (A_i;A_j)=\left|\frac{cov(A_i,A_j)}{{\sigma }_{A_i}{\sigma }_{A_j}}\right|$$

(18)

where $cov(A_i,A_j)$ represents the covariance between attributes $A_i$ and $A_j$, ${\sigma }_{A_i}$ and ${\sigma }_{A_j}$ represent the standard deviation of $A_i$ and $A_j$, respectively.

After obtaining these two indexes, the weight of each attribute $A_j$ could be calculated as Equation (19).

$$w_j=\beta \times NGain(D;A_j)-(1-\beta )\times avg\_PCC\left(A_j\right)$$

(19)

where $NGain(D;A_j)$ is expressed as the normalized value of the attribute’s information gain, and $avg\_PCC\left(A_j\right)$ represents the average degree of redundancy between the ith attribute and other attributes. The formula of $avg\_PCC\left(A_j\right)$ is shown in Equation (20).

$$avg\_PCC\left(A_j\right)=\frac{1}{n-1}\sum _{j=1\cap j\ne i}^{n}N\rho (A_i;A_j)$$

(20)

where $N\rho (A_i;A_j)$ is expressed as the normalized value between attributes $A_i$ and $A_j$.

To validate the classification performance, we compared ATFNB to the standard NB classifier and two existing state-of-the-art filter-weighted methods. In addition, the original CFW was a specific model of our framework under the regulatory factor $\beta$ = 0.5. When the regulatory factor $\beta$ of CFW could be adaptively obtained from the dataset, the original CFW evolved into CFW-$\beta$. Now, we introduce these comparisons and their abbreviations as follows:

• NB: the standard naive Bayes model [39].
• WNB: NB with gain ratio attribute weighting [31].
• CFW: NB with MI class-specific and attribute-specific attribute weighting [29].
• CFW-$\beta$: CFW with the adaptive regulatory factor $\beta$.

4.3. The Effectiveness of the Regulatory Factor $\beta$

The regulatory factor $\beta$ can be adaptively adjusted to obtain the optimal ratio for different datasets. In order to verify the effectiveness and efficiency of the regulatory factor $\beta$, we compared the RQRF method with the step-length searching (SLS) algorithm. The SLS algorithm generates $\beta$ with 0.01 as the step size. The optimal interval of regulatory factor $\beta$ by the RQRF and SLS algorithms in four datasets are shown in Figure 2.

From Figure 2, for both the SLS and RQRF algorithms, the optimal interval of $\beta$ in each dataset was biased. On abalone, the lower bound of the interval of $\beta$ was greater than 0.5. On breast-cancer and knowledge, the upper bound of the interval of $\beta$ was less than 0.5. Only the interval of $\beta$ in bupa contained 0.5. Thus, it could be concluded that the regulatory factor $\beta$ value set as 0.5 was unreasonable for all datasets. In addition, it could be clearly seen that the interval size of regulatory factor $\beta$ was inconsistent. On bupa, the interval size of $\beta$ was the largest. On the contrary, the size was the smallest on knowledge.

The optimal interval of regulatory factor $\beta$ and the runtime calculated by the SLS and RQRF algorithms are shown in

Table 4. The optimal interval of regulatory factor $\beta$ and the runtime from SLS and RQRF methods.

Dataset	The Interval of Regulatory Factor $\mathit{\beta }$		Time (s)
	SLS	RQRF	SLS	RQRF	Speed
bupa	[0.17, 0.59]	[0.1687, 0.5937]	7.4908	0.0119	×629
abalone	[0.70, 0.75]	[0.6988, 0.7521]	16.826	0.1068	×157
breast-cancer	[0.23, 0.31]	[0.2257, 0.3129]	6.8023	0.0389	×174
knowledge	[0.27, 0.29]	[0.2688, 0.2954]	6.1298	0.0229	×267

. From Table 4, we can see that two optimal intervals obtained by the SLS and RQRF algorithms had a high coincidence degree. If we reduced the step size of SLS, the coincidence degree between the two algorithms would further improve. Yet, SLS would become very inefficient. For the RQRF method, the runtime was obviously faster than that of the SLS method, and was sped up at least 150 times. Therefore, the RQRF method is not only more accurate than the SLS method, but also more efficient.

4.4. Experimental Results on UCI Datasets

Table 5. Classification accuracy comparisons for ATFNB versus NB, WNB, CFW, CFW-$\beta$ on UCI datasets.

Dataset	NB	WNB	CFW	ATFNB	CFW-$\mathit{\beta }$
abalone	0.5886	0.5871 *	0.5890 *	0.5908	0.5926
acute	0.9958	0.9521 *	0.9948	0.9635	0.9813
aggregation	0.9890	0.9882	0.9761 *	0.9875	0.9824
balance-scale	0.8592 *	0.8728	0.8312 *	0.8984	0.8581
bank	0.8765	0.8831	0.8901	0.8822	0.9076
banknote	0.8636	0.8468	0.8491	0.8498	0.8338
blood	0.7597 *	0.7733	0.7720 *	0.7847	0.7990
breast-cancer	0.7214 *	0.7059 *	0.7331	0.7472	0.7422
breast-tissue	0.5727	0.5955	0.5818 *	0.6091	0.6158
bupa	0.6232	0.5942 *	0.6174 *	0.6333	0.6299
car	0.8523	0.6965 *	0.7671 *	0.8014	0.8101
chart_Input	0.9533	0.9367	0.9558	0.9455	0.9488
climate-simulation	0.9137	0.9178	0.9174	0.9181	0.9209
congressional-voting	0.6149 *	0.6345 *	0.6253 *	0.6506	0.6614
connectionist	0.7238 *	0.7429 *	0.7214 *	0.7667	0.7560
dermatology	0.9797	0.9644	0.9757	0.9649	0.9665
diabetes	0.7377	0.6584 *	0.7403	0.7422	0.7611
Ecoli	0.8135	0.7706 *	0.7588 *	0.8245	0.7981
energy-y1	0.8874	0.8225 *	0.8701	0.8463	0.8813
fertility	0.8400 *	0.8500	0.8350 *	0.8650	0.8669
glass	0.7023	0.6837 *	0.6930	0.7193	0.7233
haberman-survival	0.7532 *	0.7468 *	0.7403 *	0.7710	0.7791
Iris	0.9133	0.9100 *	0.9167	0.9367	0.9099
Jain	0.9464	0.9368	0.9379	0.9397	0.9399
knowledge	0.7371*	0.7743 *	0.7629 *	0.8057	0.7989
libras	0.5903	0.5917	0.5847	0.5965	0.6122
low-res-spect	0.8037 *	0.8018 *	0.8131*	0.8318	0.8411
lymphography	0.8122 *	0.7889 *	0.8233	0.8334	0.8399
magic	0.7300	0.6885 *	0.7411	0.7674	0.7782
mammographic	0.8290 *	0.8394	0.8446	0.8549	0.8679
promoters	0.9091 *	0.9045 *	0.9242 *	0.9545	0.9302
splice	0.9475	0.9376 *	0.9580	0.9414	0.9677
nursery	0.9043	0.8089 *	0.8812	0.8961	0.9002
page-blocks	0.9300 *	0.9404	0.9545 *	0.9684	0.9690
pima	0.7338 *	0.6688 *	0.7330 *	0.7599	0.7613
planning	0.6000 *	0.7189	0.6919 *	0.7378	0.7500
post-operative	0.7222 *	0.8519 *	0.7593 *	0.9074	0.8489
robotnavigation	0.8760 *	0.9159	0.9095	0.9179	0.9199
seeds	0.8747	0.8622	0.8762	0.8655	0.8881
sonar	0.7625	0.7429 *	0.7571 *	0.7734	0.7662
soybean	0.9036	0.8730	0.9117	0.8781	0.9049
spect	0.6566 *	0.6604 *	0.6792 *	0.7151	0.7288
synthetic-control	0.9677	0.9458	0.9698	0.9567	0.9675
tic-tac-toe	0.7141	0.6589 *	0.7109	0.7005	0.7201
titanic	0.7782	0.6680 *	0.7751	0.7822	0.7991
twonorm	0.9384	0.9364	0.9388	0.9489	0.9346
wall-following	0.8032	0.7964	0.8137	0.7976	0.8199
waveform	0.8080 *	0.7960 *	0.8172 *	0.8355	0.8317
wilt	0.9472	0.9374 *	0.9475	0.9523	0.9538
wine	0.9694	0.9625	0.9750	0.9697	0.9622
Average	0.8146	0.8028	0.8169	0.8317	0.8345
G/W/L	9/15/35	0/2/48	8/12/38	33/ /

(*) indicates that ATFNB was significantly better than its competitors (NB, WNB, CFW) through a two-tailed t-test at the p = 0.05 significance level [40]. At the bottom of the table, G represents the number of data sets with the highest classification accuracy among the four algorithms (ATFNB, WNB, CFW, NB). W indicates the number of datasets for which the classification accuracy was higher than ATFNB, L means the opposite of W.

shows the detailed classification accuracy results of five algorithms. All classification accuracies were obtained by averaging the results of 30 independent runs. The five algorithms were used on the same training set and testing set. We conduct a group of experiments on 50 UCI datasets to compare ATFNB with NB, WNB, CFW and CFW-$\beta$ in terms of classification accuracy.

Compared with WNB, CFW, and NB, the accuracy of ATFNB on 33 datasets was the highest, which far exceeded that of WNB (0 datasets), CFW (8 datasets), and NB (9 datasets). The average accuracy of ATFNB was 83.17%, which was significantly higher than that of the other algorithms, and the improvement on the average accuracy was approximately 3%, 2%, and 2%, respectively.

In addition, the average accuracy of CFW-$\beta$ increased by 1.76% compared with that of CFW. This meant that the adaptive regulatory factor could improve the existing two-index NB model. Compared with ATFNB, the average accuracy of CFW-$\beta$ was higher than that of ATFNB. The reason was that the mutual information (class–attribute) and mutual information (attribute–attribute) were included in CFW-$\beta$, which had a more powerful representation than the information gain and Pearson correlation coefficient in ATFNB. In Section 5.3, models generated by different combinations of indexes are discussed in detail.

Base on the accuracy result, we used a two-tailed t-test at the $p=0.05$ level to compare each pair of algorithms beside CFW-$\beta$.

Table 6. Summary of two-tailed t-test results of classification accuracy with regard to ATFNB on UCI datasets.

Algorithm	ATFNB	WNB	CFW	NB
ATFNB	—	2 (0)	12 (4)	15 (5)
WNB	48 (28)	—	35 (16)	34 (19)
CFW	38 (22)	15 (5)	—	20 (8)
NB	35 (19)	16 (9)	30 (11)	—

For each i(j), i represents the number of datasets with a higher classification accuracy obtained by the column algorithm than the row algorithm, and j represents the number of datasets in which the column algorithm had a significant advantage over the row algorithm.

summarizes the comparison results on the UCI datasets. From Table 6, ATFNB had significant advantages over the other weighting algorithms. ATFNB was better than WNB (28 wins and 0 loss), CFW (22 wins and 4 losses), and NB (19 wins and 5 losses).

Based on the classification accuracy of Table 5, we utilized the Wilcoxon signed-rank test to compare the four algorithms. The Wilcoxon signed-rank test is a nonparametric statistical test, which ranks the performance differences of the two algorithms for each dataset, considering both the sign of the difference and the order of the difference.

Table 7. Ranks of the Wilcoxon test with regard to ATFNB on UCI datasets.

Algorithm	ATFNB	WNB	CFW	NB
ATFNB	—	1268	1007.5	961.5
WNB	7	—	308.5	391.5
CFW	267.5	966.5	—	771
NB	313.5	883.5	504	—

shows the ranks calculated by the Wilcoxon test. In Table 7, the numbers above the diagonal line indicate the sum of ranks for the datasets of the algorithm in the row that is better than the algorithm in the corresponding column (the sum of the ranks for the positive difference, represented by $R^{+}$). Each number below the diagonal is the sum of ranks for the datasets in which the algorithm in the column is worse than the algorithm in the corresponding row (The sum of the ranks for the negative difference, represented by $R^{-}$). According to the critical value table of the Wilcoxon test, for Table 7, when $\alpha$ = 0.05 and n = 50, if the smaller of $R^{+}$ and $R^{-}$ was equal to or less than 434, we considered that the two classifiers were significantly different, so we rejected the null hypothesis. The significant data in Table 7 (equal to or less than 434) are marked with symbols (•, ∘), as shown in

Table 8. Summary of the Wilcoxon test with regard to ATFNB on UCI datasets.

Algorithm	ATFNB	WNB	CFW	NB
ATFNB	—	∘	∘	∘
WNB	•	—	∘	∘
CFW	•	•	—
NB	•	•		—

• indicates that the algorithm in the column is improved compared to the algorithm in the corresponding row. ∘ indicates that the algorithm in the row is better than the algorithm in the corresponding column.

.

According to the results of the Wilcoxon signed rank-sum test, on the UCI datasets, ATFNB was significantly better than WNB ($R^{+}=1268,R^{-}=7$), CFW ($R^{+}=1007.5,R^{-}=267.5$) and Standard NB ($R^{+}=961.5,R^{-}=313.5$).

4.5. Experimental Results on Flavia Dataset

In order to further verify the effectiveness of ATFNB, we conducted 15 groups of experiments on the Flavia dataset. We randomly divided the data in each group of experiments 30 times and used a two-tailed t-test for the results of 30 experiments. The detailed results are shown in

Table 9. Classification accuracy comparisons for ATFNB, NB, WNB, CFW, CFW-$\beta$ on Flavia dataset. * indicates that ATFNB was significantly better than its competitors (NB, WNB, CFW) through a two-tailed t-test at the p = 0.05 significance level.

Group	NB	WNB	CFW	ATFNB	CFW-$\mathit{\beta }$
G_1	0.8253 *	0.8506 *	0.8552 *	0.8805	0.9011
G_2	0.8337	0.8629	0.8742	0.8444	0.8668
G_3	0.7874 *	0.8484	0.8312	0.8786	0.8771
G_4	0.8562 *	0.8854 *	0.8899	0.8987	0.9022
G_5	0.8016 *	0.8129	0.8050 *	0.8174	0.8177
G_6	0.8822	0.8729 *	0.8903	0.8843	0.8801
G_7	0.9134 *	0.9137 *	0.9322	0.9233	0.9400
G_8	0.9011	0.8812 *	0.8927	0.8904	0.9022
G_9	0.9122	0.9100	0.9033 *	0.9422	0.8891
G_10	0.8135 *	0.8213 *	0.8200 *	0.8422	0.8399
G_11	0.8572	0.8734	0.8534	0.8799	0.8912
G_12	0.8356	0.8132 *	0.8224	0.8233	0.8335
G_13	0.7724 *	0.7787 *	0.7732 *	0.7987	0.7887
G_14	0.8342 *	0.8344	0.8322 *	0.8458	0.8422
G_15	0.9169 *	0.9224 *	0.9243 *	0.9321	0.9095
Average	0.8495	0.8588	0.8600	0.8721	0.8720
G/W/L	2/2/13	0/1/14	3/4/11	10 / /

.

Comparing ATFNB with other existing classifiers (WNB, CFW, NB), the average accuracy of ATFNB was 87.21%, which was significantly higher than that of algorithms, and the improvement on the average accuracy was approximately 3%, 1.5%, and 1%, respectively. In 15 groups of experiments, ATFNB achieved the highest classification accuracy among 10 groups of data, which was far better than NB, WNB, and CFW.

The average accuracy of CFW-$\beta$ was slightly lower than that of ATFNB, but the average accuracy of CFW-$\beta$ was higher than that of CFW. On Flavia, the choice of indexes had little effect on the average accuracy, but adding a regulatory factor $\beta$ to the model could effectively improve the performance of model.

We summarize the results of the two-tailed test in Table 9, as shown in

Table 10. Summary two-tailed t-test results of classification accuracy with regard to ATFNB on Flavia dataset.

Algorithm	ATFNB	WNB	CFW	NB
ATFNB	—	1 (0)	4 (1)	2 (0)
WNB	14 (9)	—	8 (3)	4 (1)
CFW	11 (7)	7 (4)	—	5 (1)
NB	13 (9)	11 (6)	10 (5)	—

. In Table 10, ATFNB was better than WNB (nine wins and zero loss), CFW (seven wins and one loss), and NB (nine wins and zero loss).

On the basis of Table 9, we used the Wilcoxon signed-rank test to compare the four algorithms. According to the critical value table of the Wilcoxon test, for

Table 11. Ranks of the Wilcoxon test with regard to ATFNB on Flavia dataset.

Algorithm	ATFNB	WNB	CFW	NB
ATFNB	—	110	96	110.5
WNB	10	—	52	89
CFW	24	68	—	87
NB	9.5	31	33	—

, when $\alpha =0.05$ and $n=15$, if the smaller of $R^{+}$ and $R^{-}$ was equal to or less than 25, we considered that two classifiers were significantly different, so we rejected the null hypothesis.The significant data in Table 11 (equal to or less than 25) are marked with symbols (•,∘), as shown in

Table 12. Summary of the Wilcoxon test with regard to ATFNB on Flavia dataset.

Algorithm	ATFNB	WNB	CFW	NB
ATFNB	—	∘	∘	∘
WNB	•	—
CFW	•		—
NB	•			—

.

In the Flavia dataset, the ATFNB algorithm had obvious advantages compared with WNB ($R^{+}=110,R^{-}=10$), CFW ($R^{+}=96,R^{-}=24$), and standard NB ($R^{+}=110.5,R^{-}=9.5$).

5. Discussion

5.1. The Influence of Instance and Attribute Number

To further analyze the relationship between the performance of ATFNB and the characteristic of a dataset, we observed its performance from the two perspectives of number of instances and number of attributes. In terms of the number of instances, we divided the datasets into two categories: less than 500 instances and greater than or equal to 500 instances. Similarly, according to the number of attributes, we divided attributes into two categories: the number of attributes was less than 15, and the number of attributes was greater than or equal to 15. Then, we combined the above two criteria, which resulted in four divisions. Finally, we calculated the percentage of datasets with the highest classification accuracy of ATFNB and competitors (NB, WNB, CFW) in eight divisions. The detailed results are shown in

Table 13. ATFNB and competitors’ obtained percentage of datasets with the highest classification accuracy in each division. Bold text is to highlight the superiority of the method.

Data Characteristics		Number	ATFNB (%)	Competitors (%)
Instance number	<500	23	78.26	21.74
	≥500	27	56.25	43.75
Attribute number	<15	31	67.74	32.26
	≥15	19	63.16	36.84
Instance and attribute	<500 and <15	15	73.33	26.67
	<500 and ≥15	8	87.50	12.50
	≥500 and <15	16	62.50	37.50
	≥500 and ≥15	11	45.45	54.55

.

From Table 13, we could clearly find in which circumstances ATFNB performed better than its competitors. Here, we summarize the highlights as follows:

(1)

On 78.26% of datasets with less than 500 instances, ATFNB could achieve the highest classification accuracy. On datasets with a number of instances greater than or equal to 500, 56.25% could achieve the maximum classification accuracy. By comparison, ATFNB is more advantageous on datasets with fewer instances.

(2)

For datasets with a number of attributes less than 15, the percentage of datasets with the highest classification accuracy for ATFNB (67.74%) was also higher than that with a number of attributes greater than or equal to 15 (63.16%).

(3)

When the number of instances was less than 500 and the number of attributes was greater than 15, the percentage of datasets with the highest classification accuracy for ATFNB (87.5%) was significantly higher than that of the other three types of datasets (73.33%, 62.50%, 45.45%).

The performance of ATFNB had obvious advantages on the datasets whose number of instances was smaller than 500, especially when the number of attributes was greater than or equal to 15, such as the dataset “congressional-voting”. By contrast, ATFNB did not perform well on datasets with a large number of instances and attributes. In a word, ATFNB can be perfectly suitable for small data classification and is not limited by dimensions.

5.2. The Distribution of the Regulatory Factor $\beta$

In Section 4.3, we validated the effectiveness of the regulatory factor $\beta$ in ATFNB. Here, the distributions of the regulatory factor $\beta$ in various datasets are further analyzed. We firstly list the interval of the regulatory factor $\beta$ for the 50 UCI datasets as shown in

Table 14. The interval of regulatory factor $\beta$ on 50 UCI datasets.

Dataset	Interval ($\mathit{\beta }$)	Mark	Dataset	Interval ($\mathit{\beta }$)	Mark
abalone	[0.7122, 0.8311]	◯	libras	[0.5377, 0.8832]	◯
acute	[0.4418, 0.9433]	△	low-res-spect	[0.6552, 0.7211]	◯
aggregation	[0.5529, 0.8832]	△	lymphography	[0.3344, 0.4834]	⬜
balance-scale	[0.3233, 0.4537]	⬜	magic	[0.6733, 0.8122]	◯
bank	[0.4198, 0.7691]	△	mammographic	[0.1229, 0.3879]	⬜
banknote	[0.3144, 0.3914]	⬜	promoters	[0.4876, 0.8867]	△
blood	[0.2243, 0.5532]	△	splice	[0.0512, 0.1321]	⬜
breast-cancer	[0.2311, 0.3521]	⬜	nursery	[0.5211, 0.5908]	◯
breast-tissue	[0.3566, 0.4513]	⬜	page-blocks	[0.6322, 0.7109]	◯
bupa	[0.1533, 0.6588]	△	pima	[0.1566, 0.3118]	⬜
car	[0.3211, 0.3987]	⬜	planning	[0.1829, 0.4721]	⬜
chart_Input	[0.4592, 0.8311]	△	post-operative	[0.0187, 0.2100]	⬜
climate-simulation	[0.2301, 0.3255]	⬜	robotnavigation	[0.7122, 0.7830]	◯
congressional-voting	[0.2199, 0.3472]	⬜	seeds	[0.4288, 0.8543]	△
connectionist	[0.0912, 0.1388]	⬜	sonar	[0.0521, 0.1487]	⬜
dermatology	[0.3365, 0.8987]	△	soybean	[0.2759, 0.3108]	⬜
diabetes	[0.2355, 0.5243]	△	spect	[0.1802, 0.2499]	⬜
Ecoli	[0.7360, 0.9211]	◯	synthetic-control	[0.2480, 0.4033]	⬜
energy-y1	[0.3211, 0.9219]	△	tic-tac-toe	[0.1213, 0.1870]	⬜
fertility	[0.0511, 0.4390]	⬜	titanic	[0.4697, 0.6122]	△
glass	[0.1229, 0.1833]	⬜	twonorm	[0.5833, 0.6291]	◯
haberman-survival	[0.2166, 0.6345]	△	wall-following	[0.4128, 0.4736]	⬜
iris	[0.3522, 0.8799]	△	waveform	[0.7398, 0.7933]	◯
jain	[0.3409, 0.8577]	△	wilt	[0.6103, 0.6899]	◯
knowledge	[0.3012, 0.4522]	⬜	wine	[0.3881, 0.9220]	△

◯ indicates that the lower bound value of β interval is greater than 0.5. ⬜ indicates that the upper bound value of β interval is less than 0.5. △ indicates 0.5 is in the interval.

. From Table 14, we can summarize that the lower bound of the optimal interval in 11 datasets was greater than 0.5, the upper bound of the optimal interval in 23 datasets was less than 0.5, and the optimal interval of the remaining 16 datasets contained 0.5. In ATFNB, the information gain and Pearson correlation coefficient provided different contributions on the 50 UCI datasets. In addition, these results further demonstrated that the regulatory factor $\beta$ set as a fixed value was unreasonable.

To further investigate the relationship between the distribution of regulatory factor $\beta$ and the characteristics of the datasets, we applied the same division criteria as in Section 5.1 on the 50 UCI datasets and summarize the detailed results in

Table 15. The relationship between the regulatory factor $\beta$ in ATFNB and the characteristics of the dataset.

Data Characteristics		Number	◯ (%)	△ (%)	□ (%)
Instance number	<500	23	8.70	39.13	52.17
	≥500	27	33.33	25.93	40.74
Attribute number	<15	31	19.35	38.71	41.94
	≥15	19	26.32	21.05	52.63
Instance and attribute	<500 and <15	15	6.66	46.67	46.67
	<500 and ≥15	8	12.50	25.00	62.50
	≥500 and <15	16	31.25	31.25	37.50
	≥500 and ≥15	11	36.36	18.18	45.46

. From Table 15, we can observe the preference between the data characteristics and the distribution of the regulatory factor $\beta$, and summarize the highlights as follows:

(1)

If the number of instances was less than 500, the upper bound value of $\beta$ in 52.17% of the datasets was less than 0.5. If the number of instances was more than 500, the upper bound value of $\beta$ in 40.74% of the datasets was less than 0.5.

(2)

From the perspective of the number of attributes, regardless of the number of attributes, the upper bound value of $\beta$ was less than 0.5 in most datasets.

(3)

(3) Considering the number of instance and attributes simultaneously, the upper bound value of $\beta$ in 62.50% of the datasets with a number of instances less than 500 and number of attributes greater than 15 was less than 0.5. On the dataset with a number of instances greater than 500 and number of attributes greater than 15, the upper bound value of $\beta$ in 45.46% of the datasets was less than 0.5.

Based on the results in Table 15, the upper bound value of $\beta$ was less than 0.5 in most datasets. We can conclude that ATFNB pays attention to the Pearson correlation coefficient between attributes, especially in small instances and high-dimensional datasets.

5.3. The Impact of Different Index Combinations

The ATFNB framework contains two categories, and each category provides several popular indexes to represent the characteristic of datasets. In order to analyze the impact of different index combinations, we selected any two indexes from two categories, respectively. Excluding the gain ratio from the class–attribute category, six weighted NB models could be constructed as shown in Figure 3. Notably, ATFNB-IP and ATFNB-MM were equal to ATFNB and CFW-$\beta$, respectively.

Then, we compared the six combinations on the 50 UCI datasets, and the average accuracy of the six combinations are shown in Figure 4. From Figure 4, it can be seen that ATFNB-PP had the lowest average accuracy, but the average accuracy of ATFNB-PP outperformed the basic NB and NB (0.8146), WNB (0.8028), and CFW (0.8169). This further demonstrated the effectiveness of the ATFNB framework with an adaptive regulatory factor. In addition, compared with three indexes the from class–attribute category, the average accuracy of ATFNB-M* (denoted ATFNB-MP and ATFNB-MM) was better than ATFNB-P* and ATFNB-I*. This meant that the mutual information from the class–attribute category was more signification than the Pearson correlation coefficient and information gain.

5.4. Computation Time

The experimental platform in this paper was an AMD 5900X processor, 3.70 GHz, 32 GB memory, Windows 10 system, and the algorithm was implemented in Matlab2020A.

In order to comprehensively analyze the computation time of our framework, we split ATFNB into three stages, which were the stage of index calculation and conditional probability calculation (Stage 1), the stage of regulatory factor $\beta$ calculation (Stage 2), and the stage of classification (Stage 3).

Table 16. The computation time of each stage for various algorithms on the blood dataset (in seconds).

Algorithm	Stage 1 (s)	Stage 2 (s)	Stage 3 (s)	Total (s)
NB	0.5233	0	0.0793	0.6026
WNB	0.6811	0	0.0803	0.7614
CFW	0.8794	0	0.0647	0.9441
CFW-$\beta$	0.8794	0.1092	0.0721	1.0607
ATFNB	0.8413	0.1125	0.0823	1.0361

shows the computation time of each stage and the whole process for various algorithms on the blood dataset. From Table 16, we find that the main differences were in the Stage 1 and Stage 2. In the Stage 1, the type and number of indexes directly affected the runtime. CFW, CFW-$\beta$, and ATFNB needed to calculate two indexes, thus these three algorithms took more time than the other two single-index NB methods. In Stage 2, although CFW-$\beta$ and ATFNB introduced a regulator factor, the computation time was very short using the quick range query method. In terms of total time, CFW-$\beta$ and ATFNB only increased time by about 10% compared to CFW, but the accuracy significant improved.

6. Conclusions and Future Work

In this paper, we proposed an adaptive two-index fusion attribute-weighted NB (ATFNB) to overcome the problems of existing weighted methods, such as the poor representation ability with a single index and the fusion problem of two indexes. ATFNB could select any one index from the attribute–attribute category and the class–attribute category, respectively. Then, a regulatory factor $\beta$ was introduced to fuse two indexes and it was inferred by a range query. Finally, the weight of each attribute was calculated using the optimal value $\beta$ and integrated into an NB classifier to improve the accuracy. The experimental results on 50 benchmark datasets and the Flavia dataset showed that ATFNB outperformed the basic NB classifier and state-of-the-art filter-weighted NB models. In addition, we incorporated the regulatory factor $\beta$ into CFW. The results demonstrated the improved model CFW-$\beta$ had a significantly increased accuracy compared to CFW without the adaptive regulatory factor $\beta$.

In future work, there are two directions to further improve the NB model. Firstly, ATFNB may consider more than two indexes from different data description categories. Secondly, we hope to design more new indexes to represent the correlation of class–attribute or attribute–attribute.