Online Streaming Features Selection via Markov Blanket

Abstract

Streaming feature selection has always been an excellent method for selecting the relevant subset of features from high-dimensional data and overcoming learning complexity. However, little attention is paid to online feature selection through the Markov Blanket (MB). Several studies based on traditional MB learning presented low prediction accuracy and used fewer datasets as the number of conditional independence tests is high and consumes more time. This paper presents a novel algorithm called Online Feature Selection Via Markov Blanket (OFSVMB) based on a statistical conditional independence test offering high accuracy and less computation time. It reduces the number of conditional independence tests and incorporates the online relevance and redundant analysis to check the relevancy between the upcoming feature and target variable T, discard the redundant features from Parents-Child (PC) and Spouses (SP) online, and find PC and SP simultaneously. The performance OFSVMB is compared with traditional MB learning algorithms including IAMB, STMB, HITON-MB, BAMB, and EEMB, and Streaming feature selection algorithms including OSFS, Alpha-investing, and SAOLA on 9 benchmark Bayesian Network (BN) datasets and 14 real-world datasets. For the performance evaluation, F1, precision, and recall measures are used with a significant level of 0.01 and 0.05 on benchmark BN and real-world datasets, including 12 classifiers keeping a significant level of 0.01. On benchmark BN datasets with 500 and 5000 sample sizes, OFSVMB achieved significant accuracy than IAMB, STMB, HITON-MB, BAMB, and EEMB in terms of F1, precision, recall, and running faster. It finds more accurate MB regardless of the size of the features set. In contrast, OFSVMB offers substantial improvements based on mean prediction accuracy regarding 12 classifiers with small and large sample sizes on real-world datasets than OSFS, Alpha-investing, and SAOLA but slower than OSFS, Alpha-investing, and SAOLA because these algorithms only find the PC set but not SP. Furthermore, the sensitivity analysis shows that OFSVMB is more accurate in selecting the optimal features.

1. Introduction

In machine learning, several feature selection algorithms are essential for processing high-dimensional data. The optimal feature (feature/variable/node are same) set for a target variable T is the Markov blanket (MB) [1], composed of Parents-Child (PC), Spouses (SP), and direct causes and effects between them, as shown in Figure 1.

There are two major approaches for feature selection, i.e., traditional-based and streaming-based. Traditional-based MB discovery (discovery/learning are interchangeable in this article) assumes that all features are readily available from the beginning. This assumption, however, is often violated in real-world applications. For instance, for the problem of detecting Mars crater from high-resolution planetary images [2], it is impracticable to attain the complete feature set, which means to have a near-global coverage of the Martian surface. Several MB discovery algorithms are proposed based on traditional concepts requiring all features and instances to be available before learning. Such algorithms include Incremental Association-Based Markov Blanket (IAMB) [3], HITON-MB (HITON-MB) [4], Simultaneous Markov Blanket (STMB) [5], Balanced Markov Blanket (BAMB) [6], and Efficient and Effective Markov Blanket (EEMB) [7]. Traditional-based MB discovery algorithms do not apply to real-world scenarios, such as users of the famous microblogging website Twitter who yield over 250 million tweets every single day, which includes words and abbreviations [8], personalized recommendations [9], malware scanning [10], and ecological inspection and analysis [11] with the quality of frequent features updates and dynamic space-changing [12]. A fascinating question is whether we should wait long for all features to become accessible before starting the learning process. We need a lot of computational efforts to build such features upfront. This raises an intriguing research challenge about constructing an effective feature selection process without knowing the entire feature space. However, the existing algorithms for Markov blanket discovery commonly assume that all features must be present in advance. Therefore to tackle such issues, streaming feature selection was introduced and applied to many applications such as biology, weather forecasting, transportation, stock markets, clinical research, etc. [8]. Several algorithms based on streaming features (SF) were proposed for real scenarios including Grafting [8], Alpha-investing ($\alpha -$ investing) [13], Scalable and Accurate Online Feature Selection (SAOLA) [14], and Online Streaming Feature Selection (OSFS) [15]. However, these algorithms only focus on obtaining PC sets and do not consider the Spouses, which causes them to lose the interpretability by ignoring the causal MB discovery.

Motivated by these observations and issues, this paper presents an Online Streaming Features Selection via Markov Blanket algorithm, based on a statistical conditional independence test. The features are no longer static but flow in one by one and are analyzed as it arrives. A null-conditional test motivated it to address the streaming features, feature relevance analysis to find the true positive PC and Spouses, and feature redundancy analysis to remove the false positive/irrelevant features.

The main contributions of this study are as follows:

• OFSVMB obtains Parent-Child and Spouse feature sets simultaneously and separates them in the MB set;
• OFSVMB reduces the impact of conditionally independent tests errors since it uses fewer conditionally independent tests to learn MB;
• Sensitivity analysis of OFSVMB using three different parameters $\alpha$ values concerning the Rate of Instance $\left|RI\right|$ to analyze the performance with small and large sample sizes;
• Performance evaluation of OFSVMB algorithm on benchmark BN and real-world datasets.

The remainder of the paper is organized as follows: The related work is presented in Section 2. The preliminaries is discussed in Section 3. The proposed algorithm is presented in Section 4. The experimental findings are discussed in Section 5, and the paper is concluded in Section 6.

2. Related Work

In traditional MB discovery techniques, features must be present before learning begins. Different algorithms developed for MB learning, which is based on traditional concepts such as Incremental Association-Based Markov Blanket (IAMB) [16], Max-Min Markov Blanket (MMMB) [17], HITON-MB (HITON-MB) [18], Simultaneous Markov Blanket (STMB) [19], Iterative Parent-Child-based MB (IPCMB) [19], Balanced Markov Blanket (BAMB) [6], and Efficient and Effective Markov Blanket (EEMB) [7]. While MB is learning the Parents-Child and Spouses of the target feature, T cannot differentiate by the IAMB [16] algorithm. Additionally, when the sample size of the dataset is not quite large, the IAMB can not faithfully discover the Markov blanket of the target feature T. The Min-Max Markov blanket (MMMB) uses the Divide and Conquer strategy to minimize the size of data samples [17]. It needs to split the dilemma of finding MB into two sub-dilemmas to find the Parents-Child and the Spouses. MMMB has been modified to build HITON-MB, which excludes false-positive features from the Parents-Child set quickly by interleaving the growing and shrinking phases [18]. Under the Markov blanket assumption, MMMB [17] and HITON-MB [18] were conceptually flawed and needed a new mechanism for accurate MB discovery. The Iterative Parent-Child-based search of Markov Blanket (IPCMB) algorithm uses the same PC algorithm as PCMB [19] to identify the PC set and increases the efficiency without losing accuracy [19]. However, the symmetry constraint check makes this algorithm computationally slow. STMB [19] also followed the technique as IPCMB [19] for Parents-Child exploration. However, it consumes more time and avoids symmetry checks.

Moreover, the online feature selection methods receive features one by one dynamically. These methods include grafting [8], Alpha-investing ($\alpha -$ investing) [8], Scalable and Accurate Online Feature Selection (SAOLA) [20], and Online Streaming Feature Selection (OSFS) [21]. Grafting was the first algorithm designed by Perkins and Theiler. It attempts the streaming features selection technique, which is a stage-wise technique for gradient descent. Alpha-investing used p-value and threshold to control the feature selection process and decide to select relevant features and remove redundant ones [8]. The benefit of alpha-investing is to handle features sets of unknown sizes even up to infinity, but it fails to investigate redundant features, causing an unpredictable and low prediction accuracy. SAOLA [20] examines two features simultaneously and analyzes redundancy under a single scenario. It fails to find an optimum relevance threshold value to remove all redundant features. In contrast, OSFS [21] removes unnecessary features which are not relevant/associated with the target feature, T, using conditional independence. It uses two steps to achieve the Parents-Child relevant to the target feature T. The first step analyzes the online relevance, and the second step analyzes online redundancy. It identifies the approximate MB without Spouses. The methods of online feature selection such as Alpha-investing, SAOLA, and OSFS only identify Parents-Child (PC) features and avoid the Spouses (SP) and the causality neglect by these algorithms. Based on online feature selection and causality, the Causal Discovery From Streaming Feature (CDFSF) is used [22]. The symmetric Causal Discovery From Streaming Feature (S-CDFSF) [22] is developed, which uses conditional independence test to identify the relevant features of the target feature, T, based on streaming, which belongs to the Parents-Child and Spouses.

3. Preliminaries

In this segment, the MB discovery through streaming features is defined. In addition, the specific aspects are discussed in detail.

Table 1. Summary of Notation.

Notations	Mathematical Meanings
R	A feature set under streaming features
S	Conditional feature set within R
T	Target attribute
G	Over R, a DAG (Directed Acyclic Graph)
P	Over R, a JPD (Joint Probability Distribution)
f	False positive
X, Y, Z, N, E or $X_i,Y_i,Z_i$	A feature/a variable/a node, which belong to R
$t_i$	A time point of the $X_i$ arriving feature
$P{C}_T$	A set of Parents-Children of T
$CP{C}_T$	A candidate feature set of $P{C}_T$
non-pc${}_T$	Non-Parents-Child set of target T
$X_i⫫Y_i$	Given S, X and Y are independent
$X_i\cancel{⫫}{Y}_i$	Given S, X and Y are dependent
$S{P}_T$	A set of Spouses of T
$S{P}_T\left\{X\right\}$	T’s Spouses in relation to T’s child X
$CS{P}_T\left\{X\right\}$	a Candidate feature set of Spouse
$De{s}_T$	Descendant of Target (T)
$P{a}_T$	Parents of target T
$\alpha$	Significance levels of 0.01 and 0.05
$\rho$	p-value return by statistical $G^2$ test and Fisher’s z-test
$M{B}_T$	Markov blanket (MB) of Node T

summarized the notations used in this paper based on the new definitions.

Definition 1 (Streaming Features [15]). Involves a feature vector $\{X_1,X_2,X_3,\dots ,X_i\}$ that streams in one by one over time $\{t_i,X_{i+1},X_{i+2},\dots ,X_{i+n}\}$ while a number of training samples remain constant.

Definition 2 (Conditional Independence [6]). Feature (variable) $X_i$ is conditionally independent of feature (variable) $Y_i$ given S, if and only if $P\left(X_i\right|Y_i,S)=P\left(X_i\right|S)$.

Definition 3 (Strong relevant [23]). A feature $X_i\in R$ is strongly relevant to the target feature T, if and only if $\forall S\subseteq R\setminus \left\{X_i\right\},s.t.P\left(X_i|S\right)\ne P\left(X_i|S,T\right)$.

Definition 4 (Irrelevant [23]). A feature $X_i\in R$ is irrelevant to a target feature T, if and only if it is $\forall S\subseteq R\setminus \left\{X_i\right\},s.t.P\left(X_i|S,T\right)=P\left(S|T\right)$.

Definition 5 (Redundant [24]). A feature $X_i$ is redundant to the target variable T, if and only if it is weakly relevant to target variable T and has a Markov blanket, $MB\left(X_i\right)$, then it is a subset of the Markov blanket of $M{B}_T$.

Definition 6 (Faith-fullness Condition [25]). G denotes a Bayesian network, and P represents a joint probability distribution through feature set R. So, G is devoted to P if P captures all. Only the conditional independence is among features in G.

Definition 7 (V-structure [26]). If there is no an arrow between feature (variable) $X_i$ and and feature (variable) $Y_i$, and feature (variable) $Z_i$ has two incoming arrows from $X_i$ and $Y_i$, respectively, then $X_i$, $Z_i$, and $Y_i$ form a V-structure $X_i\to Z_i\leftarrow Y_i$.

Definition 8 (D-Separation [26]). A path D between a feature (variable) $X_i$ and feature (variable) $Y_i$ is D-separated by set of features (variables) S, if and only if:

• D includes a chain $X_i\leftarrow Z_i\to Y_i$ such that the middle one features $Z_i$ is in S.
• D includes a collider $X_i\to Z_i\leftarrow Y_i$ such that the middle one feature $Z_i$ is not in S and none of $Z_i$ successors are in S.

A feature set S is said to be D-separated $X_i$ and $Y_i$, if and only if S jammed each path D from a feature $X_i$ to a feature $Y_i$.

Theorem 1.

In a faithful Bayesian Network, an MB of the target feature T, $M{B}_T$, in a set R is an optimal set of features, composed of Parents, Children, and Spouses. All other features are not conditionally dependent of target feature T given $M{B}_T$, $\forall X_i\in R\setminus (M{B}_T\cup T)$, s.t. $X_i⫫T|M{B}_T$.

4. Online Feature Selection via Markov Blanket

This section presents the proposed algorithm for implementing the framework for feature selection with streaming features called Online features selection via a Markov blanket (OFSVMB).

4.1. Framework of OFSVMB

The framework of the Online Feature Selection via Markov blanket is shown in

Table 2. The OFSVMB framework.

1. Initialization:

The target T; Candidate Parents-Child set $\mathit{C}\mathit{P}{\mathit{C}}_{\mathit{T}}$ = ⌀; non-Parents-Child features set non_pc${}_{\mathit{T}}$ = ⌀; Candidate Spouse set $\mathit{C}\mathit{S}{\mathit{P}}_{\mathit{T}}$ = ⌀; $\mathit{M}\mathit{B}$ = ⌀;

2. Check relevancy between feature ${\mathit{X}}_{\mathit{i}}$ arrived at time ${\mathit{t}}_{\mathit{i}}$ and target feature T through null conditional independence test; if $\mathit{D}\mathit{e}\mathit{p}\left({\mathit{X}}_{\mathit{i}},\mathit{T}|\mathit{⌀}\right)$, add in $\mathit{C}\mathit{P}{\mathit{C}}_{\mathit{T}}$; if not, remove non-relevant feature through Proposition 1 and add in non_pc${}_{\mathit{T}}$ set, then enter step 3.

3. After null-conditional independence, check for relevance and redundant features, e.g., $\mathit{D}\mathit{e}\mathit{p}\left({\mathit{X}}_{\mathit{i}},\mathit{T}|\mathit{S}\right)$, through Proposition 2. If it is relevant, add to $\mathit{C}\mathit{P}{\mathit{C}}_{\mathit{T}}$ and jointly find a spouse and add in $\mathit{C}\mathit{S}{\mathit{P}}_{\mathit{T}}$. Otherwise, add it to the non_pc${}_{\mathit{T}}$ set.   It looks for non-MB successors in $\mathit{C}\mathit{P}{\mathit{C}}_{\mathit{T}}$ set through Theorem 2. Discard it from $\mathit{C}\mathit{P}{\mathit{C}}_{\mathit{T}}$ set if it is not dependent on target T.

4. After step 3, find Spouse in non_pc${}_{\mathit{T}}$ set, which is redundant in step 2 and 3, and simultaneously remove false positive spouse from spouse set through Proposition 3.

5. Steps 2–4 are repeated until no features remain.

6. Output: Selected MB of target T.

. Two conditional tests are used to check the association between features. The first is a statistical $G^2$ test (for discrete data) and the second is the statistical Fisher’s z-test (for continuous data). $Ind\left(X_i,T|S\right)$ = $\left(X_i⫫T|S\right)$ (Equal(=) sign means they are same in this article) denotes a conditional independence test between a feature $X_i$ and the target feature (variable) T, given a subset S. While $Dep\left(X_i,T|S\right)$ = $\left(X_i\cancel{⫫}T|S\right)$ represents a conditional dependence test between a feature $X_i$ and the target feature T, given a subset S.

For a new incoming feature, the OFSVMB algorithm performs the Null conditional independence analysis [26], relevance analysis, and redundancy analysis. The pseudocode of OFSVMB is shown in Algorithm 1. In Algorithm 1 RecogNC, line 5 performs null conditional independence using Proposition 1 to check the dependency between feature $X_i$ and target feature T. If $X_i$ is dependent on target feature T, then add $X_i$ to $CP{C}_T$; otherwise, add it to non_pc${}_T$.

Proposition 1.

Using null conditional independence, check the feature $X_i$ relevancy or irrelevancy with target feature T.

Proof of Proposition 1. 

Assuming that $X_i\in R$ and $Y_i\in R$, the following hold:

$$\begin{gathered} X_i⫫Y_i|[]\Rightarrow P\left(X_i|Y_i,\left[\right]\right)=P\left(X_i\left|\right[]\right)\Rightarrow P\left(X_i|Y_i\right)=P\left(X_i\right)\Rightarrow \frac{P\left(X_i,Y\right)}{P\left(Y\right)}=P\left(X_i\right)\Rightarrow \\ P\left(X_i,Y_i\right)=P\left(X_i\right)P\left(Y_i\right)\Rightarrow X_i⫫Y_i \end{gathered}$$□

Therefore, $\left(X_i⫫Y_i,\left[\right]\right)$ represents that $X_i$ and $Y_i$ are not relevant to each other.

Through the relevancy analysis based on Proposition 2, the OFSVMB analyzes features and adds them to the candidate Parents-Child set $CP{C}_T$ and the candidate Spouse set $CS{P}_T$, which are the candidate set of the Parents-Child and Spouse set. If feature $X_i$ is related to the target feature T given $S\subseteq CP{C}_T$, it is added to $CP{C}_T$; otherwise, it is removed from $CP{C}_T$ and added to non_pc${}_T$.

Furthermore, it also analyzes whether $X_i$ is a candidate spouse $CS{P}_T\left\{X_i\right\}$ from the non_pc${}_T$ set. For example, if feature $X_i\in$ non_pc${}_T$, the conditional feature set that causes $X_i$ and target independent variable T is the $se{p}_T\left\{Y_i\right\}$. If there exists a feature $N\in CP{C}_T$, $X_i$ is related to target variable T under the condition of $se{p}_T\left\{Y_i\right\}$ union N, then $X_i$ is added to the $CS{P}_T\left\{N\right\}$ as mentioned in Algorithm 1.

Proposition 2.

A current feature $X_i$ arrived at time t, and T is a target feature. If $\exists X_i\in CP{C}_T$, i.e., $X_i\cancel{⫫}T|S$, then $X_i\in CP{C}_T$.

Proof. 

The proof is as follows: If $\exists X_i\in CPC,s.t.X_i\cancel{⫫}T|\left[S\right]$, if feature $X_i$ is related to target feature T under R condition by Theorem 1, then add $X_i$ to $CP{C}_T$. □

Based on Theorem 2 and Proposition 3, using the redundancy analysis, the OFSVMB removes false-positive features separately from the candidate Parent-Child set $CP{C}_T$ and the candidate Spouse set $CS{P}_T$, which are the candidate sets of the Parents-Child and Spouse set. However, it also looks for non-MB descendant features in $CP{C}_T$ and discards them if they have been through Theorem 2, which makes the OFSVMB different from the other existing algorithms. Removing false positive Spouses through redundant analysis if $Y_i\in CP{C}_T$, if $Y_i$ is not dependent to the target feature (variable) T, giving subset S, then $Y_i$ is removed from $CP{C}_T$. By removing the irrelevant Spouse such that $N\in CP{C}_T$ and $E\in CS{P}_T\left\{N\right\}$, if E is not relevant to target feature T under the condition of $S\cup N$, then E is removed from $CS{P}_T\left\{N\right\}$.

Theorem 2.

(PC recognizes false-positives): Only successors of the target T, standing for $De{s}_T$, make up false-positives $f\in P{C}_T$.

Proof. 

$CP{C}_T$ is composed of a whole PC and a few false positives. We show that $f\subseteq De{s}_T$. $CP{C}_T$ is a candidate super-set of all true positive PCs because it should hold the true positive PCs. After an exhaustive search for the PC set, the entire parents set of target T is represented by $P{a}_T$. According to Definition 5, all non-successor nodes (features) are independent of target T provided $P{a}_T$. $f⫫T|P{a}_T$ is any non-descendant node if $f\in f$ is any non-successor nodes (feature). As a result, f will be omitted from $CP{C}_T$ due to the Markov condition $f\subseteq De{s}_T$. □

Proposition 3.

(Remove false-positives from a Spouse). In a BN, R is a feature set, assuming that $X_i$ is neighboring to $Y_i$, $Y_i$ is neighboring to T, and $X_i$ is not neighboring to T (e.g, $X_i\to Y_i\leftarrow T$). However, once feature M enters $CS{P}_T\left\{Y_i\right\}$, for any existing feature X in $CS{P}_T\left\{Y_i\right\}\setminus N$, $\exists S\subseteq CP{C}_T\cup CS{P}_T\left\{Y_i\right\}\setminus X_i$, $X_i⫫T|S\cup \left\{Y_i\right\}$, then X is a false-positives and removed from $CS{P}_T\left\{Y_i\right\}$.

Proof of Proposition 3. 

V-structure $\left(X_i\to Y_i\leftarrow T\right)$ illustrates that if $X_i$ is target T’s Spouse and $Y_i$ is the mutual Child, there occurs a subset $S\subseteq R\setminus \{X_i,T,Y_i,N\}$ so that features X and T are independent given feature S. Still, they are dependent given $S\cup Y_i$. If there is another feature, M exists to block the path between $X_i$ and $Y_i$. M has a direct effect on $Y_i$ that is not satisfied by V-structure $\left(X_i\to Y_i\leftarrow T\right)$, so in this condition, X is removed from $CS{P}_T\left\{Y_i\right\}$ and is considered as a false-positives Spouse. □

4.2. The Proposed OFSVMB Algorithm and Analysis

This paper segment explains the proposed algorithm for Online Feature Selection via Markov blanket, given in Algorithm 1. The proposed algorithm can derive the $M{B}_T$ by deleting all redundant features. The OFSVMB algorithm based on null-conditional independence test, relevance analysis, and redundant analysis. First, the null-conditional independence [26] (line 5) of Algorithm 1 is used to tackle the streaming feature. It analyze the new feature $X_i$ arrived at time $t_i$. If $X_i$ is dependent of target T given empty set, (the empty set is equal to ⌀), then include feature $X_i$ to $CP{C}_T$; otherwise, add $X_i$ to non_pc${}_T$. After null conditional independence analysis, check if either feature $X_i$ is a true positive $PC$ (line 6–9) given subset S. If it is not a true positive, remove from $CP{C}_T$ (line 10); otherwise, analyze whether it is a candidate Spouse $CS{P}_T\left\{X_i\right\}$ (line 11–19).

Algorithm 1 OFSVMB.

OFSVMB then checks non-MB successors $X_i$ in the $CP{C}_T$ that may have several pathways to the target variable T. If $X_i$ and T are independent, $X_i$ is removed from the $CP{C}_T$ (line 24). Algorithm 1 finds the Parents-Child (line 5–10) and computes the Spouses from the non_pc${}_T$ set that is discarded during the null-conditional independence step (line 5) and from false-positive PC (line 6–10).

Moreover, after checking the conditional independence test at (line 6), if the feature $X_i$ is not dependent on target feature T, it comes to (line 29), to identify the spouse from the non_pc${}_T$ set. If $X_i\cancel{⫫}T|Se{p}_T\left\{X_i\right\}\cup N$, include the $CS{P}_T\left\{N\right\}$ (line 31–34). Through redundant analysis (line 35–45) check whether the selected spouse is redundant, if yes, then discard the false positive/redundant feature from $CS{P}_T\left\{N\right\}$.

4.3. Statistical Conditional Independence Terminology in OFSVMB

The conditional independence test is used to classify irrelevant and redundant features [15], denoted by the notations $Ind(T,X_i|S)$ and $Dep(T,X_i|S)$ in Algorithm 1. The $G^2$ test, equivalent to the ${\chi }^2$ test (discrete data), and Fisher’s z-test are used.

4.3.1. Statistical $G^2$ Test for Discrete Data

The $G^2$ with three features (variables), $X_i,X_j$, and $X_k$, set $V_{ijk}^{abc}$ as the number of checks satisfying $X_i=x,X_j=y$, and $X_k=z$ in a dataset. $V_{ij}^{xy},V_{jk}^{yz}$, and $V_k^z$ are all described in the same way. If $X_i$ and $X_j$ are conditionally not dependent given $X_k$, thus, in Equation (1) below, the $G^2$ test is shown:

$$G^2=2\sum _{x,y,z}{V}_{ijk}^{xyz}ln\frac{V_{ijk}^{xyz}{V}_k^z}{V_{ik}^{xz}{V}_{jk}^{yz}}.$$

(1)

With sufficient degrees of freedom, $G^2$ is asymptotically distributed as ${\chi }^2$. In general, when checking the conditional independence of $X_i$ and $X_j$ given S, the amount of degrees of freedom $df$ used during the test is measured as:

$$df=\left(c_i-1\right)\left(c_j-1\right)\prod _{c_kϵS}{c}_k$$

(2)

where, in Equation (2), $c_i$ represents the number of distinct values of $X_i$.

4.3.2. Statistical Fisher’s z-Test for Continuous Data

Fisher’s z-test, on the other hand, calculates the degree of correlation between features as given in Equation (3). After the feature subset S is provided, the partial correlation coefficient $r\left(X_i,T|S\right)$ between the feature $X_i$ and the target variable T is expressed in the Gaussian distribution $N\left(\mu ,\sum \right)$ [14]:

$$r_{\left(X_i,T|S\right)}=\frac{r_{\left(X_i,T\right)}-r_{\left(X_i,S\right)}{r}_{\left(S,T\right)}}{\sqrt{1-r_{\left(X_i,S\right)}^2}\sqrt{1-r_{\left(T,S\right)}^2}}$$

(3)

Under the null hypothesis of conditional independence between the feature $X_i$ and the target variable T of the current feature subset S, $r_{\left(X_i,T|S\right)}=0$, according to Fisher’s z-test. Assume that $\alpha =\{0.01,0.05\}$ is a given level of significance and $\rho$ is the p-value obtained by Fisher’s z-test.

Supposing $\rho >\alpha$, $X_i$ and target variable T are not related to each other when the subset S is given, according to the null hypothesis of the conditional independence of $X_i$ and target variable T. If $\rho \le \alpha$, then $X_i$ and target variable T are both relevant.

4.4. Correctness of OFSVMB

OFSVMB produces the MB of the target feature T truly and accurately, under the faithful assumption. According to Theorem 1, in the beginning, OFSVMB finds true positive or relevant features which belong to the $CP{C}_T$. It is a candidate Parents-Child set of target feature T using null-conditional independence and adds them in the candidate parent child (CPC) set, e.g., $CP{C}_T$. In contrast, add the false positive or redundant feature in the non-Parents-Child set (line 5). Then, in the candidate Parents-Child (CPC) set, e.g., $CP{C}_T$, OFSVMB searches for a false positive or redundant feature and discards it from the candidate parent-child (CPC) set (line 9) and (line 10). The $CP{C}_T$ contains all relevant features dependent on target feature T given any subset $S\in R$.

Additionally, OFSVMB finds the Spouse (line 13–19) of target T at the same time as looking for a redundant feature in the $CP{C}_T$ set. Non-MB successors in $CP{C}_T$ are discarded by conditioning on the true positive Spouse in $CS{P}_T\left\{X_i\right\}$, combined with the true positive PC in $CP{C}_T$ in the joint set $CP{C}_T$ and $CS{P}_T\left\{X_i\right\}$ (line 20–27). Suppose the feature $X_i$ (line 6) is independent of target feature T, so the OFSVMB considers searching Spouse from the non_pc${}_T$ set (line 29–34) and simultaneously removes the redundant feature from $CS{P}_T$ through redundant analysis (line 35–38). At last, if there is no feature left, OFSVMB finds the Markov blanket (MB) of target feature T at (line 48), which is the union of true positive $P{C}_T$ = $CP{C}_T$ and true positive $S{P}_T$ = $CS{P}_T$.

4.5. Time Complexity Analysis

The time complexity, as presented in

Table 3. Time complexity of Markov blanket (MB) discovery Algorithm.

Algorithm	Time Complexity
IAMB	$O\left({\left|R\right|}^2\right)$
STMB	$O\left(\left|R\right|2^{\left|R\right|}\right)$
HITON-MB	$O\left(2^{|CP{C}_T|}\left|R\right||CP{C}_T|\right)$
BAMB	$O\left(\left|R\right|2^{|CP{C}_T|}\right)$
EEMB	$O\left(\left|R\right|2^{|CP{C}_T|}\right)$
OFSVMB	$O\left(\left|R\right|2^C\right)$

of the state-of-the-art MB discovery algorithms, counts on how many CI tests have been used in the algorithm. OFSVMB identifies the MB of target feature T through online relevance and redundancy analysis. It is assumed that R represents the total number of features that appeared with time t, where $R^{\text{'}}$ is the number of features relevant to target feature T in R. The remaining number of features in R, not relevant to target T, is represented by $R{}^{″}$. $CS{P}_T\left\{X_i\right\}$ represents the Candidate Parents-Child set, and $CS{P}_T\left\{X_i\right\}$ represents a subset of the Spouses of target T with regards to target T’s child $X_i$. When the feature $X_i$ appears at time t, the OFSVMB time complexity is explained below: The null-conditional independence has a time complexity of $O\left(1\right)$. The Parents-Child identification takes $O\left(R\left(R^{\prime }\right)\left|CP{C}_T\right|k^{CPC}\right)$ and the discovery of Candidate Spouses is $O\left(R{}^{″}\left|CP{S}_T\right\{X_i\left\}\right|\right)K^{CP{C}_T}$, where k is the maximum limit of a conditioning set that might increase. The OFSVMB is based on streaming discard redundant features in the streaming scenario, and both $CPC$ and $CSP$ become smaller. The approximate time complexity of the proposed OFSVMB algorithm is $O\left(\left|R\right|2^C\right)$, where C is the is equal to $CP{C}_T$ and $CS{P}_T$. The OFSVMB is somehow more efficient than the state-of-the-art because it handles features in real-time scenarios.

The proposed OFSVMB and EEMB accuracy are comparable. However, OFSVMB performs the streaming feature selection where features come one by one inflow, so removing the redundant features from $CP{C}_T$ and $CS{P}_T$ jointly takes a few CI tests. The STMB uses a backward strategy during PC learning and identifies the separated PC set from all other subsets of R at any repetition. It makes the STMB slower than the proposed OFSVMB algorithm, as presented in Table 3.

5. Results and Discussion

In this segment, the results of the proposed OFSVMB are discussed in detail. The results are conducted through extensive experiments and comparing them with the traditional-based MB discovery algorithms such as Iterative Associative Markov Blanket (IAMB), Simultaneous MB (STMB), HITON-MB (HITON-MB), Balanced Markov Blanket (BAMB), an Efficient and Effective MB discovery (EEMB), and streaming-based algorithms such as Alpha-investing ($\alpha$-investing), Scalable and Accurate Online Feature Selection (SAOLA), and Online Streaming Feature Selection (OSFS).

5.1. Datasets and Experiment Setup

The experimental results are computed on 9 benchmark BN datasets (https://pages.mtu.edu/lebrown/supplements/mmhc_paper/mmhc_index.html, accessed on 1 July 2021) of small and large sample-size, as shown in

Table 4. Outline of the benchmark BN datasets.

Dataset	#Features	#Edges	Max In/Out Degree	Min/Max $|\mathit{PC}-\mathit{set}|$
Child	20	25	2/7	1/8
Child3	60	79	3/7	1/8
Child10	200	126	2/7	1/9
Alarm10	370	570	4/7	1/9
Insurance	27	52	3/7	1/9
Pig	441	592	2/39	1/41
Gene	801	972	4/10	0/11
Barley	48	84	4/5	1/8
Mildew	35	46	3/3	1/5

, and 14 real-world feature selection datasets as shown in

Table 5. Outline of the real-world datasets.

Dataset	Features	Instances	Dataset	Features	Instances
arcene	10,001	100	reged1	999	500
spect	22	267	medelon	500	2600
wdbc	30	569	marti1	1024	500
ionoshpere	351	35	medical	978	1434
colon	2000	62	lymphoma	62	4027
lung	3312	203	leukemia	72	7071
prostate-GE	5967	102	sido0	4932	12,678

. The real-world datasets are selected from different domains, such as sets from the UCI machine learning repository [27]; frequently studied public microarray (wdbc) [28], ionoshpere, colon, arcene, leukemia, and madelon are from the NIPS 2003 feature selection competition [29]; lung and medical belongs to biomedical [30]; lymphoma, reged1, and marti1 [31,32]; and prostate-GE and sido0 [33,34].

The OFSVMB algorithm is implemented in Matlab R2017b. All the experimental work is conducted on Windows 10 with an Intel Core i5-6500U with 8 GB RAM. The two conditional independence (CI) tests, including the $G^2$ test (for discrete data) and the Fisher’s z-test (for continuous data) with the significance levels of 0.01 and 0.05 are used.

5.2. Evaluation Metrics

The performance of the proposed OFSVMB algorithm is evaluated using three evaluation metrics on the benchmark BN datasets. The evaluation metrics are as follows:

• Precision: The number of true positives in an algorithm’s output (e.g., features that belong to the true MB of a target feature in a DAG) divided by the number of features in the algorithm’s output yields this metric.
• Recall: This metric is calculated by dividing the number of true positives in the output by the number of true positives (the true MB of a target feature in a DAG).
• F1 = $2*\left(\left(\mathit{Precision}*\mathit{Recall}\right)/\left(\mathit{Precision}+\mathit{Recall}\right)\right)$: The harmonic mean of the precision and recall is used to calculate the F1 score. In the best-case scenario, F1 = 1 if precision and recall are both excellent. In the worst-case scenario, F1 = 0.

Each benchmark BNs contains two groups of sample size, i.e., 500 and 5000.

Results and Discussion on Benchmark BN

The efficiency and efficacy of the proposed OFSVMB algorithm is computed and compared with other state-of-the-art traditional-based MB discovery algorithms, such as IAMB, STMB, HITON-MB, BAMB, and EEMB, on 9 benchmark BN datasets. The F1, precision, recall, and running time in seconds of OFSVMB and other state-of-the-art are shown in

Table 6. F1, Precision, Recall, and Running time (s) on small and large-size benchmark BN datasets with sample size = 500.

Dataset	Algorithm	F1	Precision	Recall	Time(s)
Child	IAMB	0.78/0.67	0.86/0.67	0.77/0.77	0.13
	STMB	0.73/0.70	0.91/0.85	0.67/0.67	0.20
	HITON-MB	0.82/0.81	0.94/0.86	0.77/0.77	0.24
	BAMB	0.86/0.81	0.93/0.82	0.81/0.84	0.18
	EEMB	0.86/0.81	0.92/0.81	0.82/0.83	0.15
	OFSVMB	0.89/0.88	0.97/0.94	0.83/0.85	0.14
Child3	IAMB	0.65/0.70	0.70/0.70	0.61/0.71	0.42
	STMB	0.65/0.58	0.82/0.70	0.62/0.60	0.86
	HITON-MB	0.76/0.71	0.85/0.83	0.69/0.71	1.27
	BAMB	0.77/0.78	0.89/0.86	0.70/0.73	0.74
	EEMB	0.57/0.57	0.67/0.67	0.50/0.50	0.74
	OFSVMB	0.89/0.87	0.95/0.93	0.83/0.85	0.48
Child10	IAMB	0.62/0.62	0.60/0.61	0.65/0.64	2.11
	STMB	0.58/0.50	0.70/0.56	0.60/0.55	2.25
	HITON-MB	0.65/0.67	0.70/0.75	0.62/0.71	3.19
	BAMB	0.70/0.73	0.70/0.74	0.71/0.73	2.32
	EEMB	0.61/0.60	0.65/0.69	0.64/0.61	2.23
	OFSVMB	0.75/0.80	0.75/0.75	0.75/0.87	1.32
Alarm10	IAMB	0.60/0.62	0.75/0.75	0.51/0.53	0.31
	STMB	0.54/0.56	0.75/0.76	0.47/0.49	0.30
	HITON-MB	0.71/0.72	0.90/0.84	0.62/0.65	3.08
	BAMB	0.71/0.70	0.78/0.81	0.66/0.63	0.28
	EEMB	0.80/0.79	0.90/0.85	0.75/0.87	0.35
	OFSVMB	0.65/0.66	0.91/0.88	0.56/0.58	0.25
Insurance	IAMB	0.55/0.56	0.79/0.85	0.43/0.42	0.17
	STMB	0.46/0.46	0.81/0.82	0.34/0.35	0.24
	HITON-MB	0.56/0.57	0.89/0.89	0.44/0.45	0.33
	BAMB	0.55/0.56	0.73/0.71	0.43/0.45	0.43
	EEMB	0.62/0.67	0.75/0.76	0.58/0.59	0.56
	OFSVMB	0.78/0.79	0.90/0.90	0.70/0.71	0.15
Pig	IAMB	0.74/0.75	0.78/0.80	0.72/0.72	11.07
	STMB	0.79/0.74	0.91/0.86	0.74/0.72	11.94
	HITON-MB	0.91/0.76	0.88/0.64	0.90/0.90	14.78
	BAMB	0.86/0.84	0.88/0.84	0.86/0.86	27.3
	EEMB	0.92/0.80	0.93/0.72	0.90/0.89	18.30
	OFSVMB	0.93/0.91	0.96/0.91	0.92/0.92	9.25
Gene	IAMB	0.63/0.68	0.60/0.62	0.68/0.68	20.09
	STMB	0.56/0.18	0.44/0.12	0.73/0.73	30.24
	HITON-MB	0.80/0.66	0.79/0.56	0.90/0.92	23.34
	BAMB	0.82/0.68	0.79/0.59	0.90/0.91	23.18
	EEMB	0.82/0.67	0.78/0.57	0.90/0.92	28.50
	OFSVMB	0.87/0.89	0.85/0.86	0.91/0.93	14.88
Barley	IAMB	0.34/0.35	0.72/0.75	0.24/0.23	0.18
	STMB	0.12/0.11	0.30/0.24	0.08/0.08	0.20
	HITON-MB	0.34/0.39	0.74/0.77	0.24/0.27	0.19
	BAMB	0.28/0.31	0.30/0.35	0.28/0.28	0.21
	EEMB	0.33/0.35	0.40/0.27	0.50/0.50	0.21
	OFSVMB	0.34/0.34	0.73/0.73	0.24/0.25	0.17
Mildew	IAMB	0.30/0.32	0.79/0.75	0.19/0.21	0.11
	STMB	0.13/0.09	0.29/0.19	0.09/0.06	0.14
	HITON-MB	0.34/0.34	0.81/0.83	0.21/0.22	0.20
	BAMB	0.33/0.34	0.81/0.81	0.21/0.22	0.16
	EEMB	0.29/0.25	0.50/0.33	0.20/0.20	0.17
	OFSVMB	0.35/0.35	0.83/0.82	0.23/0.23	0.9

for 500 samples and

Table 7. F1, Precision, Recall, and Running time (s) on small and large-size benchmark BN datasets with sample size = 5000.

Dataset	Algorithm	F1	Precision	Recall	Time(s)
Child	IAMB	0.75/0.67	0.83/0.64	0.70/0.71	11.05
	STMB	0.82/0.73	0.85/0.73	0.80/0.77	69.21
	HITON-MB	0.89/0.83	0.94/0.84	0.87/0.87	41.33
	BAMB	0.93/0.86	0.96/0.83	0.91/0.91	23.25
	EEMB	0.93/0.88	0.95/0.82	0.93/0.95	17.15
	OFSVMB	0.95/0.90	0.97/0.89	0.98/0.99	11.13
Child3	IAMB	0.79/0.71	0.75/0.71	0.79/0.72	5.29
	STMB	0.87/0.75	0.90/0.79	0.88/0.87	9.13
	HITON-MB	0.89/0.83	0.94/0.84	0.87/0.87	38.54
	BAMB	0.60/0.59	0.61/0.65	0.63/0.64	14.2
	EEMB	0.70/0.70	0.67/0.67	0.74/0.74	12.06
	OFSVMB	0.95/0.91	0.97/0.89	0.95/0.95	8.50
Child10	IAMB	0.64/0.66	0.64/0.67	0.65/0.65	12.80
	STMB	0.52/0.53	0.40/0.42	0.49/0.52	18.90
	HITON-MB	0.93/0.92	0.95/0.96	0.93/0.93	21.30
	BAMB	0.64/0.63	0.71/0.68	0.59/0.59	15
	EEMB	0.61/0.62	0.65/0.65	0.64/0.60	13.80
	OFSVMB	0.63/0.69	0.62/0.75	0.65/0.65	12.60
Alarm10	IAMB	0.76/0.75	0.78/0.75	0.75/0.75	8.95
	STMB	0.77/0.80	0.81/0.90	0.71/0.75	47.33
	HITON-MB	0.87/0.86	0.88/0.85	0.85/0.87	9.85
	BAMB	0.75/0.79	0.83/0.85	0.70/0.74	11.28
	EEMB	0.79/0.78	0.89/0.85	0.77/0.77	9.02
	OFSVMB	0.91/0.89	0.97/0.91	0.87/0.88	8.86
Insurance	IAMB	0.60/0.61	0.61/0.61	0.60/0.61	8.05
	STMB	0.65/0.65	0.92/0.90	0.54/0.56	45.62
	HITON-MB	0.73/0.74	0.93/0.93	0.63/0.64	12.64
	BAMB	0.62/0.63	0.87/0.88	0.50/0.52	25.46
	EEMB	0.79/0.38	0.89/0.27	0.75/0.70	11.86
	OFSVMB	0.84/0.83	0.94/0.94	0.77/0.75	7.90
Pig	IAMB	0.63/0.61	0.62/0.62	0.65/0.70	15.22
	STMB	0.38/0.15	0.28/0.38	0.32/0.38	15.94
	HITON-MB	0.64/0.72	0.75/0.82	0.76/0.85	17.78
	BAMB	0.96/0.84	0.93/0.74	1.00/1.00	22.2
	EEMB	0.96/0.87	0.94/0.79	1.00/1.00	18.27
	OFSVMB	0.90/0.90	0.90/0.91	0.88/0.89	15.07
Gene	IAMB	0.60/0.60	0.53/0.53	0.89/0.89	24.01
	STMB	0.30/0.11	0.20/0.20	0.75/0.78	35.24
	HITON-MB	0.83/0.68	0.77/0.57	0.85/0.85	30.34
	BAMB	0.82/0.69	0.76/0.59	0.94/0.94	26.03
	EEMB	0.82/0.71	0.77/0.61	0.94/0.94	24.50
	OFSVMB	0.93/0.91	0.87/0.88	0.95/0.95	23.09
Barley	IAMB	0.35/0.36	0.63/0.65	0.25/0.25	9.70
	STMB	0.35/0.35	0.70/0.69	0.27/0.28	10.15
	HITON-MB	0.54/0.54	0.75/0.75	0.42/0.42	9.89
	BAMB	0.26/0.24	0.23/0.21	0.29/0.28	17.21
	EEMB	0.67/0.63	0.50/0.22	1.00/1.00	15.63
	OFSVMB	0.48/0.49	0.78/0.77	0.34/0.35	9.19
Mildew	IAMB	0.48/0.49	0.60/0.63	0.41/0.41	9.87
	STMB	0.21/0.20	0.39/0.38	0.15/0.14	11.45
	HITON-MB	0.42/0.40	0.77/0.76	0.41/0.41	12.09
	BAMB	0.16/0.17	0.41/0.41	0.11/0.11	14.72
	EEMB	0.29/0.27	0.19/0.18	0.60/0.60	9.89
	OFSVMB	0.53/0.53	0.77/0.75	0.42/0.42	9.66

for 5000 samples. The sign “/” represents the separation of significant level, e.g., on the left side the $\alpha =0.01$ and on the right side the $\alpha =0.05$, where $\alpha$ is the significant level. The F1, precision, and recall of sample sizes 500 and 5000 (a) shows significance levels of 0.01 and (b) shows significance levels of 0.05 in Figure 2 and Figure 3, respectively. Moreover, Figure 4 shows the running time of OFSVMB and five other state-of-the-art algorithms with different sample sizes. In the figures, the x-axis are denoted as the number of datasets (see Table 4), and the y-axis represent the F1, precision, and recall, together with running time, respectively.

According to Table 6, OFSVMB is the most accurate and fastest algorithm among the other five state-of-the-art algorithms because it is based on streaming features, which compute the features in an online manner. Meanwhile, on Child, Child3, and Insurance datasets with a sample size of 500 and significance levels of 0.01 and 0.05, OFSVMB is accurate in terms of F1, precision, and recall compared to IAMB, STMB, HITON-MB, BAMB, and EEMB, while faster than STMB, HITON-MB, BAMB, and EEMB. STMB and BAMB do not perform the symmetry check, but they must conduct an exhaustive subset analysis for the PC learning, making these two algorithms relatively slow and inefficient. The OFSVMB is slower than IAMB on Child and Child3 datasets because in each iteration, IAMB uses the entire set of presently selected features as conditioning set to identify whether or not to add or remove a feature from the currently selected features. IAMB is computationally efficient on datasets with small sample sizes, as presented in Table 6 in bold and shown in Figure 2a,b and Figure 4. On Child10 datasets, the OFSVMB is faster than other state-of-the-art algorithms, and IAMB is the second fastest, as shown in Table 6 in bold and shown in Figure 2a,b and the running time in Figure 4. Moreover, on the Child10 dataset, the HITON-MB’s accuracy is comparable with the OFSVMB in terms of precision with a significance level of 0.05, as shown in Table 6 in bold.

On the Alarm10 dataset with a sample size of 500, the EEMB is more accurate in terms of F1 and recall than the OFSVMB, IAMB, STMB, HITON-MB, and BAMB. At the same time, the OFSVMB is accurate than EEMB in terms of precision and the running time is faster as well, as presented in Table 6 in bold and shown in Figure 2a,b and Figure 4. On extensive datasets such as Pig and Gene with a sample size of 500 using significance levels of 0.01 and 0.05, OFSVMB is accurate in terms of F1, precision, and recall compared to its rivals, as shown in Figure 2a,b, while it is faster than the state-of-the-art as presented in Table 6 in bold and in Figure 4. In a dense dataset such as Barley, the HITON-MB is more accurate because HITON-MB locates the target’s Spouses, and must find the PC of each feature in the target’s discovered PC set. This method drastically reduces the number of sample data required and increases MB discovery performance, especially when dealing with high-dimensional and small data samples. When the size of the PC set of the features within the target’s PC set is enormous, therefore, this type of technique is computationally costly. The HITON-MB accuracy is higher in terms of F1 and precision than IAMB, STMB, BAMB, EEMB, and OFSVMB, but at a significance level of 0.01, the OFSVMB has comparable accuracy with HITON-MB in terms of F1. Moreover, in terms of recall, the EEMB has higher accuracy than IAMB, STMB, HITON-MB, BAMB, and OFSVMB, but the OFSVMB has higher running time than others, as shown in Table 6 in bold and Figure 2a,b and Figure 4. On Mildew, the OFSVMB shows higher accuracy than other algorithms. However, HITON-MB is more accurate than the OFSVMB and the others such as IAMB, STMB, BAMB, and EEMB at a significance level of 0.05 in terms of precision. In contrast, at a significance level of 0.01, the OFSVMB is more accurate than HITON-MB, as shown in Table 6. The OFSVMB on the Mildew dataset runs faster than the state-of-the-art as presented in Table 6 in bold and shown in Figure 2a,b and Figure 4.

On small Child and Child3 datasets with a sample size of 5000 using significance levels of 0.01 and 0.05, OFSVMB is more accurate in terms of F1, precision, and recall compared to others, as presented in Table 7 in bold and shown in Figure 3a,b. However, on the Child dataset, the OFSVMB runs faster than the HITON-MB, STMB, BAMB, and EEMB but slower than IAMB. While on the Child3 dataset, the IAMB runs faster than the OFSVMB and the other four algorithms as shown in Table 7, and the running time is shown in Figure 4. On the Child10 dataset with a sample size of 5000, HITON-MB outperforms on OFSVMB, IAMB, STMB, BAMB, and EEMB in terms of F1, precision, and recall, while OFSVMB is run faster than the other 5 algorithms as shown in Table 7 in bold. Figure 3a,b shows the accuracy of OFSVMB and other algorithms, while the running time is given in Figure 4. On the Alarm10, Insurance datasets with a sample size of 5000, OFSVMB is more accurate than IAMB, STMB, HITON-MB, BAMB, and EEMB in terms of F1, precision, and recall as shown in Table 7 in bold and in Figure 3a,b. Meanwhile, OFSVMB shows higher accuracy and better running time than others, as shown in Figure 4. On a large-size dataset with a sample size of 5000, such as the Pig, the BAMB and EEMB are more accurate in terms of F1 and precision at a significance level of 0.01 and for recall at significance levels of 0.01 and 0.05, as shown in Table 7 in bold and in Figure 3a,b. The OFSVMB is more accurate in terms of F1 and precision at significance levels of 0.05 and runs faster than its rivals as shown in Figure 4. Moreover, on an extensive Gene dataset, the OFSVMB is still more accurate and runs faster than other state-of-the-art algorithms as shown in Table 7 in bold and in Figure 3a,b, and the running time is shown in Figure 4.

In a dense dataset with a sample size of 5000, such as the Barley, the EEMB is more accurate in terms of F1 and recall at a significant level of 0.01 and 0.05, as shown in Table 7 in bold and in Figure 3a,b. Still, OFSVMB is more accurate in terms of precision at a significance level of 0.01 and 0.05 than EEMB and runs faster than others, as shown in Figure 4. While on the Mildew dataset, the OFSVMB is more accurate than IAMB, STMB, BAMB, and EEMB. The HITON-MB is more accurate than OFSVMB in terms of precision at a significance level of 0.05, while comparable accuracy in terms of precision at a significance level of 0.01, as shown in Table 7 in bold and in Figure 3a,b and runs faster than other algorithms as shown in Figure 4.

5.3. Evaluation Classifiers

The number of selected features and the prediction accuracy of OFSVMB on 14 real-world datasets with low to high dimensionality is conducted using 12 classifiers including C1 = Coarse Gaussian SVM, C2 = Coarse KNN, C3 = Coarse Tree, C4 = Cosine KNN, C5 = Fine Gaussian SVM, C6 = Fine Tree, C7 = Linear Discriminant, C8 = Linear SVM, C9 = Medium KNN, C10 = Medium Tree, C11 = Subspace Discriminant, and C12 = Subspace KNN, where C standards for classifier and saving space in

Table 8. Comparison of prediction accuracy under C1–C6 classifiers at significance level = 0.01.

		Prediction Accuracy (%)
Dataset	Algorithm	C-1	C-2	C-3	C-4	C-5	C-6
arcene	OSFS	80	56	76	82	71	74
	Alpha-investing	57	55	70	70	62	65
	SAOLA	−	−	−	−	−	−
	OFSVMB	80	65	76	82	71	74
spect	OSFS	70	70	70	46.4	96	70
	Alpha-investing	70	65.5	69.3	69.3	96	69.7
	SAOLA	70	70	70	46.4	92	70
	OFSVMB	70	82.6	72.7	91.3	97	71.9
wdbc	OSFS	96.3	93.3	92.1	96	87	93.1
	Alpha-investing	90	94	93	96.3	88	93.1
	SAOLA	91	91	91.6	90.5	88	90.5
	OFSVMB	97.3	95.8	95.5	96.1	94	93.3
ionosphere	OSFS	88.6	87.2	90	90.6	88	89.2
	Alpha-investing	88	65.8	90.3	86.9	86	91.2
	SAOLA	88.9	73.2	90.3	88.3	88	89.7
	OFSVMB	94	68.7	92	96.9	89	91.5
colon	OSFS	80.6	64.5	83.9	80.6	88	82.3
	Alpha-investing	66.1	64.5	77.4	74.2	79	71
	SAOLA	87.1	64.5	80.6	90.3	85	82.3
	OFSVMB	88.7	80	88.7	98.3	88	88.7
lung	OSFS	83.7	68.5	77.3	87.2	82	81.8
	Alpha-investing	78.3	68.5	81.8	93.6	79	82.8
	SAOLA	84.2	68.5	80.3	89.2	−	84.2
	OFSVMB	83.7	75.5	75.4	97.7	85	82.8
prostate-GE	OSFS	94.1	51	92.2	94.1	99	92.2
	Alpha-investing	94.1	51	90.2	93.1	88	90.2
	SAOLA	96.1	51	86.3	96.1	95	86.3
	OFSVMB	94.1	80	93.1	98.1	99	91.2
reged1	OSFS	98.2	88.4	97.4	98.4	90	96.8
	Alpha-investing	88.2	88.2	88	88.2	94	86.4
	SAOLA	99.4	88.2	96.8	98	93	96.6
	OFSVMB	99.2	82.2	94.6	98.4	91	95.8
medalon	OSFS	62	61.9	61.7	56	94	93
	Alpha-investing	49.8	50.5	49.4	49.2	96	50
	SAOLA	62.4	61.7	60.9	57.7	96	58.3
	OFSVMB	62.4	91	61.5	98	96	95.5
lycomphia	OSFS	98.4	67.7	95.2	98.4	83	95.2
	Alpha-investing	93.5	67.7	96.8	93.5	88	96.8
	SAOLA	98.4	67.7	87.1	85.5	80	87.1
	OFSVMB	100	92.9	96.5	96.6	70	97
marti1	OSFS	88.2	88.2	87.8	88.2	87	79.2
	Alpha-investing	88.2	88.2	86.4	88.8	66	82.4
	SAOLA	88.2	88.2	87.8	88.2	81	79.2
	OFSVMB	95.2	95.2	89.6	96.8	89	79.4
medical	OSFS	99.4	95.6	99.4	99.4	62	99.4
	Alpha-investing	99.1	95.6	99.2	98.7	49	99.4
	SAOLA	98.7	95.6	99.4	98.2	61	99.4
	OFSVMB	99.4	97.6	99.4	99	62	99.7
leukemia	OSFS	83.7	68.5	78.3	88.2	92	82.8
	Alpha-investing	78.3	68.5	77.8	94.1	50	81.8
	SAOLA	84.2	68.5	79.3	89.7	81	85.2
	OFSVMB	90.8	80	81.8	88.2	94	85.9
sido0	OSFS	83.7	68.5	74.3	88.2	88	86.9
	Alpha-investing	78.3	68.5	66.8	94.1	78	81.8
	SAOLA	80.2	68.5	72.3	89.7	85	85.2
	OFSVMB	81.8	79.5	79.8	98.2	90	89.3

,

Table 9. Comparison of prediction accuracy under C7–C12 classifiers at significance level = 0.01.

		Prediction Accuracy (%)
Dataset	Algorithm	C-7	C-8	C-9	C-10	C-11	C-12
arcene	OSFS	84	85	73	74	85	74
	Alpha-investing	76	74	67	65	72	73
	SAOLA	−	−	−	−	−	−
	OFSVMB	84	85	75	74	85	74
spect	OSFS	70	70	95	70	69.7	89.6
	Alpha-investing	70.8	67.8	96	69.7	70	48.3
	SAOLA	70	70	91	70	70	70
	OFSVMB	71.2	80	96	71.9	91	89.6
wdbc	OSFS	95.6	96.5	95	93.1	95.1	93
	Alpha-investing	95.1	97.4	66	93.1	95.3	93.1
	SAOLA	92.1	91.4	74	91.2	92.6	75.2
	OFSVMB	95.6	96.8	87	93.3	95.4	94.9
ionosphere	OSFS	87.2	87.7	77	89.2	86.3	88.9
	Alpha-investing	86.6	87.7	85	71	85.8	93.2
	SAOLA	87.2	87.7	88	90	86.6	90.3
	OFSVMB	85.5	87.9	89	91.5	89.7	93.3
colon	OSFS	83.9	83.9	83	82.3	80.6	77.4
	Alpha-investing	83.9	82.3	65	82.8	80.6	77.5
	SAOLA	87.1	85.5	65	82.3	87.1	79
	OFSVMB	88.7	88.7	88	88.7	88.7	78.2
lung	OSFS	89.2	92.5	56	81.8	88.2	88.2
	Alpha-investing	97	89.7	56	90.2	97	93.1
	SAOLA	94.6	91	−	84.2	93.1	89.7
	OFSVMB	91.4	94.1	82	82.8	91.6	90.6
prostate-GE	OSFS	95.1	95.1	99	92.2	94.1	81.4
	Alpha-investing	95.1	96.1	88	86.8	95.1	91.2
	SAOLA	95.1	95.1	98	86.3	60.8	95.1
	OFSVMB	94.1	95.1	99	91.2	94.1	94.2
reged1	OSFS	98.6	98.9	67	96.8	98.8	98.8
	Alpha-investing	88.2	88.2	68	50.7	88.2	77.2
	SAOLA	99	99.2	68	96.6	99	99.2
	OFSVMB	97.8	98.8	98	98.5	99.2	98.2
medalon	OSFS	61.3	61.6	51	61.5	61.2	55.4
	Alpha-investing	50	50.4	51	96.8	50	49.3
	SAOLA	61.8	62.1	51	60.6	61.8	57.2
	OFSVMB	61.5	75.2	95	75	61.4	57.8
lycomphia	OSFS	98.4	96.8	83	95.2	96.8	98.4
	Alpha-investing	98.4	96.8	72	83.8	96.8	93.5
	SAOLA	98.4	98.4	81	87.1	100	98.4
	OFSVMB	98.4	100	79	97.5	100	98.7
marti1	OSFS	88.2	88.2	87	84.6	88.2	77.2
	Alpha-investing	94	88.2	50	99.4	90.4	85
	SAOLA	88.2	88.2	75	84.6	88.2	77.2
	OFSVMB	87.6	88.2	88	92	94.5	85.2
madical	OSFS	−	99.4	62	99.4	99.4	96
	Alpha-investing	99.5	−	50	81.8	98.6	95.7
	SAOLA	99.4	62	−	99.4	98.1	95.8
	OFSVMB	91.3	99.4	62	99.4	98.1	95.6
leukemia	OSFS	89.2	91.6	92	82.8	91.1	89.7
	Alpha-investing	81.2	95.1	63	81.8	96.6	92.6
	SAOLA	85.5	93.1	80	85.2	85.2	91.6
	OFSVMB	89.2	94.6	91	84	91.1	88.7
sido0	OSFS	89.2	91.6	90	82.8	91.1	89.7
	Alpha-investing	85.5	95.1	88	82	82	87.6
	SAOLA	87.1	93.1	87	85.2	85.2	90.6
	OFSVMB	89.2	95.5	89	87	90.6	92.6

,

Table 10. Comparison of mean prediction accuracy under C1–C6 classifiers at significance level = 0.01.

		Prediction Accuracy (%)
Dataset	Algorithm	C-1	C-2	C-3	C-4	C-5	C-6
Mean	OSFS	86.2	73.5	82.5	85.7	86.4	86.8
	Alpha-investing	80	70.8	76.2	85	78.5	81.5
	SAOLA	−	−	−	−	−	−
	OFSVMB	88.4	83.2	85.4	95.1	86.7	88.2

and

Table 11. Comparison of mean prediction accuracy under C7–C12 classifiers at significance level = 0.01.

		Prediction Accuracy (%)
Dataset	Algorithm	C-7	C-8	C-9	C-10	C-11	C-12
Mean	OSFS	−	88.5	79.3	85.2	87.6	85.5
	Alpha-investing	85.8	−	69.6	81.1	85.6	82.2
	SAOLA	−	−	−	−	−	−
	OFSVMB	87.5	91.3	87	87.6	90.7	87.9

, the abbreviation is used with numbers. For all the datasets, cross-validation is used 10-fold to prevent bias in error estimation. Moreover, the selected number of features and running time in seconds are also reported to show the efficiency of the algorithms.

Results and Discussion on Real-World Dataset

This segment shows the comparison of selected features, prediction accuracy, and efficiency of the OFSVMB algorithm with other state-of-the-art streaming-based algorithms, such as Alpha-investing, SAOLA, and OSFS, on 14 real-world datasets with a significance level of 0.01. The following Table 8, Table 9, Table 10, Table 11 and

Table 12. #F (number of selected features) and running time (s) at significance level = 0.01.

Algorithm	OSFS		Alpha-Investing		SAOLA		OFSVMB
Dataset	#F	Time(s)	#F	Time(s)	#F	Time(s)	#F	Time(s)
arcene	5	3.02	7	0.25	−	−	6	9.05
spect	3	0.40	18	2.02	5	0.30	3	0.16
wdbc	3	0.27	20	8.65	2	0.50	9	7.01
ionosphere	5	0.2	10	0.6	5	0.9	12	19.02
colon	2	2.93	4	0.17	4	1.78	13	84.2
lung	8	51.82	45	0.71	30	1.24	10	52.88
Prostate-GE	2	2.10	12	0.59	12	0.60	5	3.75
reged1	11	34.28	1	0.06	15	0.43	10	38.05
madelon	6	1.05	1	0.12	7	0.08	9	0.25
lycomphia	3	5.13	5	0.30	35	1.99	33	70.55
marti1	3	4.99	28	5.85	1	0.50	2	2.23
medical	17	190.5	28	1.15	16	5	6	212.8
leukemia	5	1.5	2	0.01	17	1.02	13	10.55
sido	2	12.50	39	5.83	40	3	9	6.80

describe the prediction accuracy, mean prediction accuracy based on 12 classifiers, and the number of selected features and running time in seconds.

Figure 5 shows the prediction accuracy of the algorithms based on 12 classifiers where the x-axis denotes the dataset, and the y-axis represents the prediction accuracy. In contrast, Figure 6 shows the running time in seconds of the algorithms. The sign “−” denotes that the algorithm fails to select any number of features in the corresponding dataset and takes longer. The better outcomes are underlined in bold text, presented in the following Table 8, Table 9, Table 10, Table 11 and Table 12.

Table 8 and Table 9 describe the prediction accuracy of OFSVMB against OSFS, Alpha-investing, and SAOLA using 12 classifiers. As described in the tables, OFSVMB performs better than the other algorithms on most datasets. Its mean prediction accuracy is higher than others using the 12 classifiers as shown in Table 10 and Table 11. The OFSVMB searches for PC and Spouse of the target feature T. In contrast, the OSFS, Alpha-investing, and SAOLA only search for PC and ignore the Spouse set of the target feature T, which causes them to lose the interpretability.

On the arcene dataset, SAOLA fails to obtain any features, as shown in Table 8 and Table 9. OSFS and Alpha-investing have comparable accuracy with OFSVMB on very few datasets. OFSVMB somehow includes false positives, not worse than its rivals on real-world feature selection datasets. From Figure 5, the OFSVMB has better prediction accuracy compared with other algorithms on many datasets under 12 classifiers. Table 12 describes the number of selected features and the running time of the algorithms. Alpha-investing selects more features on many datasets because it does not re-evaluate the redundant features, making it ineffective but time-efficient. The OSFS selects fewer features than Alpha-investing and SAOLA and OFSVMB; when the new feature arrives, the OSFS first checks the relevancy, and then the arrived features’ redundancy, which improves accuracy against Alpha-investing and SAOLA, and selects fewer features.

The running time of OSFS is slower than Alpha-investing and SAOLA on many datasets because it considers features repeatedly to add or remove them from the feature set. The OFSVMB selects many features from many datasets and causes its running time to be slower than other algorithms, as shown in Figure 6. The OFSVMB searches for the PC and Spouse of the target feature T, while OSFS, Alpha-investing, and SAOLA only search the PC set of the target feature T.

5.4. Sensitivity Analysis

The OFSVMB algorithm is governed by $\alpha$ (Alpha $\left(\alpha \right)$ and significance level are similar) and $\left|RI\right|$. We conducted a sensitivity analysis to investigate the effect of parameter values on the model’s accuracy. This section provides the details of sensitivity experiments. Concerning different parameters, including $\alpha$ values and with different Rate of Instance $\left|RI\right|$, the analysis is practiced on six real-world feature selection datasets, where the Rate of Instance $\left|RI\right|$ is the subset of instances from the dataset. We choose three values of $\alpha$ such as $\left(0.1,0.01,0.05\right)$ concerning three different values of the $\left|RI\right|$ such as $\left(0.1,0.8,0.9\right)$.

The parameter $\alpha$ determines how well the reconstruction function preserves the original, observable feature’s values. The $\alpha$ = $\left(0.1,0.01,0.05\right)$ is used to see the effect of OFSVMB prediction accuracy. In contrast, investigating $\left|RI\right|$ = $\left(0.1,0.8,0.9\right)$ can keep important features along the instance to affect the prediction accuracy of the OFSVMB. In Figure 7, the x-axis represents the Rate of Instance $\left|RI\right|$, the y-axis represents parameter $\alpha$, and the z-axis represents the prediction accuracy (%) on six real-world datasets including (a) spect, (b) sylva, (c) madelon, (d) marti1, (e) ionosphere, and (f) reged1. These six datasets contain small and large sample sizes and sparse data.

From the results given in Figure 7c, we observe that when $\alpha$ is fixed with different $\left|RI\right|$ values, the prediction accuracy of the OFSVMB algorithm increases for $\left|RI\right|$ = 0.8 and 0.9. For some datasets given in Figure 7b,f, when the $\left|RoI\right|$ = 0.1, it can still keep the important feature along with instances and increase the prediction accuracy. We observe that the most optimal values of parameter $\alpha$ are 0.01 and 0.05, and under these values with different $\left|RI\right|$, the prediction accuracy of OFSVMB is higher. Still, the parameter $\alpha$ = 0.1 with different $\left|RI\right|$ also has an optimal contribution to the given six datasets regarding prediction accuracy. Thus, such an empirical value can be adopted in practice for future methods based on the statistical conditional independence test. These observations conclude that the OFSVMB is more accurate in selecting the optimal features regarding different $\alpha$ with different $\left|RI\right|$.

6. Conclusions

This paper proposes an Online Feature Selection via Markov Blanket (OFSVMB) algorithm that uses conditional independence $G^2$ and Fisher’s z-tests to find the MB based on streaming features. Once a feature is included in the PC and SP sets using online relevance analysis, it examines and checks for a true or false-positive feature using the online redundant analysis. OFSVMB tries to make the candidate features set of both PC and SP as small as possible, reducing the number of conditional independence tests. The proposed OFSVMB jointly identifies the Parents-Child and Spouse and separates them in streaming. The evaluation metrics such as F1, precision, and recall are used for evaluating the proposed algorithm on benchmark BN datasets and real-world datasets with 12 classifiers. Additionally, it also obtains the MB set with the highest accuracy. The results demonstrate that the OFSVMB is better than the traditional-based MB discovery algorithms such as IAMB, STMB, HITON-MB, BAMB, and EEMB on most benchmark BN datasets such as Child, Child3, Insurance, Pig, Gene with a sample size of 500 and Child3, Alarm10, Insurance, Gene, and Mildew with a sample size of 5000 with significance levels of 0.01 and 0.05. The OFSVMB also performs better on mean prediction accuracy regarding real-world datasets as compared to other streaming-based algorithms such as OSFS, Alpha-investing, and SAOLA using 12 classifiers including Fine Tree, Medium Tree, Coarse Tree, Linear Discriminant, Linear SVM, Fine Gaussian SVM, Coarse Gaussian SVM, Medium KNN, Coarse KNN, Cosine KNN, Subspace Discriminant, Subspace KNN. Furthermore, Searching feature strategies including the PC and Spouses of OFSVMB makes it a little time-consuming than OSFS, Alpha-investing, and SAOLA because these algorithms only search for the PC feature set and ignore the Spouses. OFSVMB is based on MB discovery, so it considers both PC and Spouses of the target feature T. In addition, the sensitivity analysis using two parameters $\alpha$ and Rate of Instance $\left|RoI\right|$ also shows the OFSVMB performs better.

On a large and dense network with many features, statistical hypothesis-based tests for conditional independence causes performance inconsistency in OFSVMB and reduce accuracy. Using a V-structure cause the OFSVMB running time to be slower against OSFS, Alpha-investing, and SAOLA because Alpha-investing and SAOLA select many features. However, they are still faster than our proposed algorithm.

In future work, we plan to overcome the limitations of the proposed algorithm and extend our work to address the direct causes and effects of obtaining the local causal discovery in the streaming feature of the target variable T using mutual information or Neighborhood mutual information with a combination of conditional independence tests using different structure instead of V-structure. It will help to compute more true positive PCs and Spouses. In addition, the focus will be to improve the accuracy and consistently examine the impact of causal faith-fullness violations in the streaming feature selection.