*7.5. McNemar's Statistical Test*

To evaluate whether there is any significant difference between the performance of the two logic mining models, McNemar's test is performed. According to [38], McNemar is the only test that has acceptable Type 1 error and can validate the performance of the 2SATRA model. The normal test statistics are as follows:

$$Z\_{i\bar{j}} = \frac{f\_{i\bar{j}} - f\_{\bar{j}i}}{\sqrt{f\_{i\bar{j}} + f\_{\bar{j}i}}} \tag{23}$$

where *Zij* is a measure of significance of the accuracy obtained by model *i* and *j*, while *fij* is the number of cases where logic mining is correctly classified by model i but incorrectly classified by model *j*. A similar description is given for the notation *fij*. In this experiment, a 5% level of significance is used. The null hypothesis dictates a pair from the logic mining model with no difference in disagreement. The performance of classification accuracy is said to differ significantly if *Zij* > 1.96. Note that, a positive value of *Zij* means the model *i* performs better than model *j*. Tables 14 and 15 presents the result of the McNemar's test for all the logic mining models. Several winning points for S2SATRA are discussed below.



**Table 14.** McNemar's statistical test for F1–F5.

**Table 15.** McNemar's statistical test for F6–F11.


#### **8. Discussion**

The optimal logic mining model requires pre-processing structures for neurons before the *Qbest* can be learned by HNN. Currently, the logic mining model specifically optimizes the logic extraction from the dataset without considering the optimal *Qbest*. The mechanism that optimizes the optimal neuron relationship before the learning can occur is remain unclear. In this sense, identifying a specific pair of neurons for *Qbest* will facilitate the dataset generalization and reduce computational burden.

As mentioned in the theory section, S2SATRA is not merely a modification of a conventional logic mining model, but rather it is a generalization that absorbs all the conventional models. Thus, S2SATRA not only inherits many properties from a conventional logic mining model but it adds supervised property that reduces the search space of the optimal *Q<sup>B</sup> i* . The question that we should ponder is: what is the optimal *Qbest* for the logic mining model? Therefore, it is important to discuss the properties of S2SATRA against the conventional logic mining model in extracting optimal logical rule from the dataset. According to the previous logic mining model, such as [20,21,31], the quality of attributes is not well assessed since the attributes were randomly assigned. For instance, [21] achieved high accuracy for specific combination of attributes but the quality of different combination of the attributes will result in low accuracy due to a high local minima solution. A similar neuron structure can be observed in E2SATRA, as proposed by [24], because the choice of neurons is similar during the learning phase. Practically speaking, this learning mechanism [20–22,31] is natural in real life because the neuron assignment is based on trial and error. However, the 2SATRA model needs to sacrifice the accuracy if there is no optimum neuron assignment. By adding permutation property, as carried out in Kasihmuddin et al. [30], P2SATRA is able to increase the search space of the model in the expense of higher computational complexity. To put things into perspective, 10 neurons require learning 18,900 of *Qbest* learning for each neuron combination before the model can arrive to the optimal result. Unlike our proposed model, S2SATRA can narrow down the search space by checking the degree of association among the neurons before permutation property can take place. Supervised features of S2SATRA recognized the pattern produced by the neurons and align it with the *Qbest* clause. Thus, the mutual interaction between association and permutation will optimally select the best neuron combination.

As reported in Tables 7 and 8, the number of associations for analysis required for *n* attributes to create optimal *Qbest* was reduced by <sup>1</sup> *n nC*6. In other words, the probability of P2SATRA to extract optimal *Qbest* is lower compared to the S2SATRA. As the *Qbest* supplied to the network has changed, the retrieval property of the S2SATRA model will improve. The best logic mining model demonstrates a high value of *TP* and *TP* with a minimized value of *FP* and *FN*. P2SATRA is observed to outperform the conventional logic mining in terms of performance metrics because P2SATRA can utilize the permutation attributes. In this case, the higher the number of permutations, the higher probability for the P2SATRA to achieve correct *TP* and *TN*. Despite a robust permutation feature, P2SATRA failed to disregard the non-significant attributes which leads to *Qlearn <sup>i</sup>* = 1. Despite achieving high accuracy, the retrieved final neuron state is not interpretable. E2SATRA is observed to outperform 2SATRA in terms of accuracy because the induced logic in E2SATRA is the only amount in the final state that reached global minimum energy. The dynamic of the induced logic in E2SATRA follows the convergence of the final state proposed in [22] where the final state will converge to the nearest minima. Although all the final state in E2SATRA is guaranteed to achieve global minimum energy, the choice of attribute that is embedded to the logic mining model is not well structured. Similar to 2SATRA and P2SATRA, the interpretation of the final attribute will be difficult to design. In another development, 2SATRA is observed to outperform the RA proposed by [14] in terms of all performance metric. Although the structure of RA is not similar to 2SATRA in creating the *Qlearn <sup>i</sup>* , the induced logic *<sup>Q</sup><sup>B</sup> <sup>i</sup>* has a general property of *QHORNSAT*. In this case, *QHORNSAT* is observed to create a rigid induced logic (at most 1 positive literal) and can reduce the

possible solution space of the RA. In this case, we only consider the dataset that satisfies the *QHORNSAT* that will lead to *Qtest* = 1.

In contrast, S2SATRA employs a flexible *Q*2*SAT* logic which accounts for both positive and negative literal. This structure is the main advantage over the traditional RA proposed by [14]. S2ATRA is observed to outperform the rest of the logic mining model due the optimal choice of attributes. In terms of feature, S2SATRA can capitalize the energy feature of E2SATRA and the permutation feature of P2SATRA. Hence, the induced logic obtained will always achieve global minimum energy and only relevant attribute *ρ* < *α* will be chosen to be learned in HNN. Another way to explain the effectiveness of logic mining is the ability to consistently find the correct logical rule to be learned by HNN. Initially, all logic mining models begin with HNN which has too many ineffective synaptic weights due to suboptimal features. In this case, S2SATRA can reduce the inconsistent logical rule that leads to suboptimal synaptic weight.

S2SATRA is reported to outperform almost all the existing logic mining models in terms of all performance metrics. S2SATRA has the capability to differentiate between *TP Q<sup>B</sup> <sup>i</sup>* = 1 and *TP Q<sup>B</sup> <sup>i</sup>* = −1 , which leads to high *Acc* and *F-score* values. Since S2SATRA is able to obtain more *TP Q<sup>B</sup> <sup>i</sup>* = 1 , the *Pr* and *Sen* will increase compared to the other existing methods. In terms of *Pr* and *Sen*, S2SATRA is reported to succesfully predict *Q<sup>B</sup> <sup>i</sup>* = 1 during the retrieval phase. In other words, the existing 2SATRA model is less sensitive to the positive outcome which leads to a lower value of *Pr* and *Se*. It is worth mentioning that the overfitting nature of the retrieval phase will lead to *Q<sup>B</sup> <sup>i</sup>* which can only produce more positive neuron states. This phenomenon was obvious in the existing method where the HNN tends to converge to only a few final states. This result has a good agreement with the McNemar's test where the performance of S2SATRA is significantly different from the existing method. The optimal arrangement of the *Q<sup>B</sup> <sup>i</sup>* signifies the importance of the association among the attributes towards the retrieval capability of the S2SATRA. Without proper arrangement, the obtained *Q<sup>B</sup> <sup>i</sup>* tends to overfit which leads to a high classification error. S2SATRA can only utilize correlation analysis during the pre-processing stage because correlation analysis provides preliminary connection between the attribute and *Qlearn <sup>i</sup>* .

It is worth noting that although there are many developments of the supervised learning method, such as a decision tree, a support vector machine, etc., none of these methods can provide the best approximation to the logical rule. Most of the mentioned methods are numerically compatible as an individual classification task, but not as a classification via a logical rule. For instance, a decision tree is effective in classifying the outcome of the dataset but S2SATRA is more effective in generalizing the datasets in the form of induced logics. The obtained induced logic can be utilized for a similar classification task. In term of parameter settings, S2SATRA is not dependent on any free parameter. The only parameter that can improve S2SATRA is the number of *Trial*. Increasing the number of trials will increase the number of the final state that corresponds to the *Q<sup>B</sup> <sup>i</sup>* . The main problem with this modification is that increasing the number of trials will lead to an unnecessary high computation time. Hence, in this experiment, the number of *Trial* still follows the conventional settings in [38]. It is worth noting that S2SATRA achieved the lowest accuracy for F1. This is due to imbalanced data, which leads to non-optimal induced logic. Correlation analysis cannot discriminate the correct relationship between variables and *Qlearn <sup>i</sup>* . Generally, S2SATRA improved the pre-processing phase of the logic mining which leads to an improved learning phase due to the correct combination of *Qbest <sup>i</sup>* . The correct combination of *Qbest <sup>i</sup>* will lead to optimal *<sup>Q</sup><sup>B</sup> <sup>i</sup>* which can generalize the dataset.

Finally, we would like to discuss the limitations of the study. The limitation of the S2SATRA is the computation time due to the complexity of the learning phase. Since all logic mining models utilized the same learning model to maximize the fitness of the solution, computation time is not considered as a significant factor. As the number of attribute or clausal noise increases, the learning error will exponentially increase. Hence, metaheuristics and accelerating algorithms, such as in [41], are required to effectively

minimize the cost function in Equation (5) within a shorter computational time. This phenomenon can be shown when the number of neurons *NN* ≥ 20 in the logic mining model is trapped in a trial-and-error state. In terms of satisfiability, all the proposed 2SATRA models do not consider non-satisfiable logical structure or *EQ*<sup>2</sup>*SAT* = 0, such as maximum satisfiability [42] and minimum satisfiability [43]. This is due to the nature of 2SATRA that only consider data point that leads to positive outcome or *Qlearn* = 1. In terms of network, HNN is chosen compared to other ANN structures, such as feedforward because feedback to the input is compatible to the cost function *EQ*<sup>2</sup>*SAT* . Another problem that might arise for feedforward ANN, such as within the radial basis function neural network (RBFNN), is the training choice. For instance, the work of [9,44] can produce a single induced logic due to the RBFNN structure. This will reduce the accuracy of the S2SATRA model. A convolution neural network (CNN) is not favoured because propositional logic only deals with bipolar representation and multiple layers only increase the computational cost for the S2SATRA. In another perspective, weighted satisfiability that randomly assign the negation of the neuron will reduce the generality of the induced logic. In this case, 2SATRA model must add one additional layer during the retrieval phase to obtain which logical weight yields the best accuracy. Unlike several learning environments in HNN-2SAT [45], learning iteration will not be restricted and will be terminated when *fi* = *fNC*. A restricted value of the learning iteration will lead to more induced logic trapped in local minimum energy. As a worst-case scenario, a logic mining model, such as E2SATRA, will not produce any induced logic in restricted learning environment. Hence, all the 2SATRA models exhibit the same learning rule via the Wan Abdullah method [6]. In addition, all the logic mining models, except for 

RA and conventional logic mining, follow the condition of *HPSinduced* <sup>−</sup> *<sup>H</sup>*min *PSinduced* ≤ *∂*. In

*i i* this case, only induced logic that can achieve global minimum energy will be considered during the retrieval phase. This is supported by [33] where the final state of neuron that represents the induced logic will always converge to the nearest minimum. By employing the Wan Abdullah method and HTAF [4], the number of solutions that corresponds to the local minimum solution will reduce dramatically. The neuron combination is limited to only *COMBAX* = 100 because the higher the value of *COMBAX*, the higher the learning error and HNN tends to be trapped in a trial-and-error state.

The experimental results presented above indicate that the S2SATRA improved the classification performance more than other existing logic mining model and created more solution variation. Another interesting phenomenon we discovered is that supervised learning features in S2SATRA reduce the permutation effort in finding the optimal *Qlearn <sup>i</sup>* . As a result, HNN can retrieve the logical rule to do with acquiring higher accuracy. Additionally, we observed that when a number of clausal noise was added, S2SATRA shows a better result compared to the existing model. It is expected that our work can give inspiration to other logic mining models, such as [20,21], to extract the logical rule effectively. The robust architecture of S2SATRA provides a good platform for the application of real-life bioinformatics. For instance, the proposed S2SATRA can extract the best logical rule that classifies single-nucleotide polymorphisms (SNPs) inside known genes associated with Alzheimer's disease. This can lead to large-scale S2SATRA design, which has the ability to classify and predict.
