*7.3. Error Analysis*

Tables 9 and 10 demonstrate the error evaluation for all the logic mining models. The S2SATRA model outperforms all logic mining models in terms of *RMSE* and *MAE*. Note that the improvement ratio is considered by taking into account the differences between the error value divided with the error produced by logic mining.


**Table 6.** Correlation analysis (*ρ*) for 8 sampled attributes for F1–F6.

**Table 7.** Correlation analysis (*ρ*) for 8 sampled attributes for F7–F12.


**Table 8.** Improved RA [14].


**Table 9.** *RMSE* for all logic mining models. The bracket indicates the ratio of improvement and \* indicates division by zero. A negative ratio implies the method outperform the proposed method. *P* is obtained from the paired Wilcoxon rank test and \*\* indicates the model with significant inferiority compared to the superiority model.


**Table 10.** *MAE* for all logic mining models. The bracket indicates the ratio of improvement and \* indicates division by zero. A negative ratio implies the method outperform the proposed method. *P* is obtained from the paired Wilcoxon rank test and \*\* indicates the model with significant inferiority compared to the superiority model.


A high value of *RMSE* demonstrates the high deviation of the error compared with the *Q ki* <sup>2</sup>*SAT*. S2SATRA ranks first on 12 datasets. The "+", "−", and "=" in the results column indicate that S2SATRA is superior, inferior, and equal to the comparison algorithm, respectively. The "Avg" indicates the corresponding algorithm's average of the Friedman test for 12 datasets. The rank represents the ranking of the "Avg Rank". Although the value S2SATRA is the lowest compared to other logic mining model, the *RMSE* value is

high, which shows that the error is deviated from the mean of the error for the whole *Q ki* <sup>2</sup>*SAT*. According to Tables 9 and 10, there are several winning points for S2SATRA, which are as follows.


P2SATRA is observed to achieve a competitive result where the 5 out of 12 datasets have the same error during the retrieval phase. This indicates that the conventional 2SATRA model can be further improved with a permutation operator. Despite the high permutation value (up to 1000 permutation/run) implemented in each dataset, most of the attributes in the P2SATRA are insignificant with respect to the final output. Hence, the accumulated testing error will be higher than the proposed S2SATRA. It is also worth noting that implementation of the permutation operator from P2SATRA benefits S2SATRA in terms of search space. In another perspective, an energy-based approach, E2SATRA, is able to obtain *Q ki* <sup>2</sup>*SAT* which can achieve the global minima energy but tends to get trapped in suboptimal solution. According to Tables 9 and 10, E2SATRA showed improvement in terms of error compared to the conventional 2SATRA but the induced logic only explores a limited search space. For example, the high accumulation error in F2–F8 were due to small number of *Q ki* <sup>2</sup>*SAT* produced by E2SATRA. The only advantage for E2SATRA compared to RA is the stability of the *Q ki* <sup>2</sup>*SAT* in finding the correction dataset generalisation. E2SATRA is reported to be slightly worse compared to P2SATRA, except for F8 and F10 where the error difference is 86.3% and 47.2%, respectively. Conventional 2SATRA and RA were reported to produce *Q ki* <sup>2</sup>*SAT* with the worst quality due to the wrong choice of attribute selection. Another interesting insight is that the modified RA from [14] tends to overlearn, which results in an accumulation of error. For instance, RA accumulates a large *RMSE* value in F1, F6, and F7, due to the rigid structure of *Q ki* <sup>2</sup>*SAT* during the learning phase and the testing phase of RA. Additionally, the rigid structure for *Q ki* <sup>2</sup>*SAT* in RA does not contribute to effective attribute representation. Overall, it can be seen that, compared with each comparison algorithm, S2SATRA has the greatest advantages on more than 10 datasets in terms of *RMSE* and *MAE*.

*7.4. Accuracy, Precision, Sensitivity, F1-Score, and MCC*

Figures 14 and 15 demonstrate the result for *F-score* and *Acc* for all the logic mining models. There are several winning points for S2SATRA according to both figures, which are as follows.


Tables 11 and 12 demonstrate the result for *Pr* and *Se* for all the 2SATRA models. According to Table 7 (A), there are several winning points for S2SATRA, which are as follows.


observation is reported in *Se* where the only datasets that achieved *Se* < 0.8 were F8 and F11. Hence, S2SATRA has a competitive capability to produce a positive result *Qtest* = 1 compared to other existing 2SATRA model.


**Figure 14.** *F-score* for all logic mining models.

**Figure 15.** Accuracy for all the logic mining models.

**Table 11.** Precision (*Pr*) for all models. The bracket indicates the ratio of improvement and \* indicates division by zero. A negative ratio implies the method outperform the proposed method. *P* is obtained from the paired Wilcoxon rank test and \*\* indicates the model with significant inferiority compared to the superiority model.



**Table 12.** Sensitivity (*Se*) for all logic mining models. The bracket indicates the ratio of improvement and \* indicates division by zero. A negative ratio implies the method that outperforms the proposed method. \*\* due to no positive outcome in the dataset. *P* is obtained from the paired Wilcoxon rank test and \*\* indicates a model with significant inferiority compared to the superiority model.

Table 13 demonstrates MCC analysis for all logic mining models. According to Table 13, several winning points for S2SATRA are as follows.



**Table 13.** *MCC* for all logic mining models. *P* is obtained from the paired Wilcoxon rank test and \*\* indicates the models with significant inferiority compared to the superiority model.
