4.3.1. Base Classifiers Performance Analysis
This section provides a detailed analysis of the classification performance of various drift detection algorithms, including the algorithms proposed in this paper, in terms of both accuracy and time consumption.
Table 2 presents the classification accuracy performance of NB and HT under different drift detectors on synthetic datasets. From
Table 2, it can be observed that the classification models using VFDDMs demonstrate good classification accuracy on artificial datasets. In overall accuracy comparison, VFDDMs achieve an average accuracy ranking among the top three, indicating the excellent classification performance of VFDDMs.
In the comparison among the same base classifiers, VFDDMs also achieved optimal classification performance. In NB, the average rankings of VFDDM_K, VFDDM_H, and VFDDM_M were 2.5, 3.25, and 3.75, respectively. Additionally, the best-performing algorithms besides VFDDMs were MDDM and WMDDM (ranked 4.75 and 5.5). Furthermore, the lowest ranking of the proposed algorithms on NB was 7 (see the performance of VFDDM_M on the Mixed dataset), while the rankings of the other situations for the three algorithms were all in the top 50%. Similarly, in HT, all three variants of VFDDM showed the best average performance, followed by MDDM and MWDDM. Moreover, their accuracy rankings were all in the top 50% among all artificial datasets. This analysis demonstrates that the proposed algorithm exhibits strong adaptability and can provide excellent classification performance for different base classifiers on various datasets.
Furthermore, this paper will conduct a horizontal comparison of drift detection methods using the same statistical test method and demonstrate the advantages of the proposed algorithm. In drift detection methods using the Hoeffding test, FHDDM treats window instances following a binomial distribution without distinguishing the importance of instances within the window. Therefore, its performance is not as good as the MWDDM with a weighted strategy, with a difference in overall ranking of 0.5. On the other hand, the proposed Hoeffding test drift detector VFDDM_H significantly outperforms MWDDM in terms of accuracy. This is because the VFDDM adopts a dynamic weighting strategy, where the importance difference of instances is influenced by the overall variance of the data stream. Thus, its adaptability is relatively stronger compared to MWDDM. Additionally, MWDDM employs a conservative fixed weighting strategy in the warning phase, leading to an increase in the amount of retraining, which in turn affects the accuracy performance of the base classifier.
In the comparison of drift detection methods based on the McDiarmid test, VFDDM_M and MDDM both used the same weighting strategy in the experiment. However, VFDDM_M’s weighting conservatism can be self-adjusted according to the data stream environment, resulting in superior accuracy performance. Additionally, VFDDM_M also outperforms the adaptive adjustment strategy-based WMDDM. This is because WMDDM’s adaptive adjustment strategy involves adjusting decay parameters after the warning phase to accelerate detection speed without tracking the characteristics of the data stream itself.
Additionally, the Kolmogorov test can only handle binomially distributed data instances in drift detection due to the Kolmogorov inequality constraints. Weighting cannot be applied in this part, thus failing to reflect the differences in importance among instances. Thus, only VFDDM_K with semi-stage weighting outperforms the fully weighted MWDDM, MDDM, and WMDDM algorithms, further proving the effectiveness of our proposed strategy.
Figure 7 and
Figure 8 illustrate the accuracy trends of the tested algorithms on NB and HT, respectively. The dashed vertical lines in the graphs represent drift points, with 4 drift points in the abrupt drift dataset and 3 drift points in the gradual drift dataset. The performance variation of base classifiers using different drift detectors at drift positions can be clearly compared from the graphs.
Figure 7 illustrates the accuracy trends of NB using different drift detectors on four synthetic datasets. From
Figure 7a,b, it can be observed that the performance of VFDDM remains consistently high, with a faster response to abrupt drift and minimal performance loss at most drift points, highlighting the advantage of VFDDM in detecting abrupt drifts swiftly. In
Figure 7c,d, the accuracy curve of VFDDM also ranks prominently. Although in the LED dataset, VFDDMs’ performance falls short compared to RDDM and HDDM_A, it outperforms these algorithms in other datasets.
Figure 8 illustrates the accuracy trends of HT using different drift detectors on four synthetic datasets. In Sine (
Figure 8a), it can be observed that VFDDM maintains consistently high classification accuracy, with minimal performance loss at drift points compared to other drift detectors. In LED (
Figure 8c), its performance is similar to that in
Figure 7c. In the Mixed dataset, drift detectors with conservative detection tendencies perform better, such as FHDDM and WMDDM. FHDDM, which does not use a weighting strategy, outperforms MWDDM with a strong weighting strategy. WMDDM, based on the McDiarmid inequality, uses a Sigmoid function to dynamically generate instance weights based on the current classification accuracy, achieving adaptive weighting. This approach results in a higher overall accuracy compared to MDDM. This phenomenon in HT demonstrates its inherent adaptability in handling data stream classification tasks. The stable accuracy performance of most drift detectors in the Mixed dataset further confirms this observation.
Combining observations from
Figure 7 and
Figure 8, HT clearly outperforms NB in overall accuracy, highlighting HT’s advantage in data stream classification tasks. In vertical comparisons, VFDDM_H generally outperforms FHDDM and MWDDM in most cases; similarly, VFDDM_M generally outperforms MDDM and WMDDM in most cases, validating the effectiveness of the proposed VFDDM. This framework can enhance detection performance for original drift detection methods in most scenarios. In summary, VFDDM demonstrates excellent classification accuracy in most cases, with minimal performance loss at most drift points. While it may not achieve optimal performance in certain scenarios, its performance remains consistently high across all scenarios, indicating its significant overall advantage. This underscores our VFDDM’s strong generality and competitiveness.
Table 3 presents the time consumption results of the proposed algorithm compared to other benchmark algorithms. From
Table 3, it can be observed that VFDDMs exhibit good time consumption performance in both NB and HT, comparable to FHDDM, BDDM, HDDM_A, and HDDM_W. Theoretically, for any given moment, FHDDM requires only cumulative operations to obtain the window mean, resulting in a time complexity of
O(1), which is independent of the window size. Therefore, FHDDM consistently exhibits optimal time consumption both theoretically and experimentally. VFDDM_H employs a weighting strategy, which requires a traversal of the window each time it slides, resulting in a time complexity of
O(
n). Consequently, VFDDM_H has higher time consumption compared to FHDDM. VFDDM_M exhibits similar characteristics. In contrast, VFDDM_K, which uses the Kolmogorov test in high-variance data streams, does not rely on a weighting strategy and can also use cumulative calculations to obtain the mean. Thus, VFDDM_K achieves the same theoretical time complexity as FHDDM, which is
O(1), in high-variance data stream environments. This explains why VFDDM_K performs better in terms of time consumption compared to VFDDM_H and VFDDM_M. However, the time consumption differences among the three variant algorithms are relatively low. MWDDM uses a multi-stage weighting approach, resulting in a theoretical time complexity of
O(
n) for each window slide. WMDDM, based on the McDiarmid inequality and instance weighting, also has a theoretical time complexity of
O(
n) per window slide. However, in practical implementation, WMDDM incurs additional computational overhead for calculating the McDiarmid bound and dynamically generating weight factors during each window slide. Therefore, the theoretical time complexity of WMDDM is higher than that of MWDDM. It is notable that the overall time efficiency of VFDDM_H is higher than the MWDDM algorithm, belonging to the same Hoeffding test category, demonstrating VFDDM_H’s good time performance. Furthermore, VFDDM_M exhibits better time consumption performance compared to the WMDDM based on similar tests. WMDDM’s weighting method uses the Sigmoid function, controlled by parameters
λ and
θ, resulting in higher computational overhead than VFDDM_M. This analysis demonstrates that the proposed VFDDM possesses excellent time consumption characteristics.
Finally, we compiled the ranking statistics for all algorithms on accuracy and CPU Times across all synthetic datasets, as shown in
Figure 9. In
Figure 9, the horizontal axis represents the ranking. Under the same horizontal coordinate, a larger area occupied by an algorithm indicates that the algorithm achieved that ranking more frequently in that metric. From the left graph, it can be observed that VFDDM demonstrates outstanding classification accuracy. Among the top ranks, VFDDMs’ classification accuracy occupies a significant proportion, indicating the general applicability of VFDDM across datasets. Additionally, from the right graph, it can be seen that VFDDMs also perform well in terms of time consumption, with a considerable proportion of cases showing high time efficiency rankings. Compared to other methods using the same test, VFDDMs also demonstrate strong competitiveness. This part of the results demonstrates the high efficiency of our variance quick estimation strategy. In summary, we assert that VFDDM possesses outstanding classification accuracy and time efficiency.
Table 4 shows the performance of accuracy and average drift alert count for four real datasets using NB and HT classifiers. Numerous studies indicate that concept drift in real datasets is unknown [
16], which also limits the evaluation metrics for assessing drift detectors on real datasets. Due to the unknown drift locations, metrics such as delay, true positives, false positives, and false negatives, based on drift positions, cannot be determined. Therefore, this study focuses on discussing two metrics: classification accuracy and drift warning count, and analyzes the performance of drift detectors using these two metrics, while exploring the factors affecting classification accuracy on real datasets.
From
Table 4, it can be observed that the DDM and HDDM series algorithms perform well in both NB and HT classifiers. The DDM algorithm performs best in NB-Covertype, HT-Covertype, and HT-Spam, but this performance may not solely be attributed to the drift detector. In fact, in a study by Bifet et al. [
26], experiments were conducted on real datasets Electricity and Covertype, testing a learning strategy of periodically restarting the classification model, which resulted in better classification accuracy compared to drift detectors. Bifet et al. explained this phenomenon as being due to the existence of time dependency between data stream instances, as also mentioned in the literature [
16]. Therefore, based on this conclusion, increasing the number of model restarts is often beneficial for the classification accuracy of data streams with time dependency. This also explains the phenomenon where DDM and HDDM series algorithms simultaneously achieve high levels of accuracy and average warning count on real datasets.
Based on our findings, relying solely on accuracy to evaluate drift detectors on real datasets is unconvincing. Therefore, when assessing the performance of drift detectors on real datasets, it is essential to consider both accuracy and the number of drift warnings. Adopting a controlled variable approach would be more convincing than solely focusing on accuracy. In other words, while DDM and HDDM series algorithms demonstrate excellent accuracy, this does not necessarily imply that they are the optimal drift detectors for real datasets.
In the NB classifier, BDDM and VFDDMs report almost the same number of drift risk instances in the Covertype dataset. However, VFDDMs exhibit approximately a 0.5 advantage in accuracy performance compared to BDDM, indicating that VFDDM is stronger in drift detection capability than BDDM. The performance of both algorithms in HT-Covertype further confirms this point. In the Spam dataset, although VFDDMs and DDM report the same number of drift warnings, VFDDMs achieve significantly higher accuracy than DDM, demonstrating the advantage of VFDDMs over DDM. Similarly, in the Spam dataset, it can be observed that the accuracy performance of VFDDMs is comparable to HDDM_W, while HDDM_W reports a higher number of drift warnings than VFDDMs. As previously analyzed in drift detection performance, HDDM_W has a relatively high false positive rate, which is further evidenced by its performance in NB-Spam, emphasizing the advantage of VFDDMs in having a low false positive rate.
In the HT classifier, the performance contrast of various drift detectors in the Spam dataset is most pronounced. The performance of VFDDMs is superior to MWDDM, with lower average warning counts, indicating higher precision in drift detection compared to MWDDM. Similar observations are noted for HDDM_A and HDDM_W. Although DDM performs best in this dataset, achieving a slight improvement of 0.04 compared to VFDDMs, the average warning counts reveal that DDM has a count of 0.85 while VFDDMs have a count of 0.09, and both have similar accuracy. Therefore, it can be inferred that DDM exhibits a higher false positive rate in its operation, further demonstrating the advantage of VFDDMs.
In conclusion, based on the above analysis, we believe that the VFDDM demonstrates outstanding performance in drift detection on real datasets, with high precision in detecting real concept drifts.
4.3.2. Drift Detection Performance Analysis
This section will provide a detailed analysis of the drift detection performance of various drift detection algorithms, including VFDDMs, focusing on detection delay, true positives, and false positives.
Table 5 presents the performance of VFDDMs and other comparison algorithms on the sudden drift datasets. From this table, it can be observed that regardless of the base classifier or the type of sudden drift dataset, VFDDMs successfully detected all configured drift positions with a true positive rate of 100%. Additionally, the false positive rate and detection delay remained at low levels. MWDDM, HDDM_W, BDDM, and VFDDMs exhibited shorter detection delays for concept drift in Sine and Mixed data streams, followed by MDDM, WMDDM, and FHDDM. It can be observed that drift detectors with shorter detection delays are generally based on instance weighting strategies. Among these, MWDDM applies the highest level of weighting, resulting in the lowest drift detection delay in abrupt drift data streams. However, excessive weighting can also lead to an increase in false positive rates, as shown in the table where MWDDM exhibits a high false positive rate. This indicates that an excessive focus on achieving faster detection speeds inevitably compromises the false positive rate, a point also demonstrated by HDDM_W. Among the drift detectors with faster detection speeds, VFDDMs show relatively better performance in terms of false positive rates. For example, in the Mixed dataset, VFDDMs and BDDM demonstrated significantly lower false positive rates than MWDDM and HDDM_W. BDDM, based on the Bhattacharyya distance, issues a drift alert when the Bhattacharyya distance between the current and historical optimal moment’s instance distributions exceeds a threshold. BDDM also employs an instance weighting strategy to further enhance detection speed. VFDDMs outperformed BDDMs in terms of accuracy, highlighting their superiority.
In the comparison among algorithms based on the same statistical test, VFDDM_H exhibits a larger detection delay compared to MWDDM. This is because MWDDM employs relatively large weighting parameters during the warning phase. In this study, the weighting weight during the warning phase is set to 5, which results in significant differences in the weights between new and old instances. Consequently, when the data stream encounters drift risks, the speed of drift detection increases. However, this comes at the cost of higher false positive rates, especially evident in the Mixed dataset where the false positive rate is 9 times that of VFDDM_H. This indicates that VFDDM_H demonstrates a dual advantage of low false positive rates and low latency in detecting sudden drifts. VFDDM_M and MDDM exhibit nearly identical performance in both true positive and false positive rates. However, VFDDM_M surpasses MDDM in detection delay. This is because MDDM employs fixed weighting parameters, whereas VFDDM_M adjusts its weighting strategy based on variance. In relatively stable data streams during non-drift states, VFDDM_M adopts a more lenient weighting strategy, thus enhancing drift detection speed. In terms of accuracy comparison, VFDDM_M also outperforms MDDM in average rankings on the sudden drift datasets. In summary, it can be concluded that VFDDM_M outperforms MDDM in detecting sudden drifts. The same rationale applies to WMDDM. VFDDM_M exhibits lower latency across all four sudden drift conditions. It only has slightly higher false positive rates than WMDDM on the HT-Mixed dataset.
Additionally, VFDDM_K exhibits the highest detection delay among VFDDMs. This is because the Kolmogorov test cannot weight data instances, thus failing to reflect differences in importance among them. As a result, its performance in terms of detection delay is not as good as VFDDM_H and VFDDM_M. However, its advantage lies in relatively lower false positive rates, and the delay rate is significantly improved compared to FHDDM, which also lacks weighting. Overall, in the context of sudden drift experiments, all three drift detectors demonstrate excellent performance.
Table 6 illustrates the performance of different drift detectors on the gradual drift dataset. In the LED dataset, regardless of whether the base classifier is NB or HT, VFDDMs achieve the optimal detection delay, surpassing the MWDDM algorithm with increased weighting factors. This indicates that the Bernstein test significantly helps to enhance the speed of drift detection in datasets with small variances. Moreover, VFDDMs outperform MWDDM in terms of accuracy. Additionally, VFDDMs exhibit the highest true positive rate, demonstrating the advantages of our variance feedback strategy. It is worth noting that MWDDM also achieves top detection speed in gradual drift datasets. This indicates that increasing the instance weighting strength improves drift detection speed in both abrupt and gradual drift environments, although at the cost of higher false positive rates, as shown by the performance of HDDM_W and MDDM. The performance of BDDM reveals that its advantage in detection speed is less pronounced in gradual drift environments, suggesting that the Bhattacharyya distance is more suitable for detecting concept drift in abrupt drift scenarios.
In the Circle dataset, FHDDM, MDDM, WMDDM, and VFDDMs demonstrate good performance in terms of false positive rates. In the Hoeffding test, VFDDM_H simultaneously achieves low false positive rates and detection delay rates. The McDiarmid test improves the speed of drift detection while maintaining the same false positive rate as MDDM.
Finally, we conducted a statistical analysis of the ranking distribution of various drift detectors based on two drift detection metrics: detection delay and false positive rate, as shown in
Figure 10. From the left graph, it is evident that VFDDMs generally exhibit good drift detection delay performance. In comparison with many advanced drift detection algorithms, VFDDMs rank in the top 50% in most cases. The proportion of first-place rankings for VFDDMs is second only to MWDDM, primarily due to MWDDM’s high weighting coefficient. Moreover, VFDDMs outperform other drift detectors in terms of first-place rankings.
From the right graph, it can be observed that VFDDMs maintain a low false positive rate, indicating a lower false alarm rate and reducing unnecessary reboots, thereby ensuring classifier accuracy. Hence, VFDDMs perform best in terms of classification accuracy, consistent with the findings in the classification accuracy experiment section of this paper. It is noteworthy that the second place is consistently occupied by WMDDM, but with a frequency of only 1, indicating that it achieved the second place only in the NB-Mixed scenario.
In terms of false positive rate ranking analysis, FHDDM has the lowest overall false positive rate. However, in terms of detection delay ranking, FHDDM ranks lower, indicating its conservative detection strategy. While a conservative strategy is important, excessively conservative strategies can lead to less-than-ideal detection efficiency. Conversely, MWDDM has high detection efficiency but suboptimal false positive rate performance. Although its detection delay ranks among the top, it comes at the cost of a higher false positive rate.
In summary, VFDDMs demonstrate excellent performance in balancing detection delay and false positive rate. As shown in
Figure 10, VFDDMs maintain a low false positive rate while ensuring high-ranking detection delay, placing them at the forefront among all compared algorithms. Experimental results confirm that our proposed VFDDMs possess good adaptability and can provide suitable drift detection strategies for different types of data streams, meeting our expectations.
Figure 11 illustrates the comprehensive performance of the drift detectors tested in the experiment across four key metrics: classification accuracy, CPU time consumption, detection delay, and false positive rate, under different base classifier scenarios. The algorithms extending further toward the edges of the graph exhibit superior overall performance. From
Figure 11, it can be observed that VFDDMs consistently rank within the top three in both NB and HT scenarios, demonstrating the excellent overall performance of our VFDDM.