1. Introduction
Machine learning (ML) is increasingly being deployed in large-scale software and safety-critical systems due to recent advancements in deep learning and reinforcement learning. The software applications powered by ML are applied and utilized in our daily lives: from finance and energy to health and transportation. AI-related accidents are already making headlines, from inaccurate facial recognition systems causing false arrests to unexpected racial and gender discrimination by machine learning software [
1,
2]. Also, a recent incident with an Uber autonomous car resulted in the death of a pedestrian [
3]. Thus, ensuring the reliability and conducting rigorous testing of machine learning systems is very vital to prevent any future disasters. One of the critical components of software testing is the test oracle. The mechanism for determining the correctness of the test case output is known as the test oracle [
4]. A program can be considered a non-testable program if there does not exist a test oracle or it is too difficult to determine whether the test output is correct [
5]. Hence, those programs suffer from a problem known as the test oracle problem. ML-based systems exhibit non-deterministic behavior and reason in a probabilistic manner. Also, the output is learned and predicted by a machine learning model as results do not have an expected output to compare against the actual output. As a result, the correctness of the output produced by machine learning-based applications cannot be easily determined and suffers from the test oracle problem [
6].
Metamorphic testing (MT) is a property-based software testing technique that is used to alleviate the test oracle problem. A central component of MT is a set of metamorphic relations (MRs), which are necessary properties of the target function or algorithm in relation to multiple inputs and their expected outputs [
7,
8]. An example of MT is to test the implementation of the function
by asserting the validity of the mathematical relationship
instead of manually verifying the output for an arbitrary floating point input
x. This process involves arbitrarily altering x and checking if the outputs of
x and its variant
are consistent. If the outputs are inconsistent, it indicates the presence of a fault in the implementation of
.
In recent years, especially, MT has attracted a lot of attention and detected many real-life faults in various domains. For example, Murphy et al. [
9] identified six metamorphic relations, and real bugs were identified in three machine learning tools. In a related project, Xie et al. [
10,
11] used metamorphic testing for the detection of faults in K-nearest neighbors and Naive Bayes classifiers. The results revealed that both algorithms violated some of the metamorphic relations. In addition, some real faults were detected in the open-source machine learning tool Weka [
12]. Zhang et al. [
13] applied MT to test DNN-based autonomous driving systems. MT was able to find inconsistent behaviors under different road scenes for real-world autonomous driving models.
Previous work by Srinivasan et al. [
14] has shown that several MRs detect the same type of fault and differ only in fault detection effectiveness. Also, each MR in a machine learning or a deep learning application can have multiple source and follow-up test cases; as a result, the execution time of the MRs increases drastically. The source and follow-up test cases in an MR represented as a training dataset could take several days to train and build a machine learning or deep learning model for a large training dataset. In addition, some metamorphic relations are likely to reveal faults or errors in the system based on the specific characteristics of the system and the data. Prioritizing relations that are more likely to uncover faults can help increase the overall effectiveness of the testing process. Hence, the prioritization of MRs is important for conducting effective metamorphic tests on these programs.
Techniques proposed by Huang et al. [
15], which focus on test adequacy criteria for selecting MRs, do not consider the unique challenges of ML systems, such as their dependence on diverse data inputs. Similarly, while Cao et al. [
16] propose dissimilarity metrics for test cases, their work does not extend to the prioritization of MRs, specifically for ML and deep learning. Approaches such as DeepGini [
17] and MetaTrimmer [
18] prioritize test cases rather than MRs. Nakajima et al.’s [
19] method incorporates dataset diversity but fails to address the diversity between source and follow-up test cases for MR prioritization. These gaps indicate the need for a data-driven MR prioritization approach, specifically tailored to improve fault detection in ML programs, which our work seeks to address.
A previous work by Srinivasan et al. [
20] introduced approaches to prioritize MRs based on code coverage and fault-based approach. The fault-based prioritization approach involves generating a large number of mutants for the application under test and executing the mutants with test inputs to generate mutant kill/alive information for each MR. Then, it prioritize the MRs based on the mutant kill/alive information. This fault-based prioritization process can take up to several weeks and months to complete, and as a result, it becomes costly to prioritize MRs for ML applications. The code coverage approach uses statement or branch coverage information to prioritize MRs. Furthermore, the code coverage-based approach does not work effectively for machine learning-based applications, since the data used for training determine the program logic. The machine learning program is usually just a sequence of library function invocations, and as a result, 100% code coverage can be easily achieved when at least one test input is processed [
21]. To this end, in this work, we propose metrics for
MR prioritization for machine learning programs. The MR prioritization method proposed in this work qualitatively identifies the diversity between the source and follow-up test cases in an MR.
We make the following contributions in this work:
Propose a novel data diversity-based automated method, where we propose five metrics to prioritize metamorphic relations based on diversity between the source and follow-up test cases in an MR.
Evaluate the effectiveness of our proposed MR prioritization methods on two machine learning and one deep learning algorithm implementations. Also, we perform a comparison against three baseline approaches. (1) Random baseline: represents the current practice of executing the MRs in a random order, (2) fault-based ordering: represents the MR ordering based on the fault detection effectiveness of the MRs as proposed by Srinivasan et al. [
20], and (3) neuron activation coverage-based ordering: represents the ordering of the MRs based on the number of unique neuron activations for all test inputs in a deep neural network. The neuron is activated if its output is greater than a threshold value (e.g., 0).
Our results indicate that MR prioritization reduces the number of MRs needed to execute on SUT and reduces the time taken to detect a fault by up to 163% when compared to the random ordering of MRs and up to 14% when compared to neuron coverage-based ordering.
2. Background
2.1. Metamorphic Testing
Metamorphic testing is a software testing technique that uses the idea of metamorphic relations to detect faults in software systems. The key concept behind metamorphic testing is to use a set of input–output relationships to generate new test cases that should produce outputs that satisfy certain properties. The metamorphic testing technique uses metamorphic relations to generate test cases and to verify the correctness of the software. A metamorphic relation is a mathematical or logical relationship that must hold between the inputs and outputs of a software system. If the output does not match the relation, it suggests a possible fault in the program.
Metamorphic testing works by applying a series of transformations to the inputs of a software system and then verifying that the outputs of the system satisfy the metamorphic relations. In metamorphic testing, a source test case (ST) is the initial test case that is used to generate follow-up test cases (FT). The source test case is a test case that represents an input to the software program being tested. The follow-up test cases are generated by applying a metamorphic transformation to the source test case.
For example, consider a program that calculates the sum of integers in a list. The program takes as input a list of integers and returns the sum of all integers in the list. Let L be a list of integers: L = [a1, a2, …, an]. The program takes L as input and returns the sum of the integers in L. We can define a source test case for this program by specifying a particular input list L0 and the expected output sum(L0). We can also define a metamorphic relation for this program that relates the input list L to a new transformed list L’, where the sum of the integers in L and L’ is the same. This relation states that if we reverse the order of the integers in the list L, we should obtain a new list L’ with the same sum as L. We can use this relation to generate follow-up test cases for the program. We can apply the metamorphic relation by reversing the order of the integers in the list to obtain a new list L’. We then expect the sum of L and L’ to be the same. The violation of the metamorphic relation indicates the presence of a fault in the program.
In this work, given an ML or DL system M and a dataset D, let denote the output of the system. Assume that a transformation T is applied to D and generates . Let denote the new output of the system. We refer to the original dataset D and the result as the source test case and the source output, respectively. Similarly, the transformed and the result are referred to as the follow-up test case and the follow-up output, respectively. We consider the entire dataset D and as a test case, and it is used in the ML or DL model training.
2.2. Machine Learning
Machine learning is a subfield of artificial intelligence that involves developing algorithms and models that can learn patterns and make predictions based on test data. In machine learning, the available data are divided into two separate sets: a training dataset and a test dataset. The training dataset is used to train the machine learning model. It typically consists of a large portion of the available data and is used to teach the model the patterns and relationships in the data. The test dataset is used to evaluate the performance of the trained model on unseen data.
There are three main types of machine learning: supervised learning, unsupervised learning, and reinforcement learning. Supervised learning involves training a model on labeled data, where the input data are accompanied by a corresponding output or target variable. Unsupervised learning involves training a model on unlabeled data, where the goal is to find patterns or structure in the data without a specific target variable. In our work, we applied our proposed metrics to supervised classifiers.
In contrast to traditional machine learning (ML) models like kNN, linear regression, and Naive Bayes, testing deep learning (DL) models, including Convolutional Neural Networks (CNNs), requires additional considerations due to the two-stage nature of the development pipeline. These stages are (a) training the CNN model with a training dataset on a DL platform, and (b) performing predictions on new input data using the trained CNN model to produce outputs.
Stage (a): The training process involves adjusting the weights and biases of the network based on the training dataset. In this stage, the relationship between the metamorphic relations (MRs) and the source-level mutation operators is key. The MRs in
Appendix A.3, which affect the training dataset (MRs 1, 2, 7, 8), are applied at this stage. Here, the source-level mutation operators listed in
Table 1 alter the training dataset in various ways to introduce potential faults. These modifications allow us to test how resilient the CNN model is to variations in the training data and ensure that the learned model is robust.
Stage (b): After the training phase, the trained CNN model is deployed to make predictions on new input data. This phase can be considered the “execution” of the model. At this stage, some MRs in
Appendix A.3 (MRs 3, 4, 5, 6, 9, 10) apply transformations to the input data which are related to the prediction behavior of the model. Furthermore, the model-level mutation operators in
Table 1, such as Layer Addition, Layer Removal, and Weight Shuffling, are applied directly to the structure of the trained CNN model. These mutations introduce synthetic faults at the model level, allowing us to test the robustness of the CNN’s architecture during the prediction phase.
In deep learning networks, neurons are fundamental building blocks that mimic the behavior of biological neurons in the human brain. A neuron in a neural network receives input from other neurons or from the input data. Each input is multiplied by the corresponding weight, which represents the strength of the connection. The neuron then calculates the weighted sum of the inputs with an additional bias term to determine the output of the neuron. An activation function is applied to the weighted sum. The activation function takes the weighted sum as input and produces the activation or output of the neuron. The output is between 0 and 1, which indicates the probability of the neuron being activated. In our work, we compare our proposed data diversity-based metrics with the neuron activation coverage-based approach. The neuron activation coverage indicates the number of neurons activated in the network in proportion to the total neuron in the network. By measuring neuron activation-based coverage, testers can assess whether the test inputs are sufficiently exploring the network’s various states and validating its behavior. It helps identify areas of the neural network that may not be involved during the testing and, thus, helps to focus on areas of the network that require further testing or improvement.
2.3. Mutation Testing
Mutation testing is a software testing technique that is used to evaluate the quality of test cases. It involves creating small changes or mutants in the code under test, and then running the test cases against these mutated versions of the code. Mutants are considered synthetic faults introduced into the code to test the quality of the test suite. The goal of mutation testing is to identify weaknesses in the test suite by measuring its ability to detect mutants.
Generally, combining mutation testing and metamorphic testing can help identify the fault detection capability of individual metamorphic relations. In our work, we generated mutants or synthetic faults in the training dataset and the deep learning model. Then, we used the mutants to evaluate the fault detection effectiveness of the prioritized MR ordering generated using our proposed approach.
2.4. Dataset Diversity
In this section, we discuss our approach to dataset diversity in the context of testing machine learning systems. We provide a distinction between “data diversity” and “dataset diversity”, which are key to our method but are not synonymous.
Data diversity, as originally introduced by Ammann and Knight [
22] in the context of software fault tolerance, refers to a technique where variations in input data are leveraged to improve a system’s fault tolerance. By using different variations of inputs, systems can be tested for faults that might not otherwise be detected through uniform input data. In our work, we extend this concept to metamorphic testing by prioritizing metamorphic relations (MRs) based on the diversity between source and follow-up test cases. The aim is to enhance fault detection by ensuring that MRs with higher a diversity in test cases are executed earlier, reducing the time to uncover faults.
Dataset diversity, on the other hand, is a concept more specific to the field of machine learning [
19]. It pertains to the variations within the training and testing datasets, including the distribution of attributes, labels, and instances. In machine learning systems, dataset diversity ensures that the model is exposed to a wide variety of data during testing, increasing the likelihood of revealing faults in the learning algorithm or model behavior.
Both data diversity and dataset diversity are integral to our method. While data diversity focuses on input variations to detect system faults, dataset diversity is centered on the variability within the dataset to uncover issues in machine learning models. By incorporating both aspects, our proposed approach aims to improve the effectiveness of early fault detection in machine learning systems.
6. Result and Analysis
In this section, we discuss our experimental results and provide answers to the three research questions that we list in
Section 5. For each subject program, we performed the validation procedure described in
Section 5.1 using the setup described in
Table 3. In this setup, we used the generated mutant set for evaluating the prioritized MR ordering as
and the source test cases as
. For example, in
Table 3, for IBk, Ecoli refers to the dataset we describe in
Section 4.3. Similarly, for CNN, MNIST and Fashion datasets refer to the 60,000 images used for training and testing. YearPrediction refers to the training dataset used for the linear regression program, respectively.
Figure 4 shows the average fault detection effectiveness for the evaluation runs described above vs. the MR set size used for testing each subject program. We also plot the percentage of faults detected by the random baseline for comparison. The results for each research question are provided below.
6.1. RQ1: Comparison of MR Prioritization Approaches versus Random Baseline
6.1.1. Fault Detection Effectiveness
We formulated the following statistical hypothesis to answer RQ1 in the context of fault detection effectiveness:
Hypothesis 1. For a given MR set of size m, the fault detection effectiveness of the MR set produced by the data diversity-based approach is higher than that of the random baseline.
The null hypothesis for each of the above-defined hypotheses is that the data diversity-based approaches perform equal to or worse than the random baseline.
In
Table 4a–e, we list the relative improvement in average fault detection effectiveness for the data diversity-based method over the currently used random approach of prioritizing MRs for linear regression, Naive Bayes, IBk, and CNN, respectively. To evaluate the above hypotheses, we use the paired permutation test, as it does not make any assumptions about the underlying distribution of the data [
37]. We apply the paired permutation test to each MR set size for each of the subject programs with
= 0.05, and the relative improvements that are significant are marked with a *.
As shown in
Table 4a, for IBk, the rule-based and data distribution metric shows improvements in average fault detection effectiveness over the random approach for all MR set sizes except for the last MR set size. The last MR set size does not show any improvements, as the fault detection effectiveness reaches a saturation point by the last metamorphic relation set size. In particular, among the different MR set sizes, the increase in the percentage of fault detection varies from 0% to 129%. Therefore, we reject the null hypotheses
for IBk for rule-based, anomaly, clustering-based, and data distribution-based metrics.
Similarly for linear regression,
Table 4b indicates that all metrics show significant consistent improvements in fault detection over the random baseline. The fault detection varies from 0% to 114% for the metrics. Therefore, we can reject the null hypothesis in general for linear regression.
Table 4c shows the relative improvement in the average percentage of fault detection for Naive Bayes. All metrics show improvements for MR set sizes
to
. Therefore, we can reject the null hypothesis in general for CNN only for MR set sizes
to
.
Table 4d,e show the relative improvements in the average percentage of fault detection for CNN. All metrics show improvements for all MR set sizes except MR set size
and
. Furthermore, the relative improvement between the MR prioritization methods varies between 0% and 457% for the MNIST dataset and between 0% and 108% for the Fashion dataset. Therefore, we can reject the null hypothesis in general for CNN. So, overall, the data diversity-based approach outperformed the random approach for all our subject programs.
6.1.2. Effective Number of MRs Used for Testing
In
Figure 4, the effective number of MRs using fault detection thresholds for various subject programs is illustrated as mentioned in
Section 5.2. The vertical lines indicate the size of the effective MR set for different approaches: brown for clustering-based, green for data distribution-based, violet for fault-based, and orange for the random baseline approach. The associated thresholds are annotated near each vertical line for clarity.
In the linear regression context, the clustering, data distribution, anomaly, and rule-based metrics yield an effective MR set size of 2 at the 2.5% fault detection threshold. In contrast, the random approach results in an MR set size of 5 as shown in an orange vertical line for the same threshold of 2.5%. This represents an 85% reduction in the size of the MR set when using the data diversity-based metrics. The significant reduction indicates that the data diversity metrics can achieve similar, or even superior, fault detection performance with a smaller set of MRs when compared to the random approach, thereby improving the testing efficiency.
For the IBk classifier, clustering and data distribution-based metrics produce an effective MR set size of 3 and 2, respectively, at the 2.5% threshold. Meanwhile, the random approach results in a much larger MR set size of 8. Additionally, the anomaly and rule-based approaches yield an MR set size of 3 for the 2.5% threshold. For the Naive Bayes classifier, both clustering-based and random-based metrics generate an MR set size of 1 for the 2.5% threshold, while the random approach generates an MR set size of 3. These observations underscore the effectiveness of clustering-based methods in reducing the testing effort while maintaining a high standard of fault detection.
Similarly, for CNNs trained on the MNIST dataset, the clustering-based, data distribution-based, anomaly-based, and feature-based metrics all produce an effective MR set size of 2 at the 5.0% fault detection threshold, while the random approach results in a substantially larger MR set size of 10. For CNNs trained on the Fashion–MNIST dataset, both feature-based and clustering-based metrics produce an effective MR set size of 3 at the 5.0% threshold, compared to an MR set size of 6 for the random approach.
These results demonstrate the efficiency of clustering and data-driven approaches in reducing the size of the MR set, thereby minimizing computational resources while maintaining a high rate of fault detection. This efficiency accelerates the validation process, which is crucial in iterative development cycles, particularly in machine learning projects where resource conservation and rapid validation are critical for success.
6.1.3. Average Percentage of Faults Detected
Table 5 shows the APFD for the SUT. We can observe that for all our subject programs, our proposed metrics provide a higher APFD value between 0.35 and 0.98 when compared to random-based prioritization. This leads to detecting the most important faults earlier in the testing process compared to the random approach.
6.1.4. The Average Time Taken to Detect a Fault
Table 6 shows the average time taken to detect a fault in linear regression, Naive Bayes, IBk, and CNN, respectively. The first column of the table represents the average time taken to detect a fault using our proposed metrics. As shown in these results, the use of our proposed approach resulted in significant reductions in the average time taken to detect a fault by 48% compared to the random baseline for IBk. Similarly, for linear regression, Naive Bayes, and CNN using the MNIST dataset, our proposed metrics provided 62%, 48%, and 43% reduction in time taken to detect a fault compared to random approach. Similarly, for CNN using the Fashion dataset, our proposed metrics provided a 54% reduction in the time taken to detect a fault compared to the random approach.
Table 7 illustrates the time taken to prioritize MRs using different approaches for various subject systems. Notably, the rule-based approach results in significantly longer processing times, as seen with linear regression, requiring up to 72,000 s. In contrast, anomaly-based, clustering-based, and data distribution-based approaches demonstrate considerable efficiency across all subjects, often completing tasks in a fraction of the time required by rule-based and fault-based methods. For instance, in the Naive Bayes subject, the anomaly-based approach took only 72 s, while the fault-based method took 15,000 s, indicating a substantial reduction in time.
Table 8 shows the time taken to generate the metrics.
Specifically, for the CNN models using the MNIST and Fashion datasets, the clustering-based, data distribution-based, and feature-based approaches required notably more time compared to simpler subjects, but they were still far more efficient than the fault-based approach, which took an extreme 242,000 s for MNIST and 900,000 s for the Fashion dataset. These observations confirm that our proposed approaches not only facilitate a faster prioritization of MRs but also offer a significant reduction in the time and cost associated with the MR prioritization process.
When applying the diversity metrics and prioritization techniques, it is important to note that while faster methods like anomaly and clustering-based approaches drastically reduce the time for MR prioritization, these methods still incur overhead depending on the complexity of the system and the dataset size. In CNN models, for example, the prioritization process, even when using more efficient methods, requires more time than simpler systems like Naive Bayes. This is likely due to the higher computational demands of CNNs in calculating feature diversity, distribution analysis, and anomaly detection across large and complex datasets. Thus, the overhead can vary considerably based on the method chosen and the complexity of the subject system. The proposed methods do reduce time and cost in the prioritization process, but this efficiency comes with its own computational expense, especially in models with larger feature sets or more intricate architectures like deep learning systems.
![Electronics 13 03380 i001]()
6.2. RQ2: Comparison of the Proposed MR Prioritization Approach against the Fault-Based Approach
In this research question, we evaluate whether the data diversity-based approach outperforms the fault-based approach. To answer the research question, we formulate the following hypotheses.
Hypothesis 2. For a given MR set of size m, the fault detection effectiveness of the MR set produced using a data diversity-based approach is higher than the fault detection effectiveness of the MR set produced by the fault-based prioritization.
The null hypothesis for each of the above-defined hypotheses is that the MR sets generated by the proposed method perform equal to or worse than the MR sets generated by fault-based prioritization in terms of fault detection effectiveness.
6.2.1. Fault Detection Effectiveness
We provide the answer to RQ2 in the context of the effectiveness of fault detection. In this research question, we evaluate whether the data diversity-based approach outperforms the fault-based approach.
Table 9a shows the relative improvement in the fault detection percentage between the MR sets generated by the data diversity-based approach over the fault-based approach for IBk. We observed that the data distribution metric performs equal to the fault-based approach for all the MR set sizes. Also, rule-based and clustering-based perform equally to the fault-based approach except for MR set size
. Therefore, we cannot reject the null hypothesis
for the data distribution metric.
Table 9b shows the relative improvement in the fault detection percentage between the MR sets generated by the data diversity-based approach over the fault-based approach for linear regression. The results indicate that the anomaly-based and data distribution-based metric outperform the other two metrics. Specifically for an MR set size
, both the anomaly-based and data distribution-based metrics show a relative improvement in fault detection effectiveness of 29%, while the clustering-based and rule-based show either no improvement or a decrease in effectiveness. For MR set sizes
to
, the anomaly-based, data distribution-based, and clustering-based metrics show an improvement in fault detection effectiveness, while the rule-based approaches show no improvements. For MR set sizes
to
, all approaches show no improvements in effectiveness compared to the fault-based approach. The rule-based metric performs equally as the fault-based approach except for MR set size
. Therefore, we cannot reject the null hypotheses
for the data distribution-based and anomaly-based metrics.
Table 9c shows the relative improvement in the fault detection percentage between the MR sets generated by the data diversity-based approach over the fault-based approach for Naive Bayes. The results indicate that our proposed metrics perform equally to fault-based for all MR set sizes. Therefore, we cannot reject the null hypotheses
.
Table 9d shows the relative improvement in the fault detection percentage between the MR sets generated by the data diversity-based approach over the fault-based approach for CNN using the MNIST dataset. We observe that the clustering-based, data distribution-based, and feature-based approaches performed equal to fault detection for all the MR set sizes, and the anomaly-based approach performed equally to fault detection for only MR set sizes
to
.
Table 9e shows the relative improvement in the fault detection percentage between the MR sets generated by the data diversity-based approach over the fault-based approach for CNN using the fashion dataset. We observe that the feature-based metric provided greater fault detection effectiveness when compared to the fault-based for MR set sizes
to
. However, the fault-based approach provided marginally greater or equal fault detection effectiveness when compared to anomaly, clustering, and data distribution-based metrics for all the MR set sizes. Therefore, we can reject the null hypotheses
only for the feature-based metric.
6.2.2. Effective Number of MRs Used for Testing
In
Figure 4, for IBK, we observe that the data distribution-based and anomaly-based metrics provide an effective MR set size of 2 for the 2.5% threshold, and the fault-based approach also provides an effective MR set size of 2 for the 2.5% threshold. For Naive Bayes, the rule-based and fault-based metrics provide an effective MR set size of 1 for the 2.5% threshold. Similarly, for CNN using MNIST and fashion datasets, all our metrics provide an effective MR set size of 2 for the 5.0% threshold, and the fault-based approach also provides an effective MR set size of 2 for the 5.0% threshold. In linear regression, all our proposed metrics provide an effective MR set size of 2 for the 2.5% threshold, and the fault-based approach also provides an effective MR set size of 2 for the 2.5% threshold. The results indicate that the effective number of MRs required for testing SUT using our proposed metrics did not show improvements over the fault-based approach.
6.2.3. The Average Time Taken to Detect a Fault
Table 6 shows the average time taken to detect a fault in linear regression, IBk, Naive Bayes, and CNN, respectively. As shown in these results, using our proposed metric resulted in reductions in the average time taken to detect a fault by 13.04% when compared to the fault-based approach for linear regression. This reduction in average time to detect a fault using our metrics is due to the higher fault detection effectiveness of the data distribution-, anomaly-, and clustering-based metrics when compared to the fault-based approach for certain MR set sizes. For IBk, Naive Bayes, and CNN using the MNIST and Fashion datasets, the data diversity-based approach did not reduce the time taken to detect a fault since the fault detection effectiveness of our proposed metrics was less than the fault-based approach for CNN and IBK for certain MR set sizes.
6.2.4. Average Percentage of Faults Detected
Table 5 shows the APFD for the SUT. We can observe that for all our subject programs, APFD for our proposed metrics vary from 0.21 to 0.98 when compared to the fault-based for all the subject programs. We found that the fault-based approach provides a higher APFD when compared to the anomaly-based, data distribution-based, and clustering-based approaches for IBk. Similarly, for Naive Bayes, only the clustering-based metric provided equal APFD when compared to the fault-based. This occurred due to the clustering-based metric’s capability to identify pivotal groupings or patterns in the data that align well with Naive Bayes’ probabilistic decision boundaries, thus effectively uncovering faults. For linear regression, the fault-based approach provides higher APFD when compared to data distribution and rule-based approach. Finally, for CNN using the MNIST dataset, our proposed metric provides APFD equal to the fault-based approach since the MNIST dataset consists of handwritten digit images, which have relatively simple and well-defined features. For CNN using the Fashion dataset, the fault-based approach provides a higher APFD when compared to our proposed metrics since the Fashion dataset provides inherent complexity and variability compared to the MNIST dataset.
![Electronics 13 03380 i002]()
6.3. RQ3: Comparison of the Proposed Metric with the Approach Based on Neuron Activation Cover
In this research question, we evaluate whether the data diversity-based ranking approach outperforms the neuron activation coverage-based approach. To answer the research question, we formulate the following hypothesis.
Hypothesis 3. For a given MR set of size m, the fault detection effectiveness of the MR set produced using data diversity-based approach is higher than the fault detection effectiveness of the MR set produced by the neuron activation coverage-based approach.
The null hypothesis for each of the above-defined hypotheses is that the MR sets generated by the data diversity-based method perform equally to or worse than the MR sets generated by the neuron activation coverage-based approach in terms of fault detection effectiveness.
6.3.1. Fault Detection Effectiveness
We provide the answer to RQ3 in the context of fault detection effectiveness. In this research question, we evaluate whether the data diversity-based approach outperforms the neuron activation coverage-based approach.
Table 10 and
Table 11 show the relative improvements in the percentage of fault detection between MR sets generated by the data diversity-based approach over the neuron activation coverage-based approach for CNN using the MNIST and the Fashion datasets. We observed that our proposed metrics outperformed the neuron activation coverage-based approach for all the MR set sizes. This lack of performance by the neuron activation coverage-based approach is due to the lack of variation in neuron activation in the network between the source and follow-up test cases of the MR for CNN. For CNN using the MNIST dataset, with the exception of MR6, all MRs showed the same neuron activation coverage, which resulted in MR6 being prioritized over other MRs, which were selected and ordered randomly. This randomness in the prioritization of MRs resulted in reduced fault detection effectiveness of the neuron activation coverage-based approach. However, we observed variations in neuron activation in the network between the source and follow-up test cases of MRs for CNN using the Fashion dataset. This higher complexity and diversity in the Fashion dataset led to more varied activation patterns across different neurons in the network. Therefore, we can reject the null hypothesis
for all our metrics.
6.3.2. Average Percentage of Faults Detected
Table 5 shows the APFD for the subject programs. We can observe that for CNN using the MNIST and Fashion datasets, our proposed metrics provided a higher APFD between 0.35 and 0.88 when compared to the neuron activation coverage-based approach. The superiority of the proposed metrics over the neuron activation coverage in these cases suggests that simply achieving high activation coverage does not necessarily equate to effective fault detection.
![Electronics 13 03380 i003]()
7. Discussion
Our research introduced a novel method for prioritizing metamorphic relations (MRs) in software testing, particularly for machine learning and deep learning systems, utilizing metrics based on data diversity. The proposed approach not only improved fault detection effectiveness by 5% to 550% compared to neuron activation coverage-based prioritization but also significantly reduced the time taken to detect faults by up to 62% compared to a random execution of MRs. Also, our proposed metric performed closer to fault-based approach.
The results suggest that data diversity is a critical factor in the effectiveness of MRs, aligning with the theory that diversity in test cases can expose more faults. This finding supports previous studies that highlight the importance of test case diversity, but extends it by quantifying the impact in the context of metamorphic testing. A major strength of our study is the introduction of quantifiable and practical metrics for prioritizing MRs, which can be readily applied in real-world settings. However, the study is not without limitations. The effectiveness of the proposed metrics can vary depending on the specific characteristics of the data and the ML algorithms used. Additionally, the generalizability of the results to other forms of white-box software testing remains to be explored.
The practical implications of our findings are profound, particularly for industries and sectors where rapid deployment of machine learning models is critical. Our methods can significantly reduce the time and resources spent on testing without compromising quality, thereby accelerating the pace of software development. Theoretically, this research advances our understanding of MR prioritization in the context of machine learning, offering a new perspective on how data diversity impacts the effectiveness of tests.
When selecting a metric for a subject program in practice, the choice of metric should be guided by the type of data used in the subject program. Rule-based and clustering-based metrics are suitable for structured or numerical data, while data distribution-based, clustering-based, anomaly-based, and feature-based metrics are appropriate for image or unstructured data. Domain expertise and understanding of the subject program play a crucial role in determining which metric might be most effective in detecting faults. Additionally, all our subjects show some variation in the performance of different metrics across the subject programs; it is important to note that the differences are relatively small in most cases. For example, in the CNN—Fashion dataset, the anomaly-based metric achieves the highest performance at 0.93, but the data distribution and clustering-based metrics are not far behind at 0.88 and 0.87, respectively. Similarly, for the linear regression subject program, the clustering-based and fault-based metrics both perform well at 0.97.
Prior work by Srinivasan et al. [
20] proposed a fault-based and code coverage-based approach to prioritize MRs in the context of regression testing. The fault-based approach applies a greedy algorithm on the faults detected by the MRs on the previous version of the SUT to prioritize the MRs for testing the next version of the SUT. The code coverage-based approach uses statements and branches covered by MRs to prioritize MRs. The proposed approach was applied to three diverse applications, and the result indicates that the code coverage-based approach did not work effectively for machine learning-based programs. Huang et al. [
15] proposed metamorphic relations prioritization, and the selection based on test adequacy criteria aims to improve the efficiency and effectiveness of metamorphic testing by selecting and prioritizing metamorphic relations (MRs) based on their potential to achieve test adequacy. The approach likely involves defining a set of test adequacy criteria, such as statement coverage or branch coverage, and then calculating a metric for each MR based on its contribution to meeting these criteria. MRs are prioritized and selected based on these metrics, with the goal of maximizing test adequacy while minimizing test execution time. Experiments would compare the fault detection effectiveness of MR sets selected using this approach to those selected randomly or using other criteria. Sun et al. [
38] provided path-directed source test case generation, and prioritization in metamorphic testing focuses on improving the efficiency and effectiveness of metamorphic testing by leveraging path information. This approach generates source test cases by considering the program’s control flow, aiming to increase code coverage. The generated test cases are then prioritized based on path distance, with the goal of maximizing the exploration of different program paths. By targeting specific program paths, this method aims to enhance fault detection capabilities. Experiments compared the fault detection effectiveness of path-directed test case generation with random or other test case generation techniques. However, the techniques proposed does not involve prioritizing metamorphic relations for machine learning programs based on data diversity.
Cao et al. [
16] proposed six metrics to measure the dissimilarity between the source and follow-up test cases. The authors then evaluated the effectiveness of their approach by applying it to four different metamorphic relations and comparing the results to those obtained using traditional dissimilarity metrics. The results showed that their approach is effective in identifying cases where the traditional metrics fail and that it can improve the efficiency of metamorphic testing by reducing the number of redundant test cases that need to be executed. The work does not propose metrics to prioritize metamorphic relations for machine learning and deep learning programs.
Mayer and Guderlei [
1] conducted an empirical study to assess the usefulness of MRs. Based on the result of the experiment, the authors devised several criteria to assess the quality of the MRs on the basis of potential usefulness. These criteria were defined based on the different execution paths undertaken by the source and follow-up test cases. Although this work provides some directions for selecting useful MRs, it does not directly addresses MR prioritization. In particular, we focus on developing automated methods for MR prioritization.
Hui et al. [
39] provided qualitative guidelines for MR prioritization. The guidelines could be used to select effective MR and improve MT performance. The author also provided three measurable metrics for MR selection: degree of MR, algebraic complexity of MR, and distance between test inputs of MR. The empirical result showed inconsistency between the correlation value and the metrics, indicating that other factors could influence the effectiveness of MR. Nakajima et al. [
19] focused on the use of dataset diversity, that is, the use of multiple datasets that are diverse in terms of their origin, characteristics, and distribution, to improve the effectiveness of metamorphic testing for ML software. The proposed method takes into account the dataset dependency of training results and provides a new way to generate follow-up test input. The approach was evaluated in a case of software testing of neural network programs to classify handwritten numbers. The results showed that the proposed method was able to find defects in the machine learning software that were not found by traditional metamorphic testing methods. In contrast, our work prioritizes metamorphic relations based on the diversity between the source and follow-up test cases in an MR. Sun et al. [
38] proposed a technique to generate good source test cases in metamorphic testing based on constraint solvers and symbolic execution techniques. In addition, a prioritization of source test cases was conducted on the basis of path distances among test cases. This work does not focus on the prioritization of metamorphic relations. In addition, the work does not perform a prioritization of metamorphic relations for machine learning and deep learning programs.
Feng et al. [
17] proposed a new method to identify and prioritize test cases that are likely to be most effective in improving the robustness of a deep neural network. The authors introduced a metric called DeepGini, which measures the importance of each test case in terms of its ability to activate a large number of neurons in the network. They demonstrated that using DeepGini to select a subset of test cases for training leads to improved robustness and accuracy of the network, as well as faster training times. The experimental result indicates that DeepGini outperforms existing coverage-based techniques in prioritizing tests with respect to both effectiveness and efficiency. However, the proposed technique does not focus on prioritizing metamorphic relations.
Torres et al. [
18] introduced MetaTrimmer, which generates random test data, identifies violated metamorphic relations, and derives constraints to improve their precision. Experiments on real-world programs demonstrated that MetaTrimmer effectively selects informative metamorphic relations, leading to increased fault detection rates compared to traditional random testing. By automating the selection and refinement process, MetaTrimmer significantly enhances the practicality and effectiveness of metamorphic testing. Liu et al. [
40] proposed similarity-based Metamorphic Relations Selection Strategy for Numerical Computation Programs, which focuses on improving metamorphic testing for numerical computation programs. Unlike random or data-driven approaches, this method leverages the concept of similarity between input relations of different metamorphic relations. By calculating similarities, it selects a set of metamorphic relations that are diverse and cover a wide range of program behaviors. This strategy aims to enhance test effectiveness by ensuring that the selected MRs are complementary rather than redundant. However, the proposed work focuses on MR selection based on distance metric for numerical programs in comparison to our proposed work on MR prioritization for ML programs.
Xie et al. [
41] proposed MUT Model: a metric for characterizing metamorphic relations diversity introduces a quantitative approach to measure the diversity of a set of metamorphic relations (MRs). The MUT model defines a mathematical framework to calculate diversity based on factors such as the types of operations involved in the MRs, the complexity of the input transformations, and the expected output relationships. By quantifying MR diversity, the MUT model aims to provide a foundation for selecting diverse sets of MRs, which can improve the effectiveness of metamorphic testing by increasing the likelihood of uncovering different types of faults.
8. Threats to Validity
External Validity: Our proposed metric is applied to supervised ML classifiers. Metrics such as rule-based are not suitable for unsupervised classifiers. We plan to design more metrics to cover unsupervised classifiers and non-numerical data. In this work, all experimental subjects are implementation of machine learning and the deep learning algorithm used in the Weka ML library. Although they are popular algorithms and widely used, our findings may not be generalizable to commercial machine learning projects such as autonomous vehicles. To mitigate this threat, we plan to explore the effectiveness of our proposed approach in AI systems in the health care domain and Apollo autonomous driving platform projects in the future.
Internal Validity: We used a dataset having an average of 500,000 instances and 90 attributes on our subject programs. However, the size of the dataset could be small for generating the proposed metrics and prioritizing the MRs. We used less than 300 mutants to validate our prioritized MR ordering for the CNN program. However, it is possible that the number of mutants used is low. However, despite the low number of mutants, those are generated using a set of mutation operators that are specifically designed for machine learning and deep learning and have a higher likelihood of detecting fault.
We used 10–11 MRs to test our subject programs. However, six to nine MRs are generally used for testing [
42]. In addition, the time and resources consumed in MT do not only depend on the number of MRs. Although we used MRs with single-source/follow-up test cases in this experiment, depending on the SUT, there can be MRs with multiple source and follow-up test cases. Thus, as the number of MRs increases, the number of test cases can increase exponentially. Additionally, prioritizing MRs can help you identify redundant or less important MRs that may not need to be tested. This can further reduce the testing effort and allow you to focus on the most important MRs.