1. Introduction
With the development of computer technology and the increase in custom requirements, software systems are becoming more powerful and complex. In fact, the emergence of unexpected faults in such systems is inevitable. Once the system encounters a certain fault, it is likely to fail. Failure means that the system operates with unexpected behaviors. Testing is a very necessary and significant means of system quality assurance during the product development life cycle [
1]. A report released by NIST (National Institute of Standards and Technology) in 2002 stated that software system bugs cost the U.S. economy 59.5 billion dollars annually [
2]. However, more than one third of this cost could be saved if better testing is performed [
2]. A contributing test method that can exactly find more faults with fewer test cases is urgently needed.
Combinatorial testing (CT) [
3,
4,
5,
6] is an efficient testing method through which an optimal or near optimal test suite with fewer test cases can be designed or generated. CT has proven to be an effective technique for detecting faults caused by interactions among configurations or factors in a given input space [
7]. The empirical studies of system bugs suggest that CT is equivalent to exhaustive testing in a certain sense [
8,
9]. Although CT has been widely studied and used, there are still some situations and requirements that combinatorial testing does not apply to well, such as a system under test (SUT) whose test cases need to be performed contiguously. For thorough testing, the testing requirements of this SUT are not only to cover all the interactions among factors, but also to cover all the value sequences of every factor. When CT is used to design a test suite for this SUT, only the interactions among factor values can be effectively covered to a certain level according to the
t-way combinatorial criterion; there is no effective criterion for the value sequences of every factor [
10]. For example, the “text effect” application of “Microsoft Word” has seven options for users to modify some highlighted text. These options are “subscript”, “superscript”, “strikethrough”, “double strikethrough”, “all caps”, “small caps”, and “shadow” [
9]. The font-processing function within the application correctly modifies the highlighted text on the screen according to the settings consisting of these options. When using CT, a test suite with several test cases can be generated to cover interactions among every
t options. When the test suite is executing, the font-processing function modifies the text according to the test cases in sequence. However, the sequence of settings in every factor from contiguous test cases cannot be guaranteed to be tested at a certain level, such as a sequence from text with subscript to text without subscript for the “subscript” option with binary settings.
To solve the mentioned sequence coverage requirement, a
t-way sequence coverage criterion for the requirement was proposed [
11] based on the
t-way combinatorial coverage criterion. The
t-way sequence coverage criterion was first introduced to apply to event sequence testing. In terms of an SUT with
n input events, each event can only be input once during a test. Each test case in the test suite covers
subsequences with length
t , and the covered
t events in a subsequence do not have to be neighboring. A
t-way sequence coverage test suite can cover all
subsequences that have
t different events. The size of a
t-way sequence coverage test suite is considerably less than that of the exhaustive test suite. Thus, a
t-way sequence coverage test suite can replace the exhaustive test suite, which cannot be executed in practice. Another similar form of
t-way sequence coverage is
t-wise sequence coverage, which was presented by Kruse [
12].
t-wise sequence coverage applies to SUTs with
n inputs each of which can appear more than once in a test. Additionally, the
t inputs covered in a subsequence must be contiguous. The two types of sequence coverage criteria generally meet the sequence coverage requirement. Then, for SUTs whose test cases need to be performed contiguously, test suites can cover both the interactions among factor values and the value sequences of every factor by combining
t-way combinatorial coverage and
t-wise sequence coverage. The simplest way to combine them is to generate test suites separately and then integrate them into a large test suite. Although some coverage redundancies exist in the integrated large test suite, it successfully enables testing of SUTs with test cases performed contiguously.
In practice, SUTs always involve constraints or dependencies in or among test cases [
1], such as interaction
must not appear in a test case or value
must not be input after
. If a test does not meet the constraints, the test is invalid. To automatically generate a valid test suite, the constraints should first be formally specified. Then, the formal specification of constraints can be used to direct the test suite generation. Although some formal specification methods have been used, such as linear temporal logic (LTL) and computation tree logic (CTL), the constraints among test cases or test steps have not been effectively specified, such as when the third input is
, the fifth input must be
. Because of the existence of constraints, when a
t-way combinatorial coverage test suite is used alone, it may potentially violate the constraints among test cases. When a
t-wise sequence coverage test suite is used alone, it may potentially violate the constraints in a test case. Hence, simply combining these two test suites into a large test suite is infeasible.
In this paper, we present a research work on extending covering arrays for sequence coverage, and we introduce extended covering arrays that can achieve both t-way combinatorial coverage and t-wise sequence coverage. Then, we propose a formal specification method for specifying constraints based on clocked computation tree logic (CCTL), which is an extension of CTL. The main contribution of this paper is to propose extended covering arrays with combinatorial coverage and sequence coverage for SUTs with test cases to be performed contiguously. This research has practical application value and will improve the efficiency of software testing. To evaluate the constructed test suites, a method for verifying constraints’ validity among test cases is presented corresponding to the specification method, and kernel functions are also introduced to measure the coverage for constructed test suites.
As Particle Swarm Optimization (PSO) is competitive in uniform and variable strength covering array generation [
13], we propose Particle swarm optimization based Extended covering array Generator (PEG) for constructing extended covering arrays in this paper. The performance of our proposed PEG is evaluated using several sets of benchmark experiments for some common constraints and the feasibility and usefulness of PEG is validated.
The remainder of this paper is organized as follows. First,
Section 2 reviews the theoretical background and methods for sequence coverage testing. Some relative definitions are given in
Section 3.
Section 4 introduces extended covering arrays.
Section 5 outlines the design and implementation of PEG, including its corresponding algorithms.
Section 6 presents evaluation methods for verifying constraints’ validity and measuring coverage. The results of 12 benchmark systems under test generated by PEG are presented in
Section 7. Finally, the paper is concluded with a brief summary and provides a discussion of future research in
Section 8.
2. Related Work
Here, we review previous work toward efficient solutions about sequence coverage requirements using combinatorial testing.
In terms of SUTs with
n input events, where each event occurs exactly once in a test, the CT-based testing is sequence-based
t-way testing. Kuhn is perhaps the first person to apply sequence-based
t-way testing. He proposed a “quick and dirty” (QnD) algorithm, which is based on a greedy algorithm and likely has room for improvement [
14], and he then presented Sequence Covering Arrays (SCAs) according to covering arrays [
15]. Subsequently, he proposed a modified greedy algorithm that can handle the constraints between event pairs [
16], and he used
to test labeled transition systems [
17]. He reported several algorithms for generating
with the proposed
t-way sequence coverage criterion. These algorithms represent the first effort to systematically explore possible strategies for solving the problem of
t-way test sequence generation in a general context. Zamli discussed the sequence-based fixed and variable strength testing as an extension of existing
t-way strategies and noted that there is clearly room for improvement, particularly for the
t-way sequence coverage testing [
11]. Then, a sequence-based
t-way interaction testing strategy using the bees algorithm was presented by Zamli [
18]. The proposed algorithm shows a promising result when compared to QnD through an experiment. A method for a
t-way event-driven test suite generation based on simulated annealing called
t-way Event-Driven Input Sequence Test Case (EDISTC-SA) generator was presented by Rahman [
19]. Farchi defined a test as an ordered tuple of input parameter values and introduced the ordered constraints and the ordered interaction coverage criteria [
10]. Then, an efficient algorithm for generating test suites with minimum sizes that satisfies the ordered interaction coverage criteria was proposed and evaluated on several real-life systems.
are of practical value in testing. As exhaustive testing always consists of an incredible magnitude of tests,
can reduce the cost of testing by decreasing the number of tests.
An innovative approach that combines model-based testing and combinatorial testing to design executable and feasible test sequences was described by Nguyen [
20]. The approach starts from a finite state model, and based on the model, it generates executable paths that represent sequences of events to be executed against the SUT. Then, these paths are transformed to the equivalence classes of a classification tree [
21]. The first children classifications of the root node of the tree represent the events, and the classes of a classification are the optional values of the corresponding event. Finally, the executable test cases corresponding to an executable path are generated from the classification tree using
t-way testing. The classification tree method [
21], which is a model-based black-box test design technique [
22], was proposed based on equivalence partitioning, and it is always used for systematic test design and description of test cases. Mature products based on the classification tree method to design test sequences are TESTONA [
23] and TESSY [
24,
25]. Using the classification tree method, the input domain of an SUT is regarded under various aspects assessed as equivalent by the tester. For every aspect, disjoint and complete classifications are partitioned. The subsequent partition of every aspect through classifications is a graphical representation based on the form of a tree [
21]. Classes, which are disjointed abstractions of individual input levels for test purposes [
26], derived from these classifications may be further classified even recursively [
21]. The tree is the head of the combination table corresponding to a test suite, and test cases are the body of the combination table. Test cases are constituted by combining classes from different classifications and correspond to test steps of the testing task. During the test run, test cases are generally executed in sequential order. The testing design of the classification tree method has been widely used for embedded systems [
12], embedded automotive systems [
12,
22,
26,
27,
28] and web applications [
29] in terms of functional requirements. In addition, “Modbat” and Microsoft’s ”Spec Explorer” are also model-based test case generating tools. “Modbat” can generate state transitions coverage test cases [
30]. Microsoft’s “Spec Explorer” can generate automated test cases by running traversal techniques to achieve a form of transition coverage and enable testers to find violations of the requirements with a minimum of manual effort [
31].
Generally, there are constraints among test steps in real systems [
12,
32,
33]. Once test steps in a test suite that violate the constraints exist, the test suite is invalid for testing. Thus, it is very important to specify constraints and generate valid test suites based on the specifications [
12]. Schooljan described the linear temporal logic (LTL)-based formal specification method for dependency rules [
34]. In his work, temporal logic expressions are used to validate each test step of a test sequence. The specification of dependency rules cannot describe the constraints within test steps subjected to LTL. A similar work was conducted by Fraser, in which constraints were presented by computation tree logic (CTL) [
33]. Using LTL and CTL, the constraints among test cases were specified, whereas the test cases involving constraints are restricted to neighboring test cases. Krupp and Müller proposed an innovative application of clocked CTL (CCTL) logic to describe the constraints in real-time systems [
35]. The corresponding proposed model checker can verify the validity of test sequences by combining I/O interval specifications and CCTL expressions. Some similar coverage requirements with
t-way testing criteria were proposed by Kruse [
1,
12], such as state coverage, transition coverage and state pair coverage. The state coverage is similar to the 1-wise sequence coverage, which needs all the states to be covered at least once, and constraints among test cases involving the order of states in every factor need to be avoided. The transition coverage is 2-wise sequence coverage, which needs all the transitions of states of every factor to be covered at least once. The state pair coverage is similar to the 2-way combinatorial coverage, which requires all the interactions between two factors to be covered at least once, and the constraints among test cases involving the order of states in every factor need to be avoided. Then, three algorithms to generate state coverage and transition coverage test cases were proposed [
10].
3. Background
Before we introduce extended covering arrays, we present the existing covering arrays. When using CT, the first step is to develop the input space model of the SUT [
36]. The term “input” is used here in a general sense; any factor that can have an influence on the behavior of the SUT and that can be kept under control is considered to be an “input” [
36]. The input space model implicitly defines the SUT’s valid input space [
37]. Given that an SUT has
k input factors
, and factor
has
values or levels, the input space model of the SUT can be represented as
.
P is the set of factors
, and
V is the set of numbers of values
. For each factor
, we use
to denote the set of values, abbreviated as
.
CT approaches systematically extract and produce a set of configurations that will be run in the testing phase from the input space model [
36]. A set of configurations is called a test suite and a
t-way covering array, in which each valid combination among factor values corresponding to
t different factors appears at least once.
Definition 1. Given a set with , if belongs to the set of factors P with and belongs to , the set I is defined as a t-way interaction to be covered [38]. We use the set to denote all the t-way interactions to be covered.
Definition 2. A test case is a k-dimensional vector with [38,39]. Given a test case and an interaction , if meets, then we say that T covers I, denoted as . We use to denote all the interactions covered by T.
Definition 3. Consider that A is an mixed-level covering array, denoted by [38,40], such that every column i only has elements from the set and every possible t-way interaction is at least covered in one row of A. t is the strength of . If the smallest n for exists, we also denote it by . Another alternative form of is , which indicates that there are parameters with values, parameters with values, and so forth. It is clear that . When the sizes of all the value sets are the same, , we use covering array to replace mixed covering array. If the smallest n for exists, we also denote it by or .
A single set of configurations consists of one value from every factor. However, not all combinations of factor values may be valid, as some constraints related to some certain factor values exist. Once a single set of configurations contains these values, the single set is invalid. Such existing constraints are often caused by logical relationships among factors. For a calendar example, if factor "month" takes February, then factor "date" must take no more than 29. To guarantee that a test suite avoids all the invalid combinations successfully, the constraints must be modeled and specified.
Definition 4. is a constraint covering array, where a new variable forbidden interaction F is introduced to present the set of constraints [40]. For each constraint interaction in F, there is , where x denotes the "do not care" values. The constraints are often called forbidden tuples [41,42] and forbidden edges [43]. When the k factors have different number of values, constraint mixed-level covering arrays are used.
For example, an SUT has three factors
A,
B and
C with
, and a constraint
. The constraint mixed covering array
is shown in
Table 1.
covers all the value pairs between each two factors, except
.
7. Experiments
This section describes the experimental results of PEG performed on a benchmark of SUTs. Then, more complex constraints that PEG can not support are discussed.
7.1. Experimental Results of PEG
PEG is developed in the environment that consists of a desktop computer with Windows 7 (Dell, Xiamen, China), 2.6 GHz Core 2 Duo CPU, 2 GB of RAM. It is coded and implemented in Qt Creator 4.8.1 (C++) (Digia, Helsinki, Finland).
For the experiments, we use a benchmark with 12 different SUTs. Six of them are from Ref. [
12]. They are the “Keyboard”, the “Microwave”, the “Autoradio”, the “Coffee Machine”, the “Elevator” and the “Transmission”. The reason why we choose them is that they have more than one factor and for each factor of them each value can reach to others. The “text effect“ application of “Microsoft Word” is also chosen as an SUT. Besides the seven real world SUTs, we supplement five SUTs to increase the configuration diversity of SUTs. The details of the SUTs are given in
Table 6. The configurations of factors and factor values are listed in the third column. Three kind of constraints are listed from the fourth column to the sixth column.
As PEG depends on some degree of randomness, it is non-deterministic. Thus, we performed 30 independent runs per SUT/coverage criterion for a statistical analysis. We use PEG to generate
coverage and
,
coverage, respectively. The results are shown in
Table 7 and
Table 8. As SUT2 and SUT7 have the configuration of two factors, they can not have test suites with
coverage. Thus we use the asterisks “*” to mark the results which are performed with
coverage in
Table 8. To demonstrate the performance of PEG, best generated sizes, average generated sizes, best generated time and average generated time are presented for each SUT. The average coverage of targets are also reported corresponding to factor value combinations with
coverage and value sequences of each factor with
coverage.
Generally, the generated time increases as the number of factors and factor values grows. However, the generated time of SUT1 seems longer than other SUTs. This is mainly because much time is wasted in the calls of the SAT solver under combination constraints. Based on the results obtained, PEG can generate satisfactory results with total coverage when SUTs have no constraints related to value sequences of factors. When SUTs have the constraints related to value sequences of factors, PEG can cover all the target value sequences with covering as many factor value combinations as possible. Overall, the results show that PEG is feasible and useful to generate .
7.2. Discussion
As everyone knows, a fundamental problem with software testing is that testing under all combinations of inputs and preconditions is not feasible, even with a simple product. The SUTs with test cases to be performed contiguously still face this problem. attempt to use as few test cases as possible to cover as many factor value combinations and value sequences of factors as possible. The purpose of is to find more system defects. Compared with the manual test suite generation, can design more comprehensive test suites. fill the blank of the test case generation method for SUTs with test case to be performed contiguously and are of great significance to ensure the reliability and quality of SUTs. will be widely applied to many fields with high reliability requirement, such as aviation, spaceflight and weapon industry. In these industries, most of the input instructions of components are messages that consist of some relevant elements and are needed to be performed contiguously. are very suitable for the element value combinations and value sequences of elements in the messages.
Take an input instruction of a radar as an example. A high coverage test is needed to ensure the radar works normally under various working conditions.
Table 9 shows the instruction of the radar under test. The instruction needs to be input contiguously to control its working mode. When scanning mode is “fixed point”, the scanning speed and the sector scan scope needs to be assigned invalid values, and the scan center needs to be assigned a degree in the range of
. When scanning mode is “sector scan”, the scanning speed needs to be assigned a valid speed, the sector scan scope needs to be assigned a valid scan scope, and the scan center needs to be assigned a degree in the range of
. When scanning mode is “circular scan”, the scanning speed needs to be assigned a valid speed, the sector scan scope and the scan center needs to be assigned an invalid value.
are feasible to be used as the test suites, and
can improve the test coverage dramatically compared to design test suites manually. There is a large amount of input messages in the industry like the previous instruction and high coverage test suites are needed for those messages. Therefore, we believe that
have a broad application prospect.
However, PEG still needs to be improved in practice for more complex contraints. The SUTs whose test cases need to be performed contiguously usually have the three kinds of constraints that are the prerequisites of PEG. More complex constraints may be exist though they have hardly been seen in real world systems, such as the constraints between factors values from one test step to another [
32]. Refs. [
12,
32] have put forward the requirement of generating test suites under complex constraints and described some complex constraints as follows:
If value from factor C is selected in test case , then value from factor C must be selected in the succeeding test step .
If in , then in a later .
If in , then in all to .
If in , then in all to .
If or in , then in a later .
These complex constraints can all be presented in CCTL as follows. As they do not give the configurations of factors and factor values, we could not construct their test suites:
Ref. [
33] has presented a simplified real world system with complex constraints that is a simplified controller of a car. It has two boolean inputs that represent the user’s decision to accelerate or brake. Upon acceleration, the car starts moving, with either a slow or fast velocity. Upon braking, the car immediately stops. The velocity is also a factor of the example.
Figure 3 depicts the values of the three factors that impact the car controller.
As to pedal the brake and accelerator at the same time should be avoided when driving, the value combination of “accelerate = true” and “brake = true” is a constraint for the car controller. This value combination constraint can be denoted as
. For each factor, there are also at-least and at-most constraints.
Figure 4 shows the states and the transitions of states [
33]. A constraint that can be denoted as
restricts the first state
of the car controller.
Figure 5 shows the value transitions of each factor. When “velocity = stop” holds in a test case, “velocity=fast” cannot occur in the next test case. The constraint is
. When “velocity = slow” holds in a test case, “velocity=slow” cannot occur in the next test case. The constraint is
. Integrated with seven other complex constraints that are given in [
33], all the constraints consist of the constraint set
C in
as follows:
The
of the car controller presented in
Table 10 is generated by PEG without considering complex constraints. The combinatorial coverage of this
is
, as the combination {accelerate=true, brake=true} is a constraint that is forbidden to appear. The sequence coverage of this
is
, as "velocity=stop" → "velocity=fast" and "velocity=slow" → "velocity=slow" are value sequence constraints. As PEG cannot handle complex constraints, the
in
Table 10 violates complex constraints. The first and second test cases violate the constraint
, as when the first test is (accelerate=false, brake=false, velocity=stop), the second test case violates the
. In the same way, the second and third test cases violate
. The third and fourth test cases violate
. The fourth and fifth test cases violate
. The seventh and eighth test cases violate
. The ninth and tenth test cases violate
. The tenth and eleventh test cases violate
. The eleventh and twelfth test case violate
. To illustrate an
that satisfies the complex constraints, an
is constructed manually as shown in
Table 11. As the configuration is simple, we can construct it manually. The
shown in
Table 11 satisfies all the constraints and covers all the value combinations and value sequences of each factors.
Through the analysis above, though PEG can produce satisfactory , PEG still has some limitations to generate for SUTs with complex constraints. The emerging complex constraints can be specified by CCTL actually. It is certain that more real world SUTs with complex constraints are needed and analyzed, so that complex constraints can be classified and handled reasonably.
8. Conclusions
In this paper, we have proposed extended covering arrays with t-way combinatorial coverage and t-wise sequence coverage for SUTs whose test cases need to be performed contiguously. In extended covering arrays, we have introduced the clocked computation tree logic based formal specification method for specifying constraints. A Particle swarm optimization based Extended covering array Generator (PEG) that can produce feasible and useful with common constraints has also been presented. The performance of PEG is assessed considering benchmark experiments. For generated test suites, the method for verifying constraints’ validity has been presented corresponding to the constraint specification method. Moreover, kernel functions that can measure the coverage of generated test suites have be given. Compared with the manual test suite generation, can design more comprehensive test suites, which improves the possibility of finding system defects. In a word, fill the blank of the test case generation method for SUTs with test case to be performed contiguously and are of great significance to ensure the reliability and quality of SUTs. Though some deficiencies exists, we still believe that can have a broad application prospect through continuous progress in practice.
As part of our future work, we will first optimise PEG to cover more possible value combinations under the constraints of value sequences, then try to find real world SUTs with complex constraints and extend PEG to support them. Compared to the fixed strength combinatorial testing, the variable strength combinatorial testing usually considers the actual interaction relationship in software sufficiently. Therefore, with variable strength are also worthy of study.