Next Article in Journal
Polarized Light-Induced Molecular Orientation Control of Rigid Schiff Base Ni(II), Cu(II), and Zn(II) Binuclear Complexes as Polymer Composites
Previous Article in Journal
Another Approach to Roughness of Soft Graphs with Applications in Decision Making
 
 
Font Type:
Arial Georgia Verdana
Font Size:
Aa Aa Aa
Line Spacing:
Column Width:
Background:
Article

Extended Covering Arrays for Sequence Coverage

Automatic Test and Control Institute, School of Electrical Engineering and Automation, Harbin Institute of Technology, No. 92 Xidazhi Street, Nangang District, Harbin 150001, China
*
Author to whom correspondence should be addressed.
Symmetry 2018, 10(5), 146; https://doi.org/10.3390/sym10050146
Submission received: 5 April 2018 / Revised: 3 May 2018 / Accepted: 3 May 2018 / Published: 7 May 2018

Abstract

:
Although combinatorial testing has been widely studied and used, there are still some situations and requirements that combinatorial testing does not apply to well, such as a system under test whose test cases need to be performed contiguously. For thorough testing, the testing requirements are not only to cover all the interactions among factors but also to cover all the value sequences of every factor. Generally, systems under test always involve constraints and dependencies in or among test cases. The constraints among test cases have not been effectively specified. First, we introduce extended covering arrays that can achieve both t-way combinatorial coverage and t-wise sequence coverage, and propose a clocked computation tree logic-based formal specification method for specifying constraints. Then, Particle Swarm Optimization (PSO) based Extended covering array Generator (PEG) is elaborated. To evaluate the constructed test suites, a method for verifying the constraints’ validity is presented, and kernel functions for measuring the coverage are also introduced. Finally, the performance of the proposed PEG is evaluated using several sets of benchmark experiments for some common constraints, and the feasibility and usefulness of PEG is validated.

1. Introduction

With the development of computer technology and the increase in custom requirements, software systems are becoming more powerful and complex. In fact, the emergence of unexpected faults in such systems is inevitable. Once the system encounters a certain fault, it is likely to fail. Failure means that the system operates with unexpected behaviors. Testing is a very necessary and significant means of system quality assurance during the product development life cycle [1]. A report released by NIST (National Institute of Standards and Technology) in 2002 stated that software system bugs cost the U.S. economy 59.5 billion dollars annually [2]. However, more than one third of this cost could be saved if better testing is performed [2]. A contributing test method that can exactly find more faults with fewer test cases is urgently needed.
Combinatorial testing (CT) [3,4,5,6] is an efficient testing method through which an optimal or near optimal test suite with fewer test cases can be designed or generated. CT has proven to be an effective technique for detecting faults caused by interactions among configurations or factors in a given input space [7]. The empirical studies of system bugs suggest that CT is equivalent to exhaustive testing in a certain sense [8,9]. Although CT has been widely studied and used, there are still some situations and requirements that combinatorial testing does not apply to well, such as a system under test (SUT) whose test cases need to be performed contiguously. For thorough testing, the testing requirements of this SUT are not only to cover all the interactions among factors, but also to cover all the value sequences of every factor. When CT is used to design a test suite for this SUT, only the interactions among factor values can be effectively covered to a certain level according to the t-way combinatorial criterion; there is no effective criterion for the value sequences of every factor [10]. For example, the “text effect” application of “Microsoft Word” has seven options for users to modify some highlighted text. These options are “subscript”, “superscript”, “strikethrough”, “double strikethrough”, “all caps”, “small caps”, and “shadow” [9]. The font-processing function within the application correctly modifies the highlighted text on the screen according to the settings consisting of these options. When using CT, a test suite with several test cases can be generated to cover interactions among every t options. When the test suite is executing, the font-processing function modifies the text according to the test cases in sequence. However, the sequence of settings in every factor from contiguous test cases cannot be guaranteed to be tested at a certain level, such as a sequence from text with subscript to text without subscript for the “subscript” option with binary settings.
To solve the mentioned sequence coverage requirement, a t-way sequence coverage criterion for the requirement was proposed [11] based on the t-way combinatorial coverage criterion. The t-way sequence coverage criterion was first introduced to apply to event sequence testing. In terms of an SUT with n input events, each event can only be input once during a test. Each test case in the test suite covers n t subsequences with length t ( 0 < t n ) , and the covered t events in a subsequence do not have to be neighboring. A t-way sequence coverage test suite can cover all n t t ! subsequences that have t different events. The size of a t-way sequence coverage test suite is considerably less than that of the exhaustive test suite. Thus, a t-way sequence coverage test suite can replace the exhaustive test suite, which cannot be executed in practice. Another similar form of t-way sequence coverage is t-wise sequence coverage, which was presented by Kruse [12]. t-wise sequence coverage applies to SUTs with n inputs each of which can appear more than once in a test. Additionally, the t inputs covered in a subsequence must be contiguous. The two types of sequence coverage criteria generally meet the sequence coverage requirement. Then, for SUTs whose test cases need to be performed contiguously, test suites can cover both the interactions among factor values and the value sequences of every factor by combining t-way combinatorial coverage and t-wise sequence coverage. The simplest way to combine them is to generate test suites separately and then integrate them into a large test suite. Although some coverage redundancies exist in the integrated large test suite, it successfully enables testing of SUTs with test cases performed contiguously.
In practice, SUTs always involve constraints or dependencies in or among test cases [1], such as interaction { b 1 , c 1 } must not appear in a test case or value c 2 must not be input after c 1 . If a test does not meet the constraints, the test is invalid. To automatically generate a valid test suite, the constraints should first be formally specified. Then, the formal specification of constraints can be used to direct the test suite generation. Although some formal specification methods have been used, such as linear temporal logic (LTL) and computation tree logic (CTL), the constraints among test cases or test steps have not been effectively specified, such as when the third input is c 1 , the fifth input must be c 2 . Because of the existence of constraints, when a t-way combinatorial coverage test suite is used alone, it may potentially violate the constraints among test cases. When a t-wise sequence coverage test suite is used alone, it may potentially violate the constraints in a test case. Hence, simply combining these two test suites into a large test suite is infeasible.
In this paper, we present a research work on extending covering arrays for sequence coverage, and we introduce extended covering arrays that can achieve both t-way combinatorial coverage and t-wise sequence coverage. Then, we propose a formal specification method for specifying constraints based on clocked computation tree logic (CCTL), which is an extension of CTL. The main contribution of this paper is to propose extended covering arrays with combinatorial coverage and sequence coverage for SUTs with test cases to be performed contiguously. This research has practical application value and will improve the efficiency of software testing. To evaluate the constructed test suites, a method for verifying constraints’ validity among test cases is presented corresponding to the specification method, and kernel functions are also introduced to measure the coverage for constructed test suites.
As Particle Swarm Optimization (PSO) is competitive in uniform and variable strength covering array generation [13], we propose Particle swarm optimization based Extended covering array Generator (PEG) for constructing extended covering arrays in this paper. The performance of our proposed PEG is evaluated using several sets of benchmark experiments for some common constraints and the feasibility and usefulness of PEG is validated.
The remainder of this paper is organized as follows. First, Section 2 reviews the theoretical background and methods for sequence coverage testing. Some relative definitions are given in Section 3. Section 4 introduces extended covering arrays. Section 5 outlines the design and implementation of PEG, including its corresponding algorithms. Section 6 presents evaluation methods for verifying constraints’ validity and measuring coverage. The results of 12 benchmark systems under test generated by PEG are presented in Section 7. Finally, the paper is concluded with a brief summary and provides a discussion of future research in Section 8.

2. Related Work

Here, we review previous work toward efficient solutions about sequence coverage requirements using combinatorial testing.
In terms of SUTs with n input events, where each event occurs exactly once in a test, the CT-based testing is sequence-based t-way testing. Kuhn is perhaps the first person to apply sequence-based t-way testing. He proposed a “quick and dirty” (QnD) algorithm, which is based on a greedy algorithm and likely has room for improvement [14], and he then presented Sequence Covering Arrays (SCAs) according to covering arrays [15]. Subsequently, he proposed a modified greedy algorithm that can handle the constraints between event pairs [16], and he used S C A s to test labeled transition systems [17]. He reported several algorithms for generating S C A s with the proposed t-way sequence coverage criterion. These algorithms represent the first effort to systematically explore possible strategies for solving the problem of t-way test sequence generation in a general context. Zamli discussed the sequence-based fixed and variable strength testing as an extension of existing t-way strategies and noted that there is clearly room for improvement, particularly for the t-way sequence coverage testing [11]. Then, a sequence-based t-way interaction testing strategy using the bees algorithm was presented by Zamli [18]. The proposed algorithm shows a promising result when compared to QnD through an experiment. A method for a t-way event-driven test suite generation based on simulated annealing called t-way Event-Driven Input Sequence Test Case (EDISTC-SA) generator was presented by Rahman [19]. Farchi defined a test as an ordered tuple of input parameter values and introduced the ordered constraints and the ordered interaction coverage criteria [10]. Then, an efficient algorithm for generating test suites with minimum sizes that satisfies the ordered interaction coverage criteria was proposed and evaluated on several real-life systems. S C A s are of practical value in testing. As exhaustive testing always consists of an incredible magnitude of tests, S C A s can reduce the cost of testing by decreasing the number of tests.
An innovative approach that combines model-based testing and combinatorial testing to design executable and feasible test sequences was described by Nguyen [20]. The approach starts from a finite state model, and based on the model, it generates executable paths that represent sequences of events to be executed against the SUT. Then, these paths are transformed to the equivalence classes of a classification tree [21]. The first children classifications of the root node of the tree represent the events, and the classes of a classification are the optional values of the corresponding event. Finally, the executable test cases corresponding to an executable path are generated from the classification tree using t-way testing. The classification tree method [21], which is a model-based black-box test design technique [22], was proposed based on equivalence partitioning, and it is always used for systematic test design and description of test cases. Mature products based on the classification tree method to design test sequences are TESTONA [23] and TESSY [24,25]. Using the classification tree method, the input domain of an SUT is regarded under various aspects assessed as equivalent by the tester. For every aspect, disjoint and complete classifications are partitioned. The subsequent partition of every aspect through classifications is a graphical representation based on the form of a tree [21]. Classes, which are disjointed abstractions of individual input levels for test purposes [26], derived from these classifications may be further classified even recursively [21]. The tree is the head of the combination table corresponding to a test suite, and test cases are the body of the combination table. Test cases are constituted by combining classes from different classifications and correspond to test steps of the testing task. During the test run, test cases are generally executed in sequential order. The testing design of the classification tree method has been widely used for embedded systems [12], embedded automotive systems [12,22,26,27,28] and web applications [29] in terms of functional requirements. In addition, “Modbat” and Microsoft’s ”Spec Explorer” are also model-based test case generating tools. “Modbat” can generate state transitions coverage test cases [30]. Microsoft’s “Spec Explorer” can generate automated test cases by running traversal techniques to achieve a form of transition coverage and enable testers to find violations of the requirements with a minimum of manual effort [31].
Generally, there are constraints among test steps in real systems [12,32,33]. Once test steps in a test suite that violate the constraints exist, the test suite is invalid for testing. Thus, it is very important to specify constraints and generate valid test suites based on the specifications [12]. Schooljan described the linear temporal logic (LTL)-based formal specification method for dependency rules [34]. In his work, temporal logic expressions are used to validate each test step of a test sequence. The specification of dependency rules cannot describe the constraints within test steps subjected to LTL. A similar work was conducted by Fraser, in which constraints were presented by computation tree logic (CTL) [33]. Using LTL and CTL, the constraints among test cases were specified, whereas the test cases involving constraints are restricted to neighboring test cases. Krupp and Müller proposed an innovative application of clocked CTL (CCTL) logic to describe the constraints in real-time systems [35]. The corresponding proposed model checker can verify the validity of test sequences by combining I/O interval specifications and CCTL expressions. Some similar coverage requirements with t-way testing criteria were proposed by Kruse [1,12], such as state coverage, transition coverage and state pair coverage. The state coverage is similar to the 1-wise sequence coverage, which needs all the states to be covered at least once, and constraints among test cases involving the order of states in every factor need to be avoided. The transition coverage is 2-wise sequence coverage, which needs all the transitions of states of every factor to be covered at least once. The state pair coverage is similar to the 2-way combinatorial coverage, which requires all the interactions between two factors to be covered at least once, and the constraints among test cases involving the order of states in every factor need to be avoided. Then, three algorithms to generate state coverage and transition coverage test cases were proposed [10].

3. Background

Before we introduce extended covering arrays, we present the existing covering arrays. When using CT, the first step is to develop the input space model of the SUT [36]. The term “input” is used here in a general sense; any factor that can have an influence on the behavior of the SUT and that can be kept under control is considered to be an “input” [36]. The input space model implicitly defines the SUT’s valid input space [37]. Given that an SUT has k input factors P 1 , P 2 , , P k , and factor P i ( 1 i k ) has v i values or levels, the input space model of the SUT can be represented as M = < P , V > . P is the set of factors P = { P 1 , P 2 , , P k } , and V is the set of numbers of values V = { v 1 , v 2 , , v k } . For each factor P i , we use { 0 , 1 , , v i 1 } to denote the set of values, abbreviated as [ 0 , v i 1 ] .
CT approaches systematically extract and produce a set of configurations that will be run in the testing phase from the input space model [36]. A set of configurations is called a test suite and a t-way covering array, in which each valid combination among factor values corresponding to t different factors appears at least once.
Definition 1.
Given a set I = { ( P i 1 , a i 1 ) , ( P i 2 , a i 2 ) , , ( P i t , a i t ) } with i j [ 1 , k ] ( j = 1 , 2 , , t ) , if P i j belongs to the set of factors P with | P | = k and a i j belongs to [ 0 , v i j 1 ] , the set I is defined as a t-way interaction to be covered [38].
We use the set H t = { I | I = { ( P i 1 , a i 1 ) , ( P i 2 , a i 2 ) , , ( P i t , a i t ) } } to denote all the t-way interactions to be covered.
Definition 2.
A test case T = { t 1 , t 2 , , t k } is a k-dimensional vector with t i [ 0 , v i 1 ] ( i = 1 , 2 , , k ) [38,39].
Given a test case T = { t 1 , t 2 , , t k } and an interaction I = { ( P i 1 , a i 1 ) , ( P i 2 , a i 2 ) , , ( P i t , a i t ) } , if T [ i j ] = t i j = a i j meets, then we say that T covers I, denoted as T I . We use H T , t = { I | I = { ( P i 1 , a i 1 ) , ( P i 2 , a i 2 ) , , ( P i t , a i t ) } T I } to denote all the interactions covered by T.
Definition 3.
Consider that A is an n × k mixed-level covering array, denoted by M C A ( n ; t , ( v 1 , v 2 , , v k ) ) [38,40], such that every column i only has elements from the set [ 0 , v i 1 ] and every possible t-way interaction I = { ( P i 1 , a i 1 ) , ( P i 2 , a i 2 ) , , ( P i t , a i t ) } is at least covered in one row of A. t is the strength of M C A ( n ; t , ( v 1 , v 2 , , v k ) ) .
If the smallest n for M C A ( n ; t , ( v 1 , v 2 , , v k ) ) exists, we also denote it by M C A N ( t , ( v 1 , v 2 , , v k ) ) . Another alternative form of M C A is M C A ( n ; t , s 1 y 1 s 2 y 2 s u y u ) , which indicates that there are y 1 parameters with v 1 values, y 2 parameters with v 2 values, and so forth. It is clear that 1 l u y l = k . When the sizes of all the value sets are the same, v 1 = v 2 = = v k = v , we use covering array C A ( n ; t , k , v ) to replace mixed covering array. If the smallest n for C A ( n ; t , k , v ) exists, we also denote it by C A N ( t , k , v ) or C A N ( n ; t , v k ) .
A single set of configurations consists of one value from every factor. However, not all combinations of factor values may be valid, as some constraints related to some certain factor values exist. Once a single set of configurations contains these values, the single set is invalid. Such existing constraints are often caused by logical relationships among factors. For a calendar example, if factor "month" takes February, then factor "date" must take no more than 29. To guarantee that a test suite avoids all the invalid combinations successfully, the constraints must be modeled and specified.
Definition 4.
C C A ( n ; t , k , v , F ) is a constraint covering array, where a new variable forbidden interaction F is introduced to present the set of constraints [40]. For each constraint interaction I = ( a 1 , a 2 , , a k ) in F, there is a i [ 0 , v i 1 ] [ x ] ( 1 i k ) , where x denotes the "do not care" values. The constraints are often called forbidden tuples [41,42] and forbidden edges [43].
When the k factors have different number of values, constraint mixed-level covering arrays C M C A s are used.
For example, an SUT has three factors A, B and C with v A = v B = 2 , v C = 3 , and a constraint F = { ( 0 , 0 , x ) } . The constraint mixed covering array C M C A ( 7 ; 2 , 2 2 3 1 , F = ( 0 , 0 , x ) ) is shown in Table 1. C M C A ( 7 ; 2 , 2 2 3 1 , F = ( 0 , 0 , x ) ) covers all the value pairs between each two factors, except ( A = 0 , B = 0 ) .

4. Extended Covering Arrays

In this section, we introduce extended covering arrays.

4.1. E C A s

When C M C A ( 7 ; 2 , 2 2 3 1 , F = ( 0 , 0 , x ) ) shown in Table 1 is used to test an SUT contiguously, 2-way combinations are covered. If the SUT has a sequence coverage requirement, then the two-value sequences of each factor are not completely covered. As in the test sequence { 0 0 0 1 1 1 1 } of factor A, only three two-value sequences { 0 0 } , { 0 1 } , and { 1 1 } are covered. If we use a 2-wise sequence coverage test suite as shown in Table 2, although all the sequences of two values of each factor are covered, some value combinations are not covered, such as ( A = 1 , B = 1 ) and ( B = 0 , C = 2 ) . If both the sequence coverage and combinatorial coverage are required, then the only way to satisfy this requirement is to combine them into a test suite. However, combining them into a test suite may lead to some redundancies and a larger size. To design test suites that can meet both sequence coverage and combinatorial coverage with smaller sizes, extended covering arrays are defined.
Definition 5.
Consider that a n × k array A is an extended mixed-level covering array, denoted by E M C A ( n ; t c , t s , v 1 , v 2 , , v k ) , where t c is combinatorial coverage strength and t s is sequence coverage strength, such that every column i only has elements from the set [ 0 , v i 1 ] and the following two conditions are met:
for each interaction I = { ( P i 1 , a i 1 ) , ( P i 2 , a i 2 ) , , ( P i t c , a i t c ) } with length t c ( 1 t c k ), there is at least one row r ( 1 r n ), A [ r , i j ] = a i j ( j = 1 , 2 , , t c ) ;
in each column i ( 1 i k ) , for every possible value sequence with length t s ( 1 t s min ( v 1 , v 2 , , v k ) ) such as e 1 i , e 2 i , , e t s i [ 0 , v i 1 ] , there are t s contiguous rows r 1 , r 2 , , r t s with A [ r j , i ] = e j i ( 1 j t s ) .
If the smallest n for E M C A ( n ; t c , t s , v 1 , v 2 , , v k ) exists, we also denote it by E M C A N ( t c , t s , v 1 , v 2 , , v k ) . Another alternative form of E M C A is E M C A ( n ; t c , t s , s 1 y 1 s 2 y 2 s u y u ) . When the sizes of all the value sets are the same, v 1 = v 2 = = v k = v , we denote the extended covering array as E C A ( n ; t c , t s , k , v ) or E C A ( n ; t c , t s , v k ) .
For the example presented above, an E M C A is shown in Table 3. The extended covering array covers all combinatorial pairs and two value sequences only with size 10.

4.2. E C C A s

In real systems, in addition to the constraints among factor values, there are constraints that involve sequences of factor values. Such a sequence of factor values is also often identified by logically specifying constraints. These two types of constraints must be avoided in final test suites; otherwise, test suites may be invalid. Thus, the specification and model of the constraints are the preconditions to design and generate test suites. On the one hand, the specification of constraints can help to avoid constraints in the process of constructing test suites. On the other hand, the specification of constraints can help to verify whether the designed test suites are valid. We also define the extended constraint covering array as E C C A ( n ; t c , t s , k , v , C ) . As there exist constraints that involve value sequences of each factor, some interactions of factor values may not appear in one test case in practice. See Section 5 for details. This produces a conceptual differences between E C C A and E C A . In other words, E C C A may violate the first condition in Definition 5. Then, we modify the first condition of E C A to fit E C C A . The modified condition is that E C C A could cover as many t-way interactions as possible. The t-way interactions covered in E C C A s can be measured in the method described in SubSection 6.2. When the k factors have different number of values, extended constraint mixed-level covering arrays ( E C M C A s ) are used. E C C A s have the same expression forms with E C A s .
The set C is the set of constraint statements specified by CCTL. CCTL is a variant of timed CTL based on I/O-interval structures introduced by Ruf [44,45] in the context of the real-time system model checker. The I/O-interval structures are used in state transition systems to express time annotations. A time annotation is a constraint that assigns a [ m i n , m a x ] -time interval for a state transition [44]. [ m i n , m a x ] -time intervals can help CCTL to precisely model time in which state transitions occur compared to LTL and CTL [44]. For this reason, [ m i n , m a x ] -time intervals can also support describing the temporal relationship among inputs based on time steps precisely. This is the maximum benefit of CCTL. Thus, we use CCTL to model the constraint relationship.
The CCTL syntax is defined as follows [46]:
ϕ = A P | ¬ ϕ | ϕ ϕ | ϕ ϕ | ϕ ϕ | ϕ ϕ | ϕ ϕ | E X [ n ] ϕ | E F [ m , n ] ϕ | E G [ m , n ] ϕ | E ( ϕ U [ m , n ] ϕ ) | A X [ n ] ϕ | A F [ m , n ] ϕ | A G [ m , n ] ϕ | A ( ϕ U [ m , n ] ϕ ) ,
where A P is an atomic proposition and m , n N + are time bounds with m n . ¬, ∧, →, ⊕, ∨ and ⊗ are the classical logic operators included in CCTL syntax. ϕ is the CCTL formula. X, F, U and G are the temporal operators, where X is the “next” operator, F is the “final“ operator, G is the “always” operator, and U is the “until” operator. A and E are path quantifiers. A “path” is an infinite sequence of states, denoted as ρ . If a CCTL formula ϕ is true in path ρ , then we write ρ ϕ . Because there are potentially many paths in a system, E means that “at least one path exists that satisfies the temporal operator”, and A means “for all paths that satisfy the temporal operator”. In testing, the generated E C C A is equivalent to the path because it should satisfy the constraints presented by CCTL formulas, denoted as E C C A ϕ . As there exists at least a test suite which is used to test, we use the path quantifier E in this paper to specify constraints. Based on a constraint which belongs to a test case or involves test cases, the specification of constraints are divided into the following two parts.

4.2.1. The Specification of Constraints in a Test Case

The constraints in a test case indicate the restrictions among factor values. These types of constraints correspond to the traditional covering array. The constraints forbid the appearance of some certain value combinations that are composed of different factors. Because generating a test case that avoids all the constraints is a boolean satisfiability problem [47], a formal specification is needed in the process of automatically generating test cases. Generally, the constraints are often represented in conjunctive normal form (CNF) [47].
For a forbidden tuple I = { ( P i 1 , a i 1 ) , ( P i 2 , a i 2 ) , , ( P i t , a i t ) } , the formal specification is E G ( ¬ P i 1 = a i 1 ¬ P i 2 = a i 2 ¬ P i t = a i t ) . In addition, there are always two types of internal constraints for an SUT to limit each test case to having one and only one value from every factor. They are the at-least and at-most constraints [47]. The at-least constraints are needed to ensure that there is no less than one value of each factor in a test case, and the at-most constraints are needed to ensure that no more than one value is assigned to a factor in a test case.
  • at-most constraints: for each factor P i with its value set [ 0 , v i 1 ] and a test case T = ( t 1 , t 2 , , t k ) , there exists that a i m , a i n [ 0 , v i 1 ] and a i m a i n , and the at-most constraints can be denoted as E G ( ¬ P i = a i m ¬ P i = a i n ) . The at-most constraints mean that each two values of one factor must not appear in a same test case.
  • at-least constraints: for each factor P i with its value set [ 0 , v i 1 ] and a test case T = ( t 1 , t 2 , , t k ) , the at-least constraints can be denoted as E G ( P i = 0 P i = 1 P i = v i 1 ) . The at-least constraints mean that there must be at least one value of a factor appearing in a test case.

4.2.2. The Specification of Constraints among Test Cases

The constraints involving value sequences of factors are constraints among test cases or test steps. The constraints are generally divided into absolute constraints and relative constraints according to whether constraints have the logical operator “→”. Absolute constraints do not contain the logical operator “→”, whereas relative constraints have the logical operator “→”. The relative constraints in front and behind of “→” are composed of the absolute constraints presented by E X , E G , E U and E F . Relative constraints means that when the condition in front of “→” is tenable, the condition behind of “→” must be tenable.
Each temporal operator can represent a type of absolute constraint. A formal description of the temporal operators is presented below. Given E C C A ( n ; t c , t s , k , v , C ) and u , w N + with 0 < u w n :
  • E C C A E X [ u ] ϕ : There exists an E C C A in which the uth test case satisfies ϕ . It is the same as E C C A [ u , : ] ϕ ;
  • E C C A E G [ u , w ] ϕ : u i w , and for all 1 j i : E C C A [ j , : ] ϕ ;
  • E C C A E ( ϕ U [ u , w ] ψ ) : u i w , 1 j < i , E C C A [ j , : ] ϕ and E C C A [ i , : ] ψ ;
  • E C C A E F [ u , w ] ϕ : There exists at least one i with u i w : E C C A [ i , : ] ϕ .
We always omit “ E C C A ” in the specification of constraints. The constraints specified above are the absolute constraints. Then, we introduce the relative constraints. We classify the relative constraints according to the symbol in front of “→”. We define that when the constraint has the form of E X [ u ] in front of other temporal operators, then the u + 1 th test case is regarded as the first test case for the following representations. Given E C C A ( n ; t c , t s , k , v , C ) , x , y , z , u , w N + with x y and u w :
(1)
E X
  • E C C A E G ( E X [ u ] ϕ E X [ u + x ] ψ ) : When the uth test case satisfies ϕ , E C C A [ u , : ] ϕ , the ( u + x ) th test case holds ψ , E C C A [ u + x , : ] ψ ;
  • E C C A E G ( E X [ u ] ϕ E X [ z ] E G [ x , y ] ψ ) : When the uth test case satisfies ϕ , E C C A [ u , : ] ϕ , the ( u + z + 1 ) th test case is regarded as the first test case for E G [ x , y ] ψ as the E X [ z ] . Then, x i y , and for all 1 j i : E C C A [ u + z + j , : ] ψ ;
  • E C C A E G ( E X [ u ] ϕ E X [ z ] E ψ U [ x , y ] σ ) : When the uth test case satisfies ϕ , E C C A [ u , : ] ϕ , the ( u + z + 1 ) th test case is regarded as the first test case for E ψ U [ x , y ] σ as the E X [ z ] . Then, x i y , 1 j < i , E C C A [ u + z + j , : ] ψ and E C C A [ u + z + i , : ] σ ;
  • E C C A E G ( E X [ u ] ϕ E F [ x , y ] ψ ) : When the uth test case satisfies ϕ , E C C A [ u , : ] ϕ : At least one i exists with x i y : E C C A [ i , : ] ψ with x > u .
(2)
E G
  • E C C A E G ( E G [ u , w ] ϕ E X [ x ] ψ ) : u i w and for all 1 j i , E C C A [ j , : ] ϕ exists: E C C A [ x , : ] ψ with x > w ;
  • E C C A E G ( E G [ u , w ] ϕ E F [ x , y ] ψ ) : u i w and for all 1 j i , E C C A [ j , : ] ϕ exists: At least one l exists with x l y : E C C A [ l , : ] ψ with x > w .
(3)
E U
  • E C C A E G ( E ϕ U [ u , w ] ψ E X [ x ] σ ) : u i w , 1 j < i , E C C A [ j , : ] ϕ and E C C A [ i , : ] ψ exists: E C A [ x , : ] σ with x > w ;
  • E C C A E G ( E ϕ U [ u , w ] ψ E F [ x , y ] σ ) : u i w , 1 j < i , E C C A [ j , : ] ϕ and E C C A [ i , : ] ψ exists: At least one l exists with x l y : E C C A [ l , : ] σ with x > w .
(4)
E F
  • E C C A E G ( E F [ u , w ] ϕ E X [ x ] ψ ) : At least one i exists with u i w : E C C A [ i , : ] ϕ : E C C A [ x , : ] ψ with x > w ;
  • E C C A E G ( E F [ u , w ] ϕ E F [ x , y ] ψ ) : At least one i exists with u i w : E C C A [ i , : ] ϕ : At least one l exists with x l y : E C C A [ l , : ] ψ with x > w .
(5)
ϕ
  • E C C A E G ( ϕ E X [ u ] E G [ x , y ] ψ ) : For each test case in E C C A , if there is a test case i that holds ϕ , the ( i + u + 1 ) th test case is regarded as the first test case for E G [ x , y ] ψ as E X [ u ] : x l y for all 1 j l : E C C A [ i + u + j , : ] ψ ;
  • E C C A E G ( ϕ E X [ u ] E ψ U [ x , y ] σ ) : For each test case in E C C A , if there is a test case i that holds ϕ , the ( i + u + 1 ) th test case is regarded as the first test case for E ψ U [ x , y ] σ as E X [ u ] . Then, x l y , 1 j < l : E C C A [ i + u + j , : ] ψ and E C C A [ i + u + l , : ] σ ;

4.3. A Case Study

Two real SUTs are analyzed in this subsection.
The first real system under test is the “VideoGame”, which is described in Ref [1]. “VideoGame” has two factors: one is “startingGame” and the other is “Pause”. Factor “startingGame” has four levels and they are “startingGame”, “startup”, “controlling” and “gameOver”, and factor “Pause” has two levels and they are “running” and “paused”. In terms of factor “startingGame”, “startingGame” level can transit to itself and “startup”, “startup” level can transit to itself and “controlling”, “controlling” level can transit to itself and “gameOver”, and “gameOver” level can transit to itself and “startingGame”. In terms of factor “Pause”, “running” level can only transit to “paused” and “paused” level can only transit to “running”. The limitations that restrict levels not to transit to others are constraints. Combined with the at-most constraints and at-least constraints, the constraints of “VideoGame” are obtained. Thus, the E C M C A ( 9 ; 2 , 2 , 4 1 2 1 , C ) of the “VideoGame” can be constructed as shown in Table 4 and its constraint set C is as follows. The E C M C A has covered all the eight possible level combinations and ten level transitions:
C = { E G ( ¬ s t a r t i n g G a m e = s t a r t i n g G a m e ¬ s t a r t i n g G a m e = s t a r t u p )   E G ( ¬ s t a r t i n g G a m e = s t a r t i n g G a m e ¬ s t a r t i n g G a m e = c o n t r o l l i n g )   E G ( ¬ s t a r t i n g G a m e = s t a r t i n g G a m e ¬ s t a r t i n g G a m e = g a m e O v e r )   E G ( ¬ s t a r t i n g G a m e = s t a r t u p ¬ s t a r t i n g G a m e = c o n t r o l l i n g )   E G ( ¬ s t a r t i n g G a m e = s t a r t u p ¬ s t a r t i n g G a m e = g a m e O v e r )   E G ( ¬ s t a r t i n g G a m e = c o n t r o l l i n g ¬ s t a r t i n g G a m e = g a m e O v e r )   E G ( s t a r t i n g G a m e = s t a r t i n g G a m e s t a r t i n g G a m e = s t a r t u p s t a r t i n g G a m e = c o n t r o l l i n g   s t a r t i n g G a m e = s t a r t u p )   E G ( ¬ P a u s e = r u n n i n g ¬ P a u s e = p a u s e d )   E G ( P a u s e = r u n n i n g P a u s e = p a u s e d )   E G ( s t a r t i n g G a m e = s t a r t i n g G a m e E X ( s t a r t i n g G a m e = s t a r t i n g G a m e s t a r t i n g G a m e = s t a r t u p ) )   E G ( s t a r t i n g G a m e = s t a r t u p E X ( s t a r t i n g G a m e = s t a r t u p s t a r t i n g G a m e = c o n t r o l l i n g ) )   E G ( s t a r t i n g G a m e = c o n t r o l l i n g E X ( s t a r t i n g G a m e = c o n t r o l l i n g s t a r t i n g G a m e = g a m e O v e r ) )   E G ( s t a r t i n g G a m e = g a m e O v e r E X ( s t a r t i n g G a m e = s t a r t i n g G a m e s t a r t i n g G a m e = g a m e O v e r ) ) } .
The second real system under test is the “text effect” application of “Microsoft Word”, which has seven options for users to modify some highlighted text. These options are “subscript”, “superscript”, “strikethrough”, “double strikethrough”, “all caps”, “small caps”, and “shadow” [9]. Each option has optional values: one is “true” and the other is “false”. Three value combination constraints exist, and they are ( s u b s c r i p t = t r u e , s u p e r s c r i p t = t r u e ) , ( s t r i k e t h r o u g h = t r u e , d o u b l e s t r i k e t h r o u g h = t r u e ) and ( a l l c a p s = t r u e , s m a l l c a p s = t r u e ) . The font-processing function within the application correctly modifies the highlighted text on the screen according to the settings consisting of these options. When tests are executing, the font-processing function modifies the text according to the test cases in sequence. Thus, the E C M C A ( 9 ; 2 , 2 , 2 7 , C ) of the “text effect” application can be constructed as shown in Table 5 and its constraint set C is as follows. The E C M C A has covered all the 81 possible value combinations and 28 value transitions:
C = { E G ( ¬ s u b s c r i p t = t r u e ¬ s u b s c r i p t = f a l s e )   E G ( s u b s c r i p t = t r u e s u b s c r i p t = f a l s e )   E G ( ¬ s u p e r s c r i p t = t r u e ¬ s u p e r s c r i p t = f a l s e )   E G ( s u p e r s c r i p t = t r u e s u p e r s c r i p t = f a l s e )   E G ( ¬ s t r i k e t h r o u g h = t r u e ¬ s t r i k e t h r o u g h = f a l s e )   E G ( s t r i k e t h r o u g h = t r u e s t r i k e t h r o u g h = f a l s e )   E G ( ¬ d o u b l e s t r i k e t h r o u g h = t r u e ¬ d o u b l e s t r i k e t h r o u g h = f a l s e )   E G ( d o u b l e s t r i k e t h r o u g h = t r u e d o u b l e s t r i k e t h r o u g h = f a l s e )   E G ( ¬ a l l c a p s = t r u e ¬ a l l c a p s = f a l s e )   E G ( a l l c a p s = t r u e a l l c a p s = f a l s e )   E G ( ¬ s m a l l c a p s = t r u e ¬ s m a l l c a p s = f a l s e )   E G ( s m a l l c a p s = t r u e s m a l l c a p s = f a l s e )   E G ( ¬ s h a d o w = t r u e ¬ s h a d o w = f a l s e )   E G ( s h a d o w = t r u e s h a d o w = f a l s e )   E G ( ¬ s u b s c r i p t = t r u e ¬ s u p e r s c r i p t = t r u e )   E G ( ¬ s t r i k e t h r o u g h = t r u e ¬ d o u b l e s t r i k e t h r o u g h = t r u e )   E G ( ¬ a l l c a p s = t r u e ¬ s m a l l c a p s = t r u e ) } .

5. The Construction of Extended Covering Arrays

5.1. Particle Swarm Optimization

The Particle Swarm Optimization (PSO) was originally put forward by Kennedy [48] as an optimization technique inspired by the swarm behavior of birds in 1995. The swarm of particles always moves towards the optimal position in the process of optimization. The position X i t = ( x i 1 t , x i 2 t , , x i k t ) of a particle indicates the solution under optimization. The speed V i t = ( v i 1 t , v i 2 t , , v i k t ) of a particle indicates the tendency of evolution and the degree of variation. The fitness factor value of a particle indicates the degree of optimization. Each particle remembers its coordinates in the solution space where it has found its best solution so far, which is called p B e s t . In addition to p B e s t , particles track the overall best value obtained by any particle in the population, called g B e s t . The process of evolution will continue untill the iteration time is reached. As a result of the discrete values in parameters, we adopt the discrete version of PSO (DPSO), which has been used in covering array generation [13]. The particle X i t updates its coordinate x i j t according to Equations (2) and (3):
v i j t + 1 = ω v i j t + c 1 r 1 ( p B e s t i j t x i j t ) + c 2 r 2 ( g B e s t j t x i j t ) ,
x i j t + 1 = x i j t + v i j t + 1 ,
where t is the current iteration number, j is the component of the dimension k, i is the particle index, ( c 1 , c 2 ) are the acceleration coefficients to adjust the weight between components, ω is the inertia weight in the range of ( 0 , 1 ) , and ( r 1 , r 2 ) are two random factors ranged in ( 0 , 1 ) . According to the equations above, each particle updates its velocity by following its p B e s t and g B e s t in order to produce a movement towards a better region in the search space.

5.2. The PEG Strategy

5.2.1. The Interaction and Constraint Maps Generation Algorithm

In PEG, fitness factor values that indicate the degree of optimization are used to choose best particles. Hence, in order to compute the fitness factor values of particles conveniently, effective data structures are necessary. We propose Combinatorial Interaction Maps ( C I M s ) and Sequence Interaction Maps ( S I M s ) as the structures that are used to choose best particles. Similarly, to verify the constraint validity of each particle, effective data structures are also necessary. We propose Combinatorial Constraint Maps ( C C M s ) and Sequence Constraint Maps ( S C M s ) as the structures that are used to validate constraint validity. The “Map” here is an associative container in C++ that stores elements formed by a combination of a key value and a mapped value, following a specific order. Algorithm 1 shows the corresponding generation algorithm of them. The algorithm receives t c , t s , v 1 , v 2 , , v k and C as inputs, and generates C C M , C I M , S C M and S I M one by one.
Algorithm 1 The interaction and constraint maps generation algorithm.
Input: t c , t s , v 1 , v 2 , , v k , C
Output: C C M , C I M , S C M and S I M
1:
generate C C M ;
2:
generate C I M based on C C M ;
3:
generate S C M ;
4:
generate S I M based on S C M ;
Figure 1 shows an example of generating C C M , C I M , S C M and S I M from t s = t c = 2 , v 1 = v 2 = 2 , v 3 = 3 and C = { E G ( ¬ A = 0 ¬ B = 0 ) , E G ( C = 0 ¬ E X ( C = 2 ) ) } . Here, the at-most and at-least constraints are omitted, as they are only used for constraint validity verification. The keys of the four maps are factors or factor combinations, and the values mapped to the keys are a set of values of factors. The values in combinatorial maps represent factor value combinations, whereas the values in sequence maps represent factor value sequences. In the algorithm, the C C M is generated first. Then all the value combinations of each t c factors are generated in C I M . When C I M is generated, each factor value combination that appears in C C M must be removed from it. For example, the value combination { A = 0 , B = 0 } is a constraint in C C M . Thus, it must not appear in C I M . The generation process of S C M and S I M is similar with the generation of C C M and C I M . The S C M is generated first. Then, all the value sequences with length t s of each factor are generated in S I M . When S I M is generated, each value sequence that appears in S C M must be removed from it.

5.2.2. The E C M C A Generation Algorithm

The E C M C A generation algorithm is performed immediately after the generation of C C M , C I M , S C M and S I M . The use of C I M and S I M is essential for computing fitness factor values. The use of C C M and S C M is essential for validating the constraint validity of test cases. The main idea of the algorithm is that the algorithm makes fully use of the ability of seeking excellent solutions to generate good results. As usually constraints that are related to sequences are very complex, this algorithm supports three kinds of most common constraints. The three kinds of common constraints are: ① initial value constraints of factors that are presented by E X 1 ; ② combinatorial value constraints among factors; and ③ value transition constraints of any factor. The algorithm is shown as Algorithm 2.
Algorithm 2 The E C M C A generation algorithm.
Input: t c , t s , v 1 , v 2 , , v k , C ; c 1 , c 2 , ω , R, P; C C M , C I M , S C M and S I M
Output: E C M C A
1:
initialize the SAT solver;
2:
if there is E X [ 1 ] in C then
3:
  generate a test and update C I M and S I M ;
4:
end if
5:
while TRUE do
6:
  initialize the population with a population size of P;
7:
  for R repetitions do
8:
   update the population;
9:
   disturb the population;
10:
  end for
11:
  if the fitness factor value of g B e s t > 0 then
12:
   generate a test and update C I M and S I M ;
13:
  else
14:
   if S C M = then
15:
    if S I M = and C I M = then
16:
     break while;
17:
    end if
18:
   else
19:
    if S I M = then
20:
     break while;
21:
    end if
22:
   end if
23:
   generate a test with the searching strategy and update C I M and S I M ;
24:
  end if
25:
end while
The algorithm is explained in the following aspects.
(1)
Processing of the initial value constraint
When an SUT has an initial value constraint that is presented by E X [ 1 ] , then an initial test case in which each factor is assigned its initial value is generated. Then, the factor value combinations and value sequences of factors covered in the test case are removed from C I M and S I M in line 3. As the generated test case is the first test case in E C M C A , only if t s = 1 , it covers value sequences in S I M . Otherwise, it does not cover value sequences in S I M .
(2)
Population initialization
The E C M C A generation algorithm is initialized by generating a random population position space and a random population velocity space for each particle in line 6. The position of each particle takes the form of a k-dimensional vector, X i 0 = ( x i 1 0 , x i 2 0 , , x i k 0 ) , where each dimension x i j 0 represents a random integer number from the value set [ 0 , v j 1 ] . The velocity of each particle also takes the form of a k-dimensional vector, V i 0 = ( v i 1 0 , v i 2 0 , , v i k 0 ) , where each dimension v i j 0 is also simultaneously initialized with a random integer number between ( v j 1 ) and ( v j 1 ) .
(3)
Population update
During the iteration of the algorithm, velocities and positions of the population particles are updated in line 8 according to Equations 2 and 3. After each iteration, if x i j t + 1 is out of the range [ 0 , v j 1 ] , then x i j t + 1 is updated with the previous value x i j t . If v i j t + 1 is out of the range [ ( v j 1 ) , ( v j 1 ) ] , then v i j t + 1 is updated with a random value in the range. Each particle updates its p B e s t with the solution space where it has its largest fitness factor value thus far. The global best solution g B e s t is updated with the particle that has the larger fitness factor value and satisfies the constraint validity. We use the SAT solver zChaff to verify the combinatorial constraint validity. The SAT solver is initialized in line 1 and the at-most and at-least constraints are generated as the initialization parameters of the SAT solver. The value transition constraints of factors are verified by comparing with S C M .
(4)
Fitness factor value
Fitness factor values are used with PSO in a greedy fashion to identify better particles. A fitness factor value of a test case is the sum of the number of factor value combinations with size t c covered in C I M and the number of value sequences of each factor covered in S I M composed with generated t s 1 test cases.
(5)
Population disturbance
To guarantee the avoidance of the local optimum, after each iteration, one random position in X i t is updated by a random value in line 9. We denote the new particle as X i t * . If the fitness factor value of X i t * is larger than that of X i t , replace X i t with X i t * .
(6)
Searching strategy
After the iteration progresses, if the fitness factor value of g B e s t is 0 , a searching strategy is applied in line 23. At this time, almost a large proportion of factor value combinations are covered and some value sequences of factors may be left. To find the remaining value sequences of factors as early as possible, we present a searching strategy. In the searching strategy, a new test case is constructed one by one factor value to guarantee that it has the potential to cover as many as value sequences of factors or to help the subsequent test case to cover value sequences of factors. The searching strategy is divided into two situations. The first situation is to judge each value of the last generated test case whether belongs to a value sequence uncovered in S I M . If there exists a value sequence in S I M , the value must be guaranteed that it is not the last value in the value sequence. Then, the subsequent value is inserted into a new test case, such that the value a in the last generated test case belongs to a uncovered value sequence ( a , b ) and the value b is inserted into the new test case. If the first situation does not exist, a second situation is performed. A schematic shown in Figure 2 is used to illustrate it. The precondition of the second condition is that each value can reach other values for each factor. Given a generated value x i j in a last generated test case, a traversal hierarchy is constructed based on the values that can be reached. Suppose that ( v j 1 , v j 2 ) is the uncovered value sequence. A path ( x i j , 0 , v j 1 ) can reach v j 1 , and then the value 0 is inserted into the new test case. This process guarantees that, if the fitness factor value of g B e s t is still less than 0 after the next iteration progress, then the first situation of the searching strategy works. It should be noted that when a new value is inserted into the new test case, the new test case’s constraint validity must be satisfied, otherwise a random value that can satisfy the constraint validity is inserted. The searching strategy can guarantee that at least one factor can be updated towards the direction, where the fitness factor value of g B e s t is greater than 0.
(7)
End condition
Whether S C M is empty, the algorithm has two end conditions. If S C M is empty, the algorithm is terminated when all the value sequences of factors and factor value combinations are covered in lines from 15 to 17. If S C M is not empty, the algorithm is terminated only when all the value sequences of factors are covered in lines from 19 to 21. The reason is that, owning to the existence of the constraints of value sequences, some factor value combinations can perhaps not appear in one test case. For example, an SUT has factors A and B, and each factor has values 0, 1 and 2. The input value sequence of each factor is restricted in circles from 0 to 2. Thus, there are only three value combinations ( A = 0 , B = 0 ) , ( A = 1 , B = 1 ) and ( A = 2 , B = 2 ) that exist. When t c = 2 , C I M has 3 × 3 = 9 value combinations and six of them can never be covered. Therefore, to avoid an infinite loop, the end condition is only S I M = .

6. Evaluation Methods of Extended Covering Arrays

6.1. Verification of Constraints

An important problem is the verification of constraints for constructed extended covering arrays. The extended covering arrays that are verified to be valid can be used. According to the two types of constraints, the verification is also divided into two types. One is the verification of constraints in a test case and the other is the verification of constraints among test cases.

6.1.1. Verification of Constraints in a Test Case

The constraints in a test case can be presented in conjunctive normal form and CCTL, such as ( ¬ s u b s c r i p t = t r u e ¬ s u p e r s c r i p t = t r u e ) and E G ( ¬ s u b s c r i p t = t r u e ¬ s u p e r s c r i p t = t r u e ) . The constraint verification is essentially the boolean satisfiability problem, irrespective of which presentation is used. There are two types of tools according to the presentations. For the conjunctive normal form, the verification tools are zChaff [49], Simple Theorem Prover (STP) [50], and MiniSAT [51]. For the CCTL form, the verification tools are Spin [52] and NuSMV [53]. The constraints of factor value combinations, the at-least and at-most constraints are all the inputs of the tools. The tools provide a “true” or “false” result for each test case at the end of the verification process.

6.1.2. Verification of Constraints among Test Cases

Although there is not an effective tool that can be used for CCTL, some formulas can be used for directing the verification of constraints among test cases. For different absolute constraints, different verification measures should be taken. Given m , n N + and m n ,
E C M C A E X [ n ] ϕ
Verify whether the nth test case E C M C A [ n , : ] satisfies ϕ directly.
E C M C A E G [ m , n ] ϕ
We can use the recursion formula E G [ m , n ] ϕ = ϕ E X E G [ m 1 , n 1 ] ϕ to verify E G [ m , n ] ϕ . First, verify whether the first test case is ϕ . Second, regard the next test case as the first test case as the E X . Then, execute the verification process to verify E G [ m 1 , n 1 ] ϕ by repeating the process above.
E C M C A E ( ϕ U [ m , n ] ψ )
First, use the recursion formula E [ ϕ U [ m , n ] ψ ] = ϕ E X E [ ϕ U [ m 1 , n 1 ] ψ ] to transform E [ ϕ U [ m , n ] ψ ] to E [ ϕ U [ 1 , n m ] ψ ] by repeating the process. Then, verify E [ ϕ U [ 1 , n m ] ψ ] with the recursion formula E [ ϕ U [ 1 , n m ] ψ ] = ψ ( ϕ E X E [ ϕ U [ 1 , n m 1 ] ψ ] ) .
E C M C A E F [ m , n ] ϕ
Verify whether there exists a positive integer i with m i n that makes that the ith test case holds ϕ .
To verify the relative constraints, the constraints need to be split into two parts according to the logical operator “→”. Then, verify the condition on the left of “→” first. If the condition holds, verify the constraint on the right of the operator “→”. Once there is one step in the verification process where the E C M C A violates the constraints, the E C M C A is invalid.

6.2. Coverage Measurement

Given a test suite, we expect that the test suite covers all the value combinations among factors and value sequences of each factor with a size that is as small as possible. We can use Formula (4) to calculate the coverage, where the symbol "Covered" indicates the number of targets that have already been covered and "Total" indicates the total targets to be covered in theory:
C o v e r a g e = C o v e r e d T o t a l × 100 % .
Consider the E C A ( n ; t c , t s , k , v ) . It has ( k t c ) v t c factor value combinations to be covered and ( v t s ) t s ! k value sequences with size t s . Thus, there are ( k t c ) v t c + ( v t s ) t s ! k targets to be covered. When considering the constraints, the number of targets to be covered should be less than the targets without considering constraints.
Because coverage measurement is essentially pattern analysis, which has been widely used in many domains [54], we define the coverage measurement with combinatorial coverage and sequence coverage.

6.2.1. Measurement of Combinatorial Coverage

Given a test suite T S and an interaction I = { ( P i 1 , a i 1 ) , ( P i 2 , a i 2 ) , , ( P i t c , a i t c ) } with 1 i j k , 1 j t c , δ t c : T S ( δ I t c ( T S ) ) I H t c N 0 indicates the number of t c -way combinations:
δ I t c ( T S ) = { | r : T S [ r , i j ] = a i j , 1 i j k , 1 j t c | } , I H t c .
δ I t c ( T S ) indicates the number of interactions covered in T S , and H t c is the set of all interactions with 0 δ I t c ( T S ) n . Γ c 1 is a characteristic function used for comparing with δ I t c :
θ I t c ( T S ) = m i n { δ I t c ( T S ) , Γ c } .
Another form of θ I t c ( T S ) is the following:
θ I t c ( T S ) = 0 , δ I t c ( T S ) < Γ c , 1 , δ I t c ( T S ) Γ c .
The kernel function based on θ I t c ( T S ) is as follows:
κ t c ( T S , T S ) = < θ I t c ( T S ) , θ I t c ( T S ) > = I H t c ( θ I t c ( T S ) ) 2 .
There is 0 κ t c ( T S , T S ) | H t c | for the kernel function. If there are no constraints, then 0 κ t c ( T S , T S ) | H t c | = k t c v t c . Thus, the formula to calculate combinatorial coverage is as follows:
C o m b i n a t o r i a l c o v e r a g e = κ t c ( T S , T S ) | H t c | × 100 % = I H t c ( θ I t c ( T S ) ) 2 ( k t c ) v t c × 100 % .

6.2.2. Measurement of Sequence Coverage

Given a test suite T S and for each sequence S i = { e 1 i , e 2 i , , e t s i } ( e j i [ 0 , v i 1 ] , 1 j t s ) of factor P i , the mapping δ t s : T S ( δ S i t s ( T S ) ) S i H t s i N 0 indicates the number of sequences from column i in T S :
δ S i t s ( T S ) = { | { ( S 1 , S 2 ) : T S [ : , i ] = S 1 , S i , S 2 } | } , S i H t s i .
The sequence T S [ : , i ] is viewed as a series of three subsequences S 1 , S i and S 2 . [ 0 , v i 1 ] is the value set of factor P i , and H t s i = [ 0 , v i 1 ] t s is the set of all the sequences with length t s . Γ s 1 is a characteristic function used for comparing with δ S i t s :
θ S i t s ( T S ) = m i n { δ S i t s , Γ s } .
Another form of θ S i t s ( T S ) is as follows:
θ S i t s ( T S ) = 0 , δ S i t s < Γ s , 1 , δ S i t s Γ s .
The kernel function based on θ S i t s ( T S ) is as follows:
κ t s i ( T S , T S ) = < θ S i t s ( T S ) , θ S i t s ( T S ) > = S i H t s i ( θ S i t s ( T S ) ) 2 .
There is 0 κ t s i ( T S , T S ) | H t s i | for the kernel function. If there are no constraints, then 0 κ t s i ( T S , T S ) | H t s i | = v t s t s ! . Thus, the formula to calculate sequence coverage is the following:
S e q u e n c e c o v e r a g e = i = 1 k κ t s i ( T S , T S ) i = 1 k | H t s i | × 100 % = i = 1 k S i H t s i ( θ S i t s ( T S ) ) 2 k ( v t s ) t s ! × 100 % .

7. Experiments

This section describes the experimental results of PEG performed on a benchmark of SUTs. Then, more complex constraints that PEG can not support are discussed.

7.1. Experimental Results of PEG

PEG is developed in the environment that consists of a desktop computer with Windows 7 (Dell, Xiamen, China), 2.6 GHz Core 2 Duo CPU, 2 GB of RAM. It is coded and implemented in Qt Creator 4.8.1 (C++) (Digia, Helsinki, Finland).
For the experiments, we use a benchmark with 12 different SUTs. Six of them are from Ref. [12]. They are the “Keyboard”, the “Microwave”, the “Autoradio”, the “Coffee Machine”, the “Elevator” and the “Transmission”. The reason why we choose them is that they have more than one factor and for each factor of them each value can reach to others. The “text effect“ application of “Microsoft Word” is also chosen as an SUT. Besides the seven real world SUTs, we supplement five SUTs to increase the configuration diversity of SUTs. The details of the SUTs are given in Table 6. The configurations of factors and factor values are listed in the third column. Three kind of constraints are listed from the fourth column to the sixth column.
As PEG depends on some degree of randomness, it is non-deterministic. Thus, we performed 30 independent runs per SUT/coverage criterion for a statistical analysis. We use PEG to generate t c = t s = 2 coverage and t c 2 , t s = 3 coverage, respectively. The results are shown in Table 7 and Table 8. As SUT2 and SUT7 have the configuration of two factors, they can not have test suites with t c > 2 coverage. Thus we use the asterisks “*” to mark the results which are performed with t c = 2 coverage in Table 8. To demonstrate the performance of PEG, best generated sizes, average generated sizes, best generated time and average generated time are presented for each SUT. The average coverage of targets are also reported corresponding to factor value combinations with t c coverage and value sequences of each factor with t s coverage.
Generally, the generated time increases as the number of factors and factor values grows. However, the generated time of SUT1 seems longer than other SUTs. This is mainly because much time is wasted in the calls of the SAT solver under combination constraints. Based on the results obtained, PEG can generate satisfactory results with total coverage when SUTs have no constraints related to value sequences of factors. When SUTs have the constraints related to value sequences of factors, PEG can cover all the target value sequences with covering as many factor value combinations as possible. Overall, the results show that PEG is feasible and useful to generate E C M C A s .

7.2. Discussion

As everyone knows, a fundamental problem with software testing is that testing under all combinations of inputs and preconditions is not feasible, even with a simple product. The SUTs with test cases to be performed contiguously still face this problem. E C A s attempt to use as few test cases as possible to cover as many factor value combinations and value sequences of factors as possible. The purpose of E C A s is to find more system defects. Compared with the manual test suite generation, E C A s can design more comprehensive test suites. E C A s fill the blank of the test case generation method for SUTs with test case to be performed contiguously and are of great significance to ensure the reliability and quality of SUTs. E C A s will be widely applied to many fields with high reliability requirement, such as aviation, spaceflight and weapon industry. In these industries, most of the input instructions of components are messages that consist of some relevant elements and are needed to be performed contiguously. E C A s are very suitable for the element value combinations and value sequences of elements in the messages.
Take an input instruction of a radar as an example. A high coverage test is needed to ensure the radar works normally under various working conditions. Table 9 shows the instruction of the radar under test. The instruction needs to be input contiguously to control its working mode. When scanning mode is “fixed point”, the scanning speed and the sector scan scope needs to be assigned invalid values, and the scan center needs to be assigned a degree in the range of [ 0 , 360 ] . When scanning mode is “sector scan”, the scanning speed needs to be assigned a valid speed, the sector scan scope needs to be assigned a valid scan scope, and the scan center needs to be assigned a degree in the range of [ 0 , 360 ] . When scanning mode is “circular scan”, the scanning speed needs to be assigned a valid speed, the sector scan scope and the scan center needs to be assigned an invalid value. E C A s are feasible to be used as the test suites, and E C A s can improve the test coverage dramatically compared to design test suites manually. There is a large amount of input messages in the industry like the previous instruction and high coverage test suites are needed for those messages. Therefore, we believe that E C A s have a broad application prospect.
However, PEG still needs to be improved in practice for more complex contraints. The SUTs whose test cases need to be performed contiguously usually have the three kinds of constraints that are the prerequisites of PEG. More complex constraints may be exist though they have hardly been seen in real world systems, such as the constraints between factors values from one test step to another [32]. Refs. [12,32] have put forward the requirement of generating test suites under complex constraints and described some complex constraints as follows:
  • If value c i from factor C is selected in test case t n , then value c j from factor C must be selected in the succeeding test step t n + 1 .
  • If C = c i in t n , then C = c j in a later t n + m .
  • If C = c i in t n , then C = c j in all t n + 1 to t n + m .
  • If C = c i in t n , then C = c j in all t n + m to t n + o .
  • If C = c i or B = b k in t n , then D d in a later t n + m .
These complex constraints can all be presented in CCTL as follows. As they do not give the configurations of factors and factor values, we could not construct their test suites:
  • E G ( E X [ n ] ( C = c i ) E X [ n + 1 ] ( C = c j ) ) ,
  • E G ( E X [ n ] ( C = c i ) E X [ n + m ] ( C = c j ) ) ,
  • E G ( E X [ n ] ( C = c i ) E X E G [ 1 , m ] ( C = c j ) ) ,
  • E G ( E X [ n ] ( C = c i ) E X [ m 1 ] E G [ 1 , o m ] ( C = c j ) ) ,
  • E G ( E X [ n ] ( C = c i B = b k ) ¬ E X [ n + m ] ( D = d j ) ) .
Ref. [33] has presented a simplified real world system with complex constraints that is a simplified controller of a car. It has two boolean inputs that represent the user’s decision to accelerate or brake. Upon acceleration, the car starts moving, with either a slow or fast velocity. Upon braking, the car immediately stops. The velocity is also a factor of the example. Figure 3 depicts the values of the three factors that impact the car controller.
As to pedal the brake and accelerator at the same time should be avoided when driving, the value combination of “accelerate = true” and “brake = true” is a constraint for the car controller. This value combination constraint can be denoted as E G ( ¬ a c c e l e r a t e = t r u e ¬ b r a k e = t r u e ) . For each factor, there are also at-least and at-most constraints. Figure 4 shows the states and the transitions of states [33]. A constraint that can be denoted as E G E X [ 1 ] ( a c c e l e r a t e = f a l s e b r a k e = f a l s e v e l o c i t y = s t o p ) restricts the first state S 0 of the car controller. Figure 5 shows the value transitions of each factor. When “velocity = stop” holds in a test case, “velocity=fast” cannot occur in the next test case. The constraint is E G ( v e l o c i t y = s t o p ¬ E X ( v e l o c i t y = f a s t ) ) . When “velocity = slow” holds in a test case, “velocity=slow” cannot occur in the next test case. The constraint is E G ( v e l o c i t y = s l o w ¬ E X ( v e l o c i t y = s l o w ) ) . Integrated with seven other complex constraints that are given in [33], all the constraints consist of the constraint set C in E C M C A ( n ; t c , t s , 2 2 3 1 , C ) as follows:
C = { E G ( ¬ a c c e l e r a t e = t r u e ¬ b r a k e = t r u e )   E G ( ¬ a c c e l e r a t e = t r u e ¬ a c c e l e r a t e = f a l s e )   E G ( a c c e l e r a t e = t r u e a c c e l e r a t e = f a l s e )   E G ( ¬ b r a k e = t r u e ¬ b r a k e = f a l s e )   E G ( b r a k e = t r u e b r a k e = f a l s e )   E G ( ¬ v e l o c i t y = s t o p ¬ v e l o c i t y = s l o w )   E G ( ¬ v e l o c i t y = f a s t ¬ v e l o c i t y = s l o w )   E G ( ¬ v e l o c i t y = f a s t ¬ v e l o c i t y = s t o p )   E G ( v e l o c i t y = f a s t v e l o c i t y = s t o p v e l o c i t y = s l o w )   E G E X [ 1 ] ( a c c e l e r a t e = f a l s e b r a k e = f a l s e v e l o c i t y = s t o p )   E G ( v e l o c i t y = s t o p ¬ E X ( v e l o c i t y = f a s t ) )   E G ( v e l o c i t y = s l o w ¬ E X ( v e l o c i t y = s l o w ) )   E G ( a c c e l e r a t e = t r u e b r a k e = f a l s e v e l o c i t y = s t o p E X ( v e l o c i t y = s l o w ) )   E G ( a c c e l e r a t e = t r u e b r a k e = f a l s e v e l o c i t y = s l o w E X ( v e l o c i t y = f a s t ) )   E G ( a c c e l e r a t e = t r u e b r a k e = f a l s e v e l o c i t y = f a s t E X ( v e l o c i t y = f a s t ) )   E G ( a c c e l e r a t e = f a l s e b r a k e = f a l s e v e l o c i t y = f a s t E X ( v e l o c i t y = s l o w ) )   E G ( a c c e l e r a t e = f a l s e b r a k e = f a l s e v e l o c i t y = s l o w E X ( v e l o c i t y = s t o p ) )   E G ( a c c e l e r a t e = f a l s e b r a k e = f a l s e v e l o c i t y = s t o p E X ( v e l o c i t y = s t o p ) )   E G ( b r a k e = t r u e E X ( v e l o c i t y = s t o p ) ) } .
The E C M C A ( 12 ; 2 , 2 , 2 2 3 1 , C ) of the car controller presented in Table 10 is generated by PEG without considering complex constraints. The combinatorial coverage of this E C M C A ( 12 ; 2 , 2 , 2 2 3 1 , C ) is κ 1 ( E C M C A ( 12 ; 2 , 2 , 2 2 3 1 , C ) , E C M C A ( 12 ; 2 , 2 , 2 2 3 1 , C ) ) | H 2 | × 100 % = 15 16 = 93.75 % , as the combination {accelerate=true, brake=true} is a constraint that is forbidden to appear. The sequence coverage of this E C M C A ( 12 ; 2 , 2 , 2 2 3 1 , C ) is i = 1 3 κ 2 i ( E C M C A ( 12 ; 2 , 2 , 2 2 3 1 , C ) , E C M C A ( 12 ; 2 , 2 , 2 2 3 1 , C ) ) i = 1 3 | H 2 i | × 100 % = 15 17 × 100 % = 88.24 % , as "velocity=stop" → "velocity=fast" and "velocity=slow" → "velocity=slow" are value sequence constraints. As PEG cannot handle complex constraints, the E C M C A ( 12 ; 2 , 2 , 2 2 3 1 , C ) in Table 10 violates complex constraints. The first and second test cases violate the constraint E G ( a c c e l e r a t e = f a l s e b r a k e = f a l s e v e l o c i t y = s t o p E X ( v e l o c i t y = s t o p ) ) , as when the first test is (accelerate=false, brake=false, velocity=stop), the second test case violates the E X ( v e l o c i t y = s t o p ) . In the same way, the second and third test cases violate E G ( b r a k e = t r u e E X ( v e l o c i t y = s t o p ) ) . The third and fourth test cases violate E G ( a c c e l e r a t e = t r u e b r a k e = f a l s e v e l o c i t y = f a s t E X ( v e l o c i t y = f a s t ) ) . The fourth and fifth test cases violate E G ( a c c e l e r a t e = t r u e b r a k e = f a l s e v e l o c i t y = s l o w E X ( v e l o c i t y = f a s t ) ) . The seventh and eighth test cases violate E G ( b r a k e = t r u e E X ( v e l o c i t y = s t o p ) ) . The ninth and tenth test cases violate E G ( a c c e l e r a t e = f a l s e b r a k e = f a l s e v e l o c i t y = s t o p E X ( v e l o c i t y = s t o p ) ) . The tenth and eleventh test cases violate E G ( a c c e l e r a t e = f a l s e b r a k e = f a l s e v e l o c i t y = s l o w E X ( v e l o c i t y = s t o p ) ) . The eleventh and twelfth test case violate E G ( a c c e l e r a t e = f a l s e b r a k e = f a l s e v e l o c i t y = f a s t E X ( v e l o c i t y = s l o w ) ) . To illustrate an E C M C A that satisfies the complex constraints, an E C M C A ( 12 ; 2 , 2 , 2 2 3 1 , C ) is constructed manually as shown in Table 11. As the configuration is simple, we can construct it manually. The E C M C A shown in Table 11 satisfies all the constraints and covers all the value combinations and value sequences of each factors.
Through the analysis above, though PEG can produce satisfactory E M C A s , PEG still has some limitations to generate E C M C A s for SUTs with complex constraints. The emerging complex constraints can be specified by CCTL actually. It is certain that more real world SUTs with complex constraints are needed and analyzed, so that complex constraints can be classified and handled reasonably.

8. Conclusions

In this paper, we have proposed extended covering arrays with t-way combinatorial coverage and t-wise sequence coverage for SUTs whose test cases need to be performed contiguously. In extended covering arrays, we have introduced the clocked computation tree logic based formal specification method for specifying constraints. A Particle swarm optimization based Extended covering array Generator (PEG) that can produce feasible and useful E C M C A s with common constraints has also been presented. The performance of PEG is assessed considering benchmark experiments. For generated test suites, the method for verifying constraints’ validity has been presented corresponding to the constraint specification method. Moreover, kernel functions that can measure the coverage of generated test suites have be given. Compared with the manual test suite generation, E C A s can design more comprehensive test suites, which improves the possibility of finding system defects. In a word, E C A s fill the blank of the test case generation method for SUTs with test case to be performed contiguously and are of great significance to ensure the reliability and quality of SUTs. Though some deficiencies exists, we still believe that E C A s can have a broad application prospect through continuous progress in practice.
As part of our future work, we will first optimise PEG to cover more possible value combinations under the constraints of value sequences, then try to find real world SUTs with complex constraints and extend PEG to support them. Compared to the fixed strength combinatorial testing, the variable strength combinatorial testing usually considers the actual interaction relationship in software sufficiently. Therefore, E C A s with variable strength are also worthy of study.

Author Contributions

The authors contributed equally to this work and wrote this paper together.

Acknowledgments

The authors also gratefully acknowledge the helpful comments and suggestions of the reviewers, which have improved the presentation.

Conflicts of Interest

The authors declare no conflict of interest.

References

  1. Ferrer, J.; Kruse, P.M.; Chicano, F.; Alba, E. Search based algorithms for test sequence generation in functional testing. Inf. Softw. Technol. 2014, 58, 419–432. [Google Scholar] [CrossRef]
  2. Tassey, G. The economic impacts of inadequate infrastructure for software testing. Natl. Inst. Stand. Technol. 2002, 15, 125. [Google Scholar]
  3. Kacker, R.N.; Kuhn, D.R.; Lei, Y.; Lawrence, J.F. Combinatorial testing for software: An adaptation of design of experiments. Measurement 2013, 46, 3745–3752. [Google Scholar] [CrossRef]
  4. Nie, C.; Leung, H. A survey of combinatorial testing. ACM Comput. Surv. 2011, 43, 33–63. [Google Scholar] [CrossRef]
  5. Kuhn, D.R.; Kacker, R.N.; Lei, Y. Practical combinatorial testing. Nist Spec. Publ. 2010, 10, 19–23. [Google Scholar]
  6. Kuhn, D.R.; Bryce, R.; Duan, F.; Ghandehari, L.S.; Lei, Y.; Kacker, R.N. Combinatorial testing: Theory and practice. Adv. Comput. 2015, 99, 1–66. [Google Scholar]
  7. Cohen, D.M.; Dalal, S.R.; Fredman, M.L.; Patton, G.C. The aetg system: An approach to testing based on combinatorial design. IEEE Trans. Softw. Eng. 1997, 23, 437–444. [Google Scholar] [CrossRef]
  8. Kuhn, D.R.; Wallace, D.R.; Gallo, A.M. Software fault interactions and implications for software testing. IEEE Trans. Softw. Eng. 2004, 30, 418–421. [Google Scholar] [CrossRef]
  9. Kuhn, D.R.; Kacker, R.N.; Lei, Y. Introduction to Combinatorial Testing; Springer: Berlin/Heidelberg, Germany, 2013. [Google Scholar]
  10. Farchi, E.; Segall, I.; Tzorefbrill, R.; Zlotnick, A. Combinatorial testing with order requirements. In Proceedings of the IEEE Seventh International Conference on Software Testing, Verification and Validation Workshops, Cleveland, OH, USA, 31 March–4 April 2014; pp. 118–127. [Google Scholar]
  11. Zamli, K.Z.; Othman, R.R.; Zabil, M.H.M. On sequence based interaction testing. In Computers and Informatics; Universiti Tenaga Nasional: Selangor, Malaysia, 2011; pp. 662–667. [Google Scholar]
  12. Kruse, P.M. Enhanced Test Case Generation With the Classification Tree Method; Freie Universität Berlin: Berlin, Germany, 2014. [Google Scholar]
  13. Ahmed, B.S.; Zamli, K.Z.; Lim, C.P. Application of particle swarm optimization to uniform and variable strength covering array construction. Appl. Soft Comput. 2012, 12, 1330–1347. [Google Scholar] [CrossRef]
  14. Qnd. Available online: http://csrc.nist.gov/groups/SNS/acts/sequencecovarrays.html (accessed on 5 February 2018).
  15. Kuhn, D.R.; Higdon, J.M.; Lawrence, J.F.; Kacker, R.N.; Lei, Y. Efficient methods for interoperability testing using event sequences. Crosstalk J. Def. Softw. Eng. 2012, 25, 15–18. [Google Scholar]
  16. Kuhn, D.R.; Higdon, J.M.; Lawrence, J.F.; Kacker, R.N.; Lei, Y. Combinatorial methods for event sequence testing. In Proceedings of the IEEE Fifth International Conference on Software Testing, Verification and Validation, Montreal, QC, Canada, 17–21 April 2012; pp. 601–609. [Google Scholar]
  17. Yu, L.; Lei, Y.; Kacker, R.N.; Kuhn, D.R.; Lawrence, J. Efficient algorithms for t-way test sequence generation. In Proceedings of the International Conference on Engineering of Complex Computer Systems, Paris, France, 18–20 July 2012; pp. 220–229. [Google Scholar]
  18. Hazli, M.Z.M.; Zamli, K.Z.; Othman, R.R. Sequence-based interaction testing implementation using bees algorithm. In Proceedings of the 2012 IEEE Symposium on Computers & Informatics (ISCI), Penang, Malaysia, 18–20 March 2012; pp. 81–85. [Google Scholar]
  19. Rahman, M.M.; Othman, R.R.; Ahmad, R.B.; Rahman, M.M. A meta heuristic search based t-way event driven input sequence test case generator. Int. J. Simul. Syst. Sci. Technol. 2014, 15, 65–71. [Google Scholar]
  20. Nguyen, C.D.; Marchetto, A.; Tonella, P. Combining model-based and combinatorial testing for effective test case generation. In Proceedings of the International Symposium on Software Testing and Analysis, Minneapolis, MN, USA, 15–20 July 2012; pp. 100–110. [Google Scholar]
  21. Grochtmann, M.; Grimm, K. Classification trees for partition testing. Softw. Test.Verif. Reliab. 1993, 3, 63–82. [Google Scholar] [CrossRef]
  22. Conrad, M.; Fey, I.; Sadeghipour, S. Systematic model-based testing of embedded automotive software. Electron. Notes Theor. Comput. Sci. 2005, 111, 13–26. [Google Scholar] [CrossRef]
  23. Mattner, B. Testona. Available online: http://www.testona.net/ (accessed on 1 March 2018).
  24. Razorcat, Tessy. Available online: http://www.razorcat.com/downloads-tessy.html (accessed on 1 March 2018).
  25. Büchner, F. Test case design using the classification tree method. ATZelektronik Worldw. 2007, 2, 15–17. [Google Scholar]
  26. Lamberg, K.; Beine, M.; Eschmann, M.; Otterbach, R.; Conrad, M.; Fey, I. Model-based testing of embedded automotive software using mtest. J. Passeng. Carselectron. Electr. Syst. 2004, 7, 132–140. [Google Scholar]
  27. Mjeda, A.; Mcelligott, P.; Ryan, K.; Thiel, S. Reactive Model-Based Testing Design for Embedded Automotive Software; Sae International: Hongkong, China, 2008. [Google Scholar]
  28. Conrad, M.; Krupp, A. An extension of the classification-tree method for embedded systems for the description of events. Electron. Notes Theor. Comput. Sci. 2006, 164, 3–11. [Google Scholar] [CrossRef]
  29. Kruse, P.M.; Nasarek, J.; Fernandez, N.C. Systematic testing of web applications with the classification tree method. In Proceedings of the XVII Iberoamerican Conference on Software Engineering, Pucón, Chile, 23–25 April 2014; pp. 219–232. [Google Scholar]
  30. Artho, C.V.; Biere, A.; Hagiya, M.; Platon, E.; Seidl, M.; Tanabe, Y.; Yamamoto, M. Modbat: A Model-based API tester for event-driven systems, verification and testing. In Proc. 9th Haifa Verification Conference; Springer: Cham, Switzerland, 2013; pp. 112–128. [Google Scholar]
  31. Anand, S.; Burke, E.K.; Chen, T.Y.; Clark, J.; Cohen, M.B.; Grieskamp, W.; Harman, M.; Harrold, M.J.; Mcminn, P. An orchestrated survey on automated software test case generation. J. Syst. Softw. 2013, 86, 1978–2001. [Google Scholar] [CrossRef]
  32. Kruse, P.M.; Wegener, J. Test sequence generation from classification trees. In Proceedings of the IEEE Fifth International Conference on Software Testing, Verification and Validation, Montreal, QC, Canada, 17–21 April 2012; pp. 539–548. [Google Scholar]
  33. Fraser, G.; Wotawa, F.; Ammann, P.E. Testing with model checkers: A survey. Softw. Test. Verif. Reliab. 2009, 19, 215–261. [Google Scholar] [CrossRef]
  34. Schooljan, H. Test Sequence Validation and Generation Using Classification Trees; Delft University of Technology: Delft, The Netherlands, 2013. [Google Scholar]
  35. Krupp, A.; Mueller, W. Modelchecking von klassifikationsbaum-testsequenzen. In Proceedings of the GI/ITG/GMM Workshop “Methoden und Beschreibungssprachen zur Modellierung und Verifikation von Schaltungen und Systemen”, Munich, Germany, 5–7 April 2005. [Google Scholar]
  36. Yilmaz, C.; Fouche, S.; Cohen, M.B.; Porter, A.; Demiroz, G.; Koc, U. Moving forward with combinatorial interaction testing. Computer 2014, 47, 37–45. [Google Scholar] [CrossRef] [Green Version]
  37. Yilmaz, C. Test case-aware combinatorial interaction testing. IEEE Trans. Softw. Eng. 2013, 39, 684–706. [Google Scholar] [CrossRef]
  38. Martínez, C.; Moura, L.; Panario, D.; Stevens, B. Locating errors using eLAs, covering arrays, and adaptive testing algorithms. Siam J. Discret. Math. 2010, 23, 1776–1799. [Google Scholar] [CrossRef]
  39. Sheng, Y.; Wei, C.; Jiang, S. Constraint test cases generation based on particle swarm optimization. Int. J. Reliab. Qual. Saf. Eng. 2017, 24, 1750021-1–1750021-21. [Google Scholar] [CrossRef]
  40. Alsewari, A.R.A.; Zamli, K.Z. Design and implementation of a harmony-search-based variable-strength t-way testing strategy with constraints support. Inf. Softw. Technol. 2012, 54, 553–568. [Google Scholar] [CrossRef]
  41. Yu, L.; Lei, Y.; Nourozborazjany, M.; Kacker, R.N.; Kuhn, D.R. An efficient algorithm for constraint handling in combinatorial test generation. In Proceedings of the 2013 IEEE Sixth International Conference on Software Testing, Verification and Validation, Luxembourg, Luxembourg, 18–22 March 2013; pp. 242–251. [Google Scholar]
  42. Yu, L.; Duan, F.; Lei, Y.; Kacker, R.N. Constraint handling in combinatorial test generation using forbidden tuples. In Proceedings of the IEEE Eighth International Conference on Software Testing, Verification and Validation Workshops, Graz, Austria, 13–17 April 2015; pp. 1–9. [Google Scholar]
  43. Danziger, P.; Mendelsohn, E.; Moura, L.; Stevens, B. Covering arrays avoiding forbidden edges. Theor. Comput. Sci. 2008, 410, 5403–5414. [Google Scholar] [CrossRef]
  44. Ruf, J.; Kropf, T. Symbolic model checking for a discrete clocked temporal logic with intervals. In Advances in Hardware Design and Verification; Springer: Boston, MA, USA, 1997; pp. 146–163. [Google Scholar]
  45. Ruf, J.; Kropf, T. Modeling and Checking Networks of Communicating Real-Time Processes. In Correct Hardware Design and Verification Methods; Springer: Berlin/Heidelberg, Germany, 2008; pp. 267–279. [Google Scholar]
  46. Flake, S.; Müller, W.; Ruf, J. Mapping of Structured English Sentences To Cctl Formulae; University of Paderborn: Paderborn, Germany, 2000. [Google Scholar]
  47. Cohen, M.B.; Dwyer, M.B.; Shi, J. Constructing interaction test suites for highly-configurable systems in the presence of constraints: A greedy approach. IEEE Trans. Softw. Eng. 2008, 34, 633–650. [Google Scholar] [CrossRef]
  48. Kennedy, J.; Eberhart, R. Particle swarm optimization. In Proceedings of the IEEE International Conference on Neural Networks, Perth, Australia, 27 November–1 December 1995; pp. 1942–1948. [Google Scholar]
  49. S. R. Group, Zchaff. Available online: https://www.princeton.edu/~chaff/zchaff.html (accessed on 20 February 2018).
  50. Cadar, C.; Ganesh, V.; Pawlowski, P.M.; Dill, D.L.; Engler, D.R. Exe: Automatically generating inputs of death. In ACM Transactions on Information and System Security (TISSEC); ACM: New York, NY, USA, 2008; Volume 12, pp. 1–38. [Google Scholar]
  51. Niklas, E.; Niklas, S. Minisat. Available online: http://minisat.se/ (accessed on 20 February 2018).
  52. Spin. Available online: http://spinroot.com/spin/whatispin.html (accessed on 20 February 2018).
  53. Nusmv. Available online: http://nusmv.fbk.eu/ (accessed on 20 February 2018).
  54. Last, M. Kernel Methods for Pattern Analysis; China Machine Press: Beijing, China, 2005. [Google Scholar]
Figure 1. The generation of C C M , C I M , S C M and S I M .
Figure 1. The generation of C C M , C I M , S C M and S I M .
Symmetry 10 00146 g001
Figure 2. A schematic of the second situation of the searching strategy.
Figure 2. A schematic of the second situation of the searching strategy.
Symmetry 10 00146 g002
Figure 3. The factor values of the simplified car controller presented as a classification tree.
Figure 3. The factor values of the simplified car controller presented as a classification tree.
Symmetry 10 00146 g003
Figure 4. The state transitions of the car controller.
Figure 4. The state transitions of the car controller.
Symmetry 10 00146 g004
Figure 5. The value transitions of each factor.
Figure 5. The value transitions of each factor.
Symmetry 10 00146 g005
Table 1. C M C A ( 7 ; 2 , 2 2 3 1 , F = ( 0 , 0 , x ) ) .
Table 1. C M C A ( 7 ; 2 , 2 2 3 1 , F = ( 0 , 0 , x ) ) .
ABC
1010
2011
3012
4100
5101
6102
7110
Table 2. A 2-wise sequence coverage test suite.
Table 2. A 2-wise sequence coverage test suite.
ABC
1100
2100
3012
4012
5101
6101
7100
8011
9012
10100
Table 3. E M C A ( 7 ; 2 , 2 , 2 2 3 1 ) .
Table 3. E M C A ( 7 ; 2 , 2 , 2 2 3 1 ) .
ABC
1100
2100
3012
4012
5101
6011
7010
8011
9102
10110
Table 4. An E C M C A ( 9 ; 2 , 2 , 4 1 2 1 , C ) of the “VideoGame”.
Table 4. An E C M C A ( 9 ; 2 , 2 , 4 1 2 1 , C ) of the “VideoGame”.
startingGamePause
1startingGamerunning
2startingGamepaused
3startuprunning
4startuppaused
5controllingrunning
6controllingpaused
7gameOverrunning
8gameOverpaused
9startingGamerunning
Table 5. An E C M C A ( 9 ; 2 , 2 , 2 7 , C ) of the “text effect” application.
Table 5. An E C M C A ( 9 ; 2 , 2 , 2 7 , C ) of the “text effect” application.
SubscriptSuperscriptStrikethroughDouble StrikethroughAll CapsSmall CapsShadow
1truefalsefalsetruefalsetruetrue
2falsetruetruefalsetruefalsetrue
3falsetruefalsetruefalsefalsefalse
4truefalsetruefalsefalsetruefalse
5truefalsefalsefalsetruefalsefalse
6falsefalsefalsetruetruefalsetrue
7falsetruefalsetruefalsetruetrue
8falsetruetruefalsefalsetruetrue
9truefalsetruefalsetruefalsetrue
Table 6. General characteristics of systems under test.
Table 6. General characteristics of systems under test.
SUTNameConfigurationConstraints
Combination ConstraintsSequence ConstraintsInitial Value Constraints
1text effect 2 7 300
2Keyboard 2 2 002
3Microwave 7 1 4 1 2 1 0333
4Autoradio 11 1 2 1 3 1 0843
5Coffee Machine 9 1 3 2 0633
6Elevator 5 1 2 1 4 1 2 1 0314
7Transmission 4 1 3 1 0152
8 5 5 000
9 6 6 000
10 8 8 000
11 3 2 4 2 2 3 5 1 000
12 4 6 5 6 000
Table 7. Results of PEG with t c = 2 and t s = 2 .
Table 7. Results of PEG with t c = 2 and t s = 2 .
SUTSizeTime (Second)Factor Value CombinationsValue Sequences of Factors
Best SizeAverage SizeBest TimeAverage Time | CIM | Average Covered | SIM | Covered
189.239.21124.4681812828
255.90.230.284488
345473.413.545048.93636
48284.46.256.46158.15050
56063.74.534.826358.53636
61013.31.131.496047.11818
779.40.20.28125.81010
83638.63.243.48250250125125
95758.76.757.04540540216216
10110113.221.8122.3817921792512512
113031.910.2711.072692698787
124748.533.3334.2113351335246246
Table 8. Results of PEG with t c 2 and t s = 3 .
Table 8. Results of PEG with t c 2 and t s = 3 .
SUTSizeTime (Second)Factor Value CombinationsValue Sequences of Factors
Best SizeAverage SizeBest TimeAverage Time | CIM | Average Covered | SIM | Covered
12022.4509.66995.72502505656
21113.22.560.674 *4 *1616
3175183.612.6213.095655.3113113
435335625.0825.546661.6173173
5123142.78.810.28165.57777
62934.53.34.0311677.62828
71314.70.420.4812 *6 *1616
8195199.222.4923.1512501250625625
9374383.174.3476.144320432012961296
1010881096.7578.37587.7228,67228,67240964096
11140148102.12107.3316271627331331
12285291.1740.88766.2419,98019,98011341134
* t c = 2 .
Table 9. An instruction of a radar under test.
Table 9. An instruction of a radar under test.
Scanning ModeScanning SpeedSector Scan ScopeScan Center
1fixed pointinvalid valueinvalid valueinvalid value
2sector scan1 degree per second 2 90 degreea degree in [ 0 , 360 ]
3circular scan2 degree per second 2 90 degree
4 5 degree per second 0 275 degree
5 10 degree per second 0 275 degree
6 12 degree per second
Table 10. An E C M C A ( 12 ; 2 , 2 , 2 2 3 1 , C ) generated by PEG without complex constraints.
Table 10. An E C M C A ( 12 ; 2 , 2 , 2 2 3 1 , C ) generated by PEG without complex constraints.
AccelerateBrakeVelocity AccelerateBrakeVelocity
1falsefalsestop7falsetrueslow
2falsetrueslow8falsetruefast
3truefalsefast9falsefalsestop
4truefalseslow10falsefalseslow
5falsetruestop11falsefalsefast
6truefalsestop12falsefalsefast
Table 11. An E C M C A ( 12 ; 2 , 2 , 2 2 3 1 , C ) generated manually.
Table 11. An E C M C A ( 12 ; 2 , 2 , 2 2 3 1 , C ) generated manually.
AccelerateBrakeVelocity AccelerateBrakeVelocity
1falsefalsestop7falsetruefast
2falsetruestop8truefalsestop
3falsetruestop9truefalseslow
4truefalsestop10falsefalsefast
5truefalseslow11falsetrueslow
6truefalsefast12falsefalsestop

Share and Cite

MDPI and ACS Style

Sheng, Y.; Sun, C.; Jiang, S.; Wei, C. Extended Covering Arrays for Sequence Coverage. Symmetry 2018, 10, 146. https://doi.org/10.3390/sym10050146

AMA Style

Sheng Y, Sun C, Jiang S, Wei C. Extended Covering Arrays for Sequence Coverage. Symmetry. 2018; 10(5):146. https://doi.org/10.3390/sym10050146

Chicago/Turabian Style

Sheng, Yunlong, Chao Sun, Shouda Jiang, and Chang’an Wei. 2018. "Extended Covering Arrays for Sequence Coverage" Symmetry 10, no. 5: 146. https://doi.org/10.3390/sym10050146

Note that from the first issue of 2016, this journal uses article numbers instead of page numbers. See further details here.

Article Metrics

Back to TopTop