MVSCA: Multi-Valued Sequence Covering Array

This paper discusses the limitation of both Sequence Covering Array (SCA) and Covering Array (CA) for testing reactive system when the order of parameter-values is sensitive. In doing so, this paper proposes a new model to take the sequence values into consideration. Accordingly, by superimposing the CA onto SCA yields another type of combinatorial test suite termed MultiValued Sequence Covering Array (MVSCA) in a more generalized form. This superimposing is a challenging process due to NP-Hardness for both SCA and CA. Motivated by such a challenge, this paper presents the MVSCA with a working illustrative example to show the similarities and differences among combinatorial testing methods. Consequently, the MVSCA is a new trend that can be a research vehicle for researchers to develop new and/or modify existing combinatorial strategies to deal with the combinatorial explosion problem raised by the MVSCA.


INTRODUCTION
Exhaustive system testing is a beneficial activity for quality and reliability enhancements; however, due to resources and timing constraints, exhaustive system testing is prohibitively impossible , Zamli and Younis, 2010. Earlier works adopted combinatorial testing as an efficient black-box technique for system testing for any system under test (SUT) that reduces the size of an exhaustive test suite significantly in a systematic manner. The generation of test suite involves covering the interaction elements (for parameter values) of a certain degree of parameter interaction strength (t). Currently, there are two types of combinatorial interaction testing, namely: covering array (CA) and sequence covering array (SCA) ,Nasser, et al., 2018.
With the advent of the Internet of Things (IoT), enormous sensors and actuators produce many dynamic events. These events may have multiple values. For instance, a temperature sensor may read a temperature less than or equal to a threshold value, or greater than a threshold value. Similar notes could be observed for other sensors and actuators ,Younis and Hussein, 2018. In addition, the events with their corresponding values can be entered asynchronously out of order (i.e., the parameter-values are non-adjacent) which fits the demands of the derivation test beds and quality assurance test suites for IoT systems. This raises the need for a new combinatorial testing array that takes into consideration the sequences and their parameter-values, we call it a Multi-Valued Sequence Covering Array (MVSCA). This paper illustrates the similarities and differences between CA, SCA, and the proposed MVSCA. The remaining sections of this paper are organized as follows. Section 2 gives a mathematical presentation for combinatorial testing. Section 3, gives a step-by-step-example and further discussion regarding non-uniform parameters' values. Section 4 summarizes the trends in related works. Finally, Section 5 states the conclusion and gives future trends in combinatorial test data generation.

MATHEMATICAL NOTATIONS
Mathematically, the CA takes parameters of N, t, p, and v witch corresponding to the size, strength of coverage, number of parameters, and uniform-value respectively , Cohen, 2004;Lei, et al., 2008; Younis and Zamli, 2011. The number of parameter interaction is determined by Eq. (1). The number of t-way interaction elements (tuples) per parameter interaction is determined by Eq. (2). Thus, the total number of tuples is determined by Eq. (3). The CA is an N*p matrix in which each t-way combination tuples occurs at least one row. It should be mentioned that the parameters are in order (i.e., adjacent). The SCA takes the parameters of N, t, and p , Kuhn, et al., 2012;Chee, et al., 2013. Similar to CA the number of parameter interaction is determined by Eq. (1). However, the SCA takes the permutation of parameters (sequences), thus the number of tuples per parameter interaction is given in Eq. (4). Thus, the total number of tuples is determined by Eq. (5). The SCA is an N*p matrix in which each t-way permutation tuples occurs at least one row; the t symbols in the permutation may be out of order (i.e., are not required to be adjacent).
The MVCA like CA, MVSCA takes parameters of N, t, p, and v. Similar to both CA and SCA the number of parameter interaction is determined by Eq. (1). Unlike both CA and SCA, MVSCA takes every interaction elements and their permutation to generate t-way tuples. Hence, the number of tuples per parameter interaction is given in Eq. (6). Thus, the total number of tuples is determined by Eq. (7). The MVSCA is an N*p matrix in which each t-way permutation tuples occurs at least one row; the t-way permutation of the interaction elements may be out of order (i.e., are not required to be adjacent).
Equations 2,4, and 6 represent the lower bound for CA, SCA, and MVCA respectively.

Test Case Number
Looking at Tables 1 and 2, it is clear that the generated tuples of CA are a subset of the MVSCA's tuples (i.e., the first eight tuples in Table 1 is identical to Table 2). In fact, if only adjacent tuples of the MVSCA is taken into consideration, the MVSCA degrades to CA. Looking at Tables 1 and  3, if only one parameter-value is taken into consideration (e.g., the values a0, b0, and c0), the MVSCA degrades to CA. Taking the exhaustive space into account, it is clear that the number of tuples in MVSCA is very large as compared to both CA and SCA. In fact, the number of tuples in MVSCA is the product of CA's tuples and SCA's tuples. In order to reduce the exhaustive test suites when relaxing the interaction coverage to 2-way (pairwise), we adopt an exhaustive greedy strategy (EGS). In EGS, the exhaustive test suite is generated as a candidate test cases pool. In addition, the generation of all t-way interaction elements (tuples) is followed by searching for test cases that cover the maximum number of uncovered tuples. The generated test case is added to the test suite in one-test-at-a-time fashion. This process is repeated for each test case until all tuples are covered. Moreover, EGS takes an intelligent parameter (combinatorial types) to control the generation of tuples as well as the exhaustive pool, and hence the generated test suite to be either CA, SCA, or MVSCA. Return to our working example, the interaction parameters are unique as specified by Eq. (1) (i.e., AB, AC, and BC). However, the tuples are different according to the type of combinatorial testing method as tabulated in Table 4. Moreover, the exhaustive test suites are different also as tabulated in Tables 1, 2, and 3. The number of tuples per parameter interaction is determined by equations 2,4, and 6 for CA, SCA, and MVSCA respectively (i.e., 4, 2, and 8 respectively). The total number of tuples are determined by equations 3,5, and 7 for CA, SCA, and MVSCA respectively (i.e., 12, 6, and 24 respectively). Following the steps of the EGS, the generated test suites and the corresponding covered tuples are tabulated in Tables 5,6, and 7 for MVSCA, CA, and SCA respectively.    To illustrate the construction of test cases for MVSCA, the candidate test cases pool is given in Table 1. The maximum number of tuples per test case can be determined by Eq. (1) (i.e., three in our example). The first candidate test case in Table 1 (a0b0c0) covers three tuples; therefore; it is added to the test set and the tuples covered are deleted from the interaction elements set. Next, it is deleted from the pool. The next candidate test case is a0b0c1 it only covers two tuples (i.e., a0c1 and b0c1 because the tuple a0b0 already covered). The same observation is true for the test case It should be mentioned that the parameters'-values could be non-uniformed for the system under the test. In this situation, the total number of tuples is determined as the sum of multiplications for each t-way interaction values. For instance, let's consider a system with seven parameters with mixed configuration values denoted by CA (N, 3, (5 2 4 2 2 3 )). In this example, there are two parameters with five values, two parameters with four values, and three parameters with two values. An alternative representation of this system could be CA (N, 3, (5 5 4 4 2 2 2)). According to Eq. (1) there are 35 3-way parameter interactions. Thus, the total number of tuples in CA (N, 3, (5 2 4 2 2 3 )) = [5*5*4+5*5*4+5*5*2+5*5*2+5*5*2+5*4*4+5*4*2+5*4*2+5*4*2+5*4*2+5*4*2+5*4*2+5*2 *2+5*2*2+5*2*2+5*4*4+5*4*2+5*4*2+5*4*2+5*4*2+5*4*2+5*4*2+5*2*2+5*2*2+5*2*2+4 *4*2+4*4*2+4*4*2+4*2*2+4*2*2+4*2*2+4*2*2+4*2*2+4*2*2+2*2*2] =1310. When take the t-way sequence into account, for the MVSCA (N, 3, (5 2 4 2 2 3 )); there are 3! =6 possible permutations for each 3-way interaction elements, hence, the total number of tuples in this case= 1310 *6=7860 tuples.

RESEARCH TRENDS IN COMBINATORIAL TESTING
To the best of author knowledge, the notation of MVSCA is not discussed in the literature. In short, earlier works in combinatorial testing mentioned that the generation of a test suite is considered as NP-