Orthogonal array
Encyclopedia
Orthogonal array testing is a black box testing
technique which is a systematic, statistical way of software testing
.It is used when the number of inputs to the system is relatively small, but too large to allow for exhaustive testing of every possible input to the systems
. It is particularly effective in finding errors associated with faulty logic
within computer
software systems.Orthogonal arrays can be applied in user interface
testing, system testing
, regression
testing, configuration testing and performance testing
.
The permutations of factor levels comprising a single treatment are so chosen that their responses are uncorrelated & hence each treatment gives unique piece of information
. The net effect of organizing the experiment in such treatments is that the same piece of information is gathered in the minimum number of experiments.
. Orthogonal vectors exhibit the following properties:
The difference is that Orthogonal Array test cases are uniformly distributed over the test domain and this provides significant cost advantages when you look at the end-to-end testing process. Pairwise testing can sometimes yield fewer test points than Orthogonal Array testing; however, the additional cost of defect analysis can far outweigh the potential reduction in execution cost. Since test cases created by the pairwise method can be unbalanced, it requires significantly more time to isolate the root causes of defects. In addition, it becomes very challenging to effectively assess performance in terms of the statistical properties like mean and variance. In fact, to effectively conduct analysis of faults and results, users have to run several additional test cases if they originally ran Pairwise tests. Orthogonal Arrays distribute test cases uniformly in the multidimensional test domain and defect analysis and performance assessments are very straightforward. This significantly reduces the expert labor effort required to analyze test results
Given that assumption, the table shows the set of nine combination of parameters which are sufficient to catch the fault considering the interaction of the input parameters, which is very effective and economical. The array is orthogonal because all possible pair-wise combinations between parameters occurs only once.
The given L9 Orthogonal Array assess result of test cases as follows,
Single Mode Faults - Single mode faults occur only due to one parameter. For example, in above Orthogonal array if test cases 7, 8 and 9 show error, we can expect that value 3 of parameter 1 is causing the error.Likewise we can detect as well as isolate the error.
Double Mode Fault - Double mode fault is caused by the two specific parameters values interacting together. Such a interaction is a harmful interaction between interacting parameters.
Multimode Faults - More than two interacting components if produce the consistent erroneous output, then it is a multimode fault. Orthogonal array detects the multimode faults.
Black box testing
Black-box testing is a method of software testing that tests the functionality of an application as opposed to its internal structures or workings . Specific knowledge of the application's code/internal structure and programming knowledge in general is not required...
technique which is a systematic, statistical way of software testing
Software testing
Software testing is an investigation conducted to provide stakeholders with information about the quality of the product or service under test. Software testing can also provide an objective, independent view of the software to allow the business to appreciate and understand the risks of software...
.It is used when the number of inputs to the system is relatively small, but too large to allow for exhaustive testing of every possible input to the systems
Software system
A software system is a system based on software forming part of a computer system . The term "software system" is often used as a synonym of computer program or software; is related to the application of systems theory approaches in software engineering context and are used to study large and...
. It is particularly effective in finding errors associated with faulty logic
Logic
In philosophy, Logic is the formal systematic study of the principles of valid inference and correct reasoning. Logic is used in most intellectual activities, but is studied primarily in the disciplines of philosophy, mathematics, semantics, and computer science...
within computer
Computer
A computer is a programmable machine designed to sequentially and automatically carry out a sequence of arithmetic or logical operations. The particular sequence of operations can be changed readily, allowing the computer to solve more than one kind of problem...
software systems.Orthogonal arrays can be applied in user interface
User interface
The user interface, in the industrial design field of human–machine interaction, is the space where interaction between humans and machines occurs. The goal of interaction between a human and a machine at the user interface is effective operation and control of the machine, and feedback from the...
testing, system testing
System testing
System testing of software or hardware is testing conducted on a complete, integrated system to evaluate the system's compliance with its specified requirements. System testing falls within the scope of black box testing, and as such, should require no knowledge of the inner design of the code or...
, regression
Regression analysis
In statistics, regression analysis includes many techniques for modeling and analyzing several variables, when the focus is on the relationship between a dependent variable and one or more independent variables...
testing, configuration testing and performance testing
Performance testing
In software engineering, performance testing is in general testing performed to determine how a system performs in terms of responsiveness and stability under a particular workload...
.
The permutations of factor levels comprising a single treatment are so chosen that their responses are uncorrelated & hence each treatment gives unique piece of information
Information
Information in its most restricted technical sense is a message or collection of messages that consists of an ordered sequence of symbols, or it is the meaning that can be interpreted from such a message or collection of messages. Information can be recorded or transmitted. It can be recorded as...
. The net effect of organizing the experiment in such treatments is that the same piece of information is gathered in the minimum number of experiments.
Orthogonal Vector
All orthogonal vectors exhibit orthogonalityOrthogonality
Orthogonality occurs when two things can vary independently, they are uncorrelated, or they are perpendicular.-Mathematics:In mathematics, two vectors are orthogonal if they are perpendicular, i.e., they form a right angle...
. Orthogonal vectors exhibit the following properties:
- Each of the vectors conveys information different from that of any other vector in the sequence, i.e., each vector conveys unique information therefore avoiding redundancy.
- On a linear addition, the signals may be separated easily.
- Each of the vectors is statistically independent of the others, i.e., the correlation between them is nil.
- When linearly added, the resultant is the arithmetic sum of the individual components.
Orthogonal Array testing vs. Pairwise testing
Pairwaise testing describes a process for defining test vectors that ensure that all two parameter combinations are covered and it is often confused with Orthogonal Array testing. They both have a key similarity in that they ensure that all pairs of two parameters are covered, but they also have a key difference.The difference is that Orthogonal Array test cases are uniformly distributed over the test domain and this provides significant cost advantages when you look at the end-to-end testing process. Pairwise testing can sometimes yield fewer test points than Orthogonal Array testing; however, the additional cost of defect analysis can far outweigh the potential reduction in execution cost. Since test cases created by the pairwise method can be unbalanced, it requires significantly more time to isolate the root causes of defects. In addition, it becomes very challenging to effectively assess performance in terms of the statistical properties like mean and variance. In fact, to effectively conduct analysis of faults and results, users have to run several additional test cases if they originally ran Pairwise tests. Orthogonal Arrays distribute test cases uniformly in the multidimensional test domain and defect analysis and performance assessments are very straightforward. This significantly reduces the expert labor effort required to analyze test results
Technique
Consider a system which has 3 parameters and each of them has 3 values. To test the all possible combination (i.e. exhaustive testing) we will need a set of 27 test cases.But instead of testing the system for each combination of parameters, orthogonal array select the combination effectively and uniformly. With orthogonal array technique we can maximize the test coverage while minimum number of test cases..We here assume that the pair that the test that maximizes interaction between the parameters, will have more defects and the technique works.| Test case ↓ | Parameter 1 | Parameter 2 | Parameter 3 |
|---|---|---|---|
| 1 | 1 | 1 | 3 |
| 2 | 1 | 2 | 2 |
| 3 | 1 | 3 | 1 |
| 4 | 2 | 1 | 2 |
| 5 | 2 | 2 | 1 |
| 6 | 2 | 3 | 3 |
| 7 | 3 | 1 | 1 |
| 8 | 3 | 2 | 3 |
| 9 | 3 | 3 | 2 |
Given that assumption, the table shows the set of nine combination of parameters which are sufficient to catch the fault considering the interaction of the input parameters, which is very effective and economical. The array is orthogonal because all possible pair-wise combinations between parameters occurs only once.
The given L9 Orthogonal Array assess result of test cases as follows,
Single Mode Faults - Single mode faults occur only due to one parameter. For example, in above Orthogonal array if test cases 7, 8 and 9 show error, we can expect that value 3 of parameter 1 is causing the error.Likewise we can detect as well as isolate the error.
Double Mode Fault - Double mode fault is caused by the two specific parameters values interacting together. Such a interaction is a harmful interaction between interacting parameters.
Multimode Faults - More than two interacting components if produce the consistent erroneous output, then it is a multimode fault. Orthogonal array detects the multimode faults.
Benefits
- Reduces testing cycle time and analysis is simpler.
- Balanced test cases ensure straightforward defect isolation and performance assessments. This provides a significant cost savings over pair-wise testing
Important Notes
- The only means to guarantee coverage of all defects is to test all possible combinations. As this is not feasible within budget and schedule, testers generally select a "sample: of the total test domain. Orthogonal arrays provide a means to effectively generate high coverage samples to validate the test domain.