Pairwise Testing Example 1: Classic Example
A classic testing problem discussed in many classes, tutorials and books is the so-called Weinberg-Myers Triangle problem, or the triangle problem for short. Glenford J. Myers himself used the problem in the classic The Art of Software Testing already in 1979.
Pairwise testing might not be the best technique for this testing problem, but being a kind of Hello World problem for testing, it can be an introductory example to understand the basics of pairwise testing and to see how pairwise testing is different than other testing techniques.
The Testing Problem
The triangle problem is quite simple. The system under test is a function that takes three parameters and returns a string. The three parameters are the three sides of a triangle and the return value is a classification of the triangle; for example, a triangle might be classified as “Right”, “Isosceles”, “Equilateral”, “Scalene”, “Degenerate” or “Invalid”.
Classification of Valid Triangles
There are also two kinds of invalid triangle classifications:
- If two of the sides together are longer than the third, the three line segments will not form a triangle shape.
- if two sides are exactly as long as the third, then the triangle will look like a line-segment, and is therefore classified as degenerate, as it degenerates to a line segment.
The System under Test
There are many implementations of the triangle problem available online, so let us use one of them as a system under test. This JavaScript implementation by Quality Tree Software is the implementation we will use.
The implementation to test is found in triangle.js. It depends on raphael.js and jQuery.js.
The implementation uses an object-oriented style, so let us wrap it in a function that we can test.
function classifyTriangle(a, b, c){
var triangle = new Triangle();
triangle.setSides(a, b, c);
var type = triangle.getTriangleType();
return type;
}
Testing Setup
In order to run tests, we need to setup a testing environment. As the system under test is written in JavaScript, we can simply create a small HTML page (let us call it triangleTest.html) that will include the system under test and the script with the tests (let us call it triangleTester.js) that will output the results of the tests to the HTML page itself.
<!DOCTYPE html>
<html lang="en">
<head>
<meta charset="utf-8">
<title>Tringle Tester</title>
<script src="https://ajax.googleapis.com/ajax/libs/jquery/2.1.3/jquery.min.js"></script>
<script src="http://practice.agilistry.com/javascripts/raphael.js" type="text/javascript"></script>
<script src="http://practice.agilistry.com/javascripts/triangle.js" type="text/javascript"></script>
<script src="triangleTester.js" type="text/javascript"></script>
</head>
<body>
<h1>Tringle Tester</h1>
<pre id="results"></pre>
<div id="canvas"></div>
</body>
</html>
Here, the system under test as well as its dependencies has been included from their original locations.
The pre called results will show the results, and the div called canvas is a requirement of the system under test.
function classifyTriangle(a, b, c){
var triangle = new Triangle();
triangle.setSides(a, b, c);
var type = triangle.getTriangleType();
return type;
}
var exampleTests = [
[5, 5, 5, "Equilateral"],
[5, 4, 4, "Isosceles"],
[5, 4, 3, "Right"],
[5, 4, 6, "Scalene"],
[0, 4, 5, "Invalid"],
[0, 1, 1, "Degenerate"],
];
$( document ).ready( function (){
testTriangle(exampleTests);
})
function testTriangle(tests){
var testResults = "", pass = 0;
for(var i = 0; i < tests.length; i++){
var a = tests[i][0], b = tests[i][1];
var c = tests[i][2], expected = tests[i][3];
var result = classifyTriangle(a, b, c);
testResults += "Test "+(i+1)+": "+a+", "+b+", "+c+" - ";
if(result == expected){
testResults += "pass \n";
pass++;
}else{
testResults += "failed - expected "+expected+" got "+result+" \n";
}
}
var text = pass + " / " + tests.length + " passed" + "\n";
text += (tests.length-pass) + " failed" + "\n";
text += testResults;
$('#results').text(text);
}
First, we see our wrapper function. Then follows an array with tests called exampleTests. We will generate such an array with tests later using pairwise testing. Then follows a jQuery call that is invoked when the whole page has been loaded, which is a good time to execute our tests. Then follows a function that runs each test, logs the result and outputs the results to the pre element of the HTML page.
Opening triangleTest.html in a browser will show the following:
Tringle Tester 6 / 6 passed 0 failed Test 1: 5, 5, 5 - pass Test 2: 5, 4, 4 - pass Test 3: 5, 4, 3 - pass Test 4: 5, 4, 6 - pass Test 5: 0, 4, 5 - pass Test 6: 0, 1, 1 - pass
Generating Tests
We are now ready to generate tests. This is the stage of a project when test generation is usually considered. We have our system under test and we have a basic test setup. The test setup may or may not include the use of a testing framework (for example xUnit).
Finding Parameters
The first step in using pairwise testing is defining the parameters of the system under test, which, in the case of the example discussed in this article is function classifyTriangle(a, b, c). Therefore, our parameters are the three sides, a, b and c.
Picking Values
The three parameters, a, b, and c, are all floating point numbers. Being 32 bits, there are about 4 billion possible values for each floating-point variable. Exhaustive testing is seldom feasible, which is also the case for the triangle problem.
One way to reduce the number of values is to select a few values that approximately represents the rest of the values. This selection is similar to equivalence class testing or boundary value testing.
First of all, the sides of a triangle cannot be negative. A representative of the negative value is -1, so let us include that number. Another special value for lengths is the lowest possible length, namely 0. As for the positive lengths, let us include 1. Let us also include 5 as it is more than the twice of 1. Let us include 3, 4 and 6 so that we can get all the possible kinds of triangles by combining three of the chosen values.
Defining Parameters with Pairwiser
Now we are ready to define out parameters in a pairwise testing tool. In this article, we will use Pairwiser.
Add three parameters and add the values -1, 0, 1, 3, 4, 5, 6 to each:
There are no relevant constraints to add as the system under test is expected to accept all combinations of parameter values.
Include Existing Tests
We already have six existing tests, so there is no need to retest those parts of the system that the existing tests have already tested. In Pairwiser, we can add the six tests as required tests:
In Pairwiser, it will look like this:
Generating Tests
We are now ready to generate pairwise tests.
Let us first generate a 1-wise test suite. 1-wise means that every parameter value is included in at least one test of the test suite. Make sure to include the required tests in order to avoid redundant test cases.
In Pairwiser, the settings and generated 1-wise tests looks like this:
From the analysis of the tests, we can see that the first six tests covered 76% of the values. An additional five tests was needed to achieve full 1-wise coverage.
Formatting the Tests
We are now ready to format the test so that we can add them to our testing program. Let us export the tests as a JavaScript array that we can add to our JavaScript setup. The following test script template is sufficient. It will generate one line for each test case. For each test case, the concrete value of the three parameters a, b and c will be inserted. Notice the syntax: The bar symbols enclose the parameter name signaling to Pairwiser that the concrete value for the parameter is to be inserted there.
Running this template on the test cases produced the following output. The first six tests are our existing test cases.
[5, 5, 5, ""], [5, 4, 4, ""], [5, 3, 4, ""], [5, 4, 6, ""], [0, 4, 5, ""], [0, 1, 1, ""], [3, 1, 0, ""], [4, 3, 6, ""], [1, 0, -1, ""], [-1, -1, 3, ""], [6, 6, 0, ""],
Let us add the five new test cases to our test setup and run it. It gives the following output:
6 / 11 passed 5 failed Test 1: 5, 5, 5 - pass Test 2: 5, 4, 4 - pass Test 3: 5, 4, 3 - pass Test 4: 5, 4, 6 - pass Test 5: 0, 4, 5 - pass Test 6: 0, 1, 1 - pass Test 7: 3, 1, 0 - failed - expected got Invalid Test 8: 4, 3, 6 - failed - expected got Scalene Test 9: 1, 0, -1 - failed - expected got Invalid Test 10: -1, -1, 3 - failed - expected got Invalid Test 11: 6, 6, 0 - failed - expected got Degenerate
There are some things to notice:
- The system under test did not crash.
- The system under test gave a result for each set of inputs.
We can confirm that the results are correct manually by adding them as expected outputs. And, indeed, the five new answers are correct.
Pairwise Testing – a.k.a. 2-wise, 2-way or all-pairs testing
Now, 1-wise is the least thorough kind of combinatorial testing. 1-wise makes sure every one value of every parameter is included in at least one test. For example, in the new five tests above, we can see that the value of -1 was not in any of the original six tests, so they are included in tests 9 and 10.
Now, pairwise, or 2-wise, ensures that every pair of values of every pair of parameters are include together in at least one test. This excludes combinations that cannot occur because of some constraint of course.
To generate pairwise tests in Pairwiser, simply select 2-wise and press generate. In Pairwiser, we now get 63 tests. Now, if you think this is a high number of tests for such a simple problem, you are right. Pairwise testing is not well suited for a testing problem with a few parameters with relatively many combinations. As we will see in other examples, pairwise testing performs very well when we have more parameters, which the most interesting, real industrial cases surely tend to have!
The first pairwise tests for the triangle problem looks like this:
The following graph shows the increase in pairwise coverage as more tests are added. The first five tests cover about 55% of the pairs.
The graph is typical for a pairwise test suite. It gets harder and harder to cover the remaining pairs. This is what makes pairwise testing thorough but it also makes the pairwise tests hard to compute. Trying to make a pairwise test suite by hand is impossible even for the smaller cases, let alone the industrial cases. Thus, pairwise testing really leverages the computational power to do thorough testing.
Let us now export the tests as a JavaScript array by running the test script template on the generated tests. We get the following tests which we can add to our test setup.
[5, 5, 5, ""], [5, 4, 4, ""], [5, 3, 4, ""], [5, 4, 6, ""], [0, 4, 5, ""], [0, 1, 1, ""], [1, -1, 0, ""], [-1, 3, 6, ""], [6, 4, 0, ""], [6, -1, 3, ""], [-1, 0, 1, ""], [4, -1, -1, ""], [6, 6, 5, ""], [-1, 5, 1, ""], [1, 4, -1, ""], [0, -1, 6, ""], [4, 6, 4, ""], [3, 6, -1, ""], [6, 5, 6, ""], [6, 3, 6, ""], [5, 1, 0, ""], [4, 0, 3, ""], [3, 4, 1, ""], [1, -1, 5, ""], [0, 3, -1, ""], [5, 0, 3, ""], [1, 0, 6, ""], [-1, 5, -1, ""], [-1, 4, 3, ""], [0, -1, 1, ""], [0, 5, 3, ""], [3, 5, 6, ""], [4, 3, 6, ""], [1, 3, 1, ""], [1, 1, 4, ""], [6, 1, 5, ""], [4, 5, 4, ""], [5, -1, 1, ""], [4, 0, 5, ""], [-1, -1, 4, ""], [3, -1, 5, ""], [3, 3, 0, ""], [1, 5, 3, ""], [6, 0, 4, ""], [5, 6, 6, ""], [0, 5, 0, ""], [4, 1, -1, ""], [3, 0, 3, ""], [0, 0, 4, ""], [-1, 1, 3, ""], [4, 4, 1, ""], [1, 6, 0, ""], [-1, 6, 5, ""], [3, 1, 6, ""], [0, 6, 3, ""], [0, 3, 5, ""], [-1, 0, 0, ""], [5, 0, -1, ""], [6, 3, 3, ""], [6, 6, 1, ""], [3, 5, 4, ""], [6, 6, -1, ""], [4, 6, 0, ""],
Running these gives the following output.
6 / 63 passed 57 failed Test 1: 5, 5, 5 - pass Test 2: 5, 4, 4 - pass Test 3: 5, 4, 3 - pass Test 4: 5, 4, 6 - pass Test 5: 0, 4, 5 - pass Test 6: 0, 1, 1 - pass Test 7: 1, -1, 0 - failed - expected got Invalid Test 8: -1, 3, 6 - failed - expected got Invalid Test 9: 6, 4, 0 - failed - expected got Invalid Test 10: 6, -1, 3 - failed - expected got Invalid Test 11: -1, 0, 1 - failed - expected got Invalid Test 12: 4, -1, -1 - failed - expected got Invalid Test 13: 6, 6, 5 - failed - expected got Isosceles Test 14: -1, 5, 1 - failed - expected got Invalid Test 15: 1, 4, -1 - failed - expected got Invalid Test 16: 0, -1, 6 - failed - expected got Invalid Test 17: 4, 6, 4 - failed - expected got Isosceles Test 18: 3, 6, -1 - failed - expected got Invalid Test 19: 6, 5, 6 - failed - expected got Isosceles Test 20: 6, 3, 6 - failed - expected got Isosceles Test 21: 5, 1, 0 - failed - expected got Invalid Test 22: 4, 0, 3 - failed - expected got Invalid Test 23: 3, 4, 1 - failed - expected got Degenerate Test 24: 1, -1, 5 - failed - expected got Invalid Test 25: 0, 3, -1 - failed - expected got Invalid Test 26: 5, 0, 3 - failed - expected got Invalid Test 27: 1, 0, 6 - failed - expected got Invalid Test 28: -1, 5, -1 - failed - expected got Invalid Test 29: -1, 4, 3 - failed - expected got Invalid Test 30: 0, -1, 1 - failed - expected got Invalid Test 31: 0, 5, 3 - failed - expected got Invalid Test 32: 3, 5, 6 - failed - expected got Scalene Test 33: 4, 3, 6 - failed - expected got Scalene Test 34: 1, 3, 1 - failed - expected got Invalid Test 35: 1, 1, 4 - failed - expected got Invalid Test 36: 6, 1, 5 - failed - expected got Degenerate Test 37: 4, 5, 4 - failed - expected got Isosceles Test 38: 5, -1, 1 - failed - expected got Invalid Test 39: 4, 0, 5 - failed - expected got Invalid Test 40: -1, -1, 4 - failed - expected got Invalid Test 41: 3, -1, 5 - failed - expected got Invalid Test 42: 3, 3, 0 - failed - expected got Degenerate Test 43: 1, 5, 3 - failed - expected got Invalid Test 44: 6, 0, 4 - failed - expected got Invalid Test 45: 5, 6, 6 - failed - expected got Isosceles Test 46: 0, 5, 0 - failed - expected got Invalid Test 47: 4, 1, -1 - failed - expected got Invalid Test 48: 3, 0, 3 - failed - expected got Degenerate Test 49: 0, 0, 4 - failed - expected got Invalid Test 50: -1, 1, 3 - failed - expected got Invalid Test 51: 4, 4, 1 - failed - expected got Isosceles Test 52: 1, 6, 0 - failed - expected got Invalid Test 53: -1, 6, 5 - failed - expected got Invalid Test 54: 3, 1, 6 - failed - expected got Invalid Test 55: 0, 6, 3 - failed - expected got Invalid Test 56: 0, 3, 5 - failed - expected got Invalid Test 57: -1, 0, 0 - failed - expected got Invalid Test 58: 5, 0, -1 - failed - expected got Invalid Test 59: 6, 3, 3 - failed - expected got Degenerate Test 60: 6, 6, 1 - failed - expected got Isosceles Test 61: 3, 5, 4 - failed - expected got Right Test 62: 6, 6, -1 - failed - expected got Invalid Test 63: 4, 6, 0 - failed - expected got Invalid
Again, we can note that the system under test did not crash, and that it gave results for all test cases. We can go through the list to confirm that the answers are correct and add them to the expected result to see all test cases pass.
Also note that we got all classifications produced except Equilateral, which might be argued is the simplest case.
We now have a pairwise test suite for the triangle problem that we can rerun in the future when we enhance or refactor the code.
No bugs yet but …
We have not found any bugs yet. It is not a surprise, as the solution to the triangle problem is not only easy, but also well understood and solved repeatedly. Now, even though we have not found any bugs, having performed testing has a few benefits:
- We have exercised our system, and increased our confidence in its correctness.
- We have a test suite we can easily rerun in the future.
- We have documented that the system performs correctly for many varied cases.
3-wise testing
Now, we can perform even more thorough testing. 3-wise testing is a kind of combinatorial testing stronger than pairwise testing. It ensures that all combinations of three values of every three parameters occur in at least one test.
Now, as we only have three parameters, when every combination of three values of three parameters is included in at least one test, we get exhaustive testing (with respect to the selected values). As we have three parameters with seven values each, the total number of possible combinations is 7^3 = 343, and indeed that is what we get if we generate the 3-wise tests.
Conclusion
In this article, we saw how to apply pairwise testing (or in general combinatorial testing) to one of the classic testing problems, the Weinberg-Myers Triangle problem. We saw how to utilize many features of a pairwise testing tool, in this case Pairwiser, to generate the pairwise tests and how to export them using a template to a JavaScript testing setup.
Pairwise testing was combined with a kind of equivelence class selection technique to reduce the number of values to include for generating the pairwise tests. This is a quite common technique to combine with pairwise testing. For this example, the selection of value was what reduced the size of the testing problem the most. The pairwise tests were 18% of all possible combinations of the selected values, so it also reduced the number a bit.
For larger and more relevant examples, pairwise testing reduces the number of tests significantly. Check out the other examples to see how to use pairwise testing on its home turf: It is when we have many parameters and many constraints.
Relevant Links
- http://www.testing-challenges.org/Weinberg-Myers+Triangle+Problem
- http://www.testingeducation.org/conference/wtst3_collard4.pdf
- http://www.softwaretestpro.com/Item/4533/Nifty-Triangle-Test-Example/Software-Testing
- http://www.testinggeek.com/software-testing-testometer-triangle-test