Pairwise Testing Example 2: Testing a Website

In this example, we are testing a website. Pairwise testing is very well suited for this testing problem.

The Testing Problem

A common problem for website developers and designer is website compatibility. In this example, we will test a website’s compatibility on mobile and desktop, on various browsers, operating systems, window sizes and server settings such as minification and caching.

Interaction faults between these things are a common problem. For example, that using https on a certain mobile browser gives an error, a pairwise fault, or that minification causes the website to look wrong in Internet Explorer, another pairwise fault. Pairwise testing ensures that all of these pairs is exercised.

System under Test

The system under test could have been any website, and the pairwise tests generated below is probably applicable for many websites.

For this example, however, we will test the Pairwiser home page inductive.no/pairwiser. Below we can see the website accessed on a few different devices and browsers.

 

Defining Parameters

In order to generate pairwise tests we will use Pairwiser.

The primary parameter for a website is which device it will be used on. Let us test on two primary devices: Mobile and desktop. This the first parameter is “Device” and its values are are “Mobile” and “Desktop”.

Device: Mobile, Desktop

The second parameter is browser. We will differentiate between mobile browsers and desktop browsers as browsers are typically specially designed for each device type. Let us chose the most common desktop and mobile browsers:

Desktop Browser: Chrome, Firefox, Internet Explorer, Opera, Safari, n/a
Mobile Browser: Chrome, Android, Opera, Safari, n/a

Notice that we added an n/a value to each. This is because a mobile browser is not applicable (n/a) when the device is not a mobile. To tell Pairwiser this, we need to express this as constraints:

always together "Device" is not "Mobile" with "Mobile Browser" is "n/a" 
always together "Device" is not "Desktop" with "Desktop Browser" is "n/a"

What these mean is that, for example, a device not being mobile always must occur together with the Mobile Browser being n/a.

Let’s do the same for connection types and OSes:

Desktop Connection: wifi, cable, n/a
Mobile Connection: edge, 4g, wifi, n/a
Desktop OS: Windows, Linux, Mac, n/a
Mobile OS: iOS, Android, n/a

And we need some constraints again.

always together "Device" is not "Desktop" with "Desktop OS" is "n/a" 
always together "Device" is not "Mobile" with "Mobile OS" is "n/a"

always together "Device" is not "Desktop" with "Desktop Connection" is "n/a" 
always together "Device" is not "Mobile" with "Mobile Connection" is "n/a"

The android browser is only available on Android and Internet Explorer is only available on Windows, so we need to tell this to Pairwiser using two if constraints:

if "Mobile Browser" is "Android" then "Mobile OS" is "Android"
if "Desktop Browser" is "Internet Explorer" then "Desktop OS" is "Windows"

An if-constraint says that if the first parameter has a certain value, then the second parameter must have a certain value. Note the difference with the always-together-constraints in that they need not always occur together.

There are several options available to the client. It can use IP version 4 or 6, it can request http or https, it can use www in front of the domain name or not, it can have cookies enabled or not, the windows dimensions can be wide, tall or square, the window size can be large or small and the system language can be English or non-English.

IP version: IPv4, IPv6
protocol: http, https
www in front: yes, no
Cookies: enabled, disabled
Window Dimentions: wide, tall, square
Windows Size: large, small
Language: english, non-english

For mobiles, the windows size is small. We can express that with an if-constraint.

if "Device" is "Mobile" then "Windows Size" is "Small"

Finally, we add two server configurations that often cause problems, but that also varies throughout the life time of a website: caching and minifcation of CSS and JS. Both of these are typically on, but might be turned off during development and testing, and we want to make sure that the website keeps functioning even if one or both are off.

Server Caching: on, off
Server Minification: on, off

This is how the final parameters and constraints look in Pairwiser:

Required Tests

As developers regularly test their website on their desktop computer and on their mobile phones, let’s include them as required tests. This will ensure that the remaining tests do not have to include the combinations already covered by these tests.

This is how the required tests look in Pairwiser:

Pairwise Tests

We can now generate the pairwise tests. In Pairwiser, check the box for including the required tests and select 2-wise from the drop-down. Pairwiser generates 31 tests that include every pair of interaction among the parameter values just discussed, except, of course, the ones that violate the constraints.

Here are the ten first tests from Pairwiser:

Analysis of the Tests

The tests generated by Pairwiser has the following accumulated coverage. Notice that the first 12 tests covers 95% of the pairs of interactions.

There are in total 7 564 pairs that needs to be covered. Pairwise calls this T-sets, and as web can see, Pairwiser’s test suite covers them all.

Test Script Template

Checking that a website looks good is something that is not necessarily easy to automate. For the 30 pairwise tests generated for the website, we can do them manually.

Let us then generate test scripts for a person to follow. We will start the template with the test number.

Test |testnr|.

This tells Pairwiser to replace the two bar-symbols with “testnr” in between with the current test number.

Next, we instruct the tester to select a device with a certain OS.

- Get a |Device| in front of you with |if "Device" is "Mobile"||Mobile OS||else||Desktop OS||endif|.

Notice the if-statement within the bars. This tells Pairwiser to include the Mobile OS if we are on a mobile device and if not, the desktop OS.

We can now instruct the tester to configure the network connection and the system language.

- Set network connection to |if "Device" is "Mobile"||Mobile Connection||else||Desktop Connection||endif| and make sure it is connected.

- In the network options, make sure |IP version| will be used.

- On the |if "Device" is "Mobile"|mobile phone|else|desktop machine|endif|, set system language to |Language|.

It is now time to configure the server.

|if "Server Caching" is "yes"|
- Activate Server chaching.
|else|
- Disable Server chaching.
|endif|

|if "Server Minification" is "yes"|
- Activate Server minification of JavaScript and CSS.
|else|
- Disable Server minification.
|endif|

We can now instruct the tester to open the correct browser and set the correct window size.

- Open |if "Device" is "Mobile"||Mobile Browser||else||Desktop Browser||endif|.
- Set window size to |Window Dimentions| and |Window Size|.

Finally, direct the tester to the correct address. He er she can then verify whether the website looks acceptable.

- Open URL "|protocol|://|if "www in front" is "yes"|www.|endif|inductive.no/pairwiser".
- Check that everything looks good.

Test Script

Here is the complete test script generated by Pairwiser:

Test 1.

– Get a Desktop in front of you with Windows.
– In the network options, make sure IPv4 will be used.
– Set network connection to cable and make sure it is connected.
– On the desktop machine, set system language to english.
– Open Chrome.
– Set window size to wide and large.
– Open URL “https://inductive.no/pairwiser”.
– Check that everything looks good.

Test 2.

– Get a Mobile in front of you with Android.
– In the network options, make sure IPv4 will be used.
– Set network connection to wifi and make sure it is connected.
– On the mobile phone, set system language to english.
– Open Chrome.
– Set window size to tall and small.
– Open URL “https://inductive.no/pairwiser”.
– Check that everything looks good.

Test 3.

– Get a Mobile in front of you with iOS.
– In the network options, make sure IPv6 will be used.
– Set network connection to edge and make sure it is connected.
– On the mobile phone, set system language to non-english.
– Open Opera.
– Set window size to wide and small.
– Open URL “http://inductive.no/pairwiser”.
– Check that everything looks good.

Test 4.

– Get a Desktop in front of you with Windows.
– In the network options, make sure IPv6 will be used.
– Set network connection to cable and make sure it is connected.
– On the desktop machine, set system language to english.
– Open Internet Explorer.
– Set window size to wide and large.
– Open URL “http://www.inductive.no/pairwiser”.
– Check that everything looks good.

Test 5.

– Get a Desktop in front of you with Linux.
– In the network options, make sure IPv6 will be used.
– Set network connection to wifi and make sure it is connected.
– On the desktop machine, set system language to non-english.
– Open Firefox.
– Set window size to tall and large.
– Open URL “https://inductive.no/pairwiser”.
– Check that everything looks good.

Test 6.

– Get a Mobile in front of you with iOS.
– In the network options, make sure IPv4 will be used.
– Set network connection to wifi and make sure it is connected.
– On the mobile phone, set system language to non-english.
– Open Safari.
– Set window size to tall and small.
– Open URL “http://www.inductive.no/pairwiser”.
– Check that everything looks good.

Test 7.

– Get a Mobile in front of you with Android.
– In the network options, make sure IPv4 will be used.
– Set network connection to edge and make sure it is connected.
– On the mobile phone, set system language to english.
– Open Android.
– Set window size to tall and small.
– Open URL “https://inductive.no/pairwiser”.
– Check that everything looks good.

Test 8.

– Get a Desktop in front of you with Mac.
– In the network options, make sure IPv4 will be used.
– Set network connection to wifi and make sure it is connected.
– On the desktop machine, set system language to non-english.
– Open Safari.
– Set window size to wide and large.
– Open URL “https://www.inductive.no/pairwiser”.
– Check that everything looks good.

Test 9.

– Get a Mobile in front of you with Android.
– In the network options, make sure IPv6 will be used.
– Set network connection to wifi and make sure it is connected.
– On the mobile phone, set system language to non-english.
– Open Android.
– Set window size to wide and small.
– Open URL “http://www.inductive.no/pairwiser”.
– Check that everything looks good.

Test 10.

– Get a Desktop in front of you with Windows.
– In the network options, make sure IPv6 will be used.
– Set network connection to cable and make sure it is connected.
– On the desktop machine, set system language to non-english.
– Open Opera.
– Set window size to wide and large.
– Open URL “http://www.inductive.no/pairwiser”.
– Check that everything looks good.

Test 11.

– Get a Mobile in front of you with iOS.
– In the network options, make sure IPv4 will be used.
– Set network connection to 4g and make sure it is connected.
– On the mobile phone, set system language to english.
– Open Safari.
– Set window size to wide and small.
– Open URL “https://www.inductive.no/pairwiser”.
– Check that everything looks good.

Test 12.

– Get a Desktop in front of you with Linux.
– In the network options, make sure IPv4 will be used.
– Set network connection to cable and make sure it is connected.
– On the desktop machine, set system language to english.
– Open Safari.
– Set window size to wide and large.
– Open URL “http://inductive.no/pairwiser”.
– Check that everything looks good.

Test 13.

– Get a Mobile in front of you with Android.
– In the network options, make sure IPv6 will be used.
– Set network connection to edge and make sure it is connected.
– On the mobile phone, set system language to non-english.
– Open Chrome.
– Set window size to wide and small.
– Open URL “http://www.inductive.no/pairwiser”.
– Check that everything looks good.

Test 14.

– Get a Desktop in front of you with Mac.
– In the network options, make sure IPv6 will be used.
– Set network connection to wifi and make sure it is connected.
– On the desktop machine, set system language to english.
– Open Opera.
– Set window size to wide and large.
– Open URL “http://inductive.no/pairwiser”.
– Check that everything looks good.

Test 15.

– Get a Desktop in front of you with Windows.
– In the network options, make sure IPv4 will be used.
– Set network connection to cable and make sure it is connected.
– On the desktop machine, set system language to non-english.
– Open Chrome.
– Set window size to tall and large.
– Open URL “http://www.inductive.no/pairwiser”.
– Check that everything looks good.

Test 16.

– Get a Mobile in front of you with Android.
– In the network options, make sure IPv6 will be used.
– Set network connection to 4g and make sure it is connected.
– On the mobile phone, set system language to english.
– Open Chrome.
– Set window size to tall and small.
– Open URL “https://inductive.no/pairwiser”.
– Check that everything looks good.

Test 17.

– Get a Desktop in front of you with Linux.
– In the network options, make sure IPv4 will be used.
– Set network connection to cable and make sure it is connected.
– On the desktop machine, set system language to english.
– Open Firefox.
– Set window size to wide and large.
– Open URL “http://www.inductive.no/pairwiser”.
– Check that everything looks good.

Test 18.

– Get a Desktop in front of you with Windows.
– In the network options, make sure IPv4 will be used.
– Set network connection to wifi and make sure it is connected.
– On the desktop machine, set system language to non-english.
– Open Opera.
– Set window size to tall and large.
– Open URL “https://www.inductive.no/pairwiser”.
– Check that everything looks good.

Test 19.

– Get a Mobile in front of you with Android.
– In the network options, make sure IPv4 will be used.
– Set network connection to wifi and make sure it is connected.
– On the mobile phone, set system language to english.
– Open Opera.
– Set window size to tall and small.
– Open URL “https://inductive.no/pairwiser”.
– Check that everything looks good.

Test 20.

– Get a Desktop in front of you with Mac.
– In the network options, make sure IPv6 will be used.
– Set network connection to wifi and make sure it is connected.
– On the desktop machine, set system language to non-english.
– Open Safari.
– Set window size to tall and large.
– Open URL “https://inductive.no/pairwiser”.
– Check that everything looks good.

Test 21.

– Get a Mobile in front of you with Android.
– In the network options, make sure IPv6 will be used.
– Set network connection to edge and make sure it is connected.
– On the mobile phone, set system language to non-english.
– Open Safari.
– Set window size to tall and small.
– Open URL “http://inductive.no/pairwiser”.
– Check that everything looks good.

Test 22.

– Get a Desktop in front of you with Mac.
– In the network options, make sure IPv6 will be used.
– Set network connection to cable and make sure it is connected.
– On the desktop machine, set system language to english.
– Open Chrome.
– Set window size to tall and large.
– Open URL “http://www.inductive.no/pairwiser”.
– Check that everything looks good.

Test 23.

– Get a Mobile in front of you with iOS.
– In the network options, make sure IPv4 will be used.
– Set network connection to 4g and make sure it is connected.
– On the mobile phone, set system language to non-english.
– Open Chrome.
– Set window size to wide and small.
– Open URL “http://www.inductive.no/pairwiser”.
– Check that everything looks good.

Test 24.

– Get a Desktop in front of you with Windows.
– In the network options, make sure IPv4 will be used.
– Set network connection to wifi and make sure it is connected.
– On the desktop machine, set system language to non-english.
– Open Internet Explorer.
– Set window size to tall and large.
– Open URL “https://inductive.no/pairwiser”.
– Check that everything looks good.

Test 25.

– Get a Mobile in front of you with Android.
– In the network options, make sure IPv4 will be used.
– Set network connection to 4g and make sure it is connected.
– On the mobile phone, set system language to english.
– Open Android.
– Set window size to tall and small.
– Open URL “https://www.inductive.no/pairwiser”.
– Check that everything looks good.

Test 26.

– Get a Mobile in front of you with Android.
– In the network options, make sure IPv4 will be used.
– Set network connection to 4g and make sure it is connected.
– On the mobile phone, set system language to non-english.
– Open Opera.
– Set window size to tall and small.
– Open URL “https://www.inductive.no/pairwiser”.
– Check that everything looks good.

Test 27.

– Get a Desktop in front of you with Linux.
– In the network options, make sure IPv6 will be used.
– Set network connection to wifi and make sure it is connected.
– On the desktop machine, set system language to english.
– Open Opera.
– Set window size to tall and large.
– Open URL “http://www.inductive.no/pairwiser”.
– Check that everything looks good.

Test 28.

– Get a Desktop in front of you with Linux.
– In the network options, make sure IPv6 will be used.
– Set network connection to wifi and make sure it is connected.
– On the desktop machine, set system language to english.
– Open Chrome.
– Set window size to wide and large.
– Open URL “http://www.inductive.no/pairwiser”.
– Check that everything looks good.

Test 29.

– Get a Desktop in front of you with Mac.
– In the network options, make sure IPv4 will be used.
– Set network connection to cable and make sure it is connected.
– On the desktop machine, set system language to non-english.
– Open Firefox.
– Set window size to wide and large.
– Open URL “https://inductive.no/pairwiser”.
– Check that everything looks good.

Test 30.

– Get a Desktop in front of you with Windows.
– In the network options, make sure IPv4 will be used.
– Set network connection to cable and make sure it is connected.
– On the desktop machine, set system language to non-english.
– Open Firefox.
– Set window size to wide and large.
– Open URL “https://www.inductive.no/pairwiser”.
– Check that everything looks good.

Test 31.

– Get a Desktop in front of you with Windows.
– In the network options, make sure IPv4 will be used.
– Set network connection to cable and make sure it is connected.
– On the desktop machine, set system language to non-english.
– Open Safari.
– Set window size to wide and large.
– Open URL “http://www.inductive.no/pairwiser”.
– Check that everything looks good.

Results and Bugs Found

For the system under test, inductive.no, http redirects to https and www.inductive.no redirects to inductive.no.

During testing, it was found that the web host hosting the system under test does not support IPv6, so all the tests with only it activated will fail. This is an example of a 1-wise fault: The test fails because of one particular value of one particular parameter. As it is known in advance that this value will cause a test to fail, all the tests involving it was changed to IPv4.

A 3-wise interaction fault was also discovered. When allowing both http and https access to the website, the caching system caches pages with http links during http access and serves these cached pages when accessing the page through https. This is an example of a 3-wise fault: The interaction between caching, http and https.

Let us look at a few results.