Forum Replies Created
-
AuthorPosts
-
Martin F. Johansen
KeymasterIt is up again. Note that this is a free service for the community, so there is a limit to how much compute power the server has.
KeymasterHi, try again now.
KeymasterHi Gordon, the Pairwiser server sitting behind the web-interface accepts general constrains written as propositional logic or CNF. This is not currently supported in the web-interface.
We could accomodate for it in the web-interface. Could you elaborate some more on your case?
KeymasterHi Bret, yes, you can now export the tests to Excel!
KeymasterHi Rajesh!
Have you taken a look at the two examples in our knowledge base?
Example 1: Classic Example
Example 2: Testing a WebsiteThese discuss the issues you mention in relation to two different examples.
The significance and applicability of 1, 2 and 3-wise testing depends on the case you are applying it to. If you describe your testing problem a bit, I can describe the significance and applicability to it.
KeymasterHi Alex!
> Is T-sets a complete number of all various combinations of all parameters?
Yes, you are right: A t-set is an interaction: For example, if we have Parameter 1 with values A and B and Parameter 2 with values C and D, all the (pairwise) t-sets are: {A, C}, {A, D}, {B, C} and {B, D}. Notice that {A, B} and {C, D} are illegal and invalid t-sets.
In the numbers you see (21424 / 21424 (0)), the first number is the number of t-sets covered by the tests. The second number is the total number of legal and valid t-sets. The third number is the difference between those numbers: The uncovered t-sets. If the coverage is 100% the number of uncovered t-sets is 0.
We have planned to add some explanation of those numbers in addition to adding other interesting metrics.
> How did you figure out that 293 tests covers all necessary situations?
It is basically a greedy set cover algorithm. For an overview, you can read this paper:
http://martinfjohansen.com/papers/Johansen2012.pdf
The Pairwiser algorithm is based on this algorithm with some significant advancements that makes it several orders of magnitudes faster.
> How do you evaluate for example that 16 test-cases in this situation will cover 82% of functionality?
It is done using the algorithm and associated algorithms mentioned in the previous answer.
KeymasterI did run the test generation task submitted on 2015-06-29 12:24:53 (CET) on a larger server.
The result has 63 759 test cases and took 21 494 937 CPU ms to generate (approx 6 CPU hours). On a machine with 8 CPUs, this will take less than 1 hour and on a machine with 64 CPUs less than 10 minutes (probably).
KeymasterHi Alex!
Pairwiser can indeed handle your models, but it is a question of the size of the servers we are running on. Not many people run such large models (yet!) so the servers you ran on was not fast enough for a swift response to such a large model. (By the way, we will soon provide products which will run on larger servers. We will also put up a progress bar during test generation, so you can see the progress.)
Let me first see if I have found the correct model. Your model has, among other things, two parameters called “Include countries” and “Exclude countries”, each with a long list of countries?
This is an unusual way to set up a model for Pairwise testing. It will necessarily cause the number of tests to be very high. It will be 10 000 if not 100 000 tests.
Pairwise testing performs best with models with many parameters (even 10 000s) but with relatively few values each. (e.g. less than 10.) This will cause a few tests to cover a lot of interactions. With a few parameters and many values (such as in your model), each tests can only cover a few interactions, causing a large amount of tests and slow generation speeds.
I would be happy to look at you testing problem and assist you in setting it up! There are techniques that can be applied, such as equivalence classes, to make the testing problem perform well with pairwise testing. We can either discuss it here on the forums or just send me a mail on [email protected], and I will assist you in setting it up and making sure that you get your tests successfully generated, even if your models are very large.
-
AuthorPosts