In two earlier post, Performance Report for Pairwiser alpha Pairwise Generation Algorithm and Performance Report for Pairwiser beta Pairwise Algorithm, we wrote about the performance of the Pairwiser pairwise (2-wise) covering array generation algorithm. This report is on the Pairwiser 3-wise algorithm.
You can try Pairwiser by registering for the beta version. You can run Pairwiser either on the command line using the command line tool or programmatically using the Pairwiser API.
Test Machine
Pairwiser runs on the cloud service Amazon Web Services. We have done performance tests running on a machine able to run 32 threads in parallel (AWS instance type c3.8xlarge). Here are the specifications:
- CPU: Intel Xeon E5-2680 v2 (Ivy Bridge) @ 2.80 GHz; it can run 32 threads in parallel.
- Memory: 60.0 GB
Test Results
Here are the test result for the Pairwiser 3-wise algorithm (internally named Algorithm 244). Note that Model 9 to 13 were run in one thread because of their low run-time.
The models are available for download here and here.
| Nr. | System | Parame-ters | Constraints | Cases (Rows) | Wall_Clock Time (s) |
|---|---|---|---|---|---|
| 1 | Linux Kernel (x86) | 5 701 | 187 193 | 11 375 | 26 713,1 |
| 2 | Buildroot | 3 672 | 45 575 | 984 | 1 798,8 |
| 3 | uClinux | 3 144 | 31 614 | 318 | 589,7 |
| 4 | EmbToolkit | 2 136 | 160 353 | 629 | 868,7 |
| 5 | BusyBox | 1 690 | 17 784 | 286 | 81,9 |
| 7 | coreboot | 1 386 | 47 090 | 224 | 134,5 |
| 8 | eCos | 1 244 | 2 768 | 858 | 72,8 |
| 6 | FreeBSD kernel | 1 050 | 17 352 | 1 109 | 70,6 |
| 9 | GCC | 220 | 41 | 148 | 8,4 |
| 10 | Apache | 210 | 8 | 245 | 9,7 |
| 11 | Spin Verifier | 93 | 50 | 356 | 2,7 |
| 12 | Bugzilla | 61 | 6 | 81 | 0,2 |
| 13 | Eclipse IDE | 48 | 615 | 135 | 0,4 |
Please note that Pairwiser is in beta, and that the stability and quality might be less than the final product.
Plot
This plot shows test parameters vs. wall clock time (s).
