In an earlier post, Performance Report for Pairwiser alpha Pairwise Generation Algorithm, we wrote about the performance of the alpha version of Pairwiser. We have now made a beta version available, which runs on a cluster of machines in the cloud. You can try Pairwiser by registering for the beta version. You can run Pairwiser either on the command line using the command line tool or programmatically using the Pairwiser API.
Test Machine
Pairwiser runs on the cloud service Amazon Web Services. We have done performance tests running on a machine able to run 32 threads in parallel (AWS instance type c3.8xlarge). Here are the specifications:
- CPU: Intel Xeon E5-2680 v2 (Ivy Bridge) @ 2.80 GHz; it can run 32 threads in parallel.
- Memory: 60.0 GB
Test Results
Here are the test result for the Pairwiser algorithm, internally named Algorithm 237, the same as the previous report.
| Nr. | System | Parame-ters | Constraints | Cases (Rows) | Wall-Clock Time (s) |
|---|---|---|---|---|---|
| 1 | Linux Kernel (23 platforms) | 13 836 | 246 724 | 336 | 701,15 |
| 2 | Freetz | 6 848 | 101 877 | 166 | 71,66 |
| 3 | Linux Kernel (x86) | 5 701 | 187 193 | 794 | 60,79 |
| 4 | Buildroot | 3 672 | 45 575 | 212 | 17,76 |
| 5 | uClinux | 3 144 | 31 614 | 115 | 5,51 |
| 6 | EmbToolkit | 2 136 | 160 353 | 170 | 23,47 |
| 7 | BusyBox | 1 690 | 17 784 | 109 | 1,29 |
| 8 | FreeBSD kernel | 1 050 | 17 352 | 196 | 0,99 |
As we remarked in the previous post, Model 3 (the Linux Kernel model for x86) has a best published time of 33 702 seconds (or 9.4 hours). This was improved in the alpha version of Pairwiser to 316 seconds (or 5.3 minutes). For Pairwiser beta, we can see that the new record time is just 61 seconds (or 1 minute)!
Note that there will be some overhead in using the Pairwiser client or API. The input model must be transferred to the machine and the result back. Also, the online service double-checks the coverage to ensure a correct measurement.
Please note that Pairwiser is in beta, and that the stability and quality might be less than the final product.
Plot
This plot shows test parameters vs. wall clock time (s).
References
- Martin F. Johansen et al., An Algorithm for Generating t-wise Covering Arrays from Large Feature Models (2012)
