![Page 1: Main Effects Screening: A Distributed Continuous Quality Assurance Process for Monitoring Performance Degradation in Evolving Software Systems Cemal Yilmaz,](https://reader036.vdocument.in/reader036/viewer/2022081519/56649d385503460f94a11233/html5/thumbnails/1.jpg)
Main Effects Screening: A Distributed Continuous Quality Assurance Process for Monitoring Performance Degradation in Evolving Software Systems
Cemal Yilmaz, Arvind S. Krishnaz, Atif Memon, Adam Porter, Douglas C. Schmidt, Aniruddha Gokhale, Balachandran Natarajan
Presented By: Walaa El-Din M. Moustafa
![Page 2: Main Effects Screening: A Distributed Continuous Quality Assurance Process for Monitoring Performance Degradation in Evolving Software Systems Cemal Yilmaz,](https://reader036.vdocument.in/reader036/viewer/2022081519/56649d385503460f94a11233/html5/thumbnails/2.jpg)
Objective
As software systems change, developers often run regression tests to detect unintended functional side effects.
QA efforts can be confounded by the enormous configuration space.
Time and resource constraints severely limit the number of configurations that can be examined.
![Page 3: Main Effects Screening: A Distributed Continuous Quality Assurance Process for Monitoring Performance Degradation in Evolving Software Systems Cemal Yilmaz,](https://reader036.vdocument.in/reader036/viewer/2022081519/56649d385503460f94a11233/html5/thumbnails/3.jpg)
Main Effects Screening
Screening Designs are Highly economical Can reveal important low order effects
that strongly affect performance.
![Page 4: Main Effects Screening: A Distributed Continuous Quality Assurance Process for Monitoring Performance Degradation in Evolving Software Systems Cemal Yilmaz,](https://reader036.vdocument.in/reader036/viewer/2022081519/56649d385503460f94a11233/html5/thumbnails/4.jpg)
Low Order Effects
First-, second-, or third-order effects. nth-order effect is an effect caused by
the simultaneous interaction of n factors.
![Page 5: Main Effects Screening: A Distributed Continuous Quality Assurance Process for Monitoring Performance Degradation in Evolving Software Systems Cemal Yilmaz,](https://reader036.vdocument.in/reader036/viewer/2022081519/56649d385503460f94a11233/html5/thumbnails/5.jpg)
Low Order Effects Example For certain web server applications:
A 1st-order effect might be that performance slows considerably when logging is turned on and another might be that it also slows when few server threads are used.
A 2nd order effect involves the interaction of two options, e.g., web server performance may slow down when caching is turned off and the server performs blocking reads.
![Page 6: Main Effects Screening: A Distributed Continuous Quality Assurance Process for Monitoring Performance Degradation in Evolving Software Systems Cemal Yilmaz,](https://reader036.vdocument.in/reader036/viewer/2022081519/56649d385503460f94a11233/html5/thumbnails/6.jpg)
Factorial Designs
Full factorial design involving k binary factors. Such a design exhaustively tests all combinations.
Fractional factorial designs. These designs use only a carefully selected fraction (such as 1/2 or 1/4) of a full factorial design.
![Page 7: Main Effects Screening: A Distributed Continuous Quality Assurance Process for Monitoring Performance Degradation in Evolving Software Systems Cemal Yilmaz,](https://reader036.vdocument.in/reader036/viewer/2022081519/56649d385503460f94a11233/html5/thumbnails/7.jpg)
Fractional Factorial Designs
Save money Do so by giving up the ability to
measure some higher-order effects.
![Page 8: Main Effects Screening: A Distributed Continuous Quality Assurance Process for Monitoring Performance Degradation in Evolving Software Systems Cemal Yilmaz,](https://reader036.vdocument.in/reader036/viewer/2022081519/56649d385503460f94a11233/html5/thumbnails/8.jpg)
Fractional Factorial Design Example
To show how screening designs are computed, we present a hypothetical example of a performance-intensive software system: 4 binary configuration options, A
through D. No inter-option constraints. The full configuration space therefore
has 24 = 16 configurations.
![Page 9: Main Effects Screening: A Distributed Continuous Quality Assurance Process for Monitoring Performance Degradation in Evolving Software Systems Cemal Yilmaz,](https://reader036.vdocument.in/reader036/viewer/2022081519/56649d385503460f94a11233/html5/thumbnails/9.jpg)
Fractional Factorial Design Example
8 configurations. Will focus only on 1st-order effects. This design is referred to as a 24 -1
design, where 4 refers to the total number of options we will examine and the -1 (2-1 = 1/2) indicates the fraction of the full factorial over which we will collect data.
![Page 10: Main Effects Screening: A Distributed Continuous Quality Assurance Process for Monitoring Performance Degradation in Evolving Software Systems Cemal Yilmaz,](https://reader036.vdocument.in/reader036/viewer/2022081519/56649d385503460f94a11233/html5/thumbnails/10.jpg)
Fractional Factorial Design Example
We can extend the design and estimate the effect of option D without going to a 24 full factorial design.
![Page 11: Main Effects Screening: A Distributed Continuous Quality Assurance Process for Monitoring Performance Degradation in Evolving Software Systems Cemal Yilmaz,](https://reader036.vdocument.in/reader036/viewer/2022081519/56649d385503460f94a11233/html5/thumbnails/11.jpg)
Fractional Factorial Design Example
Design Generator: Specifies the aliasing patterns used to
build the design For this example, we select the design
generator D = ABC.
![Page 12: Main Effects Screening: A Distributed Continuous Quality Assurance Process for Monitoring Performance Degradation in Evolving Software Systems Cemal Yilmaz,](https://reader036.vdocument.in/reader036/viewer/2022081519/56649d385503460f94a11233/html5/thumbnails/12.jpg)
Fractional Factorial Design Example
![Page 13: Main Effects Screening: A Distributed Continuous Quality Assurance Process for Monitoring Performance Degradation in Evolving Software Systems Cemal Yilmaz,](https://reader036.vdocument.in/reader036/viewer/2022081519/56649d385503460f94a11233/html5/thumbnails/13.jpg)
Fractional Factorial Design Example
The design we described above is a resolution IV design.
In resolution R designs, no effects involving i factors are aliased with effects involving less than R - i factors.
![Page 14: Main Effects Screening: A Distributed Continuous Quality Assurance Process for Monitoring Performance Degradation in Evolving Software Systems Cemal Yilmaz,](https://reader036.vdocument.in/reader036/viewer/2022081519/56649d385503460f94a11233/html5/thumbnails/14.jpg)
Fractional Factorial Design Example Run the experiment For binary options (with settings - or +),
the main effect of option A, ME(A), isME(A) = z(A-) - z(A+)where z(A-) and z(A+) are the mean values of the observed data over all runs where option A is (-) and where option A is (+), respectively.
If desired, 2nd order effects can be calculated in a similar way. The interaction effect of option A and B, INT(A, B) is:= ½ (ME(B|A+) - ME(B|A-)) = ½ (ME(A|B+) - ME(A|B-))
![Page 15: Main Effects Screening: A Distributed Continuous Quality Assurance Process for Monitoring Performance Degradation in Evolving Software Systems Cemal Yilmaz,](https://reader036.vdocument.in/reader036/viewer/2022081519/56649d385503460f94a11233/html5/thumbnails/15.jpg)
The Experiment
ACE v5.4 + TAO v1.4 + CIAO v0.4. 14 binary run-time options 214 =
16,384 different configurations. For each task measure:
Message latency. Throughput between the client and the
server.
![Page 16: Main Effects Screening: A Distributed Continuous Quality Assurance Process for Monitoring Performance Degradation in Evolving Software Systems Cemal Yilmaz,](https://reader036.vdocument.in/reader036/viewer/2022081519/56649d385503460f94a11233/html5/thumbnails/16.jpg)
The Experiment
![Page 17: Main Effects Screening: A Distributed Continuous Quality Assurance Process for Monitoring Performance Degradation in Evolving Software Systems Cemal Yilmaz,](https://reader036.vdocument.in/reader036/viewer/2022081519/56649d385503460f94a11233/html5/thumbnails/17.jpg)
The Full Data Set
![Page 18: Main Effects Screening: A Distributed Continuous Quality Assurance Process for Monitoring Performance Degradation in Evolving Software Systems Cemal Yilmaz,](https://reader036.vdocument.in/reader036/viewer/2022081519/56649d385503460f94a11233/html5/thumbnails/18.jpg)
Evaluating Screening Designs
Can screening designs correctly identify the important options discovered during the analysis of the full data set?
![Page 19: Main Effects Screening: A Distributed Continuous Quality Assurance Process for Monitoring Performance Degradation in Evolving Software Systems Cemal Yilmaz,](https://reader036.vdocument.in/reader036/viewer/2022081519/56649d385503460f94a11233/html5/thumbnails/19.jpg)
Evaluating Screening Designs Scr32 is a 214 - 9 IV with design generators
F = ABC, G = ABD, H = ACD, I = BCD, J = ABE, K = ACE, L = BCE, M = ADE, N = BDE.
Scr64 is a 214 - 8 IV with design generators G = ABC, H = ABD, I = ABE, J = ACDE, K = ABF, L = ACDF, M = ACEF, N = ADEF.
Scr128 is a 214 - 7 IV with design generators H = ABC, I = ABDE, J = ABDF, K = ACEF, L = ACDG, M = ABEFG, N = BCDEFG.
![Page 20: Main Effects Screening: A Distributed Continuous Quality Assurance Process for Monitoring Performance Degradation in Evolving Software Systems Cemal Yilmaz,](https://reader036.vdocument.in/reader036/viewer/2022081519/56649d385503460f94a11233/html5/thumbnails/20.jpg)
Evaluating Screening Designs
![Page 21: Main Effects Screening: A Distributed Continuous Quality Assurance Process for Monitoring Performance Degradation in Evolving Software Systems Cemal Yilmaz,](https://reader036.vdocument.in/reader036/viewer/2022081519/56649d385503460f94a11233/html5/thumbnails/21.jpg)
Evaluating Screening Designs
![Page 22: Main Effects Screening: A Distributed Continuous Quality Assurance Process for Monitoring Performance Degradation in Evolving Software Systems Cemal Yilmaz,](https://reader036.vdocument.in/reader036/viewer/2022081519/56649d385503460f94a11233/html5/thumbnails/22.jpg)
Evaluating Screening Designs
Screening designs can detect important options at a large fraction of the cost of exhaustive testing.
The smaller the effect, the larger the run size needed to identify it.
![Page 23: Main Effects Screening: A Distributed Continuous Quality Assurance Process for Monitoring Performance Degradation in Evolving Software Systems Cemal Yilmaz,](https://reader036.vdocument.in/reader036/viewer/2022081519/56649d385503460f94a11233/html5/thumbnails/23.jpg)
Estimating Performance with Screening Suites
The estimates are generated by examining all combinations of the most important options, while defaulting the settings of the unimportant options
Make the estimates based on benchmarking either 4 (all combinations of options B and J) or 32 (all combinations of options B, J, C, I, and F) configurations.
![Page 24: Main Effects Screening: A Distributed Continuous Quality Assurance Process for Monitoring Performance Degradation in Evolving Software Systems Cemal Yilmaz,](https://reader036.vdocument.in/reader036/viewer/2022081519/56649d385503460f94a11233/html5/thumbnails/24.jpg)
Estimating Performance with Screening Suites
![Page 25: Main Effects Screening: A Distributed Continuous Quality Assurance Process for Monitoring Performance Degradation in Evolving Software Systems Cemal Yilmaz,](https://reader036.vdocument.in/reader036/viewer/2022081519/56649d385503460f94a11233/html5/thumbnails/25.jpg)
Estimating Performance with Screening Suites
The distributions of the top-5 and top-2 screening suites closely track the overall performance data.
![Page 26: Main Effects Screening: A Distributed Continuous Quality Assurance Process for Monitoring Performance Degradation in Evolving Software Systems Cemal Yilmaz,](https://reader036.vdocument.in/reader036/viewer/2022081519/56649d385503460f94a11233/html5/thumbnails/26.jpg)
Screening Suites vs. Random Sampling
![Page 27: Main Effects Screening: A Distributed Continuous Quality Assurance Process for Monitoring Performance Degradation in Evolving Software Systems Cemal Yilmaz,](https://reader036.vdocument.in/reader036/viewer/2022081519/56649d385503460f94a11233/html5/thumbnails/27.jpg)
Screening Suites vs. Random Sampling
These graphs suggest the obvious weakness of random sampling.
![Page 28: Main Effects Screening: A Distributed Continuous Quality Assurance Process for Monitoring Performance Degradation in Evolving Software Systems Cemal Yilmaz,](https://reader036.vdocument.in/reader036/viewer/2022081519/56649d385503460f94a11233/html5/thumbnails/28.jpg)
Dealing with Evolving Systems Detect
performance degradations in evolving systems quickly.
![Page 29: Main Effects Screening: A Distributed Continuous Quality Assurance Process for Monitoring Performance Degradation in Evolving Software Systems Cemal Yilmaz,](https://reader036.vdocument.in/reader036/viewer/2022081519/56649d385503460f94a11233/html5/thumbnails/29.jpg)
Dealing with Evolving Systems
Developer records and problem reports indicate that problems were noticed on 12/14/03.
![Page 30: Main Effects Screening: A Distributed Continuous Quality Assurance Process for Monitoring Performance Degradation in Evolving Software Systems Cemal Yilmaz,](https://reader036.vdocument.in/reader036/viewer/2022081519/56649d385503460f94a11233/html5/thumbnails/30.jpg)
Higher Order Effects
Resolution VI design Increased the run size to 2,048 to
capture the second-order effects.
![Page 31: Main Effects Screening: A Distributed Continuous Quality Assurance Process for Monitoring Performance Degradation in Evolving Software Systems Cemal Yilmaz,](https://reader036.vdocument.in/reader036/viewer/2022081519/56649d385503460f94a11233/html5/thumbnails/31.jpg)
Higher Order Effects
![Page 32: Main Effects Screening: A Distributed Continuous Quality Assurance Process for Monitoring Performance Degradation in Evolving Software Systems Cemal Yilmaz,](https://reader036.vdocument.in/reader036/viewer/2022081519/56649d385503460f94a11233/html5/thumbnails/32.jpg)
Higher Order Effects
The important interaction effects involve only options that are already considered important by themselves
The screening design correctly identifies the 5 most important pairwise interactions at 1/8th the cost of exhaustive testing.
![Page 33: Main Effects Screening: A Distributed Continuous Quality Assurance Process for Monitoring Performance Degradation in Evolving Software Systems Cemal Yilmaz,](https://reader036.vdocument.in/reader036/viewer/2022081519/56649d385503460f94a11233/html5/thumbnails/33.jpg)
Thank you