03.03.2014, copenhagen
DESCRIPTION
Multi- Objective Test Suite Optimization for Incremental Product Family Testing ( joint work with Hauke Baller, Sascha Lity and Ina Schaefer). 03.03.2014, Copenhagen. Contents. „ What is a good ordering for testing the members of a product family ?“ - PowerPoint PPT PresentationTRANSCRIPT
© author(s) of these slides 2014 including research results of the research network ES and TU Darmstadt otherwise as specified at the respective slide
Malte Lochau
TU Darmstadt
ES Real-Time Systems Lab
Prof. Dr. rer. nat. Andy Schürr
Dept. of Electrical Engineering and Information Technology
Dept. of Computer Science (adjunct Professor)
www.es.tu-darmstadt.de
Multi-Objective Test Suite Optimization for Incremental Product Family Testing
(joint work with Hauke Baller, Sascha Lity and Ina Schaefer)
03.03.2014, Copenhagen
ES – Real-Time Systems Lab
2 | 03.03.2014 | PLE
Contents „What is a good ordering for testing the members of a product
family?“
Extension of the classical Test Suite Minimization Problem Weights Bounds Variability
Approx. optimal solution(s) for Multi-Objectives via ILP Solving
Incremental heuristic solution based on dynamic programming to appear at ICST‘2014
ES – Real-Time Systems Lab
3 | 03.03.2014 | PLE
Test Suite Minimization Problem [Harrold et al., 1993]
t1
r1 r2 r3 r4 r5 r6
t2 t3 t4…
Test Cases
Test Requirements
Complete Test
Suite
ES – Real-Time Systems Lab
4 | 03.03.2014 | PLE
All-Transitions Coverage R = {r1,r2, …, r8}
Complete Test Suite TS (4 Test Cases)Minimal Complete Test Suite TS‘ (1 Test Case)
Example: Model-Based Testing
r1
r2
r3
r4 r5
r6
r7
r8
test
model
Test Suites
TS = {(r1r2),(r1r4),(r3r7r8),(r3r5r6)}
TS
‘ = {(r1r2r3r5r4r7r8r5r6)}
ES – Real-Time Systems Lab
5 | 03.03.2014 | PLE
Minimum Set Cover Problem
Given:
Set of Requirements
Set of Test Cases
Sub sets satisfying Requirement
Find:
Minimum sub set such that
NP-Complete
Assumptions:
All Test Cases are relevant:
Each Requirement is satisfiable:
ii TtRrTt ::
Ø: ii TRr
ES – Real-Time Systems Lab
6 | 03.03.2014 | PLE
Cost/Profit Test Suite Optimization [Malishevsky et al. 2006, Xu et al. 2012, Rout et al. 2013]
t1
r1 r2 r3 r4 r5 r6
t2 t3 t4…
Test Cases
Test Requirements
Adequate Test
Suite
2 1 4 3 1 2 Testing Profits pTS
4 2 1 4 Testing Costs cTS
+ + + + + =
+ + + =
≥ Profit Goal kp
≤ Cost Bound kc
∑=7Minimal Test Suite ∑=8
ES – Real-Time Systems Lab
7 | 03.03.2014 | PLE
Test Suite Profits pTS
TS = {(r1r2),(r1r4),(r3r7r8),(r3r5r6)}
pTS=(2+1)+(2+1)+(2+2+4)+(2+2+1)=15
TS
‘= {(r1r2r3r5r4r7r8r5r6)}
pTS=(2+1+2+2+1+2+4+2+1)=15
TS
‘‘ = {(r1r4r5r2),(r3r7)}
pTS=(2+1+2+1)+(2+2)=10
TS‘‘
is Adequate Test Suite for kp = 10
Example: Model-Based Testing with Costs/Profits
Test Suite Costs cTS
TS = {(r1r2),(r1r4),(r3r7r8),(r3r5r6)} cTS = 2+2+3+3 = 10
TS‘= {(r1r2r3r5r4r7r8r5r6)} cTS = 9
TS‘‘ = {(r1r4r5r2),(r3r7)} cTS = 5+2 = 7
r1(2)
r2(1)
r3(2)
r4(1) r5(2)
r6(1)
r7(2)
r8(4)
Profits are counted once
kp=10
ES – Real-Time Systems Lab
8 | 03.03.2014 | PLE
Minimum Partial Weighted Set Cover Problem [Könemann et al., 2011]
Given:
Set of Weighted Requirements R
Set of Weighted Test Cases T
Profit goal kp
Find:
Sub set with minimal such that
pTS ≥ kp
where and
TSt
TTS twc
ØTST
RriRTS
i
i
rwp
ES – Real-Time Systems Lab
9 | 03.03.2014 | PLE
Product Line Analysis
ES – Real-Time Systems Lab
10 | 03.03.2014 | PLE
Sample-based Product Line Analysis
[Cohen et al. 2006], [Gustafsson 2007], [Oster et al., 2010], [Perrouin et al, 2010], [Kim et al. 2011], [Johansen et al., 2012], [Henard et al., 2013], [Haslinger et al., 2013],
….
ES – Real-Time Systems Lab
11 | 03.03.2014 | PLE
Family-based Product Line Analysis
[Krishnamurti et al., 2006], [Gruler et al., 2010], [Classen et al. 2010], [Gnesi et al. 2011], [Cordy et al., 2013], …
𝑂 (?∗𝜑 )
ES – Real-Time Systems Lab
12 | 03.03.2014 | PLE
Incremental Product Line Analysis
∅
[Batory et al. 2007], [Engström et al. 2011], [Lochau et al. 2012], …
ES – Real-Time Systems Lab
13 | 03.03.2014 | PLE
T={t1, …, t15}
t7=(r1r6r8r7)
t8=(r1r4r5r2)
t11=(r1r4r7)
t12=(r9r10r8)
t13=(r3r11)
t14=(r1r4r11)
t15=(r3r7)}
Family-based Product Line Test Suite Design
4 Product Variants
r1(2)
r2(1)
r3(2)
r4(1) r5(2)
r6(1)
r7(2)
r8(4)
r9(2)
r10(2)
r11(8)
ES – Real-Time Systems Lab
14 | 03.03.2014 | PLE
Complete Minimal Product Line Test Suite [Cichos et al. 2011]
Assumptionsif then satisfies on
Given:
Set of Requirements R
Set of Test Cases T
Set of Product Variants P
Sub sets satisfying Requirement
Sub sets executable on product variant
Sub sets relevant for product variant
Find:
Minimum sub set such that
ES – Real-Time Systems Lab
15 | 03.03.2014 | PLE
Constraint Minimum Set Cover Problem
t1
r1 r2 r3 r4 r5 r6
t2 t3 t4…
Test Cases
Test Requirements
p1
Product Variants
p2 p3
5 2 3Product Costs/Profits ppTS
2 1 4 3 1 2
4 2 4 1 Testing Costs
Testing Profits
+ + =≥ Profit Goal kpp
pTS ≥ kp
rpTS ≥ krp
ES – Real-Time Systems Lab
16 | 03.03.2014 | PLE
Constraint (Minimum) Partial Weighted Set Cover Problem
Given:
Set of Weighted Requirements
Set of Weighted Test Cases
Set of Weighted Product Variants
Profit Goals
(Cost Bound
Find:
Sub set (with minimal such that
and and is an Adequat Product Line Test Suite
Ø:, TSTTRrPp
RriRTS
ijj
ij
i
rwrp
Ø: TSTTRr
PpjpTS
ijj
i
j
pwpp
ES – Real-Time Systems Lab
17 | 03.03.2014 | PLE
Integer Linear Programming (ILP)
(Integer) Decision variables Linear objective function Linear constraints
ES – Real-Time Systems Lab
18 | 03.03.2014 | PLE
ILP for Product Line Test Suite Optimization
Product pj selected xj=1
Test Case t selected into TS yt=1
Requirement ri satisfied zi=1
Binary Variables
ppTS
Product selected Requirements must be satisfied
pj select Test Case from Test Case Set
NOT pj OR One Test Case from Test Case Set
tp
For all Requirements
(of a Product)
Requirement satisfied For all Products, a Test Case must be selected
Again (ri Set) (NOT ri OR Set)
r t
Test Cases must be applied to a Product
tp
Priotization
ppTS ≥ kpp ppTS - Δpp = kpp
ppTS ≥ kpp
cTS ≤ kc
rpTS ≥ krp
Differences must not be negativ
Normalization Value
ES – Real-Time Systems Lab
19 | 03.03.2014 | PLE
Re-Formulation as Product Selection Problem
ILP finds Adequate Test Suite (if exists), but Problem is NP-hardAnother Point-of-View: Incremental Product Selection
t1
r1 r2 r3 r4 r5 r6
t2 t3 t4
p1 p2 p3
rpTS
cTS
ppTS
ES – Real-Time Systems Lab
20 | 03.03.2014 | PLE
Adequate Product Sub Set
Given:
Set of Weighted Requirements
Set of Weighted Test Cases
Set of Weighted Product Variants
Profit Goals
(Cost Bound
Find:
Sub set P such that an adequate test suite exists with
ES – Real-Time Systems Lab
21 | 03.03.2014 | PLE
Incremental Product Selection
p1
p2p3
p4
𝑃𝑆𝑖 𝑃𝑆𝑖+1
p1
p2p3
p4
p5
P∖𝑃𝑆𝑖
p7
p6p8
p9
p5
is adequate
is adequate
𝑇 𝑆𝑖
+
𝑇 𝑆𝑖
+
𝑇
profit
goals reached?
select next product
cost bounds
exceeded?
no adequate solution
found
ES – Real-Time Systems Lab
22 | 03.03.2014 | PLE
Product Selection Heuristic1. Are Profit Goals reached? (rpTS ≥ krp and ppTS ≥ kpp)
2. No:a. For each unselected Product
i. While unsatisfied Requirements exist, select one: With high profit and uniqueness/rareness Select a satisfying Test Case :
With the ability to satisfy many other open Requirements for many other Products and low cost
b. Rank Products and Select By: Product profit, resulting Test Case costs, and Requirement profit (weighted)
c. Recalculate ppTS, rpTS and cTS
d. Is Test budget excelled (cTS > kc)
i. Yes: “Error” Ende. Goto 1.
3. Yes: “Success” End
r
n
?
?
tn
?
?
?
cf. [Harrold et al., 1993]
[Harrold et al., 1993]
+ Products
ES – Real-Time Systems Lab
23 | 03.03.2014 | PLE
Experimental ResultsVarying: Number of: Requirements, Test Cases, and Products Mapping: Requirements/Test Cases and Test Cases/Products Weights
In Total: 51 generated Data SetsTime Limit: 24h
t1
r1 r2 r3 r4 r5 r6
t2 t3 t4
p1 p2 p3
50, 100, and 500 Test Cases
50, 100, and 500 Products
50, 100, and 500 Requirements
Mean and Variance
Mean and Variance
-/o/+
-/o/+
Heuristic is 21.5 times faster
68.6% less
accuracy
Efficency improvement factor: 127.48
ES – Real-Time Systems Lab
24 | 03.03.2014 | PLE
Discussion & Future Work
Summary Heuristic produces useful results and scales Feasible tradeoff between accuracy and efficiency
Ongoing and Future Work Backtracking if bounds are exceeded Search-based optimization techniques Methods and criteria to find appropriate weights and bounds Beyond ILP: Dynamic weight function, On-the-fly generation of test artifacts, …
Application to Real-World Case Studies
ES – Real-Time Systems Lab
25 | 03.03.2014 | PLE
Industrial Case Study: Heidelberg Ion-Beam Therapy Center
DCU Case Study forBeam Cycle Creation
Complex Automation Engineering Software
Highly-Configurableand Runtime Adaptive
Re-Engineered as (D)SPL ~ 200 Features (Boolean
and Non-Boolean) 150% Test Model
~ 640 Transitions SPL Implementation in C
~ 175.000 LOCTreatment
Rooms
Gantry
DCU-Z DCU-R DCU-R DCU-P
ES – Real-Time Systems Lab
26 | 03.03.2014 | PLE
Model-Driven Development of DCU Code
Parameters
+
User Data.h
Hand-written
ECA Code
.c .h+
States
Tra
nsiti
on T
rigge
r
.hAutomata
Stubs.c
insert
deploy
generate
SFT
ES – Real-Time Systems Lab
27 | 03.03.2014 | PLE
DCU SPL Test Model Specification
Device Control Unit
Z
SD T
TS
R
RB
RK
Device ControllerManual
Experiment
Quality Assurance
Adjustment
Therapy
Standby
Idle
Version
Checksum
Timing Master Parameter
EnableSyncSupplyFrequence
EnableGenRTBMasterClock
Device Control Parameter
EnableSwitchOn
EnableShutdown
BeamRequestTakeOverRefValues
DelayTimeActivatedelay: [0,100]
DelayTimeFeedbackdelay: [0,100]
TimeoutFeedbacktimeout: [0,100]
RefValueTHoldMaxdelay [0,100]
AddRtbDelayTimedelay : [0,100]
ErrorMonEn
LowCurrentTimertimer: [0, 100]
Mode 0
Mode 1
Mode 2
Mode 3
Mode 4
DCU Type Parameter
DCU Master Parameter
Type
bDevice Control
b
P
Ramped
bbbbb
Operation Mode
MeasuringMode
RtbTimingEn
150% SFT
+
ES – Real-Time Systems Lab
28 | 03.03.2014 | PLE
Thank Your For Your Attention!
Integrated Model-based Testing
of Continuously Evolving Software Product
Lines
Multi-Mechanismen Adaption für das
künftige Internet