norman vincent peale, - larc...
TRANSCRIPT
![Page 1: Norman Vincent Peale, - LARC Researchresearch.larc.smu.edu.sg/ase2016/slides/ase2016keynote20160907... · – Norman Vincent Peale, The Power of Posi,ve Thinking “Do not build up](https://reader030.vdocument.in/reader030/viewer/2022020108/5ab91a1f7f8b9ad5338d9bf9/html5/thumbnails/1.jpg)
National University of Singapore
![Page 2: Norman Vincent Peale, - LARC Researchresearch.larc.smu.edu.sg/ase2016/slides/ase2016keynote20160907... · – Norman Vincent Peale, The Power of Posi,ve Thinking “Do not build up](https://reader030.vdocument.in/reader030/viewer/2022020108/5ab91a1f7f8b9ad5338d9bf9/html5/thumbnails/2.jpg)
– Norman Vincent Peale, The Power of Posi,ve Thinking
“Do not build up obstacles in your imagina:on.”
![Page 3: Norman Vincent Peale, - LARC Researchresearch.larc.smu.edu.sg/ase2016/slides/ase2016keynote20160907... · – Norman Vincent Peale, The Power of Posi,ve Thinking “Do not build up](https://reader030.vdocument.in/reader030/viewer/2022020108/5ab91a1f7f8b9ad5338d9bf9/html5/thumbnails/3.jpg)
Pop Quiz What Do These Things Have in Common?
www.nbcnews.com
An Earthquake
![Page 4: Norman Vincent Peale, - LARC Researchresearch.larc.smu.edu.sg/ase2016/slides/ase2016keynote20160907... · – Norman Vincent Peale, The Power of Posi,ve Thinking “Do not build up](https://reader030.vdocument.in/reader030/viewer/2022020108/5ab91a1f7f8b9ad5338d9bf9/html5/thumbnails/4.jpg)
Pop Quiz What Do These Things Have in Common?
www.healthina:on.com
A Heart Attack
![Page 5: Norman Vincent Peale, - LARC Researchresearch.larc.smu.edu.sg/ase2016/slides/ase2016keynote20160907... · – Norman Vincent Peale, The Power of Posi,ve Thinking “Do not build up](https://reader030.vdocument.in/reader030/viewer/2022020108/5ab91a1f7f8b9ad5338d9bf9/html5/thumbnails/5.jpg)
Pop Quiz What Do These Things Have in Common?
www.freerepublic.com
President Trump
![Page 6: Norman Vincent Peale, - LARC Researchresearch.larc.smu.edu.sg/ase2016/slides/ase2016keynote20160907... · – Norman Vincent Peale, The Power of Posi,ve Thinking “Do not build up](https://reader030.vdocument.in/reader030/viewer/2022020108/5ab91a1f7f8b9ad5338d9bf9/html5/thumbnails/6.jpg)
Pop Quiz What Do These Things Have in Common?
✓ They are governed by stochas:c phenomena
✓We have a reasonable understanding of their causes
✓We base our understanding on past observa:ons - At various levels of abstrac:on
(Simple) Answer: They are events to which
experts assign a probability based on models
Software is like this too!
![Page 7: Norman Vincent Peale, - LARC Researchresearch.larc.smu.edu.sg/ase2016/slides/ase2016keynote20160907... · – Norman Vincent Peale, The Power of Posi,ve Thinking “Do not build up](https://reader030.vdocument.in/reader030/viewer/2022020108/5ab91a1f7f8b9ad5338d9bf9/html5/thumbnails/7.jpg)
Certainty inSoKware Engineering
“My program is correct.” “The specifica,on is sa,sfied.”
A simplistic viewpoint, which permeates most of our models, techniques and tools
![Page 8: Norman Vincent Peale, - LARC Researchresearch.larc.smu.edu.sg/ase2016/slides/ase2016keynote20160907... · – Norman Vincent Peale, The Power of Posi,ve Thinking “Do not build up](https://reader030.vdocument.in/reader030/viewer/2022020108/5ab91a1f7f8b9ad5338d9bf9/html5/thumbnails/8.jpg)
Example Model Checking
! ¬p→◊q( )∧"( )
Model
Checker
✓✕
State Machine
Model
Temporal Property
Results
Counterexample
Trace
System
Requirements
![Page 9: Norman Vincent Peale, - LARC Researchresearch.larc.smu.edu.sg/ase2016/slides/ase2016keynote20160907... · – Norman Vincent Peale, The Power of Posi,ve Thinking “Do not build up](https://reader030.vdocument.in/reader030/viewer/2022020108/5ab91a1f7f8b9ad5338d9bf9/html5/thumbnails/9.jpg)
Example Model Checking
! ¬p→◊q( )∧"( )
Model
Checker
✕
State Machine
Model
Temporal Property
Results
Counterexample
Trace
System
Requirements
![Page 10: Norman Vincent Peale, - LARC Researchresearch.larc.smu.edu.sg/ase2016/slides/ase2016keynote20160907... · – Norman Vincent Peale, The Power of Posi,ve Thinking “Do not build up](https://reader030.vdocument.in/reader030/viewer/2022020108/5ab91a1f7f8b9ad5338d9bf9/html5/thumbnails/10.jpg)
Why Apply aProbabilis:c Viewpoint?
![Page 11: Norman Vincent Peale, - LARC Researchresearch.larc.smu.edu.sg/ase2016/slides/ase2016keynote20160907... · – Norman Vincent Peale, The Power of Posi,ve Thinking “Do not build up](https://reader030.vdocument.in/reader030/viewer/2022020108/5ab91a1f7f8b9ad5338d9bf9/html5/thumbnails/11.jpg)
Why Apply aProbabilis:c Viewpoint?
![Page 12: Norman Vincent Peale, - LARC Researchresearch.larc.smu.edu.sg/ase2016/slides/ase2016keynote20160907... · – Norman Vincent Peale, The Power of Posi,ve Thinking “Do not build up](https://reader030.vdocument.in/reader030/viewer/2022020108/5ab91a1f7f8b9ad5338d9bf9/html5/thumbnails/12.jpg)
Why Apply aProbabilis:c Viewpoint?
✓ Perfect and complete requirements are improbable
✓ Execu:on and tes:ng are akin to sampling … and we use tes:ng to increase confidence!
✓ The behavior of the execu:on environment is
random and unpredictable
✓ Frequency of execu:on failures is (hopefully) low
There are many random phenomena and “shades of grey”
in software engineering
But our models, techniques and tools rarely capture this
![Page 13: Norman Vincent Peale, - LARC Researchresearch.larc.smu.edu.sg/ase2016/slides/ase2016keynote20160907... · – Norman Vincent Peale, The Power of Posi,ve Thinking “Do not build up](https://reader030.vdocument.in/reader030/viewer/2022020108/5ab91a1f7f8b9ad5338d9bf9/html5/thumbnails/13.jpg)
NATO Conference on SERome, 1969
h[p://homepages.cs.ncl.ac.uk/brian.randell/NATO/N1969/DIJKSTRA.html
– Edsger W. Dijkstra
(on more than one occasion!)
“Tes:ng shows the presence,
not the absence of bugs.”
![Page 14: Norman Vincent Peale, - LARC Researchresearch.larc.smu.edu.sg/ase2016/slides/ase2016keynote20160907... · – Norman Vincent Peale, The Power of Posi,ve Thinking “Do not build up](https://reader030.vdocument.in/reader030/viewer/2022020108/5ab91a1f7f8b9ad5338d9bf9/html5/thumbnails/14.jpg)
Probabili:es at Garmisch, 1968 John Nash, IBM Hursley
h[p://homepages.cs.ncl.ac.uk/brian.randell/NATO/N1968/GROUP1.html
Naur & Randell, SoDware Engineering: Report on a Conference sponsored by the NATO Science CommiLee, Garmisch, Germany, 7th to 11th October 1968, January 1969.
![Page 15: Norman Vincent Peale, - LARC Researchresearch.larc.smu.edu.sg/ase2016/slides/ase2016keynote20160907... · – Norman Vincent Peale, The Power of Posi,ve Thinking “Do not build up](https://reader030.vdocument.in/reader030/viewer/2022020108/5ab91a1f7f8b9ad5338d9bf9/html5/thumbnails/15.jpg)
Some Previous Efforts with
Probabilis:c Approaches• Performance Engineering (many)
• Cleanroom SoKware Engineering (Mills)
• Opera:onal Profiles and SoKware Reliability Engineering (Musa, …)
• Quan:ta:ve Goal Reasoning in KAOS (Lamsweerde, Le:er)
• Sta:s:cal Debugging (Harrold, Orso, Liblit, …)
• Probabilis:c Programming & Analysis (Poole, Pfeffer, Dwyer, Visser, …)
• Probabilis:c and Sta:s:cal Model Checking (many)
![Page 16: Norman Vincent Peale, - LARC Researchresearch.larc.smu.edu.sg/ase2016/slides/ase2016keynote20160907... · – Norman Vincent Peale, The Power of Posi,ve Thinking “Do not build up](https://reader030.vdocument.in/reader030/viewer/2022020108/5ab91a1f7f8b9ad5338d9bf9/html5/thumbnails/16.jpg)
Probabilis:c Model Checking
! ¬p→◊q( )∧"( )
Model
Checker
✓✕
State Machine
Model
Temporal Property
Results
Counterexample
Trace
System
Requirements
P≥0.95 [ ]
0.4
0.6
Probabilis:c
Probabilis:c
![Page 17: Norman Vincent Peale, - LARC Researchresearch.larc.smu.edu.sg/ase2016/slides/ase2016keynote20160907... · – Norman Vincent Peale, The Power of Posi,ve Thinking “Do not build up](https://reader030.vdocument.in/reader030/viewer/2022020108/5ab91a1f7f8b9ad5338d9bf9/html5/thumbnails/17.jpg)
Probabilis:c Model Checking
! ¬p→◊q( )∧"( )
Model
Checker
✓✕
State Machine
Model
Temporal Property
Results
Counterexample
Trace
System
Requirements
P=? [ ]
0.4
0.6
Quan:ta:ve Results
0.9732Probabilis:c
Probabilis:c
![Page 18: Norman Vincent Peale, - LARC Researchresearch.larc.smu.edu.sg/ase2016/slides/ase2016keynote20160907... · – Norman Vincent Peale, The Power of Posi,ve Thinking “Do not build up](https://reader030.vdocument.in/reader030/viewer/2022020108/5ab91a1f7f8b9ad5338d9bf9/html5/thumbnails/18.jpg)
Example The Zeroconf Protocol
s1 s0 s2 s3
q
1
1
{ok} {error}
{start} s4
s5
s6
s7
s8
1
1-q
1-p
1-p
1-p 1-p
p p p
p
1
Pr(unsuccessful message probe)Pr(new address in use)
from the PRISM group(Kwiatkowska et al.)
P≤0.05 [ true U error ]
0.1 0.9
0.5
0.5
0.10.10.1
0.9
0.9
0.9
![Page 19: Norman Vincent Peale, - LARC Researchresearch.larc.smu.edu.sg/ase2016/slides/ase2016keynote20160907... · – Norman Vincent Peale, The Power of Posi,ve Thinking “Do not build up](https://reader030.vdocument.in/reader030/viewer/2022020108/5ab91a1f7f8b9ad5338d9bf9/html5/thumbnails/19.jpg)
Some Previous Efforts with
Probabilis:c Approaches• Cleanroom SoKware Engineering
• Opera:onal Profiles & SoKware Reliability Engineering
• Quan:ta:ve Goal/Requirements Reasoning in KAOS
• Performance Engineering
• Sta:s:cal Debugging
• Probabilis:c Programming and Analysis
• Probabilis:c Model Checking
We lack an holistic approach
for the whole software development lifecycle
![Page 20: Norman Vincent Peale, - LARC Researchresearch.larc.smu.edu.sg/ase2016/slides/ase2016keynote20160907... · – Norman Vincent Peale, The Power of Posi,ve Thinking “Do not build up](https://reader030.vdocument.in/reader030/viewer/2022020108/5ab91a1f7f8b9ad5338d9bf9/html5/thumbnails/20.jpg)
Challenges in Taking a
Probabilis:c Viewpoint
1. Some Things Are Certain, Or Should Be
2. Educa:on and Training
3. Popula:on Sizes and Sample Sizes
4. Determining the Probabili:es
5. Pinpoin:ng the Root Cause of Uncertainty
![Page 21: Norman Vincent Peale, - LARC Researchresearch.larc.smu.edu.sg/ase2016/slides/ase2016keynote20160907... · – Norman Vincent Peale, The Power of Posi,ve Thinking “Do not build up](https://reader030.vdocument.in/reader030/viewer/2022020108/5ab91a1f7f8b9ad5338d9bf9/html5/thumbnails/21.jpg)
Challenges Some Things Are Certain, Or Should Be
![Page 22: Norman Vincent Peale, - LARC Researchresearch.larc.smu.edu.sg/ase2016/slides/ase2016keynote20160907... · – Norman Vincent Peale, The Power of Posi,ve Thinking “Do not build up](https://reader030.vdocument.in/reader030/viewer/2022020108/5ab91a1f7f8b9ad5338d9bf9/html5/thumbnails/22.jpg)
Challenges Some Things Are Certain, Or Should Be
![Page 23: Norman Vincent Peale, - LARC Researchresearch.larc.smu.edu.sg/ase2016/slides/ase2016keynote20160907... · – Norman Vincent Peale, The Power of Posi,ve Thinking “Do not build up](https://reader030.vdocument.in/reader030/viewer/2022020108/5ab91a1f7f8b9ad5338d9bf9/html5/thumbnails/23.jpg)
Challenges Some Things Are Certain, Or Should Be
Need to mix probabilistic and non-probabilistic approaches
![Page 24: Norman Vincent Peale, - LARC Researchresearch.larc.smu.edu.sg/ase2016/slides/ase2016keynote20160907... · – Norman Vincent Peale, The Power of Posi,ve Thinking “Do not build up](https://reader030.vdocument.in/reader030/viewer/2022020108/5ab91a1f7f8b9ad5338d9bf9/html5/thumbnails/24.jpg)
Challenges Educa:on and Training
![Page 25: Norman Vincent Peale, - LARC Researchresearch.larc.smu.edu.sg/ase2016/slides/ase2016keynote20160907... · – Norman Vincent Peale, The Power of Posi,ve Thinking “Do not build up](https://reader030.vdocument.in/reader030/viewer/2022020108/5ab91a1f7f8b9ad5338d9bf9/html5/thumbnails/25.jpg)
Challenges Educa:on and Training
![Page 26: Norman Vincent Peale, - LARC Researchresearch.larc.smu.edu.sg/ase2016/slides/ase2016keynote20160907... · – Norman Vincent Peale, The Power of Posi,ve Thinking “Do not build up](https://reader030.vdocument.in/reader030/viewer/2022020108/5ab91a1f7f8b9ad5338d9bf9/html5/thumbnails/26.jpg)
Challenges Educa:on and Training
![Page 27: Norman Vincent Peale, - LARC Researchresearch.larc.smu.edu.sg/ase2016/slides/ase2016keynote20160907... · – Norman Vincent Peale, The Power of Posi,ve Thinking “Do not build up](https://reader030.vdocument.in/reader030/viewer/2022020108/5ab91a1f7f8b9ad5338d9bf9/html5/thumbnails/27.jpg)
Challenges Educa:on and Training
![Page 28: Norman Vincent Peale, - LARC Researchresearch.larc.smu.edu.sg/ase2016/slides/ase2016keynote20160907... · – Norman Vincent Peale, The Power of Posi,ve Thinking “Do not build up](https://reader030.vdocument.in/reader030/viewer/2022020108/5ab91a1f7f8b9ad5338d9bf9/html5/thumbnails/28.jpg)
Challenges Popula:on Sizes and Sample Sizes
![Page 29: Norman Vincent Peale, - LARC Researchresearch.larc.smu.edu.sg/ase2016/slides/ase2016keynote20160907... · – Norman Vincent Peale, The Power of Posi,ve Thinking “Do not build up](https://reader030.vdocument.in/reader030/viewer/2022020108/5ab91a1f7f8b9ad5338d9bf9/html5/thumbnails/29.jpg)
Challenges Determining the Probabili:es
! ¬p→◊q( )∧"( )
Model
Checker
✓
State Machine
Model
Temporal Property
ResultsSystem
Requirements
P≥0.95 [ ]
0.4
0.6
Quan:ta:ve Results
0.9732Probabilis:c
Probabilis:c
![Page 30: Norman Vincent Peale, - LARC Researchresearch.larc.smu.edu.sg/ase2016/slides/ase2016keynote20160907... · – Norman Vincent Peale, The Power of Posi,ve Thinking “Do not build up](https://reader030.vdocument.in/reader030/viewer/2022020108/5ab91a1f7f8b9ad5338d9bf9/html5/thumbnails/30.jpg)
Challenges Determining the Probabili:es
! ¬p→◊q( )∧"( )
Model
Checker
✕
State Machine
Model
Temporal Property
Results
Counterexample
Trace
System
Requirements
P≥0.95 [ ]
Quan:ta:ve Results
Probabilis:c
Probabilis:c
0.41
0.59
0.6211
![Page 31: Norman Vincent Peale, - LARC Researchresearch.larc.smu.edu.sg/ase2016/slides/ase2016keynote20160907... · – Norman Vincent Peale, The Power of Posi,ve Thinking “Do not build up](https://reader030.vdocument.in/reader030/viewer/2022020108/5ab91a1f7f8b9ad5338d9bf9/html5/thumbnails/31.jpg)
Example The Zeroconf Protocol Revisited
s1 s0 s2 s3
q
1
1
{ok} {error}
{start} s4
s5
s6
s7
s8
1
1-q
1-p
1-p
1-p 1-p
p p p
p
1
from the PRISM group(Kwiatkowska et al.)
The packet-loss rate is determined by an
empirically es,mated probability distribu,on
Pr(packet loss)
0.1 0.9
0.5
0.5
0.10.10.1
0.9
0.9
0.9
![Page 32: Norman Vincent Peale, - LARC Researchresearch.larc.smu.edu.sg/ase2016/slides/ase2016keynote20160907... · – Norman Vincent Peale, The Power of Posi,ve Thinking “Do not build up](https://reader030.vdocument.in/reader030/viewer/2022020108/5ab91a1f7f8b9ad5338d9bf9/html5/thumbnails/32.jpg)
Perturbed Probabilis:c Systems (Current Research)
• Discrete-Time Markov Chains (DTMCs) • “Small” perturba:ons of probability parameters
• Reachability proper:es P≤p[ ] • DRA proper:es
• Linear, quadra:c bounds on verifica:on impact
see papers at ICFEM 2013, ICSE 2014, CONCUR 2014,ATVA 2014, FASE 2016, ICSE 2016, IEEE TSE 2016
• Markov Decision Processes (MDPs)
• Con:nuous-Time Markov Chains (CTMCs)
S? U S!
![Page 33: Norman Vincent Peale, - LARC Researchresearch.larc.smu.edu.sg/ase2016/slides/ase2016keynote20160907... · – Norman Vincent Peale, The Power of Posi,ve Thinking “Do not build up](https://reader030.vdocument.in/reader030/viewer/2022020108/5ab91a1f7f8b9ad5338d9bf9/html5/thumbnails/33.jpg)
Asympto:cPerturba:on Bounds
• Perturba:on Func:on
where A is the transi:on probability sub-matrix for S?
and b is the vector of one-step probabili:es from S? to S!
• Condi:on Number
• Predicted varia:on to probabilis:c verifica:onresult p due to perturba:on Δ:
ρ x( ) = ι? i A x i ib x( )− Ai ib( )( )i=0
∞
∑
κ = limδ→0
sup ρ(x − r)δ
: x − r ≤ δ ,δ > 0⎧⎨⎩
⎫⎬⎭
p̂ = p ±κΔ
![Page 34: Norman Vincent Peale, - LARC Researchresearch.larc.smu.edu.sg/ase2016/slides/ase2016keynote20160907... · – Norman Vincent Peale, The Power of Posi,ve Thinking “Do not build up](https://reader030.vdocument.in/reader030/viewer/2022020108/5ab91a1f7f8b9ad5338d9bf9/html5/thumbnails/34.jpg)
Case Study Results Noisy Zeroconf (35000 Hosts, PRISM)
p Actual Collision Probability
Predicted Collision Probability
0.095 -19.8% -21.5%
0.096 -16.9% -17.2%
0.097 -12.3% -12.9%
0.098 -8.33% -8.61%
0.099 -4.23% -4.30%
0.100 1.8567 ✕ 10-4 —0.101 +4.38% +4.30%
0.102 +8.91% +8.61%
0.103 +13.6% +12.9%
0.104 +18.4% +17.2%
0.105 +23.4% +21.5%
![Page 35: Norman Vincent Peale, - LARC Researchresearch.larc.smu.edu.sg/ase2016/slides/ase2016keynote20160907... · – Norman Vincent Peale, The Power of Posi,ve Thinking “Do not build up](https://reader030.vdocument.in/reader030/viewer/2022020108/5ab91a1f7f8b9ad5338d9bf9/html5/thumbnails/35.jpg)
Challenges Pinpoin:ng the Root Cause of Uncertainty
“There are known knowns; there are things we know we know. We also know
there are known unknowns; that is to say, we know there are some things we
do not know. But there are also unknown unknowns – the ones we don’t
know we don’t know.”
— Donald Rumsfeld
![Page 36: Norman Vincent Peale, - LARC Researchresearch.larc.smu.edu.sg/ase2016/slides/ase2016keynote20160907... · – Norman Vincent Peale, The Power of Posi,ve Thinking “Do not build up](https://reader030.vdocument.in/reader030/viewer/2022020108/5ab91a1f7f8b9ad5338d9bf9/html5/thumbnails/36.jpg)
The Changing Nature ofSoKware Engineering
✓ Autonomous Vehicles
✓ Cyber Physical Systems
✓ Internet of Things
see
Deep Learning and Understandability versusSoDware Engineering and Verifica,on
by Peter Norvig, Director of Research at Google
h[p://www.youtube.com/watch?v=X769cyzBNVw
![Page 37: Norman Vincent Peale, - LARC Researchresearch.larc.smu.edu.sg/ase2016/slides/ase2016keynote20160907... · – Norman Vincent Peale, The Power of Posi,ve Thinking “Do not build up](https://reader030.vdocument.in/reader030/viewer/2022020108/5ab91a1f7f8b9ad5338d9bf9/html5/thumbnails/37.jpg)
Example Affec:ve Compu:ng
![Page 38: Norman Vincent Peale, - LARC Researchresearch.larc.smu.edu.sg/ase2016/slides/ase2016keynote20160907... · – Norman Vincent Peale, The Power of Posi,ve Thinking “Do not build up](https://reader030.vdocument.in/reader030/viewer/2022020108/5ab91a1f7f8b9ad5338d9bf9/html5/thumbnails/38.jpg)
Example Affec:ve Compu:ng
When is an incorrect emo,on
classifica,on a bug, and when is it a
“feature”?
And how do you know?
![Page 39: Norman Vincent Peale, - LARC Researchresearch.larc.smu.edu.sg/ase2016/slides/ase2016keynote20160907... · – Norman Vincent Peale, The Power of Posi,ve Thinking “Do not build up](https://reader030.vdocument.in/reader030/viewer/2022020108/5ab91a1f7f8b9ad5338d9bf9/html5/thumbnails/39.jpg)
Uncertainty in Tes:ng (Current Research)
TestExecu:on
System
Under Test
Result Interpreta,on
Acceptable✓
![Page 40: Norman Vincent Peale, - LARC Researchresearch.larc.smu.edu.sg/ase2016/slides/ase2016keynote20160907... · – Norman Vincent Peale, The Power of Posi,ve Thinking “Do not build up](https://reader030.vdocument.in/reader030/viewer/2022020108/5ab91a1f7f8b9ad5338d9bf9/html5/thumbnails/40.jpg)
Uncertainty in Tes:ng (Current Research)
TestExecu:on
System
Under Test
Result Interpreta,on
Unacceptable
Acceptable✓
✕
![Page 41: Norman Vincent Peale, - LARC Researchresearch.larc.smu.edu.sg/ase2016/slides/ase2016keynote20160907... · – Norman Vincent Peale, The Power of Posi,ve Thinking “Do not build up](https://reader030.vdocument.in/reader030/viewer/2022020108/5ab91a1f7f8b9ad5338d9bf9/html5/thumbnails/41.jpg)
Uncertainty in Tes:ng (Current Research)
TestExecu:on
System
Under Test
Result Interpreta,on
Unacceptable
Acceptable
Acceptable
✓
✕
✕
![Page 42: Norman Vincent Peale, - LARC Researchresearch.larc.smu.edu.sg/ase2016/slides/ase2016keynote20160907... · – Norman Vincent Peale, The Power of Posi,ve Thinking “Do not build up](https://reader030.vdocument.in/reader030/viewer/2022020108/5ab91a1f7f8b9ad5338d9bf9/html5/thumbnails/42.jpg)
Uncertainty in Tes:ng (Current Research)
TestExecu:on
System
Under Test
Result Interpreta,on
Unacceptable
Acceptable
Acceptable
✓
✕
✕
Acceptable misbehaviors can mask real faults!
![Page 43: Norman Vincent Peale, - LARC Researchresearch.larc.smu.edu.sg/ase2016/slides/ase2016keynote20160907... · – Norman Vincent Peale, The Power of Posi,ve Thinking “Do not build up](https://reader030.vdocument.in/reader030/viewer/2022020108/5ab91a1f7f8b9ad5338d9bf9/html5/thumbnails/43.jpg)
One Possible Solu:on Distribu:on Fi{ng
System
Under Test
TrainingData WEKA
Sebastian Elbaum and David S. Rosenblum, “Known Unknowns: Testing in the Presence of Uncertainty”, Proc. FSE 2014.
![Page 44: Norman Vincent Peale, - LARC Researchresearch.larc.smu.edu.sg/ase2016/slides/ase2016keynote20160907... · – Norman Vincent Peale, The Power of Posi,ve Thinking “Do not build up](https://reader030.vdocument.in/reader030/viewer/2022020108/5ab91a1f7f8b9ad5338d9bf9/html5/thumbnails/44.jpg)
One Possible Solu:on Distribu:on Fi{ng
System
Under Test
WEKA
Sebastian Elbaum and David S. Rosenblum, “Known Unknowns: Testing in the Presence of Uncertainty”, Proc. FSE 2014.
![Page 45: Norman Vincent Peale, - LARC Researchresearch.larc.smu.edu.sg/ase2016/slides/ase2016keynote20160907... · – Norman Vincent Peale, The Power of Posi,ve Thinking “Do not build up](https://reader030.vdocument.in/reader030/viewer/2022020108/5ab91a1f7f8b9ad5338d9bf9/html5/thumbnails/45.jpg)
One Possible Solu:on Distribu:on Fi{ng
System
Under Test
Result Interpreta,on
Acceptablep < 0.99
TestExecu:on
WEKA
Sebastian Elbaum and David S. Rosenblum, “Known Unknowns: Testing in the Presence of Uncertainty”, Proc. FSE 2014.
![Page 46: Norman Vincent Peale, - LARC Researchresearch.larc.smu.edu.sg/ase2016/slides/ase2016keynote20160907... · – Norman Vincent Peale, The Power of Posi,ve Thinking “Do not build up](https://reader030.vdocument.in/reader030/viewer/2022020108/5ab91a1f7f8b9ad5338d9bf9/html5/thumbnails/46.jpg)
One Possible Solu:on Distribu:on Fi{ng
System
Under Test
Result Interpreta,on
Unacceptable
Acceptablep < 0.99
TestExecu:on
WEKA
p < 0.0027
Sebastian Elbaum and David S. Rosenblum, “Known Unknowns: Testing in the Presence of Uncertainty”, Proc. FSE 2014.
![Page 47: Norman Vincent Peale, - LARC Researchresearch.larc.smu.edu.sg/ase2016/slides/ase2016keynote20160907... · – Norman Vincent Peale, The Power of Posi,ve Thinking “Do not build up](https://reader030.vdocument.in/reader030/viewer/2022020108/5ab91a1f7f8b9ad5338d9bf9/html5/thumbnails/47.jpg)
One Possible Solu:on Distribu:on Fi{ng
System
Under Test
Result Interpreta,on
Unacceptable
Acceptable
Inconclusive
p < 0.99
TestExecu:on
WEKA
p < 0.37
p < 0.0027
Sebastian Elbaum and David S. Rosenblum, “Known Unknowns: Testing in the Presence of Uncertainty”, Proc. FSE 2014.
![Page 48: Norman Vincent Peale, - LARC Researchresearch.larc.smu.edu.sg/ase2016/slides/ase2016keynote20160907... · – Norman Vincent Peale, The Power of Posi,ve Thinking “Do not build up](https://reader030.vdocument.in/reader030/viewer/2022020108/5ab91a1f7f8b9ad5338d9bf9/html5/thumbnails/48.jpg)
Conclusion
There is poten:ally much to be gained by relaxing the
tradi:onal absolu:st view of soKware engineering
And there are great research opportuni:es inapplying a probabilis:c viewpoint
![Page 49: Norman Vincent Peale, - LARC Researchresearch.larc.smu.edu.sg/ase2016/slides/ase2016keynote20160907... · – Norman Vincent Peale, The Power of Posi,ve Thinking “Do not build up](https://reader030.vdocument.in/reader030/viewer/2022020108/5ab91a1f7f8b9ad5338d9bf9/html5/thumbnails/49.jpg)
– Norman Vincent Peale, The Power of Posi,ve Thinking
“Do not build up obstacles in your imagina:on.”
![Page 50: Norman Vincent Peale, - LARC Researchresearch.larc.smu.edu.sg/ase2016/slides/ase2016keynote20160907... · – Norman Vincent Peale, The Power of Posi,ve Thinking “Do not build up](https://reader030.vdocument.in/reader030/viewer/2022020108/5ab91a1f7f8b9ad5338d9bf9/html5/thumbnails/50.jpg)
National University of Singapore