![Page 1: Even the Very Wise Cannot See All Ends: Many … › ... › Keynote-TH-Tse-slides.pdfEven the Very Wise Cannot See All Ends: Many Facets of the Test Oracle Problem Prof. T.H. Tse](https://reader035.vdocument.in/reader035/viewer/2022070813/5f0c4f817e708231d434c453/html5/thumbnails/1.jpg)
Even the Very Wise Cannot See All Ends:
Many Facets of the Test Oracle Problem
Prof. T.H. Tse The University of Hong Kong Pokfulam, Hong Kong
![Page 2: Even the Very Wise Cannot See All Ends: Many … › ... › Keynote-TH-Tse-slides.pdfEven the Very Wise Cannot See All Ends: Many Facets of the Test Oracle Problem Prof. T.H. Tse](https://reader035.vdocument.in/reader035/viewer/2022070813/5f0c4f817e708231d434c453/html5/thumbnails/2.jpg)
Formula for A Successful Keynote Speech
{ ? ! " " _ ^ ∉ -> :-) }
2
![Page 3: Even the Very Wise Cannot See All Ends: Many … › ... › Keynote-TH-Tse-slides.pdfEven the Very Wise Cannot See All Ends: Many Facets of the Test Oracle Problem Prof. T.H. Tse](https://reader035.vdocument.in/reader035/viewer/2022070813/5f0c4f817e708231d434c453/html5/thumbnails/3.jpg)
Formula for A Successful Keynote Speech
{ ? ! " " _ ^ ∉ -> :-) }
3
Comprehensive coverage
![Page 4: Even the Very Wise Cannot See All Ends: Many … › ... › Keynote-TH-Tse-slides.pdfEven the Very Wise Cannot See All Ends: Many Facets of the Test Oracle Problem Prof. T.H. Tse](https://reader035.vdocument.in/reader035/viewer/2022070813/5f0c4f817e708231d434c453/html5/thumbnails/4.jpg)
Formula for A Successful Keynote Speech
{ ? ! " " _ ^ ∉ -> :-) }
4
Comprehensive coverage
Bold questions
![Page 5: Even the Very Wise Cannot See All Ends: Many … › ... › Keynote-TH-Tse-slides.pdfEven the Very Wise Cannot See All Ends: Many Facets of the Test Oracle Problem Prof. T.H. Tse](https://reader035.vdocument.in/reader035/viewer/2022070813/5f0c4f817e708231d434c453/html5/thumbnails/5.jpg)
Formula for A Successful Keynote Speech
{ ? ! " " _ ^ ∉ -> :-) }
5
Comprehensive coverage
Bold questions
Provocative assertions
![Page 6: Even the Very Wise Cannot See All Ends: Many … › ... › Keynote-TH-Tse-slides.pdfEven the Very Wise Cannot See All Ends: Many Facets of the Test Oracle Problem Prof. T.H. Tse](https://reader035.vdocument.in/reader035/viewer/2022070813/5f0c4f817e708231d434c453/html5/thumbnails/6.jpg)
Formula for A Successful Keynote Speech
{ ? ! " " _ ^ ∉ -> :-) }
6
Comprehensive coverage
Bold questions
Provocative assertions
Learn from gurus
![Page 7: Even the Very Wise Cannot See All Ends: Many … › ... › Keynote-TH-Tse-slides.pdfEven the Very Wise Cannot See All Ends: Many Facets of the Test Oracle Problem Prof. T.H. Tse](https://reader035.vdocument.in/reader035/viewer/2022070813/5f0c4f817e708231d434c453/html5/thumbnails/7.jpg)
Formula for A Successful Keynote Speech
{ ? ! " " _ ^ ∉ -> :-) }
7
Solid foundations
![Page 8: Even the Very Wise Cannot See All Ends: Many … › ... › Keynote-TH-Tse-slides.pdfEven the Very Wise Cannot See All Ends: Many Facets of the Test Oracle Problem Prof. T.H. Tse](https://reader035.vdocument.in/reader035/viewer/2022070813/5f0c4f817e708231d434c453/html5/thumbnails/8.jpg)
Formula for A Successful Keynote Speech
{ ? ! " " _ ^ ∉ -> :-) }
8
Solid foundations
Advanced concepts
![Page 9: Even the Very Wise Cannot See All Ends: Many … › ... › Keynote-TH-Tse-slides.pdfEven the Very Wise Cannot See All Ends: Many Facets of the Test Oracle Problem Prof. T.H. Tse](https://reader035.vdocument.in/reader035/viewer/2022070813/5f0c4f817e708231d434c453/html5/thumbnails/9.jpg)
Formula for A Successful Keynote Speech
{ ? ! " " _ ^ ∉ -> :-) }
9
Solid foundations
Advanced concepts
Avoid formalism
![Page 10: Even the Very Wise Cannot See All Ends: Many … › ... › Keynote-TH-Tse-slides.pdfEven the Very Wise Cannot See All Ends: Many Facets of the Test Oracle Problem Prof. T.H. Tse](https://reader035.vdocument.in/reader035/viewer/2022070813/5f0c4f817e708231d434c453/html5/thumbnails/10.jpg)
Formula for A Successful Keynote Speech
{ ? ! " " _ ^ ∉ -> :-) }
10
Real-world applications
![Page 11: Even the Very Wise Cannot See All Ends: Many … › ... › Keynote-TH-Tse-slides.pdfEven the Very Wise Cannot See All Ends: Many Facets of the Test Oracle Problem Prof. T.H. Tse](https://reader035.vdocument.in/reader035/viewer/2022070813/5f0c4f817e708231d434c453/html5/thumbnails/11.jpg)
Formula for A Successful Keynote Speech
{ ? ! " " _ ^ ∉ -> :-) }
11
Interesting stories
Real-world applications
![Page 12: Even the Very Wise Cannot See All Ends: Many … › ... › Keynote-TH-Tse-slides.pdfEven the Very Wise Cannot See All Ends: Many Facets of the Test Oracle Problem Prof. T.H. Tse](https://reader035.vdocument.in/reader035/viewer/2022070813/5f0c4f817e708231d434c453/html5/thumbnails/12.jpg)
Formula for A Successful Keynote Speech
{ ? ! " " _ ^ ∉ -> :-) }
12
Inspiring conclusions.
Interesting stories
Real-world applications
![Page 13: Even the Very Wise Cannot See All Ends: Many … › ... › Keynote-TH-Tse-slides.pdfEven the Very Wise Cannot See All Ends: Many Facets of the Test Oracle Problem Prof. T.H. Tse](https://reader035.vdocument.in/reader035/viewer/2022070813/5f0c4f817e708231d434c453/html5/thumbnails/13.jpg)
Presentation Outline Formula for a successful keynote speech A successful keynote speech.
13
![Page 14: Even the Very Wise Cannot See All Ends: Many … › ... › Keynote-TH-Tse-slides.pdfEven the Very Wise Cannot See All Ends: Many Facets of the Test Oracle Problem Prof. T.H. Tse](https://reader035.vdocument.in/reader035/viewer/2022070813/5f0c4f817e708231d434c453/html5/thumbnails/14.jpg)
Presentation Outline Background
14
![Page 15: Even the Very Wise Cannot See All Ends: Many … › ... › Keynote-TH-Tse-slides.pdfEven the Very Wise Cannot See All Ends: Many Facets of the Test Oracle Problem Prof. T.H. Tse](https://reader035.vdocument.in/reader035/viewer/2022070813/5f0c4f817e708231d434c453/html5/thumbnails/15.jpg)
Presentation Outline Background Many facets of the test
oracle problem
15
![Page 16: Even the Very Wise Cannot See All Ends: Many … › ... › Keynote-TH-Tse-slides.pdfEven the Very Wise Cannot See All Ends: Many Facets of the Test Oracle Problem Prof. T.H. Tse](https://reader035.vdocument.in/reader035/viewer/2022070813/5f0c4f817e708231d434c453/html5/thumbnails/16.jpg)
Presentation Outline Background Many facets of the test
oracle problem Expected outcome
= actual execution result
16
![Page 17: Even the Very Wise Cannot See All Ends: Many … › ... › Keynote-TH-Tse-slides.pdfEven the Very Wise Cannot See All Ends: Many Facets of the Test Oracle Problem Prof. T.H. Tse](https://reader035.vdocument.in/reader035/viewer/2022070813/5f0c4f817e708231d434c453/html5/thumbnails/17.jpg)
Presentation Outline Background Many facets of the test
oracle problem Expected outcome
= actual execution result Expected outcome
= actual execution result
17
![Page 18: Even the Very Wise Cannot See All Ends: Many … › ... › Keynote-TH-Tse-slides.pdfEven the Very Wise Cannot See All Ends: Many Facets of the Test Oracle Problem Prof. T.H. Tse](https://reader035.vdocument.in/reader035/viewer/2022070813/5f0c4f817e708231d434c453/html5/thumbnails/18.jpg)
Presentation Outline Background Many facets of the test
oracle problem Expected outcome
= actual execution result Expected outcome
= actual execution result Expected outcome
= actual execution result 18
![Page 19: Even the Very Wise Cannot See All Ends: Many … › ... › Keynote-TH-Tse-slides.pdfEven the Very Wise Cannot See All Ends: Many Facets of the Test Oracle Problem Prof. T.H. Tse](https://reader035.vdocument.in/reader035/viewer/2022070813/5f0c4f817e708231d434c453/html5/thumbnails/19.jpg)
Presentation Outline Background Many facets of the test
oracle problem Expected outcome
= actual execution result Expected outcome
= actual execution result Expected outcome
= actual execution result
Jungle of proposals
19
![Page 20: Even the Very Wise Cannot See All Ends: Many … › ... › Keynote-TH-Tse-slides.pdfEven the Very Wise Cannot See All Ends: Many Facets of the Test Oracle Problem Prof. T.H. Tse](https://reader035.vdocument.in/reader035/viewer/2022070813/5f0c4f817e708231d434c453/html5/thumbnails/20.jpg)
Presentation Outline Background Many facets of the test
oracle problem Expected outcome
= actual execution result Expected outcome
= actual execution result Expected outcome
= actual execution result
Jungle of proposals Empirical studies?
20
![Page 21: Even the Very Wise Cannot See All Ends: Many … › ... › Keynote-TH-Tse-slides.pdfEven the Very Wise Cannot See All Ends: Many Facets of the Test Oracle Problem Prof. T.H. Tse](https://reader035.vdocument.in/reader035/viewer/2022070813/5f0c4f817e708231d434c453/html5/thumbnails/21.jpg)
Presentation Outline Background Many facets of the test
oracle problem Expected outcome
= actual execution result Expected outcome
= actual execution result Expected outcome
= actual execution result
Jungle of proposals Empirical studies? What do other
researchers do?
21
![Page 22: Even the Very Wise Cannot See All Ends: Many … › ... › Keynote-TH-Tse-slides.pdfEven the Very Wise Cannot See All Ends: Many Facets of the Test Oracle Problem Prof. T.H. Tse](https://reader035.vdocument.in/reader035/viewer/2022070813/5f0c4f817e708231d434c453/html5/thumbnails/22.jpg)
Presentation Outline Background Many facets of the test
oracle problem Expected outcome
= actual execution result Expected outcome
= actual execution result Expected outcome
= actual execution result
Jungle of proposals Empirical studies? What do other
researchers do? Trim the tree or
tame the forest? .
22
![Page 23: Even the Very Wise Cannot See All Ends: Many … › ... › Keynote-TH-Tse-slides.pdfEven the Very Wise Cannot See All Ends: Many Facets of the Test Oracle Problem Prof. T.H. Tse](https://reader035.vdocument.in/reader035/viewer/2022070813/5f0c4f817e708231d434c453/html5/thumbnails/23.jpg)
Presentation Outline Background Many facets of the test
oracle problem Expected outcome
= actual execution result Expected outcome
= actual execution result Expected outcome
= actual execution result
Jungle of proposals Empirical studies? What do other
researchers do? Trim the tree or
tame the forest?
23
![Page 24: Even the Very Wise Cannot See All Ends: Many … › ... › Keynote-TH-Tse-slides.pdfEven the Very Wise Cannot See All Ends: Many Facets of the Test Oracle Problem Prof. T.H. Tse](https://reader035.vdocument.in/reader035/viewer/2022070813/5f0c4f817e708231d434c453/html5/thumbnails/24.jpg)
24
Summary of My Work Testing of object-oriented software
Black and White [ACM TOSEM 1998] TACCLE [ACM TOSEM 2001] VITAMIN [Communications of the ACM 2007] Equality to Equals and Unequals
[IEEE TSE 2013].
![Page 25: Even the Very Wise Cannot See All Ends: Many … › ... › Keynote-TH-Tse-slides.pdfEven the Very Wise Cannot See All Ends: Many Facets of the Test Oracle Problem Prof. T.H. Tse](https://reader035.vdocument.in/reader035/viewer/2022070813/5f0c4f817e708231d434c453/html5/thumbnails/25.jpg)
25
Testing pervasive software [COMPSAC 2004 best paper] [FSE 2006] [ICSE 2008].
Summary of My Work
![Page 26: Even the Very Wise Cannot See All Ends: Many … › ... › Keynote-TH-Tse-slides.pdfEven the Very Wise Cannot See All Ends: Many Facets of the Test Oracle Problem Prof. T.H. Tse](https://reader035.vdocument.in/reader035/viewer/2022070813/5f0c4f817e708231d434c453/html5/thumbnails/26.jpg)
26
Testing services computing [ICSE 2008b] [FSE 2009] [WWW 2009] [IEEE TSC 2015 spotlight paper] [IEEE TSC 2015b].
Summary of My Work
![Page 27: Even the Very Wise Cannot See All Ends: Many … › ... › Keynote-TH-Tse-slides.pdfEven the Very Wise Cannot See All Ends: Many Facets of the Test Oracle Problem Prof. T.H. Tse](https://reader035.vdocument.in/reader035/viewer/2022070813/5f0c4f817e708231d434c453/html5/thumbnails/27.jpg)
27
Summary of My Work Testing based on formal specifications
Tabular Expressions [IEEE TSE 2011]
![Page 28: Even the Very Wise Cannot See All Ends: Many … › ... › Keynote-TH-Tse-slides.pdfEven the Very Wise Cannot See All Ends: Many Facets of the Test Oracle Problem Prof. T.H. Tse](https://reader035.vdocument.in/reader035/viewer/2022070813/5f0c4f817e708231d434c453/html5/thumbnails/28.jpg)
28
Summary of My Work Testing based on formal specifications
Tabular Expressions [IEEE TSE 2011]
Testing based on informal specifications CHOC’LATE [IEEE TSE 2003] [Communications of the ACM 2010] DESSERT [IEEE TSE 2012].
![Page 29: Even the Very Wise Cannot See All Ends: Many … › ... › Keynote-TH-Tse-slides.pdfEven the Very Wise Cannot See All Ends: Many Facets of the Test Oracle Problem Prof. T.H. Tse](https://reader035.vdocument.in/reader035/viewer/2022070813/5f0c4f817e708231d434c453/html5/thumbnails/29.jpg)
29
Spectrum-based fault localization [COMPSAC 2008 best paper] [FSE 2009b] [IEEE Computer 2012]
Summary of My Work
![Page 30: Even the Very Wise Cannot See All Ends: Many … › ... › Keynote-TH-Tse-slides.pdfEven the Very Wise Cannot See All Ends: Many Facets of the Test Oracle Problem Prof. T.H. Tse](https://reader035.vdocument.in/reader035/viewer/2022070813/5f0c4f817e708231d434c453/html5/thumbnails/30.jpg)
30
Spectrum-based fault localization COMPSAC 2008 best paper] [FSE 2009b] [IEEE Computer 2012]
Debugging of concurrent systems [Information Sciences 2012] [ISSTA 2012].
Summary of My Work
![Page 31: Even the Very Wise Cannot See All Ends: Many … › ... › Keynote-TH-Tse-slides.pdfEven the Very Wise Cannot See All Ends: Many Facets of the Test Oracle Problem Prof. T.H. Tse](https://reader035.vdocument.in/reader035/viewer/2022070813/5f0c4f817e708231d434c453/html5/thumbnails/31.jpg)
31
Integration of testing, debugging, and proving [COMPSAC 2009 best paper] [QSIC 2011 best paper] [IEEE TSE 2011b].
Summary of My Work
![Page 32: Even the Very Wise Cannot See All Ends: Many … › ... › Keynote-TH-Tse-slides.pdfEven the Very Wise Cannot See All Ends: Many Facets of the Test Oracle Problem Prof. T.H. Tse](https://reader035.vdocument.in/reader035/viewer/2022070813/5f0c4f817e708231d434c453/html5/thumbnails/32.jpg)
32
Selected PhD Graduates
Dr Ke Zhai, Goldman Sachs.
Dr W.K. Chan, Associate Professor, City University of Hong Kong
Dr Bo Jiang, Associate Professor, Beihang University
Dr Zhenyu Zhang, Associate Professor, Institute of Software, Chinese Academy of Sciences
Dr Lijun Mei, IBM Research China
![Page 33: Even the Very Wise Cannot See All Ends: Many … › ... › Keynote-TH-Tse-slides.pdfEven the Very Wise Cannot See All Ends: Many Facets of the Test Oracle Problem Prof. T.H. Tse](https://reader035.vdocument.in/reader035/viewer/2022070813/5f0c4f817e708231d434c453/html5/thumbnails/33.jpg)
Dr Yan Cai, Associate Professor, Institute of Software, Chinese Academy of Sciences.
33
Selected PhD Graduates
Dr Ke Zhai, Goldman Sachs
Dr W.K. Chan, Associate Professor, City University of Hong Kong
Dr Bo Jiang, Associate Professor, Beihang University
Dr Zhenyu Zhang, Associate Professor, Institute of Software, Chinese Academy of Sciences
Dr Lijun Mei, IBM Research China
![Page 34: Even the Very Wise Cannot See All Ends: Many … › ... › Keynote-TH-Tse-slides.pdfEven the Very Wise Cannot See All Ends: Many Facets of the Test Oracle Problem Prof. T.H. Tse](https://reader035.vdocument.in/reader035/viewer/2022070813/5f0c4f817e708231d434c453/html5/thumbnails/34.jpg)
Presentation Outline Background Many facets of the test
oracle problem Expected outcome
= actual execution result Expected outcome
= actual execution result Expected outcome
= actual execution result
Jungle of proposals Empirical studies? What do other
researchers do? Trim the tree or
tame the forest?
34
![Page 35: Even the Very Wise Cannot See All Ends: Many … › ... › Keynote-TH-Tse-slides.pdfEven the Very Wise Cannot See All Ends: Many Facets of the Test Oracle Problem Prof. T.H. Tse](https://reader035.vdocument.in/reader035/viewer/2022070813/5f0c4f817e708231d434c453/html5/thumbnails/35.jpg)
The Test Oracle Problem
Even the Very Wise Cannot See All Ends:
Many Facets of the Test Oracle Problem
Prof. T.H. TseThe University of Hong KongPokfulam, Hong Kong
Present 20 years of work in 45 minutes
35
![Page 36: Even the Very Wise Cannot See All Ends: Many … › ... › Keynote-TH-Tse-slides.pdfEven the Very Wise Cannot See All Ends: Many Facets of the Test Oracle Problem Prof. T.H. Tse](https://reader035.vdocument.in/reader035/viewer/2022070813/5f0c4f817e708231d434c453/html5/thumbnails/36.jpg)
Even the Very Wise Cannot See All Ends:
Many Facets of the Test Oracle Problem
Prof. T.H. TseThe University of Hong KongPokfulam, Hong Kong
Many Facets of the Test Oracle Problem
J.R.R. Tolkien The Lord of the Rings
Even the Very Wise Cannot See All Ends
36
![Page 37: Even the Very Wise Cannot See All Ends: Many … › ... › Keynote-TH-Tse-slides.pdfEven the Very Wise Cannot See All Ends: Many Facets of the Test Oracle Problem Prof. T.H. Tse](https://reader035.vdocument.in/reader035/viewer/2022070813/5f0c4f817e708231d434c453/html5/thumbnails/37.jpg)
Rawlinson and Bosworth Professor of Anglo-Saxon, University of Oxford (1925−1945)
Merton Professor of English Language and Literature, University of Oxford (1945−1959).
Even the Very Wise Cannot See All Ends:
Many Facets of the Test Oracle Problem
Prof. T.H. TseThe University of Hong KongPokfulam, Hong Kong
Many Facets of the Test Oracle Problem
J.R.R. Tolkien The Lord of the Rings
Even the Very Wise Cannot See All Ends
![Page 38: Even the Very Wise Cannot See All Ends: Many … › ... › Keynote-TH-Tse-slides.pdfEven the Very Wise Cannot See All Ends: Many Facets of the Test Oracle Problem Prof. T.H. Tse](https://reader035.vdocument.in/reader035/viewer/2022070813/5f0c4f817e708231d434c453/html5/thumbnails/38.jpg)
Oracle A message supposedly from
God’s inspiration, usually given by a priest
![Page 39: Even the Very Wise Cannot See All Ends: Many … › ... › Keynote-TH-Tse-slides.pdfEven the Very Wise Cannot See All Ends: Many Facets of the Test Oracle Problem Prof. T.H. Tse](https://reader035.vdocument.in/reader035/viewer/2022070813/5f0c4f817e708231d434c453/html5/thumbnails/39.jpg)
Oracle A message supposedly from
God’s inspiration, usually given by a priest
An utterance of deep import or wisdom; an opinion or declaration regarded as authoritative and infallible; undeniable truth.
39
![Page 40: Even the Very Wise Cannot See All Ends: Many … › ... › Keynote-TH-Tse-slides.pdfEven the Very Wise Cannot See All Ends: Many Facets of the Test Oracle Problem Prof. T.H. Tse](https://reader035.vdocument.in/reader035/viewer/2022070813/5f0c4f817e708231d434c453/html5/thumbnails/40.jpg)
Oracle A message supposedly from
God’s inspiration, usually given by a priest
An utterance of deep import or wisdom; an opinion or declaration regarded as authoritative and infallible; undeniable truth.
![Page 41: Even the Very Wise Cannot See All Ends: Many … › ... › Keynote-TH-Tse-slides.pdfEven the Very Wise Cannot See All Ends: Many Facets of the Test Oracle Problem Prof. T.H. Tse](https://reader035.vdocument.in/reader035/viewer/2022070813/5f0c4f817e708231d434c453/html5/thumbnails/41.jpg)
41
Test Oracle A test oracle is a mechanism to check whether
expected outcome according to the specification = actual result of executing the implementation.
![Page 42: Even the Very Wise Cannot See All Ends: Many … › ... › Keynote-TH-Tse-slides.pdfEven the Very Wise Cannot See All Ends: Many Facets of the Test Oracle Problem Prof. T.H. Tse](https://reader035.vdocument.in/reader035/viewer/2022070813/5f0c4f817e708231d434c453/html5/thumbnails/42.jpg)
42
Real-Life Search Engine Example
![Page 43: Even the Very Wise Cannot See All Ends: Many … › ... › Keynote-TH-Tse-slides.pdfEven the Very Wise Cannot See All Ends: Many Facets of the Test Oracle Problem Prof. T.H. Tse](https://reader035.vdocument.in/reader035/viewer/2022070813/5f0c4f817e708231d434c453/html5/thumbnails/43.jpg)
43
Real-Life Search Engine Example Is 2.77M results correct? .
![Page 44: Even the Very Wise Cannot See All Ends: Many … › ... › Keynote-TH-Tse-slides.pdfEven the Very Wise Cannot See All Ends: Many Facets of the Test Oracle Problem Prof. T.H. Tse](https://reader035.vdocument.in/reader035/viewer/2022070813/5f0c4f817e708231d434c453/html5/thumbnails/44.jpg)
Challenge 1: Expected outcome = actual execution result
44
Many Facets of the Test Oracle Problem
![Page 45: Even the Very Wise Cannot See All Ends: Many … › ... › Keynote-TH-Tse-slides.pdfEven the Very Wise Cannot See All Ends: Many Facets of the Test Oracle Problem Prof. T.H. Tse](https://reader035.vdocument.in/reader035/viewer/2022070813/5f0c4f817e708231d434c453/html5/thumbnails/45.jpg)
Metamorphic Testing
45
sin 0.9876 ? Expected 0.8347 Actual
![Page 46: Even the Very Wise Cannot See All Ends: Many … › ... › Keynote-TH-Tse-slides.pdfEven the Very Wise Cannot See All Ends: Many Facets of the Test Oracle Problem Prof. T.H. Tse](https://reader035.vdocument.in/reader035/viewer/2022070813/5f0c4f817e708231d434c453/html5/thumbnails/46.jpg)
Metamorphic Testing
46
sin 0.9876 ? Expected 0.8347 Actual
Cannot be verified because we do not know what to expect
![Page 47: Even the Very Wise Cannot See All Ends: Many … › ... › Keynote-TH-Tse-slides.pdfEven the Very Wise Cannot See All Ends: Many Facets of the Test Oracle Problem Prof. T.H. Tse](https://reader035.vdocument.in/reader035/viewer/2022070813/5f0c4f817e708231d434c453/html5/thumbnails/47.jpg)
Metamorphic Testing
47
sin 0.9876 ? Expected 0.8347 Actual
? Expected 0.8347 Actual sin (π – 0.9876)
Take a follow-up test case:
Cannot be verified because we do not know what to expect
![Page 48: Even the Very Wise Cannot See All Ends: Many … › ... › Keynote-TH-Tse-slides.pdfEven the Very Wise Cannot See All Ends: Many Facets of the Test Oracle Problem Prof. T.H. Tse](https://reader035.vdocument.in/reader035/viewer/2022070813/5f0c4f817e708231d434c453/html5/thumbnails/48.jpg)
Metamorphic Testing
48
sin 0.9876
sin (π – 0.9876)
? Expected 0.8347 Actual
? Expected 0.8347 Actual
![Page 49: Even the Very Wise Cannot See All Ends: Many … › ... › Keynote-TH-Tse-slides.pdfEven the Very Wise Cannot See All Ends: Many Facets of the Test Oracle Problem Prof. T.H. Tse](https://reader035.vdocument.in/reader035/viewer/2022070813/5f0c4f817e708231d434c453/html5/thumbnails/49.jpg)
Metamorphic Testing
49
sin 0.9876
sin (π – 0.9876)
? Expected 0.8347 Actual
? Expected 0.8347 Actual
![Page 50: Even the Very Wise Cannot See All Ends: Many … › ... › Keynote-TH-Tse-slides.pdfEven the Very Wise Cannot See All Ends: Many Facets of the Test Oracle Problem Prof. T.H. Tse](https://reader035.vdocument.in/reader035/viewer/2022070813/5f0c4f817e708231d434c453/html5/thumbnails/50.jpg)
Metamorphic Testing
50
sin (π – 0.9876) ? Expected 0.8347 Actual
sin 0.9876 = sin (π – 0.9876)
Expected metamorphic relation
sin 0.9876 ? Expected 0.8347 Actual
![Page 51: Even the Very Wise Cannot See All Ends: Many … › ... › Keynote-TH-Tse-slides.pdfEven the Very Wise Cannot See All Ends: Many Facets of the Test Oracle Problem Prof. T.H. Tse](https://reader035.vdocument.in/reader035/viewer/2022070813/5f0c4f817e708231d434c453/html5/thumbnails/51.jpg)
Metamorphic Testing
51
sin (π – 0.9876) ? Expected 0.8347 Actual
sin 0.9876 = sin (π – 0.9876)
Expected metamorphic relation
sin 0.9876 ? Expected 0.8347 Actual
![Page 52: Even the Very Wise Cannot See All Ends: Many … › ... › Keynote-TH-Tse-slides.pdfEven the Very Wise Cannot See All Ends: Many Facets of the Test Oracle Problem Prof. T.H. Tse](https://reader035.vdocument.in/reader035/viewer/2022070813/5f0c4f817e708231d434c453/html5/thumbnails/52.jpg)
Metamorphic Testing
52
sin (π – 0.9876) ? Expected 0.8347 Actual
sin 0.9876 = sin (π – 0.9876)
Expected metamorphic relation
0.8347 = 0.8347
Actual relation
sin 0.9876 ? Expected 0.8347 Actual
![Page 53: Even the Very Wise Cannot See All Ends: Many … › ... › Keynote-TH-Tse-slides.pdfEven the Very Wise Cannot See All Ends: Many Facets of the Test Oracle Problem Prof. T.H. Tse](https://reader035.vdocument.in/reader035/viewer/2022070813/5f0c4f817e708231d434c453/html5/thumbnails/53.jpg)
53
Real-Life Search Engine Example Apply Metamorphic Testing
Real-Life ExampleIs 2.77Mresults correct? .
Metamorphic Relation: More refined search should produce fewer number of entries.
![Page 54: Even the Very Wise Cannot See All Ends: Many … › ... › Keynote-TH-Tse-slides.pdfEven the Very Wise Cannot See All Ends: Many Facets of the Test Oracle Problem Prof. T.H. Tse](https://reader035.vdocument.in/reader035/viewer/2022070813/5f0c4f817e708231d434c453/html5/thumbnails/54.jpg)
Real-Life Search Engine
More entries indicate failure in search engine.
Refine to “children’s hospital of chicago” .
![Page 55: Even the Very Wise Cannot See All Ends: Many … › ... › Keynote-TH-Tse-slides.pdfEven the Very Wise Cannot See All Ends: Many Facets of the Test Oracle Problem Prof. T.H. Tse](https://reader035.vdocument.in/reader035/viewer/2022070813/5f0c4f817e708231d434c453/html5/thumbnails/55.jpg)
Real-Life Search Engine Example Is 4.54M results correct? .
![Page 56: Even the Very Wise Cannot See All Ends: Many … › ... › Keynote-TH-Tse-slides.pdfEven the Very Wise Cannot See All Ends: Many Facets of the Test Oracle Problem Prof. T.H. Tse](https://reader035.vdocument.in/reader035/viewer/2022070813/5f0c4f817e708231d434c453/html5/thumbnails/56.jpg)
Real-Life Search Engine
More entries indicate failure in search engine.
Refine to “children’s hospital of chicago” .
![Page 57: Even the Very Wise Cannot See All Ends: Many … › ... › Keynote-TH-Tse-slides.pdfEven the Very Wise Cannot See All Ends: Many Facets of the Test Oracle Problem Prof. T.H. Tse](https://reader035.vdocument.in/reader035/viewer/2022070813/5f0c4f817e708231d434c453/html5/thumbnails/57.jpg)
Real-Life Search Engine Example Human Issues
Microsoft Research selected our project for a Virtual Earth award
Google asked us to submit a bug report Each of my brothers has 2 doctoral degrees I have only one doctoral degree.
57
![Page 58: Even the Very Wise Cannot See All Ends: Many … › ... › Keynote-TH-Tse-slides.pdfEven the Very Wise Cannot See All Ends: Many Facets of the Test Oracle Problem Prof. T.H. Tse](https://reader035.vdocument.in/reader035/viewer/2022070813/5f0c4f817e708231d434c453/html5/thumbnails/58.jpg)
Metamorphic Testing Other Real-World Applications
58
Services computing Ubiquitous computing Concurrent systems Graphic applications Numerical programs.
![Page 59: Even the Very Wise Cannot See All Ends: Many … › ... › Keynote-TH-Tse-slides.pdfEven the Very Wise Cannot See All Ends: Many Facets of the Test Oracle Problem Prof. T.H. Tse](https://reader035.vdocument.in/reader035/viewer/2022070813/5f0c4f817e708231d434c453/html5/thumbnails/59.jpg)
Metamorphic Testing Recent Track Records
59
Compilers Based on metamorphic testing, an “equivalence
modulo inputs” technique identified 147 unique faults in GCC and LLVM using one metamorphic relation
![Page 60: Even the Very Wise Cannot See All Ends: Many … › ... › Keynote-TH-Tse-slides.pdfEven the Very Wise Cannot See All Ends: Many Facets of the Test Oracle Problem Prof. T.H. Tse](https://reader035.vdocument.in/reader035/viewer/2022070813/5f0c4f817e708231d434c453/html5/thumbnails/60.jpg)
Metamorphic Testing Recent Track Records
60
Compilers Based on metamorphic testing, an “equivalence
modulo inputs” technique identified 147 unique faults in GCC and LLVM using one metamorphic relation
Siemens Suite Detected 3 new faults in Siemens suite after its long
history of test case studies.
![Page 61: Even the Very Wise Cannot See All Ends: Many … › ... › Keynote-TH-Tse-slides.pdfEven the Very Wise Cannot See All Ends: Many Facets of the Test Oracle Problem Prof. T.H. Tse](https://reader035.vdocument.in/reader035/viewer/2022070813/5f0c4f817e708231d434c453/html5/thumbnails/61.jpg)
Challenge 2:
61
Many Facets of the Test Oracle Problem
Expected outcome = actual execution result
![Page 62: Even the Very Wise Cannot See All Ends: Many … › ... › Keynote-TH-Tse-slides.pdfEven the Very Wise Cannot See All Ends: Many Facets of the Test Oracle Problem Prof. T.H. Tse](https://reader035.vdocument.in/reader035/viewer/2022070813/5f0c4f817e708231d434c453/html5/thumbnails/62.jpg)
Challenge 2:
62
Consider the testing of real-life object-oriented software.
Many Facets of the Test Oracle Problem
Expected outcome = actual execution result
![Page 63: Even the Very Wise Cannot See All Ends: Many … › ... › Keynote-TH-Tse-slides.pdfEven the Very Wise Cannot See All Ends: Many Facets of the Test Oracle Problem Prof. T.H. Tse](https://reader035.vdocument.in/reader035/viewer/2022070813/5f0c4f817e708231d434c453/html5/thumbnails/63.jpg)
Testing of Object-Oriented Software Our Real-Life Experience
63
Technology-transfer project for ASM, the world’s largest supplier of assembly and packaging equipment for the semiconductor industry.
![Page 64: Even the Very Wise Cannot See All Ends: Many … › ... › Keynote-TH-Tse-slides.pdfEven the Very Wise Cannot See All Ends: Many Facets of the Test Oracle Problem Prof. T.H. Tse](https://reader035.vdocument.in/reader035/viewer/2022070813/5f0c4f817e708231d434c453/html5/thumbnails/64.jpg)
Testing of Object-Oriented Software
64
Robotic arm after handling problem chip
Home afresh Expected
Home afresh Actual
Simple concept in specification
Complex implementation
![Page 65: Even the Very Wise Cannot See All Ends: Many … › ... › Keynote-TH-Tse-slides.pdfEven the Very Wise Cannot See All Ends: Many Facets of the Test Oracle Problem Prof. T.H. Tse](https://reader035.vdocument.in/reader035/viewer/2022070813/5f0c4f817e708231d434c453/html5/thumbnails/65.jpg)
Challenge 2: Expected outcome = actual execution result Consider the testing of real-life object-oriented
software
65
Many Facets of the Test Oracle Problem
Cannot define “=” at two different levels of abstraction.
![Page 66: Even the Very Wise Cannot See All Ends: Many … › ... › Keynote-TH-Tse-slides.pdfEven the Very Wise Cannot See All Ends: Many Facets of the Test Oracle Problem Prof. T.H. Tse](https://reader035.vdocument.in/reader035/viewer/2022070813/5f0c4f817e708231d434c453/html5/thumbnails/66.jpg)
Testing of Object-Oriented Software Mimic Metamorphic Testing?
66
Robotic arm after handling problem chip
Home afresh Expected
Home afresh Actual
![Page 67: Even the Very Wise Cannot See All Ends: Many … › ... › Keynote-TH-Tse-slides.pdfEven the Very Wise Cannot See All Ends: Many Facets of the Test Oracle Problem Prof. T.H. Tse](https://reader035.vdocument.in/reader035/viewer/2022070813/5f0c4f817e708231d434c453/html5/thumbnails/67.jpg)
Testing of Object-Oriented Software Mimic Metamorphic Testing?
67
Robotic arm after handling problem chip
Home afresh Expected
Home afresh Actual
Robotic arm after handling standard chip
![Page 68: Even the Very Wise Cannot See All Ends: Many … › ... › Keynote-TH-Tse-slides.pdfEven the Very Wise Cannot See All Ends: Many Facets of the Test Oracle Problem Prof. T.H. Tse](https://reader035.vdocument.in/reader035/viewer/2022070813/5f0c4f817e708231d434c453/html5/thumbnails/68.jpg)
Testing of Object-Oriented Software Mimic Metamorphic Testing?
68
Robotic arm after handling problem chip
Home afresh Expected
Home afresh Actual
Robotic arm after handling standard chip
Home afresh Expected
![Page 69: Even the Very Wise Cannot See All Ends: Many … › ... › Keynote-TH-Tse-slides.pdfEven the Very Wise Cannot See All Ends: Many Facets of the Test Oracle Problem Prof. T.H. Tse](https://reader035.vdocument.in/reader035/viewer/2022070813/5f0c4f817e708231d434c453/html5/thumbnails/69.jpg)
Testing of Object-Oriented Software Mimic Metamorphic Testing?
69
Robotic arm after handling problem chip
Home afresh Expected
Home afresh Actual
Robotic arm after handling standard chip
Home afresh Expected Home
afresh Actual
![Page 70: Even the Very Wise Cannot See All Ends: Many … › ... › Keynote-TH-Tse-slides.pdfEven the Very Wise Cannot See All Ends: Many Facets of the Test Oracle Problem Prof. T.H. Tse](https://reader035.vdocument.in/reader035/viewer/2022070813/5f0c4f817e708231d434c453/html5/thumbnails/70.jpg)
Testing of Object-Oriented Software Mimic Metamorphic Testing?
70
Robotic arm after handling problem chip
Home afresh Expected
Home afresh Actual
Robotic arm after handling standard chip
Home afresh Expected Home
afresh Actual
![Page 71: Even the Very Wise Cannot See All Ends: Many … › ... › Keynote-TH-Tse-slides.pdfEven the Very Wise Cannot See All Ends: Many Facets of the Test Oracle Problem Prof. T.H. Tse](https://reader035.vdocument.in/reader035/viewer/2022070813/5f0c4f817e708231d434c453/html5/thumbnails/71.jpg)
Testing of Object-Oriented Software Mimic Metamorphic Testing?
71
Robotic arm after handling problem chip
Home afresh Expected
Home afresh Actual
Robotic arm after handling standard chip
Home afresh Expected Home
afresh Actual
![Page 72: Even the Very Wise Cannot See All Ends: Many … › ... › Keynote-TH-Tse-slides.pdfEven the Very Wise Cannot See All Ends: Many Facets of the Test Oracle Problem Prof. T.H. Tse](https://reader035.vdocument.in/reader035/viewer/2022070813/5f0c4f817e708231d434c453/html5/thumbnails/72.jpg)
Testing of Object-Oriented Software Mimic Metamorphic Testing?
72
Robotic arm after handling problem chip
Home afresh Expected
Home afresh Actual
Robotic arm after handling standard chip
Home afresh Expected Home
afresh Actual
Equivalent
Expected relation
![Page 73: Even the Very Wise Cannot See All Ends: Many … › ... › Keynote-TH-Tse-slides.pdfEven the Very Wise Cannot See All Ends: Many Facets of the Test Oracle Problem Prof. T.H. Tse](https://reader035.vdocument.in/reader035/viewer/2022070813/5f0c4f817e708231d434c453/html5/thumbnails/73.jpg)
Testing of Object-Oriented Software Mimic Metamorphic Testing?
73
Robotic arm after handling problem chip
Home afresh Expected
Home afresh Actual
Robotic arm after handling standard chip
Home afresh Expected Home
afresh Actual
Equivalent
Expected relation
![Page 74: Even the Very Wise Cannot See All Ends: Many … › ... › Keynote-TH-Tse-slides.pdfEven the Very Wise Cannot See All Ends: Many Facets of the Test Oracle Problem Prof. T.H. Tse](https://reader035.vdocument.in/reader035/viewer/2022070813/5f0c4f817e708231d434c453/html5/thumbnails/74.jpg)
Testing of Object-Oriented Software Mimic Metamorphic Testing?
74
Robotic arm after handling problem chip
Home afresh Expected
Home afresh Actual
Robotic arm after handling standard chip
Home afresh Expected Home
afresh Actual
Equivalent
Expected relation
Equivalent
Actual relation
![Page 75: Even the Very Wise Cannot See All Ends: Many … › ... › Keynote-TH-Tse-slides.pdfEven the Very Wise Cannot See All Ends: Many Facets of the Test Oracle Problem Prof. T.H. Tse](https://reader035.vdocument.in/reader035/viewer/2022070813/5f0c4f817e708231d434c453/html5/thumbnails/75.jpg)
An implementation P is correct with respect to the specification Sp if and only if For any pair of equivalent sequences of
operations in Sp, the actual objects resulting from P must be equivalent .
Target of OO Software Testing
75
![Page 76: Even the Very Wise Cannot See All Ends: Many … › ... › Keynote-TH-Tse-slides.pdfEven the Very Wise Cannot See All Ends: Many Facets of the Test Oracle Problem Prof. T.H. Tse](https://reader035.vdocument.in/reader035/viewer/2022070813/5f0c4f817e708231d434c453/html5/thumbnails/76.jpg)
76
Challenge 3:
Many Facets of the Test Oracle Problem
What is object equivalence?
Object A = object B
![Page 77: Even the Very Wise Cannot See All Ends: Many … › ... › Keynote-TH-Tse-slides.pdfEven the Very Wise Cannot See All Ends: Many Facets of the Test Oracle Problem Prof. T.H. Tse](https://reader035.vdocument.in/reader035/viewer/2022070813/5f0c4f817e708231d434c453/html5/thumbnails/77.jpg)
What is Object Equivalence? Real-Life Word Processing Example
![Page 78: Even the Very Wise Cannot See All Ends: Many … › ... › Keynote-TH-Tse-slides.pdfEven the Very Wise Cannot See All Ends: Many Facets of the Test Oracle Problem Prof. T.H. Tse](https://reader035.vdocument.in/reader035/viewer/2022070813/5f0c4f817e708231d434c453/html5/thumbnails/78.jpg)
What is Object Equivalence? Real-Life Word Processing Example
![Page 79: Even the Very Wise Cannot See All Ends: Many … › ... › Keynote-TH-Tse-slides.pdfEven the Very Wise Cannot See All Ends: Many Facets of the Test Oracle Problem Prof. T.H. Tse](https://reader035.vdocument.in/reader035/viewer/2022070813/5f0c4f817e708231d434c453/html5/thumbnails/79.jpg)
What is Object Equivalence? Real-Life Word Processing Example
Call this object O.
![Page 80: Even the Very Wise Cannot See All Ends: Many … › ... › Keynote-TH-Tse-slides.pdfEven the Very Wise Cannot See All Ends: Many Facets of the Test Oracle Problem Prof. T.H. Tse](https://reader035.vdocument.in/reader035/viewer/2022070813/5f0c4f817e708231d434c453/html5/thumbnails/80.jpg)
What is Object Equivalence? Real-Life Word Processing Example
Click “Hidden”.
![Page 81: Even the Very Wise Cannot See All Ends: Many … › ... › Keynote-TH-Tse-slides.pdfEven the Very Wise Cannot See All Ends: Many Facets of the Test Oracle Problem Prof. T.H. Tse](https://reader035.vdocument.in/reader035/viewer/2022070813/5f0c4f817e708231d434c453/html5/thumbnails/81.jpg)
What is Object Equivalence? Real-Life Word Processing Example
Call this object H.
![Page 82: Even the Very Wise Cannot See All Ends: Many … › ... › Keynote-TH-Tse-slides.pdfEven the Very Wise Cannot See All Ends: Many Facets of the Test Oracle Problem Prof. T.H. Tse](https://reader035.vdocument.in/reader035/viewer/2022070813/5f0c4f817e708231d434c453/html5/thumbnails/82.jpg)
T.H. Tse is an Honorary Professor in Computer Science at The University of Hong Kong. He is a Steering Committee Chair of the IEEE International Conference on Software Quality, Reliability & Security.
82
What is Object Equivalence? Output from Object H
![Page 83: Even the Very Wise Cannot See All Ends: Many … › ... › Keynote-TH-Tse-slides.pdfEven the Very Wise Cannot See All Ends: Many Facets of the Test Oracle Problem Prof. T.H. Tse](https://reader035.vdocument.in/reader035/viewer/2022070813/5f0c4f817e708231d434c453/html5/thumbnails/83.jpg)
What is Object Equivalence? Real-Life Word Processing Example
Take object O again.
![Page 84: Even the Very Wise Cannot See All Ends: Many … › ... › Keynote-TH-Tse-slides.pdfEven the Very Wise Cannot See All Ends: Many Facets of the Test Oracle Problem Prof. T.H. Tse](https://reader035.vdocument.in/reader035/viewer/2022070813/5f0c4f817e708231d434c453/html5/thumbnails/84.jpg)
What is Object Equivalence? Real-Life Word Processing Example
Delete.
![Page 85: Even the Very Wise Cannot See All Ends: Many … › ... › Keynote-TH-Tse-slides.pdfEven the Very Wise Cannot See All Ends: Many Facets of the Test Oracle Problem Prof. T.H. Tse](https://reader035.vdocument.in/reader035/viewer/2022070813/5f0c4f817e708231d434c453/html5/thumbnails/85.jpg)
What is Object Equivalence? Real-Life Word Processing Example
Call this object D.
![Page 86: Even the Very Wise Cannot See All Ends: Many … › ... › Keynote-TH-Tse-slides.pdfEven the Very Wise Cannot See All Ends: Many Facets of the Test Oracle Problem Prof. T.H. Tse](https://reader035.vdocument.in/reader035/viewer/2022070813/5f0c4f817e708231d434c453/html5/thumbnails/86.jpg)
86
What is Object Equivalence? Output from Object D
T.H. Tse is an Honorary Professor in Computer Science at The University of Hong Kong. He is a Steering Committee Chair of the IEEE International Conference on Software Quality, Reliability & Security.
![Page 87: Even the Very Wise Cannot See All Ends: Many … › ... › Keynote-TH-Tse-slides.pdfEven the Very Wise Cannot See All Ends: Many Facets of the Test Oracle Problem Prof. T.H. Tse](https://reader035.vdocument.in/reader035/viewer/2022070813/5f0c4f817e708231d434c453/html5/thumbnails/87.jpg)
What is Object Equivalence? Attributive Equivalence
Two objects will be attributively equivalent if they have the exactly the same visible attributes
= ? H D
87
![Page 88: Even the Very Wise Cannot See All Ends: Many … › ... › Keynote-TH-Tse-slides.pdfEven the Very Wise Cannot See All Ends: Many Facets of the Test Oracle Problem Prof. T.H. Tse](https://reader035.vdocument.in/reader035/viewer/2022070813/5f0c4f817e708231d434c453/html5/thumbnails/88.jpg)
T.H. Tse is an Honorary Professor in Computer Science at The University of Hong Kong. He is a Steering Committee Chair of the IEEE International Conference on Software Quality, Reliability & Security.
T.H. Tse is an Honorary Professor in Computer Science at The University of Hong Kong. He is a Steering Committee Chair of the IEEE International Conference on Software Quality, Reliability & Security.
= H D
88
What is Object Equivalence? Attributive Equivalence
Simple to test But the definition is too weak to be useful ... Why? .
![Page 89: Even the Very Wise Cannot See All Ends: Many … › ... › Keynote-TH-Tse-slides.pdfEven the Very Wise Cannot See All Ends: Many Facets of the Test Oracle Problem Prof. T.H. Tse](https://reader035.vdocument.in/reader035/viewer/2022070813/5f0c4f817e708231d434c453/html5/thumbnails/89.jpg)
What is Object Equivalence? Real-Life Word Processing Example
Take object H again (with hidden text).
![Page 90: Even the Very Wise Cannot See All Ends: Many … › ... › Keynote-TH-Tse-slides.pdfEven the Very Wise Cannot See All Ends: Many Facets of the Test Oracle Problem Prof. T.H. Tse](https://reader035.vdocument.in/reader035/viewer/2022070813/5f0c4f817e708231d434c453/html5/thumbnails/90.jpg)
What is Object Equivalence? Real-Life Word Processing Example
Click “¶”.
![Page 91: Even the Very Wise Cannot See All Ends: Many … › ... › Keynote-TH-Tse-slides.pdfEven the Very Wise Cannot See All Ends: Many Facets of the Test Oracle Problem Prof. T.H. Tse](https://reader035.vdocument.in/reader035/viewer/2022070813/5f0c4f817e708231d434c453/html5/thumbnails/91.jpg)
What is Object Equivalence? Real-Life Word Processing Example
Unclick “Hidden”.
![Page 92: Even the Very Wise Cannot See All Ends: Many … › ... › Keynote-TH-Tse-slides.pdfEven the Very Wise Cannot See All Ends: Many Facets of the Test Oracle Problem Prof. T.H. Tse](https://reader035.vdocument.in/reader035/viewer/2022070813/5f0c4f817e708231d434c453/html5/thumbnails/92.jpg)
What is Object Equivalence? Real-Life Word Processing Example
We get an object that behaves like the original object O.
![Page 93: Even the Very Wise Cannot See All Ends: Many … › ... › Keynote-TH-Tse-slides.pdfEven the Very Wise Cannot See All Ends: Many Facets of the Test Oracle Problem Prof. T.H. Tse](https://reader035.vdocument.in/reader035/viewer/2022070813/5f0c4f817e708231d434c453/html5/thumbnails/93.jpg)
What is Object Equivalence? Real-Life Word Processing Example
Take object D again (with deleted text).
![Page 94: Even the Very Wise Cannot See All Ends: Many … › ... › Keynote-TH-Tse-slides.pdfEven the Very Wise Cannot See All Ends: Many Facets of the Test Oracle Problem Prof. T.H. Tse](https://reader035.vdocument.in/reader035/viewer/2022070813/5f0c4f817e708231d434c453/html5/thumbnails/94.jpg)
What is Object Equivalence? Real-Life Word Processing Example
Click “¶”.
![Page 95: Even the Very Wise Cannot See All Ends: Many … › ... › Keynote-TH-Tse-slides.pdfEven the Very Wise Cannot See All Ends: Many Facets of the Test Oracle Problem Prof. T.H. Tse](https://reader035.vdocument.in/reader035/viewer/2022070813/5f0c4f817e708231d434c453/html5/thumbnails/95.jpg)
What is Object Equivalence? Real-Life Word Processing Example
We do not get an object that behaves like the original O.
![Page 96: Even the Very Wise Cannot See All Ends: Many … › ... › Keynote-TH-Tse-slides.pdfEven the Very Wise Cannot See All Ends: Many Facets of the Test Oracle Problem Prof. T.H. Tse](https://reader035.vdocument.in/reader035/viewer/2022070813/5f0c4f817e708231d434c453/html5/thumbnails/96.jpg)
What is Object Equivalence? Observational Equivalence Two objects will be observationally equivalent
if they have the exactly the same visible attributes and behavior
≠ H D
![Page 97: Even the Very Wise Cannot See All Ends: Many … › ... › Keynote-TH-Tse-slides.pdfEven the Very Wise Cannot See All Ends: Many Facets of the Test Oracle Problem Prof. T.H. Tse](https://reader035.vdocument.in/reader035/viewer/2022070813/5f0c4f817e708231d434c453/html5/thumbnails/97.jpg)
What is Object Equivalence? Observational Equivalence Ideal criterion in object-oriented software testing But very very very very very very very very very
very very very very very very very very very very very very very very very very very very very very difficult to verify: Need to check an infinite combination of
possible behavior for every test case.
97
![Page 98: Even the Very Wise Cannot See All Ends: Many … › ... › Keynote-TH-Tse-slides.pdfEven the Very Wise Cannot See All Ends: Many Facets of the Test Oracle Problem Prof. T.H. Tse](https://reader035.vdocument.in/reader035/viewer/2022070813/5f0c4f817e708231d434c453/html5/thumbnails/98.jpg)
Cannot distinguish between visible and hidden attributes and behavior
Cannot distinguish between attributive and observational equivalence
UML State Machine Not Truly Suitable for OO Specification
98
= ?
![Page 99: Even the Very Wise Cannot See All Ends: Many … › ... › Keynote-TH-Tse-slides.pdfEven the Very Wise Cannot See All Ends: Many Facets of the Test Oracle Problem Prof. T.H. Tse](https://reader035.vdocument.in/reader035/viewer/2022070813/5f0c4f817e708231d434c453/html5/thumbnails/99.jpg)
99
Algebraic Specification A ground term is a sequence of operations
show(HKU QRS hide(TSE))
An axiom is a rule that specifies the refinement of a term hide(D D’) = hide(D) hide(D’) show(D D’) = show(D) show(D’) show(hide(D)) = show(D) show(W) = W .
![Page 100: Even the Very Wise Cannot See All Ends: Many … › ... › Keynote-TH-Tse-slides.pdfEven the Very Wise Cannot See All Ends: Many Facets of the Test Oracle Problem Prof. T.H. Tse](https://reader035.vdocument.in/reader035/viewer/2022070813/5f0c4f817e708231d434c453/html5/thumbnails/100.jpg)
Algebraic Specification A ground term may be transformed into another
using axioms as left-to-right rewrite rules, until it reaches a normal form
100
![Page 101: Even the Very Wise Cannot See All Ends: Many … › ... › Keynote-TH-Tse-slides.pdfEven the Very Wise Cannot See All Ends: Many Facets of the Test Oracle Problem Prof. T.H. Tse](https://reader035.vdocument.in/reader035/viewer/2022070813/5f0c4f817e708231d434c453/html5/thumbnails/101.jpg)
A ground term may be transformed into another using axioms as left-to-right rewrite rules, until it reaches a normal form show(HKU QRS hide(TSE))
Algebraic Specification
101
![Page 102: Even the Very Wise Cannot See All Ends: Many … › ... › Keynote-TH-Tse-slides.pdfEven the Very Wise Cannot See All Ends: Many Facets of the Test Oracle Problem Prof. T.H. Tse](https://reader035.vdocument.in/reader035/viewer/2022070813/5f0c4f817e708231d434c453/html5/thumbnails/102.jpg)
A ground term may be transformed into another using axioms as left-to-right rewrite rules, until it reaches a normal form show(HKU QRS hide(TSE)) ➔ show(HKU) show(QRS) show(hide(TSE))
102
show(D D’) = show(D) show (D’)
Algebraic Specification
![Page 103: Even the Very Wise Cannot See All Ends: Many … › ... › Keynote-TH-Tse-slides.pdfEven the Very Wise Cannot See All Ends: Many Facets of the Test Oracle Problem Prof. T.H. Tse](https://reader035.vdocument.in/reader035/viewer/2022070813/5f0c4f817e708231d434c453/html5/thumbnails/103.jpg)
A ground term may be transformed into another using axioms as left-to-right rewrite rules, until it reaches a normal form show(HKU QRS hide(TSE)) ➔ show(HKU) show(QRS) show(hide(TSE)) ➔ show(HKU) show(QRS) show(TSE)
Algebraic Specification
103
show(hide(D)) = show(D)
![Page 104: Even the Very Wise Cannot See All Ends: Many … › ... › Keynote-TH-Tse-slides.pdfEven the Very Wise Cannot See All Ends: Many Facets of the Test Oracle Problem Prof. T.H. Tse](https://reader035.vdocument.in/reader035/viewer/2022070813/5f0c4f817e708231d434c453/html5/thumbnails/104.jpg)
A ground term may be transformed into another using axioms as left-to-right rewrite rules, until it reaches a normal form show(HKU QRS hide(TSE)) ➔ show(HKU) show(QRS) show(hide(TSE)) ➔ show(HKU) show(QRS) show(TSE) ➔ HKU QRS TSE
Algebraic Specification
104
show(W) = W
![Page 105: Even the Very Wise Cannot See All Ends: Many … › ... › Keynote-TH-Tse-slides.pdfEven the Very Wise Cannot See All Ends: Many Facets of the Test Oracle Problem Prof. T.H. Tse](https://reader035.vdocument.in/reader035/viewer/2022070813/5f0c4f817e708231d434c453/html5/thumbnails/105.jpg)
A ground term may be transformed into another using axioms as left-to-right rewrite rules, until it reaches a normal form show(HKU QRS hide(TSE)) ➔ show(HKU) show(QRS) show(hide(TSE)) ➔ show(HKU) show(QRS) show(TSE) ➔ HKU QRS TSE
Algebraic Specification
Normal form. 105
![Page 106: Even the Very Wise Cannot See All Ends: Many … › ... › Keynote-TH-Tse-slides.pdfEven the Very Wise Cannot See All Ends: Many Facets of the Test Oracle Problem Prof. T.H. Tse](https://reader035.vdocument.in/reader035/viewer/2022070813/5f0c4f817e708231d434c453/html5/thumbnails/106.jpg)
Algebraic Specification Denotational Semantics Mathematical meaning Like simultaneous equations
2x + 3y = 4 5x + 6y = 7 “=” means “equals” The order of the equations is not important.
106
![Page 107: Even the Very Wise Cannot See All Ends: Many … › ... › Keynote-TH-Tse-slides.pdfEven the Very Wise Cannot See All Ends: Many Facets of the Test Oracle Problem Prof. T.H. Tse](https://reader035.vdocument.in/reader035/viewer/2022070813/5f0c4f817e708231d434c453/html5/thumbnails/107.jpg)
Algebraic Specification Operational Semantics How the statements should be executed Like C programs j = 1; j = j + 2;
Replace the variable on the left-hand side by the value on the right-hand side.
107
![Page 108: Even the Very Wise Cannot See All Ends: Many … › ... › Keynote-TH-Tse-slides.pdfEven the Very Wise Cannot See All Ends: Many Facets of the Test Oracle Problem Prof. T.H. Tse](https://reader035.vdocument.in/reader035/viewer/2022070813/5f0c4f817e708231d434c453/html5/thumbnails/108.jpg)
Canonical Algebraic Specification An algebraic specification is canonical if every
sequence of rewrites of a ground term produces a unique normal form
108
![Page 109: Even the Very Wise Cannot See All Ends: Many … › ... › Keynote-TH-Tse-slides.pdfEven the Very Wise Cannot See All Ends: Many Facets of the Test Oracle Problem Prof. T.H. Tse](https://reader035.vdocument.in/reader035/viewer/2022070813/5f0c4f817e708231d434c453/html5/thumbnails/109.jpg)
Canonical Algebraic Specification An algebraic specification is canonical if every
sequence of rewrites of a ground term produces a unique normal form show(HKU) show(QRS) show(hide(TSE)) ➔ show(HKU) show(QRS) show(TSE) ➔ HKU QRS TSE
109
![Page 110: Even the Very Wise Cannot See All Ends: Many … › ... › Keynote-TH-Tse-slides.pdfEven the Very Wise Cannot See All Ends: Many Facets of the Test Oracle Problem Prof. T.H. Tse](https://reader035.vdocument.in/reader035/viewer/2022070813/5f0c4f817e708231d434c453/html5/thumbnails/110.jpg)
Canonical Algebraic Specification An algebraic specification is canonical if every
sequence of rewrites of a ground term produces a unique normal form show(HKU) show(QRS) show(hide(TSE)) ➔ show(HKU) show(QRS) show(TSE) ➔ HKU QRS TSE show(HKU) show(QRS) show(hide(TSE)) ➔ HKU QRS show(hide(TSE)) ➔ HKU QRS TSE
110
![Page 111: Even the Very Wise Cannot See All Ends: Many … › ... › Keynote-TH-Tse-slides.pdfEven the Very Wise Cannot See All Ends: Many Facets of the Test Oracle Problem Prof. T.H. Tse](https://reader035.vdocument.in/reader035/viewer/2022070813/5f0c4f817e708231d434c453/html5/thumbnails/111.jpg)
Canonical Algebraic Specification For a canonical specification, the operational
semantics agrees with the denotational semantics.
111
![Page 112: Even the Very Wise Cannot See All Ends: Many … › ... › Keynote-TH-Tse-slides.pdfEven the Very Wise Cannot See All Ends: Many Facets of the Test Oracle Problem Prof. T.H. Tse](https://reader035.vdocument.in/reader035/viewer/2022070813/5f0c4f817e708231d434c453/html5/thumbnails/112.jpg)
An implementation P is correct with respect to the specification Sp if and only if
112
Target of OO Software Testing According to Real-World Practice
![Page 113: Even the Very Wise Cannot See All Ends: Many … › ... › Keynote-TH-Tse-slides.pdfEven the Very Wise Cannot See All Ends: Many Facets of the Test Oracle Problem Prof. T.H. Tse](https://reader035.vdocument.in/reader035/viewer/2022070813/5f0c4f817e708231d434c453/html5/thumbnails/113.jpg)
An implementation P is correct with respect to the specification Sp if and only if P satisfies the set (AE) of all attributively
equivalent pairs of ground terms in Sp
Target of OO Software Testing According to Real-World Practice
For example, hide(TSE) and delete(TSE) are attributively equivalent.
113
![Page 114: Even the Very Wise Cannot See All Ends: Many … › ... › Keynote-TH-Tse-slides.pdfEven the Very Wise Cannot See All Ends: Many Facets of the Test Oracle Problem Prof. T.H. Tse](https://reader035.vdocument.in/reader035/viewer/2022070813/5f0c4f817e708231d434c453/html5/thumbnails/114.jpg)
114
Target of OO Software Testing According to Huo Yan Chen et al. (1998)
![Page 115: Even the Very Wise Cannot See All Ends: Many … › ... › Keynote-TH-Tse-slides.pdfEven the Very Wise Cannot See All Ends: Many Facets of the Test Oracle Problem Prof. T.H. Tse](https://reader035.vdocument.in/reader035/viewer/2022070813/5f0c4f817e708231d434c453/html5/thumbnails/115.jpg)
An implementation P is correct with respect to the specification Sp if and only if: Equivalence Criterion
P satisfies the set (OE) of all observationally equivalent pairs of ground terms in Sp
115
Target of OO Software Testing According to Huo Yan Chen et al. (1998)
![Page 116: Even the Very Wise Cannot See All Ends: Many … › ... › Keynote-TH-Tse-slides.pdfEven the Very Wise Cannot See All Ends: Many Facets of the Test Oracle Problem Prof. T.H. Tse](https://reader035.vdocument.in/reader035/viewer/2022070813/5f0c4f817e708231d434c453/html5/thumbnails/116.jpg)
An implementation P is correct with respect to the specification Sp if and only if: Equivalence Criterion
P satisfies the set (OE) of all observationally equivalent pairs of ground terms in Sp
Non-Equivalence Criterion P satisfies the set (OE' ) of all observationally nonequivalent pairs of ground terms in Sp.
Target of OO Software Testing According to Huo Yan Chen et al. (1998)
For example, hide(TSE) and delete(TSE) are observationally nonequivalent.
![Page 117: Even the Very Wise Cannot See All Ends: Many … › ... › Keynote-TH-Tse-slides.pdfEven the Very Wise Cannot See All Ends: Many Facets of the Test Oracle Problem Prof. T.H. Tse](https://reader035.vdocument.in/reader035/viewer/2022070813/5f0c4f817e708231d434c453/html5/thumbnails/117.jpg)
Targets of OO Software Testing
117
AE OE U OE’
Real-world correctness, but very very very very very very very very very difficult to test.
Real-world short-cut, but too weak to be useful
?
A jungle of proposals by various researchers.
![Page 118: Even the Very Wise Cannot See All Ends: Many … › ... › Keynote-TH-Tse-slides.pdfEven the Very Wise Cannot See All Ends: Many Facets of the Test Oracle Problem Prof. T.H. Tse](https://reader035.vdocument.in/reader035/viewer/2022070813/5f0c4f817e708231d434c453/html5/thumbnails/118.jpg)
An implementation P is correct with respect to the specification Sp if and only if P satisfies the set (GI ) of all ground instances
of every axiom in Sp
Target of OO Software Testing According to Bernot et al. (1991), Le Gall and Arnould (1996), Machado (1998, 2000), Machado and Sannella (2002), Aiguier et al. (2006)
118
![Page 119: Even the Very Wise Cannot See All Ends: Many … › ... › Keynote-TH-Tse-slides.pdfEven the Very Wise Cannot See All Ends: Many Facets of the Test Oracle Problem Prof. T.H. Tse](https://reader035.vdocument.in/reader035/viewer/2022070813/5f0c4f817e708231d434c453/html5/thumbnails/119.jpg)
An implementation P is correct with respect to the specification Sp if and only if: Equivalence Criterion
P satisfies the set (RP) of all “equivalent” ground pairs such that one can be rewritten to the other using axioms in Sp as left-to-write rewrite rules
Target of OO Software Testing According to Doong and Frankl (1994), Gaudel (1995), Zhu (2003)
119
![Page 120: Even the Very Wise Cannot See All Ends: Many … › ... › Keynote-TH-Tse-slides.pdfEven the Very Wise Cannot See All Ends: Many Facets of the Test Oracle Problem Prof. T.H. Tse](https://reader035.vdocument.in/reader035/viewer/2022070813/5f0c4f817e708231d434c453/html5/thumbnails/120.jpg)
An implementation P is correct with respect to the specification Sp if and only if: Equivalence Criterion
P satisfies the set (RP) of all “equivalent” ground pairs such that one can be rewritten to the other using axioms in Sp as left-to-write rewrite rules
Non-Equivalence Criterion P satisfies the set (RP' ) of all “nonequivalent” ground pairs.
Target of OO Software Testing According to Doong and Frankl (1994), Gaudel (1995), Zhu (2003)
120
![Page 121: Even the Very Wise Cannot See All Ends: Many … › ... › Keynote-TH-Tse-slides.pdfEven the Very Wise Cannot See All Ends: Many Facets of the Test Oracle Problem Prof. T.H. Tse](https://reader035.vdocument.in/reader035/viewer/2022070813/5f0c4f817e708231d434c453/html5/thumbnails/121.jpg)
An implementation P is correct with respect to the specification Sp if and only if P satisfies the set (FP ) of all fundamental pairs
in Sp
Target of OO Software Testing According to Huo Yan Chen et al. (1998)
121
![Page 122: Even the Very Wise Cannot See All Ends: Many … › ... › Keynote-TH-Tse-slides.pdfEven the Very Wise Cannot See All Ends: Many Facets of the Test Oracle Problem Prof. T.H. Tse](https://reader035.vdocument.in/reader035/viewer/2022070813/5f0c4f817e708231d434c453/html5/thumbnails/122.jpg)
An implementation P is correct with respect to the specification Sp if and only if: Equivalence Criterion
P satisfies the set (NE) of all normally equivalent ground pairs in Sp
Target of OO Software Testing According to Huo Yan Chen et al. (1998)
122
![Page 123: Even the Very Wise Cannot See All Ends: Many … › ... › Keynote-TH-Tse-slides.pdfEven the Very Wise Cannot See All Ends: Many Facets of the Test Oracle Problem Prof. T.H. Tse](https://reader035.vdocument.in/reader035/viewer/2022070813/5f0c4f817e708231d434c453/html5/thumbnails/123.jpg)
An implementation P is correct with respect to the specification Sp if and only if: Equivalence Criterion
P satisfies the set (NE) of all normally equivalent ground pairs in Sp
Non-Equivalence Criterion P satisfies the set (NE' ) of all normally nonequivalent ground pairs in Sp
Target of OO Software Testing According to Huo Yan Chen et al. (1998)
123
![Page 124: Even the Very Wise Cannot See All Ends: Many … › ... › Keynote-TH-Tse-slides.pdfEven the Very Wise Cannot See All Ends: Many Facets of the Test Oracle Problem Prof. T.H. Tse](https://reader035.vdocument.in/reader035/viewer/2022070813/5f0c4f817e708231d434c453/html5/thumbnails/124.jpg)
An implementation P is correct with respect to the specification Sp if and only if P satisfies the set (CI ) of all ground instances of
every axiom in Sp that contains creators or constructors only.
Target of OO Software Testing According to Aiguier et al. (2006)
124
![Page 125: Even the Very Wise Cannot See All Ends: Many … › ... › Keynote-TH-Tse-slides.pdfEven the Very Wise Cannot See All Ends: Many Facets of the Test Oracle Problem Prof. T.H. Tse](https://reader035.vdocument.in/reader035/viewer/2022070813/5f0c4f817e708231d434c453/html5/thumbnails/125.jpg)
Targets of OO Software Testing
125
AE OE U OE’ GI RP U RP’ FP
NE U NE’ CI
Comparisons through empirical studies?
![Page 126: Even the Very Wise Cannot See All Ends: Many … › ... › Keynote-TH-Tse-slides.pdfEven the Very Wise Cannot See All Ends: Many Facets of the Test Oracle Problem Prof. T.H. Tse](https://reader035.vdocument.in/reader035/viewer/2022070813/5f0c4f817e708231d434c453/html5/thumbnails/126.jpg)
What is Empirical Study? Research based on experimentation or observation
to answer a specific question or to test a hypothesis Undue emphasis in software engineering, even in
first-tier publication venues.
126
![Page 127: Even the Very Wise Cannot See All Ends: Many … › ... › Keynote-TH-Tse-slides.pdfEven the Very Wise Cannot See All Ends: Many Facets of the Test Oracle Problem Prof. T.H. Tse](https://reader035.vdocument.in/reader035/viewer/2022070813/5f0c4f817e708231d434c453/html5/thumbnails/127.jpg)
What is Empirical Study? Statistical significance does not mean research
significance A hypothesis may be very probable simply
because it tells us nothing, or very little
Sir Karl Popper, Professor of Logic and Scientific Method, London School of Economics (1949−1969)
![Page 128: Even the Very Wise Cannot See All Ends: Many … › ... › Keynote-TH-Tse-slides.pdfEven the Very Wise Cannot See All Ends: Many Facets of the Test Oracle Problem Prof. T.H. Tse](https://reader035.vdocument.in/reader035/viewer/2022070813/5f0c4f817e708231d434c453/html5/thumbnails/128.jpg)
What is Empirical Study? Statistical significance does not mean research
significance A hypothesis may be very probable simply because it tells us nothing, or very little
A high degree of probability is therefore not an indication of “goodness”
Sir Karl Popper, Professor of Logic and Scientific Method, London School of Economics (1949−1969)
![Page 129: Even the Very Wise Cannot See All Ends: Many … › ... › Keynote-TH-Tse-slides.pdfEven the Very Wise Cannot See All Ends: Many Facets of the Test Oracle Problem Prof. T.H. Tse](https://reader035.vdocument.in/reader035/viewer/2022070813/5f0c4f817e708231d434c453/html5/thumbnails/129.jpg)
What is Empirical Study? Statistical significance does not mean research
significance A hypothesis may be very probable simply because it tells us nothing, or very little A high degree of probability is therefore not an indication of “goodness”
It may be merely a symptom of low informative content.
Sir Karl Popper, Professor of Logic and Scientific Method, London School of Economics (1949−1969)
![Page 130: Even the Very Wise Cannot See All Ends: Many … › ... › Keynote-TH-Tse-slides.pdfEven the Very Wise Cannot See All Ends: Many Facets of the Test Oracle Problem Prof. T.H. Tse](https://reader035.vdocument.in/reader035/viewer/2022070813/5f0c4f817e708231d434c453/html5/thumbnails/130.jpg)
Presentation Outline Background Many facets of the test
oracle problem Expected outcome
= actual execution result Expected outcome
= actual execution result Expected outcome
= actual execution result
Jungle of proposals Empirical studies? What do other
researchers do? Trim the tree or
tame the forest?
130
![Page 131: Even the Very Wise Cannot See All Ends: Many … › ... › Keynote-TH-Tse-slides.pdfEven the Very Wise Cannot See All Ends: Many Facets of the Test Oracle Problem Prof. T.H. Tse](https://reader035.vdocument.in/reader035/viewer/2022070813/5f0c4f817e708231d434c453/html5/thumbnails/131.jpg)
What Do Other Researchers Do? General Relativity luminiferous aether corpuscular model Lorentz’s theory of electrons
general covariance field equations gravitational redshift
hole argument gravitational time dilation principle of relative motion
electromagnetic mass absolute space and time light constancy
131
A jungle of proposals
![Page 132: Even the Very Wise Cannot See All Ends: Many … › ... › Keynote-TH-Tse-slides.pdfEven the Very Wise Cannot See All Ends: Many Facets of the Test Oracle Problem Prof. T.H. Tse](https://reader035.vdocument.in/reader035/viewer/2022070813/5f0c4f817e708231d434c453/html5/thumbnails/132.jpg)
What Do Other Researchers Do? General Relativity
132
So many people today — and even professional scientists — seem to me like somebody who has seen thousands of trees but has never seen a forest.
Albert Einstein The Collected Papers of Albert Einstein Princeton University Press
![Page 133: Even the Very Wise Cannot See All Ends: Many … › ... › Keynote-TH-Tse-slides.pdfEven the Very Wise Cannot See All Ends: Many Facets of the Test Oracle Problem Prof. T.H. Tse](https://reader035.vdocument.in/reader035/viewer/2022070813/5f0c4f817e708231d434c453/html5/thumbnails/133.jpg)
What Do Other Researchers Do? General Relativity
133
Trim the individual trees? What Do Others Do?General Relativityluminiferous aether corpuscular model Lorentz’s theory of electrons
general covariance field equationsgravitational redshift
hole argumentgravitational time dilationprinciple of relative motion
electromagnetic mass absolute space and time light constancy
A jungle of proposals Albert Einstein
space time curvature
Tame the forest!
![Page 134: Even the Very Wise Cannot See All Ends: Many … › ... › Keynote-TH-Tse-slides.pdfEven the Very Wise Cannot See All Ends: Many Facets of the Test Oracle Problem Prof. T.H. Tse](https://reader035.vdocument.in/reader035/viewer/2022070813/5f0c4f817e708231d434c453/html5/thumbnails/134.jpg)
Risk evaluation formulas
What Do Other Researchers Do? Spectrum-Based Fault Localization
AMPLE2 Anderberg Arithmetic Mean Binary CBI Inc. Cohen
Dice Goodman Euclid Fleiss Hamann Hamming, etc.
Jaccard Naish1 M2 Kulczynski2 Naish2 Ochiai qe
Rogers & Tanimoto Rogot1 Russel & Rao Scott Simple Matching
Wong3 Wong2 Sokal Sørensen-Dice Tarantula Wong1
134
![Page 135: Even the Very Wise Cannot See All Ends: Many … › ... › Keynote-TH-Tse-slides.pdfEven the Very Wise Cannot See All Ends: Many Facets of the Test Oracle Problem Prof. T.H. Tse](https://reader035.vdocument.in/reader035/viewer/2022070813/5f0c4f817e708231d434c453/html5/thumbnails/135.jpg)
What Do Other Reseachers Do? Spectrum-Based Fault Localization Another jungle of proposals Comparisons through empirical studies in IEEE
TSE, ICSE, and FSE.
135
![Page 136: Even the Very Wise Cannot See All Ends: Many … › ... › Keynote-TH-Tse-slides.pdfEven the Very Wise Cannot See All Ends: Many Facets of the Test Oracle Problem Prof. T.H. Tse](https://reader035.vdocument.in/reader035/viewer/2022070813/5f0c4f817e708231d434c453/html5/thumbnails/136.jpg)
Risk evaluation formulas
What Do Others Do?Spectrum-Based Fault Localization
AMPLE2 Anderberg Arithmetic Mean Binary CBI Inc. Cohen
Dice GoodmanEuclid Fleiss Hamann Hamming, etc.
Jaccard Naish1M2Kulczynski2 Naish2 Ochiai qe
Rogers & Tanimoto Rogot1 Russel & Rao Scott Simple Matching
Wong3Wong2Sokal Sørensen-Dice Tarantula Wong1
What Do Other Researchers Do? Spectrum-Based Fault Localization
136
Trim the individual trees? Tame the forest!
T.Y. Chen and Team
![Page 137: Even the Very Wise Cannot See All Ends: Many … › ... › Keynote-TH-Tse-slides.pdfEven the Very Wise Cannot See All Ends: Many Facets of the Test Oracle Problem Prof. T.H. Tse](https://reader035.vdocument.in/reader035/viewer/2022070813/5f0c4f817e708231d434c453/html5/thumbnails/137.jpg)
Spectrum-Based Fault Localization According to T.Y. Chen and Team (2013)
Theoretical framework to compare risk evaluation formulas for single-fault programs
No single formula can outperform the rest Among the formulas under study, only five are
“maximal” Most of best-known formulas are not among them.
137
![Page 138: Even the Very Wise Cannot See All Ends: Many … › ... › Keynote-TH-Tse-slides.pdfEven the Very Wise Cannot See All Ends: Many Facets of the Test Oracle Problem Prof. T.H. Tse](https://reader035.vdocument.in/reader035/viewer/2022070813/5f0c4f817e708231d434c453/html5/thumbnails/138.jpg)
Test Case Generation Techniques According to T.Y. Chen and Team (2008)
Prove that no test case generation technique can be better than random testing by more than 50% in the absence of further information on possible
locations of failure-causing inputs Adaptive random testing is close to this theoretic
limit.
138
![Page 139: Even the Very Wise Cannot See All Ends: Many … › ... › Keynote-TH-Tse-slides.pdfEven the Very Wise Cannot See All Ends: Many Facets of the Test Oracle Problem Prof. T.H. Tse](https://reader035.vdocument.in/reader035/viewer/2022070813/5f0c4f817e708231d434c453/html5/thumbnails/139.jpg)
Partition Testing Techniques According to T.Y. Chen and Team (2000)
Prove that proposed proportional sampling strategy is the only partition testing technique that ensures probability of finding at least one failure is no lower than random testing for any program.
139
![Page 140: Even the Very Wise Cannot See All Ends: Many … › ... › Keynote-TH-Tse-slides.pdfEven the Very Wise Cannot See All Ends: Many Facets of the Test Oracle Problem Prof. T.H. Tse](https://reader035.vdocument.in/reader035/viewer/2022070813/5f0c4f817e708231d434c453/html5/thumbnails/140.jpg)
140
Targets of OO Software Testing
AE OE U OE’GI RP U RP’ FP
NE U NE’ CI
Comparisons through empirical studies?
Trim the individual trees? Tame the forest!
Targets of OO Software Testing
![Page 141: Even the Very Wise Cannot See All Ends: Many … › ... › Keynote-TH-Tse-slides.pdfEven the Very Wise Cannot See All Ends: Many Facets of the Test Oracle Problem Prof. T.H. Tse](https://reader035.vdocument.in/reader035/viewer/2022070813/5f0c4f817e708231d434c453/html5/thumbnails/141.jpg)
Toward Ultimate Target of OO Software Testing Tame the Forest FP ⊂ CI ⊂ GI ⊂ RP ⊂ NE ⊂ OE ⊂ AE
141
Subset but not equal
![Page 142: Even the Very Wise Cannot See All Ends: Many … › ... › Keynote-TH-Tse-slides.pdfEven the Very Wise Cannot See All Ends: Many Facets of the Test Oracle Problem Prof. T.H. Tse](https://reader035.vdocument.in/reader035/viewer/2022070813/5f0c4f817e708231d434c453/html5/thumbnails/142.jpg)
Toward Ultimate Target of OO Software Testing Tame the Forest FP ⊂ CI ⊂ GI ⊂ RP ⊂ NE ⊂ OE ⊂ AE
142
Subset but not equal
Hence, (P satisfies AE ) ⇒ (P satisfies OE ) ⇒ (P satisfies NE ) ⇒ (P satisfies RP ) ⇒ (P satisfies GI ) ⇒ (P satisfies CI ) ⇒ (P satisfies FP ) and
(P satisfies OE ∪ OE' ) ⇒ (P satisfies OE )
Not helpful.
Not helpful.
OE ⊂ OE ∪ OE'
![Page 143: Even the Very Wise Cannot See All Ends: Many … › ... › Keynote-TH-Tse-slides.pdfEven the Very Wise Cannot See All Ends: Many Facets of the Test Oracle Problem Prof. T.H. Tse](https://reader035.vdocument.in/reader035/viewer/2022070813/5f0c4f817e708231d434c453/html5/thumbnails/143.jpg)
Ultimate Target of OO Software Testing Tame the Forest Have we made full use of algebraic specifications? Given a canonical specification Sp with proper
imports and a complete implementation P (P satisfies AE)
143
![Page 144: Even the Very Wise Cannot See All Ends: Many … › ... › Keynote-TH-Tse-slides.pdfEven the Very Wise Cannot See All Ends: Many Facets of the Test Oracle Problem Prof. T.H. Tse](https://reader035.vdocument.in/reader035/viewer/2022070813/5f0c4f817e708231d434c453/html5/thumbnails/144.jpg)
Ultimate Target of OO Software Testing Tame the Forest Have we made full use of algebraic specifications? Given a canonical specification Sp with proper
imports and a complete implementation P (P satisfies AE) ⇔ (P satisfies OE)
144
![Page 145: Even the Very Wise Cannot See All Ends: Many … › ... › Keynote-TH-Tse-slides.pdfEven the Very Wise Cannot See All Ends: Many Facets of the Test Oracle Problem Prof. T.H. Tse](https://reader035.vdocument.in/reader035/viewer/2022070813/5f0c4f817e708231d434c453/html5/thumbnails/145.jpg)
Ultimate Target of OO Software Testing Tame the Forest Have we made full use of algebraic specifications? Given a canonical specification Sp with proper
imports and a complete implementation P (P satisfies AE) ⇔ (P satisfies OE) ⇔ (P satisfies OE' )
145
![Page 146: Even the Very Wise Cannot See All Ends: Many … › ... › Keynote-TH-Tse-slides.pdfEven the Very Wise Cannot See All Ends: Many Facets of the Test Oracle Problem Prof. T.H. Tse](https://reader035.vdocument.in/reader035/viewer/2022070813/5f0c4f817e708231d434c453/html5/thumbnails/146.jpg)
Ultimate Target of OO Software Testing Tame the Forest Have we made full use of algebraic specifications? Given a canonical specification Sp with proper
imports and a complete implementation P (P satisfies AE) ⇔ (P satisfies OE) ⇔ (P satisfies OE' ) ⇔ (P satisfies OE ∪ OE' )
146
![Page 147: Even the Very Wise Cannot See All Ends: Many … › ... › Keynote-TH-Tse-slides.pdfEven the Very Wise Cannot See All Ends: Many Facets of the Test Oracle Problem Prof. T.H. Tse](https://reader035.vdocument.in/reader035/viewer/2022070813/5f0c4f817e708231d434c453/html5/thumbnails/147.jpg)
Ultimate Target of OO Software Testing Tame the Forest Have we made full use of algebraic specifications? Given a canonical specification Sp with proper
imports and a complete implementation P (P satisfies AE) ⇔ (P satisfies OE) ⇔ (P satisfies OE' ) ⇔ (P satisfies OE ∪ OE' ) ⇔ (P satisfies NE) ⇔ (P satisfies RP) ⇔ (P satisfies GI ) ⇔ (P satisfies CI) ⇔ (P satisfies FP).
147
![Page 148: Even the Very Wise Cannot See All Ends: Many … › ... › Keynote-TH-Tse-slides.pdfEven the Very Wise Cannot See All Ends: Many Facets of the Test Oracle Problem Prof. T.H. Tse](https://reader035.vdocument.in/reader035/viewer/2022070813/5f0c4f817e708231d434c453/html5/thumbnails/148.jpg)
Ultimate Target of OO Software Testing Tame the Forest
In short Given a canonical specification Sp with proper
imports and a complete implementation P (P satisfies AE) ⇔ (P satisfies OE ∪ OE' )
Real-world correctness considered too difficult to test.
Real-world short-cut considered too weak to be useful
![Page 149: Even the Very Wise Cannot See All Ends: Many … › ... › Keynote-TH-Tse-slides.pdfEven the Very Wise Cannot See All Ends: Many Facets of the Test Oracle Problem Prof. T.H. Tse](https://reader035.vdocument.in/reader035/viewer/2022070813/5f0c4f817e708231d434c453/html5/thumbnails/149.jpg)
149
Many Facets of the Test Oracle Problem
Challenge 4: Expected outcome = actual execution result
![Page 150: Even the Very Wise Cannot See All Ends: Many … › ... › Keynote-TH-Tse-slides.pdfEven the Very Wise Cannot See All Ends: Many Facets of the Test Oracle Problem Prof. T.H. Tse](https://reader035.vdocument.in/reader035/viewer/2022070813/5f0c4f817e708231d434c453/html5/thumbnails/150.jpg)
Ubiquitous Computing Computing everywhere and at any time Applications operate in dynamic environment
150
![Page 151: Even the Very Wise Cannot See All Ends: Many … › ... › Keynote-TH-Tse-slides.pdfEven the Very Wise Cannot See All Ends: Many Facets of the Test Oracle Problem Prof. T.H. Tse](https://reader035.vdocument.in/reader035/viewer/2022070813/5f0c4f817e708231d434c453/html5/thumbnails/151.jpg)
Ubiquitous Computing Computing everywhere and at any time Applications operate in dynamic environment Expected outcome = actual execution result
151
![Page 152: Even the Very Wise Cannot See All Ends: Many … › ... › Keynote-TH-Tse-slides.pdfEven the Very Wise Cannot See All Ends: Many Facets of the Test Oracle Problem Prof. T.H. Tse](https://reader035.vdocument.in/reader035/viewer/2022070813/5f0c4f817e708231d434c453/html5/thumbnails/152.jpg)
Ubiquitous Computing Computing everywhere and at any time Applications operate in dynamic environment Expected outcome = actual execution result When does a test case end?
152
![Page 153: Even the Very Wise Cannot See All Ends: Many … › ... › Keynote-TH-Tse-slides.pdfEven the Very Wise Cannot See All Ends: Many Facets of the Test Oracle Problem Prof. T.H. Tse](https://reader035.vdocument.in/reader035/viewer/2022070813/5f0c4f817e708231d434c453/html5/thumbnails/153.jpg)
Ubiquitous Computing Computing everywhere and at any time Applications operate in dynamic environment Expected outcome = actual execution result When does a test case end?
Middleware remains active and environment context keeps changing!
153
![Page 154: Even the Very Wise Cannot See All Ends: Many … › ... › Keynote-TH-Tse-slides.pdfEven the Very Wise Cannot See All Ends: Many Facets of the Test Oracle Problem Prof. T.H. Tse](https://reader035.vdocument.in/reader035/viewer/2022070813/5f0c4f817e708231d434c453/html5/thumbnails/154.jpg)
Ubiquitous Computing Expected and Actual Context Trends
Actual result
0
50
100
150
200
250
300
350
0
5
10
15
20
25
30
1 2 3 4 5 6 7 8 9 10 11 12 13 14
Con
text
ual V
alue
afte
r i-th
con
text
upd
ate
The i-th context update
qd qv
ql d
Expected Result
0
50
100
150
200
250
300
350
0
5
10
15
20
25
30
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15
Con
text
ual V
alue
afte
r i-th
con
text
upd
ate
The i-th context update
qd qv
qp d Any difference?
![Page 155: Even the Very Wise Cannot See All Ends: Many … › ... › Keynote-TH-Tse-slides.pdfEven the Very Wise Cannot See All Ends: Many Facets of the Test Oracle Problem Prof. T.H. Tse](https://reader035.vdocument.in/reader035/viewer/2022070813/5f0c4f817e708231d434c453/html5/thumbnails/155.jpg)
Identify checkpoints where the system is momentarily stable.
Ubiquitous Computing Computing everywhere and at any time Applications operate in dynamic environment Expected outcome = actual execution result When does a test case end?
Middleware remains active and environment context keeps changing!
155
![Page 156: Even the Very Wise Cannot See All Ends: Many … › ... › Keynote-TH-Tse-slides.pdfEven the Very Wise Cannot See All Ends: Many Facets of the Test Oracle Problem Prof. T.H. Tse](https://reader035.vdocument.in/reader035/viewer/2022070813/5f0c4f817e708231d434c453/html5/thumbnails/156.jpg)
Conclusion Many facets of the test oracle problem
156
![Page 157: Even the Very Wise Cannot See All Ends: Many … › ... › Keynote-TH-Tse-slides.pdfEven the Very Wise Cannot See All Ends: Many Facets of the Test Oracle Problem Prof. T.H. Tse](https://reader035.vdocument.in/reader035/viewer/2022070813/5f0c4f817e708231d434c453/html5/thumbnails/157.jpg)
Conclusion Many facets of the test oracle problem
Expected outcome = actual execution result
157
Metamorphic testing
Many facets of metamorphic testing.
![Page 158: Even the Very Wise Cannot See All Ends: Many … › ... › Keynote-TH-Tse-slides.pdfEven the Very Wise Cannot See All Ends: Many Facets of the Test Oracle Problem Prof. T.H. Tse](https://reader035.vdocument.in/reader035/viewer/2022070813/5f0c4f817e708231d434c453/html5/thumbnails/158.jpg)
Conclusion Many facets of the test oracle problem
Expected outcome = actual execution result Expected outcome = actual execution result
158
Algebraic specifications
Many facets of algebraic specifications.
![Page 159: Even the Very Wise Cannot See All Ends: Many … › ... › Keynote-TH-Tse-slides.pdfEven the Very Wise Cannot See All Ends: Many Facets of the Test Oracle Problem Prof. T.H. Tse](https://reader035.vdocument.in/reader035/viewer/2022070813/5f0c4f817e708231d434c453/html5/thumbnails/159.jpg)
Conclusion Many facets of the test oracle problem
Expected outcome = actual execution result Expected outcome = actual execution result Expected outcome = actual execution result
159
Many facets of ubiquitous computing.
![Page 160: Even the Very Wise Cannot See All Ends: Many … › ... › Keynote-TH-Tse-slides.pdfEven the Very Wise Cannot See All Ends: Many Facets of the Test Oracle Problem Prof. T.H. Tse](https://reader035.vdocument.in/reader035/viewer/2022070813/5f0c4f817e708231d434c453/html5/thumbnails/160.jpg)
Conclusion Jungle of problems Jungle of proposals Empirical studies are just an exploratory first step
rather than the ultimate goal Tame the forest rather than trimming individual
trees. 160
![Page 161: Even the Very Wise Cannot See All Ends: Many … › ... › Keynote-TH-Tse-slides.pdfEven the Very Wise Cannot See All Ends: Many Facets of the Test Oracle Problem Prof. T.H. Tse](https://reader035.vdocument.in/reader035/viewer/2022070813/5f0c4f817e708231d434c453/html5/thumbnails/161.jpg)
Your Comments are Welcome
![Page 162: Even the Very Wise Cannot See All Ends: Many … › ... › Keynote-TH-Tse-slides.pdfEven the Very Wise Cannot See All Ends: Many Facets of the Test Oracle Problem Prof. T.H. Tse](https://reader035.vdocument.in/reader035/viewer/2022070813/5f0c4f817e708231d434c453/html5/thumbnails/162.jpg)
Thank you
![Page 163: Even the Very Wise Cannot See All Ends: Many … › ... › Keynote-TH-Tse-slides.pdfEven the Very Wise Cannot See All Ends: Many Facets of the Test Oracle Problem Prof. T.H. Tse](https://reader035.vdocument.in/reader035/viewer/2022070813/5f0c4f817e708231d434c453/html5/thumbnails/163.jpg)
Load the next file please
![Page 164: Even the Very Wise Cannot See All Ends: Many … › ... › Keynote-TH-Tse-slides.pdfEven the Very Wise Cannot See All Ends: Many Facets of the Test Oracle Problem Prof. T.H. Tse](https://reader035.vdocument.in/reader035/viewer/2022070813/5f0c4f817e708231d434c453/html5/thumbnails/164.jpg)