software testing #irl [email protected]. agenda why software testing software testing...
TRANSCRIPT
Software Testing #[email protected]
Agenda
• Why Software Testing• Software Testing Principles
– Exhaustive testing is impossible– Early Testing– Testing shows the Presence of Bugs– Defect Clustering– The Pesticide Paradox– Testing is Context Dependent – Absence of errors fallacy
• Summary• Recommended reading
Other professions in Computer Science
Project Lead Developer Architect
Technical Writer Customer IT Ops
How Other professions in Computer Science sees test
Project Lead Developer Architect
Technical Writer Customer IT Ops
Agenda
• Why Software Testing• Software Testing Principles
– Exhaustive testing is impossible– Early Testing– Testing shows the Presence of Bugs– Defect Clustering– The Pesticide Paradox– Testing is Context Dependent – Absence of errors fallacy
• Summary• Recommended reading
Exhaustive Testing is Impossible
What you Think you are Testing
Developer Forgot «this.»
Off by One
Type Casting
Previous Values
Date and Time
Obey Thermodynamics – it’s also a LAW!
• 1 year of power of sun is enough energy to flip bits to count to all combinations in a 187 bit number.
• 187 bit < 24 byte.
Google: “schneier dog crypteto”http://www.schneier.com/blog/archives/2009/09/the_doghouse_cr.html
Early Testing: V-Model
Architecture
Requirements
Specification
Design
Code
Acceptance Test
System Test
Integration Test
Unit Test
Early Testing: W-Model
Architecture
Requirements
Specification
Design
Code
Acceptance Test
System Test
Integration Test
Unit Test
Design Acceptance Test
Design System Test
Design Integration Test
Design Unit Test
Cost of Defect
Introduced\Detected Reqs Arch Construction System test Production
Requirements 1X 3X 5-10X 10X 10-100X
Architecture - 1X 10X 15X 25-100X
Construction - - 1X 10X 10-25X
http://www.abeacha.com/NIST_press_release_bugs_cost.htm
Quality Attributes: ISO/IEC 9126-1:2001
• Functionality - satisfies needs.– Suitability– Accuracy– Interoperability– Security– Functionality Compliance
• Reliability - maintains operation– Maturity– Fault Tolerance– Recoverability– Reliability Compliance
• Usability - effort needed for use– Understandability– Learnability– Operability– Attractiveness– Usability Compliance
• Efficiency - performance given resources– Time Behaviour– Resource Utilisation– Efficiency Compliance
• Maintainability - make modifications– Analyzability– Changeability– Stability– Testability– Maintainability Compliance
• Portability - change environment– Adaptability– Installability– Co-Existence– Replaceability– Portability Compliance
Types of Testing
Acceptance Test
System Test
Integration Test
Unit Test
Functional Efficiency Interopability
X
X
X
X
X
X
X
Several Layers
Acceptance Test
System Test
Integration Test
Unit Test
Functional Efficiency Security
X
X
X
X
X
X
X
System Int. Test X
Testing shows the presence of bugs, but can not show that there are no defects
• 1986: Programming Pearls, binarySearch() bevist og testet i et kapittel
• 1997: Implementert i Sun java.util.Arrays.binarySearch()
• 2006: Feil funnet og fikset i Java-biblioteket
http://googleresearch.blogspot.it/2006/06/extra-extra-read-all-about-it-nearly.html
Defect Clustering
http://www.slideshare.net/andreas.zeller/myths-in-software-engineering
Bugs Found with Bug Seeding
http://www.slideshare.net/andreas.zeller/seeding-bugs-to-find-bugs-presentation
Pesticide Paradox
Age oftest
Bugs found per dayby a test
Regression Testing
Acceptance Test
System Test
Integration Test
Unit Test
Functional Efficiency Security
X
X
X
X
X
X
X
System Int. Test X
X
Leve
ls
Types
Testing is Context Dependent
Coverage
Cost
0% 50% 100%
Cost oftesting
80-85%?Cost ofdefects
Absence of Errors does not mean the System is OK
Acceptance Test
System Test
Integration Test
Unit Test95%
Performance/Stability
Agenda
• Why Software Testing• Software Testing Principles
– Exhaustive testing is impossible– Early Testing– Testing shows the Presence of Bugs– Defect Clustering– The Pesticide Paradox– Testing is Context Dependent – Absence of errors fallacy
• Summary• Recommended reading
TL;DL
• Testing #FTW #YOLO
• You can’t test everything!• Testing is Context Dependent!
• Get involved early!
• Testing doesn’t prove software is without bugs• Absence of errors does not prove the system is ok
• Defects cluster• Old tests find fewer bugs
Books you should read to learn more about…
People and testing Creative Testing Coding
Software Engineering Testing in Trondheim Fun Web
Software Testing #[email protected]
Copyrights
• Ant Hill: RukaKuusamo.com• Money bag: http://www.flickr.com/photos/68751915@N05/ CC AT SA• Book: http://www.flickr.com/photos/pat00139/ CC AT• Cables: http://www.flickr.com/photos/alq666/ CC AT SA• Bokhylle: Sehoon Lee Phote
http://design-milk.com/squaring-movable-bookshelf-by-sehoon-lee/• Hender: CC AT NM NC http://www.flickr.com/photos/johnkay/• Angel: CC AT NC http://www.flickr.com/photos/jens-h-friedrich• Guarantee: CC AT http://www.flickr.com/photos/vectorportal• Road: CC AT SA http://www.flickr.com/photos/huangjiahui/