![Page 1: Questioning Best Practices: An Experience Reportmcconda.com/wp-content/uploads/2011/06/Questioning-Best-Practic… · Context-Driven Approach •2007: –ustomer’s concern with](https://reader033.vdocument.in/reader033/viewer/2022050409/5f85e4ada1d3a8189b46dbff/html5/thumbnails/1.jpg)
Questioning Best Practices:An Experience Report
John McConda, Moser ConsultingIllustrations by Garrett VanTiem
![Page 2: Questioning Best Practices: An Experience Reportmcconda.com/wp-content/uploads/2011/06/Questioning-Best-Practic… · Context-Driven Approach •2007: –ustomer’s concern with](https://reader033.vdocument.in/reader033/viewer/2022050409/5f85e4ada1d3a8189b46dbff/html5/thumbnails/2.jpg)
About Me
• 10 years industry experience
• Consultant for a US government software developer
• Volunteer instructor for BBST online courses
• Member of the Context-Driven testing community
• Las Vegas Buffet Enthusiast
![Page 3: Questioning Best Practices: An Experience Reportmcconda.com/wp-content/uploads/2011/06/Questioning-Best-Practic… · Context-Driven Approach •2007: –ustomer’s concern with](https://reader033.vdocument.in/reader033/viewer/2022050409/5f85e4ada1d3a8189b46dbff/html5/thumbnails/3.jpg)
Rejected Titles
• Burning Best Practices
• Bashing Best Practices
• Banishing Best Practices
• Belittling Best Practices
• Booting Best Practices
• Banning Best Practices
• I Donny and Marie
![Page 4: Questioning Best Practices: An Experience Reportmcconda.com/wp-content/uploads/2011/06/Questioning-Best-Practic… · Context-Driven Approach •2007: –ustomer’s concern with](https://reader033.vdocument.in/reader033/viewer/2022050409/5f85e4ada1d3a8189b46dbff/html5/thumbnails/4.jpg)
What is a best practice?
• Any process, technique, approach, or activity touted as “best” without regard to context
![Page 5: Questioning Best Practices: An Experience Reportmcconda.com/wp-content/uploads/2011/06/Questioning-Best-Practic… · Context-Driven Approach •2007: –ustomer’s concern with](https://reader033.vdocument.in/reader033/viewer/2022050409/5f85e4ada1d3a8189b46dbff/html5/thumbnails/5.jpg)
From the 7 Principles of Context-Driven School of Software Testing
1. The value of any practice depends on its context.
2. There are good practices in context, but there are no best practices.
![Page 6: Questioning Best Practices: An Experience Reportmcconda.com/wp-content/uploads/2011/06/Questioning-Best-Practic… · Context-Driven Approach •2007: –ustomer’s concern with](https://reader033.vdocument.in/reader033/viewer/2022050409/5f85e4ada1d3a8189b46dbff/html5/thumbnails/6.jpg)
Examples in software testing
• Requirements are crucial for effective testing
• Trace requirements to tests
• Tests must have preconditions, expected results, actual results, and post-conditions
![Page 7: Questioning Best Practices: An Experience Reportmcconda.com/wp-content/uploads/2011/06/Questioning-Best-Practic… · Context-Driven Approach •2007: –ustomer’s concern with](https://reader033.vdocument.in/reader033/viewer/2022050409/5f85e4ada1d3a8189b46dbff/html5/thumbnails/7.jpg)
Examples in software testing
• Create an overall test plan
• Perform both positive and negative testing
• Formally design your tests
![Page 8: Questioning Best Practices: An Experience Reportmcconda.com/wp-content/uploads/2011/06/Questioning-Best-Practic… · Context-Driven Approach •2007: –ustomer’s concern with](https://reader033.vdocument.in/reader033/viewer/2022050409/5f85e4ada1d3a8189b46dbff/html5/thumbnails/8.jpg)
Examples in software testing
• Can you think of any “best practices” that were not right for your context?
![Page 9: Questioning Best Practices: An Experience Reportmcconda.com/wp-content/uploads/2011/06/Questioning-Best-Practic… · Context-Driven Approach •2007: –ustomer’s concern with](https://reader033.vdocument.in/reader033/viewer/2022050409/5f85e4ada1d3a8189b46dbff/html5/thumbnails/9.jpg)
OED defines a practice as:
"The habitual doing or carrying on of something.
Usual, customary, or constant action or performance; conduct."
Can we even define a best practice?
![Page 10: Questioning Best Practices: An Experience Reportmcconda.com/wp-content/uploads/2011/06/Questioning-Best-Practic… · Context-Driven Approach •2007: –ustomer’s concern with](https://reader033.vdocument.in/reader033/viewer/2022050409/5f85e4ada1d3a8189b46dbff/html5/thumbnails/10.jpg)
Can we even define a best practice?
• "Best" is a comparison to something else, so better than what other practices?
• How do you know what is "best" without context?
• How do you know what the practice actually means without context?
![Page 11: Questioning Best Practices: An Experience Reportmcconda.com/wp-content/uploads/2011/06/Questioning-Best-Practic… · Context-Driven Approach •2007: –ustomer’s concern with](https://reader033.vdocument.in/reader033/viewer/2022050409/5f85e4ada1d3a8189b46dbff/html5/thumbnails/11.jpg)
• "Every requirement should be covered by a test case."
– What kind of requirements?
• (Use Cases, user stories, design documents, etc.)
Can we even define a best practice?
![Page 12: Questioning Best Practices: An Experience Reportmcconda.com/wp-content/uploads/2011/06/Questioning-Best-Practic… · Context-Driven Approach •2007: –ustomer’s concern with](https://reader033.vdocument.in/reader033/viewer/2022050409/5f85e4ada1d3a8189b46dbff/html5/thumbnails/12.jpg)
• "Every requirement should be covered by a test case."
– What kind of tool are you using to show this coverage?
– What measurement tells you (or your auditor) that the test has tested every part of the requirement it is reported as covering?
Can we even define a best practice?
![Page 13: Questioning Best Practices: An Experience Reportmcconda.com/wp-content/uploads/2011/06/Questioning-Best-Practic… · Context-Driven Approach •2007: –ustomer’s concern with](https://reader033.vdocument.in/reader033/viewer/2022050409/5f85e4ada1d3a8189b46dbff/html5/thumbnails/13.jpg)
Bach's Definition:
A best practice (in practice) really means “a practice that I promote”.
Can we even define a best practice?
![Page 14: Questioning Best Practices: An Experience Reportmcconda.com/wp-content/uploads/2011/06/Questioning-Best-Practic… · Context-Driven Approach •2007: –ustomer’s concern with](https://reader033.vdocument.in/reader033/viewer/2022050409/5f85e4ada1d3a8189b46dbff/html5/thumbnails/14.jpg)
Experience Report: Applying the Context-Driven Approach
![Page 15: Questioning Best Practices: An Experience Reportmcconda.com/wp-content/uploads/2011/06/Questioning-Best-Practic… · Context-Driven Approach •2007: –ustomer’s concern with](https://reader033.vdocument.in/reader033/viewer/2022050409/5f85e4ada1d3a8189b46dbff/html5/thumbnails/15.jpg)
Experience Report: Applying the Context-Driven Approach
• 2007:
–Customer’s concern with quality leads to corrective action plan
–New QA manager brought in to help remediate concerns
–Chance to look at the entire development process, starting with Testing.
![Page 16: Questioning Best Practices: An Experience Reportmcconda.com/wp-content/uploads/2011/06/Questioning-Best-Practic… · Context-Driven Approach •2007: –ustomer’s concern with](https://reader033.vdocument.in/reader033/viewer/2022050409/5f85e4ada1d3a8189b46dbff/html5/thumbnails/16.jpg)
Experience Report: Applying the Context-Driven Approach
Establish the Mission
Assess Value of Practices
Eliminate practices that do not contribute to
our mission
Add practices that maximize focus on
our mission
![Page 17: Questioning Best Practices: An Experience Reportmcconda.com/wp-content/uploads/2011/06/Questioning-Best-Practic… · Context-Driven Approach •2007: –ustomer’s concern with](https://reader033.vdocument.in/reader033/viewer/2022050409/5f85e4ada1d3a8189b46dbff/html5/thumbnails/17.jpg)
Task 1: Establish the Mission of the Testing Team
Experience Report: Applying the Context-Driven Approach
![Page 18: Questioning Best Practices: An Experience Reportmcconda.com/wp-content/uploads/2011/06/Questioning-Best-Practic… · Context-Driven Approach •2007: –ustomer’s concern with](https://reader033.vdocument.in/reader033/viewer/2022050409/5f85e4ada1d3a8189b46dbff/html5/thumbnails/18.jpg)
• Customer-defined SLAs:
–Must have 85% of features with no defects logged during UAT
–Must have 95% of features with no defects logged in production
–Performance SLAs: Five transactions that must perform under a specified response time.
Experience Report: Applying the Context-Driven Approach
![Page 19: Questioning Best Practices: An Experience Reportmcconda.com/wp-content/uploads/2011/06/Questioning-Best-Practic… · Context-Driven Approach •2007: –ustomer’s concern with](https://reader033.vdocument.in/reader033/viewer/2022050409/5f85e4ada1d3a8189b46dbff/html5/thumbnails/19.jpg)
• Customer’s Statement of Work
–Describes guidelines and requirements for testing documentation
–Subject to external audit by IV & V group
Experience Report: Applying the Context-Driven Approach
![Page 20: Questioning Best Practices: An Experience Reportmcconda.com/wp-content/uploads/2011/06/Questioning-Best-Practic… · Context-Driven Approach •2007: –ustomer’s concern with](https://reader033.vdocument.in/reader033/viewer/2022050409/5f85e4ada1d3a8189b46dbff/html5/thumbnails/20.jpg)
Task 2:
Assess value of all practices in contributing to the mission
Experience Report: Applying the Context-Driven Approach
![Page 21: Questioning Best Practices: An Experience Reportmcconda.com/wp-content/uploads/2011/06/Questioning-Best-Practic… · Context-Driven Approach •2007: –ustomer’s concern with](https://reader033.vdocument.in/reader033/viewer/2022050409/5f85e4ada1d3a8189b46dbff/html5/thumbnails/21.jpg)
Impossibility of Complete Testing
“If you test completely, then at the end
of testing, there cannot be any
undiscovered errors.”
-Cem Kaner6
![Page 22: Questioning Best Practices: An Experience Reportmcconda.com/wp-content/uploads/2011/06/Questioning-Best-Practic… · Context-Driven Approach •2007: –ustomer’s concern with](https://reader033.vdocument.in/reader033/viewer/2022050409/5f85e4ada1d3a8189b46dbff/html5/thumbnails/22.jpg)
• Because complete testing is impossible, every activity is a trade-off with something else we could be doing
• We must focus our time on activities that contribute most to completing our mission
Impossibility of Complete Testing
![Page 23: Questioning Best Practices: An Experience Reportmcconda.com/wp-content/uploads/2011/06/Questioning-Best-Practic… · Context-Driven Approach •2007: –ustomer’s concern with](https://reader033.vdocument.in/reader033/viewer/2022050409/5f85e4ada1d3a8189b46dbff/html5/thumbnails/23.jpg)
Task 3: Eliminate practices not contributing to the mission:
Experience Report: Applying the Context-Driven Approach
![Page 24: Questioning Best Practices: An Experience Reportmcconda.com/wp-content/uploads/2011/06/Questioning-Best-Practic… · Context-Driven Approach •2007: –ustomer’s concern with](https://reader033.vdocument.in/reader033/viewer/2022050409/5f85e4ada1d3a8189b46dbff/html5/thumbnails/24.jpg)
• Maintaining Manual Test Cases
– Updating tests took over 60% of total testing time for many features.
– Formal test execution did not occur until tests were updated.
• Automated Regression Testing Suite
Experience Report: Applying the Context-Driven Approach
![Page 25: Questioning Best Practices: An Experience Reportmcconda.com/wp-content/uploads/2011/06/Questioning-Best-Practic… · Context-Driven Approach •2007: –ustomer’s concern with](https://reader033.vdocument.in/reader033/viewer/2022050409/5f85e4ada1d3a8189b46dbff/html5/thumbnails/25.jpg)
• Use of Word documents and Excel spreadsheets to write tests and track execution
• Use of Test Team resources for “side” projects not factored in metrics for the mission
Experience Report: Applying the Context-Driven Approach
![Page 26: Questioning Best Practices: An Experience Reportmcconda.com/wp-content/uploads/2011/06/Questioning-Best-Practic… · Context-Driven Approach •2007: –ustomer’s concern with](https://reader033.vdocument.in/reader033/viewer/2022050409/5f85e4ada1d3a8189b46dbff/html5/thumbnails/26.jpg)
Task 4: Add practices that maximize focus on the mission
Experience Report: Applying the Context-Driven Approach
![Page 27: Questioning Best Practices: An Experience Reportmcconda.com/wp-content/uploads/2011/06/Questioning-Best-Practic… · Context-Driven Approach •2007: –ustomer’s concern with](https://reader033.vdocument.in/reader033/viewer/2022050409/5f85e4ada1d3a8189b46dbff/html5/thumbnails/27.jpg)
• Mission metrics posted and updated each month
• Redesigned Performance Testing suite to model user behavior for response time SLAs
Experience Report: Applying the Context-Driven Approach
![Page 28: Questioning Best Practices: An Experience Reportmcconda.com/wp-content/uploads/2011/06/Questioning-Best-Practic… · Context-Driven Approach •2007: –ustomer’s concern with](https://reader033.vdocument.in/reader033/viewer/2022050409/5f85e4ada1d3a8189b46dbff/html5/thumbnails/28.jpg)
• Adopted an Exploratory approach to testing
–Only High-level test strategies are documented before testing
– Test cases are not maintained release to release
Experience Report: Applying the Context-Driven Approach
![Page 29: Questioning Best Practices: An Experience Reportmcconda.com/wp-content/uploads/2011/06/Questioning-Best-Practic… · Context-Driven Approach •2007: –ustomer’s concern with](https://reader033.vdocument.in/reader033/viewer/2022050409/5f85e4ada1d3a8189b46dbff/html5/thumbnails/29.jpg)
• Test automation pared down to more maintainable regression test suite and used for high volume testing
• Moved test documentation and execution to HP Quality Center
Experience Report: Applying the Context-Driven Approach
![Page 30: Questioning Best Practices: An Experience Reportmcconda.com/wp-content/uploads/2011/06/Questioning-Best-Practic… · Context-Driven Approach •2007: –ustomer’s concern with](https://reader033.vdocument.in/reader033/viewer/2022050409/5f85e4ada1d3a8189b46dbff/html5/thumbnails/30.jpg)
• Adopted an Exploratory approach to testing
– Testing can happen before requirements are complete (are they ever? )
– Test evidence remained compliant with customer’s standards
Experience Report: Applying the Context-Driven Approach
![Page 31: Questioning Best Practices: An Experience Reportmcconda.com/wp-content/uploads/2011/06/Questioning-Best-Practic… · Context-Driven Approach •2007: –ustomer’s concern with](https://reader033.vdocument.in/reader033/viewer/2022050409/5f85e4ada1d3a8189b46dbff/html5/thumbnails/31.jpg)
• You might be saying “We can’t do that!”
• This Better Software article is for you!
A Note to Regulated Testers
![Page 32: Questioning Best Practices: An Experience Reportmcconda.com/wp-content/uploads/2011/06/Questioning-Best-Practic… · Context-Driven Approach •2007: –ustomer’s concern with](https://reader033.vdocument.in/reader033/viewer/2022050409/5f85e4ada1d3a8189b46dbff/html5/thumbnails/32.jpg)
Results
• 2009-2011–Unprecedented 6 consecutive “Superior”
ratings
–100% compliance to performance SLAs
–100% compliance to UAT defect margin
–100% compliance to production defect margin
• Not just the Test Team, but a combined effort
![Page 33: Questioning Best Practices: An Experience Reportmcconda.com/wp-content/uploads/2011/06/Questioning-Best-Practic… · Context-Driven Approach •2007: –ustomer’s concern with](https://reader033.vdocument.in/reader033/viewer/2022050409/5f85e4ada1d3a8189b46dbff/html5/thumbnails/33.jpg)
Bumps in the Road
• Never believe a success story that doesn’t have some failure along the way!
![Page 34: Questioning Best Practices: An Experience Reportmcconda.com/wp-content/uploads/2011/06/Questioning-Best-Practic… · Context-Driven Approach •2007: –ustomer’s concern with](https://reader033.vdocument.in/reader033/viewer/2022050409/5f85e4ada1d3a8189b46dbff/html5/thumbnails/34.jpg)
Bumps in the Road
• Established process
• Team turnover
• “Pot Roast problem”
![Page 35: Questioning Best Practices: An Experience Reportmcconda.com/wp-content/uploads/2011/06/Questioning-Best-Practic… · Context-Driven Approach •2007: –ustomer’s concern with](https://reader033.vdocument.in/reader033/viewer/2022050409/5f85e4ada1d3a8189b46dbff/html5/thumbnails/35.jpg)
Overcoming Challenges
• “New” is not always better
• New ideas are subject to the same measure of the mission
• Context-driven is the opposite of forcing preconceived ideas on an unknown situation
• Change was gradual, still happening
![Page 36: Questioning Best Practices: An Experience Reportmcconda.com/wp-content/uploads/2011/06/Questioning-Best-Practic… · Context-Driven Approach •2007: –ustomer’s concern with](https://reader033.vdocument.in/reader033/viewer/2022050409/5f85e4ada1d3a8189b46dbff/html5/thumbnails/36.jpg)
This Presentation is Context-Driven!
• What works for you?
• What doesn’t?
• Why?
![Page 37: Questioning Best Practices: An Experience Reportmcconda.com/wp-content/uploads/2011/06/Questioning-Best-Practic… · Context-Driven Approach •2007: –ustomer’s concern with](https://reader033.vdocument.in/reader033/viewer/2022050409/5f85e4ada1d3a8189b46dbff/html5/thumbnails/37.jpg)
Sources
1. Lessons Learned in Software Testing: A Context-Driven ApproachCem Kaner, James Bach, Bret Pettichord2002 John Wiley & Sons
2. QualityPoint Technologies Blog: “Best Practices in Software Testing”http://qualitypoint.blogspot.com/2009/04/best-practices-in-in-software-testing.html
3. International Institute for Software Testing Course Description: “Industry Best Practices in Software Quality Assurance and Testing”www.testinginstitute.com/display.php?id=s_t_f_b_p_MH
4. James Bach’s Blog “No Best Practices” www.satisfice.com/blog/archives/27
5. The Seven Basic Principles of the Context-Driven Schoolwww.context-driven-testing.com/
6. Cem Kaner: “Impossibility of Complete Testing”www.kaner.com/pdfs/impossible.pdf
![Page 38: Questioning Best Practices: An Experience Reportmcconda.com/wp-content/uploads/2011/06/Questioning-Best-Practic… · Context-Driven Approach •2007: –ustomer’s concern with](https://reader033.vdocument.in/reader033/viewer/2022050409/5f85e4ada1d3a8189b46dbff/html5/thumbnails/38.jpg)
Contact
• Email me! [email protected]
• Twitter @jmcconda
• BBST Courses: associationforsoftwaretesting.org/training
• WREST Workshop tomorrow! wrestworkshop.com