how do we fix testing
TRANSCRIPT
Slide 1Logo design: Sarah Cole Design
SQGNE is made possible by the support of our sponsors:
Oracle andSun Integrated Systems
Dec 2014
WELCOME TO SEASON 21!
All-volunteer non-profit with no membership dues!
Supported entirely by our sponsors…
Over 1,275 members on LinkedIn, 875 Constant
Contact
Monthly meetings - Sept to July on 2nd Wed of
month
SQGNE Web site: www.sqgne.org
Dec 2014
WELCOME TO SEASON 21!
Officers / Hosts / Mission
Mission To promote use of engineering and management techniques that lead to delivery
of high quality software
To disseminate concepts and techniques related to software quality engineering and software engineering process
To provide a forum for discussion of concepts and techniques related to software quality engineering and the software engineering process
To provide networking opportunities for software quality professionals
Dec 2014
Current Officers: John Pustaver – Founder Steve Rakitin – President Stan Wrobel – Vice
President Barbara Wioncek –
Treasurer David Sullivan– Clerk
Our gracious
Hosts: Paul Ratty Tom Arakel Marge Shinkle Jack Guilderson
At-large
Directors: Robin
Goldsmith Marge Shinkle Jim Turner
SQGNE 2014-15 ScheduleSpeaker Affiliation Date Topic
Jim Parson
Nick Olivo
Checkpoint Tech.
SmartBearSept 10
Test Tool Bake-off featuring HP WebInspect Checkpoint Tech. and Test Complete
Harish Narayan Oct 8Leveraging lean systems development and testing to maximize business value
Michael MahQSM Associates
Nov 12Passion: What Software Teams Might Learn About Creating Lasting Change from Eco-pirates
Peter Varhol Gerie Owen
Dec 10 How do we fix testing?
Stan WrobelMass. Dept of Education
Jan 14Agile Development and the changing roles for Business Analysts and QA
Robin Goldsmith
GoPro Management
Feb 11 Is Exploratory Testing Really Better?
Mario Moreira Mar 11 Agile – Bringing Testing to the Forefront
Andrew Ambroise
Apr 8Shifting Quality to the Left; Negotiating Cleaner Results
Gerie Owen NSTAR May 13Testing the Internet of Things: Human Experience Testing
David Heimann June 10
IEEE Standard 730-2014 Software Quality Assurance
Processes
Annual Election of Officers
Everyone July 8 Annual Hot Topics Night…
Dec 2014
Tonight’s TopicHow do we fix testing?Peter Varhol and Gerie Owen
Perhaps in no other professional field is the dichotomy between theory and practice more starkly different than in the realm of software testing. Researchers and thought leaders claim that testing requires a high level of cognitive and interpersonal skills, in order to make judgments about the ability of software to fulfill its operational goals. In their minds, testing is about assessing and communicating the risks involved in deploying software in a specific state.However, in many organizations, testing remains a necessary evil, and a cost to drive down as much as possible. Testing is merely a measure of conformance to requirements, without regard to the quality of requirements or how conformance is measured. This is certainly an important measure, but tells an incomplete story about the value of software in support of our business goals.We as testers often help to perpetuate the status quo. Although in many cases we realize we can add far more value than we do, we continue to perform testing in a manner that reduces our value in the software development process.This presentation looks at the state of the art as well of the state of common practice, and attempts to provide a rationale and roadmap whereby the practice of testing can be made more exciting and stimulating to the testing professional, as well as more valuable to the product and the organization.
BiosPeter Varhol is a well-known writer and speaker on software and technology topics, having authored dozens of articles and spoken at a number of industry conferences and webcasts. He has advanced degrees in computer science, applied mathematics, and psychology, and is currently Principal at Technology Strategy Research. His past roles include technology journalist, software product manager, software developer, and university professor.Gerie Owen is a Test Architect who specializes in developing and managing test teams. She has implemented various Quality Assurance methodology models, and has developed, trained and mentored new teams from their inception. Gerie manages large, complex projects involving multiple applications, coordinates test teams across multiple time zones and delivers high quality projects on time and within budget. In her everyday working life, Gerie brings a cohesive team approach to testing. She has also presented at several conferences and authored articles on testing and quality assurance at
www.qsma.com.Dec 2014
HOW DO WE FIX TESTING?WHY WE HAVE A GREAT OPPORTUNITY TO MAKE IMPROVEMENTS
TO OUR CRAFT
THOUGHT LEADER
Peter Varhol
International speaker on technology topics
Technology Evangelist and Writer, Product Manager, University Professor
Someone who cares about getting testing right
PRACTITIONER
Gerie Owen
Quality Assurance Consultant
Speaker and Writer on Testing topics
Experienced Tester, Test Lead, & Test Architect
Marathon Runner & Running Coach
Meet Our Cats...Max, Stephanie& Monica
AGENDA
• What we do
• Why it’s not right
• What thought leaders say we should do
• What we need to do differently
• How we get there
• Summary and conclusions
OUR APPLICATIONS HAVE PROBLEMS
EVEN THE IMPORTANT ONES
WHAT HAPPENED TO TESTING?
•We don’t have time for it
• We have to get to market
• We don’t have funding to test
• We haven’t shown sufficient value
• So we let our users test
• That seems to work
INCREASINGLY THE RESULT IS FAILURE
•And everyone knows it’s going to fail
•But no one wants to say so
•So we are all complicit• Because we know better
WHAT WE DO
• Write test plans and procedures
• Analyze and deconstruct requirements
• Write test cases
• Execute test cases and compare results to requirements
• Make a determination on whether requirements have been met
WHAT WE DO
• Testing finds bugs
• Bugs are a mismatch between requirements and implementation
• It’s an investigation with a yea or nay answer
• If only it were that simple
WHAT WE DO
• Classify bugs by priority and severity
• Allowing room for negotiation and compromise
• Provide metrics on test cases executed
• Use test cases and metrics to make decisions on quality
WHAT WE DO
• Many organizations call this Quality Assurance
• But we can’t assure quality
• Quality is the result of lifecycle processes
• And we can’t build in quality
• But we can test and evaluate
• If not quality, then what?
AND WHAT IS QUALITY?
• Is it really a mismatch of requirements and reality?
• That makes several assumptions
• The requirements are accurate and complete
• The requirements are unambiguous
• The requirements are all we care about
• We talk about 100 percent test coverage
• But do we really know?
WHAT THOUGHT LEADERS SAY WE SHOULD DO
• Think
• Act
• Go beyond the requirements
• Look at the context
• Evaluate and make judgments
• Communicate and advise
AND WHAT IS QUALITY?
•Does testing to requirements enable us to find
• Inaccurate or incomplete requirements?
• Inefficient database calls?
• Memory or object leaks?
• Just plain bad design?
• External or internal
AND WHAT IS QUALITY?
• So we have to do more
• Yes, it is our problem
WHAT THOUGHT LEADERS SAY WE SHOULD DO
• Testing to requirements is a necessary part of the process
• But . . .
• It’s necessary but not sufficient
RULES OF THE ROAD
• Testing requires understanding the problem domain intimately
• Domain experts must be part of the team
• Testing requires understanding the requirements
• Testing requires thinking like a user
• And being willing to take unpopular stands
AND BY THE WAY
• A lot of testing today is done by users
• Whether they know it or not
• Whether they’re qualified or not
• And they probably have no idea of the purpose or requirements
• Let alone context
AND THAT’S A PROBLEM
• Software that should be better isn’t
• Often not even tested to requirements
• And many software organizations don’t seem to care
•We should care about that
WHAT SHOULD WE DO DIFFERENTLY?
• Understand our own Mindsets
WHAT ARE MINDSETS?
• Developed by psychologist Carol Dweck
• How we mentally approach life and its challenges
• Why brains and talent don’t bring success
• How they can stand in the way of it
• Why praising brains and talent doesn’t foster self-esteem and accomplishment, but jeopardizes them
WHAT IS A MINDSET
•Broadly categorized in one of two ways
• Fixed
•Growth
IMPLICATIONS OF A FIXED MINDSET
• I did my job right; someone else screwed up
• That’s not really a bug, so I didn’t miss it
• My test cases are complete
• I test to requirements
• You can’t ship this software
IMPLICATIONS OF A GROWTH MINDSET
• I always have more to learn
• I missed that bug, and I understand why
• I can adapt my testing approach to new trends
• I advise decision makers on risks of releasing software
• I don’t ensure quality, whatever my title is
HOW DO WE FIX TESTING?
•Personally Adopt a “Growth Mindset”
•Evangelize Testing within the Context of Your Organization
YOUR GROWTH MINDSET
• Use your intelligence
• If something seems wrong to you, it probably does to others
• Test Beyond Requirements
• Exploratory Testing
• Experiment with new test approaches• Field testing for mobile devices
• Study your craft
• Learn about new technologies and apply your knowledge
EVANGELIZE TESTING
• APM is for testers too
• Goes beyond the IT ops people
•Do root cause analysis
• Know why the problem exists
• You’re the expert; let’s hear your opinion
UNDERSTAND YOUR ORGANIZATION
• Depending on your organization
• You may be thanked for going the extra mile
• You may be criticized for exceeding your portfolio
• When you feel burned out, discouraged or bored
• Participate in the professional community
SUMMARY
• Make it personal while remaining professional
• Think beyond the requirements
• Does it feel right?
• Experiment
• Work with development early on
• You are all on the same side
• Make recommendations
THOUGHTS?
• Thank you