successful investing in software testing! - rbcs, inc · pdf filesuccessful investing in...
TRANSCRIPT
Rex BlackRBCS, Inc.31520 Beck RoadBulverde, TX 78163 USA
Phone: +1 (830) 438-4830HTTP: //www.rexblackconsulting.com
E-mail: [email protected]
Successful Investing in Software Testing!
Black bear photo © by and used with permission of of Wild Things Photography, [email protected]
Champion bull, Deets, photo © by and used with permission of of The Double D Ranch, [email protected]
Rex Black Consulting Services, Inc.Technical and Project Management Expertise for Quality
July 24, 2002 Copyright (c) 2000-2002 Rex Black Page 2
Advice for the Prudent Test ProfessionalThink of testing as an investment
The project management team foregoes expenditures elsewhere……to fund your testing effort
When you consider testing as an investment, the following questions arise
What kinds of returns can I get on my testing investment?How do I pick the right tests for my portfolio?What kinds of (quality) risks should I worry about?What kinds of techniques can I apply to build ahigh-return-on-investment test system?When should I invest in testing, who does the testing, and how else does testing affect my development process?
Let’s look at some answers…
Rex Black Consulting Services, Inc.Technical and Project Management Expertise for Quality
July 24, 2002 Copyright (c) 2000-2002 Rex Black Page 3
Cost of Quality: Your Initial CapitalWhat does quality cost?Ë Cquality = Cconformance + Cnonconformance
Conformance costs include testing (finding bugs, a.k.a. defects) and quality assurance (preventing bugs)Nonconformance costs include fixing bugs, retesting, dealing with angry customers, damage to company image, lost business, etc.
Can investing in testing reduce the cost of quality?Let’s look at a hypothetical case study
Conformance costs: manual, automated testing only (no QA)Nonconformance costs: fixing bugs only (no intangibles)Assume 1,000 must-fix bugs in each (quarterly) releaseAssume that a developer-found bug costs $10 to fix, a tester-found bug costs $100 to fix, and a customer-found bug costs $1,000 to fix
Rex Black Consulting Services, Inc.Technical and Project Management Expertise for Quality
July 24, 2002 Copyright (c) 2000-2002 Rex Black Page 4
No Formal Testing1,000 bugs in productDevelopers find 250Customers find 750 Quality costs over 3/4 million dollars--and customers are angry!What to do?Let’s introduce formal manual testing with anindependenttest team…
Rex Black Consulting Services, Inc.Technical and Project Management Expertise for Quality
July 24, 2002 Copyright (c) 2000-2002 Rex Black Page 5
Manual TestingInvest in testing teamDevelopers find 250 bugsTesters find 350 bugsCustomers find about 50% fewer bugsCosts are down...Customers are happier…Can we find more bugs and reduce costs further?Suppose we can find 40% more bugs with test automation…
Rex Black Consulting Services, Inc.Technical and Project Management Expertise for Quality
July 24, 2002 Copyright (c) 2000-2002 Rex Black Page 6
Manual and Automated TestingInvest $150,000 in tools
Amortized over twelve quarterly releases
Complement manual testing with automationCustomers find 66% fewer bugsQuality costs halvedTesting ROI445%But such returns require smartinvestment...
Rex Black Consulting Services, Inc.Technical and Project Management Expertise for Quality
July 24, 2002 Copyright (c) 2000-2002 Rex Black Page 7
Smart Test InvestmentsPicking the wrong stocks can result in a total loss of your investment--or worseYou won’t get big returns if you pick the wrong tests, tooTest investment returns can be zero--or even negative
Don’t waste money testing actions that customers will never perform, verifying configurations customers don’t use, and fixing bugs customers will never seeTo add insult to injury, such misguided test efforts give management a false sense of security
The smart test investments are in the tests that find bugs your customer would want fixed
Understand customer usage profiles to……test the system the way the customers use it
Rex Black Consulting Services, Inc.Technical and Project Management Expertise for Quality
July 24, 2002 Copyright (c) 2000-2002 Rex Black Page 8
Testing and Usage ProfilesHigh fidelity test systems mimic customer usage
Rex Black Consulting Services, Inc.Technical and Project Management Expertise for Quality
July 24, 2002 Copyright (c) 2000-2002 Rex Black Page 9
Testing and Usage ProfilesLow fidelity test systems miss key customer scenarios
Rex Black Consulting Services, Inc.Technical and Project Management Expertise for Quality
July 24, 2002 Copyright (c) 2000-2002 Rex Black Page 10
Quality Risks and Your Testing Investment
Functional: missing or broken features or functionsUse cases: features okay alone, but important workflows brokenRobustness: common errors not handled properly Performance: too slow at key points or end-to-end
Localization: errors with dates, language, money, and cultureUsability: the software works, but what a pain!Volume/capacity: can’t stand large datasets, transaction flowsReliability: system crashing, hanging, or behaving strangely
Quality risks go beyond broken functionality
Quality: fitness for use; i.e., presence of customer-satisfying behaviors and absence of customer-dissatisfying behaviorsQuality risks: the potential for dissatisfying behaviorsQuality risks fall into many categories, such as…
Rex Black Consulting Services, Inc.Technical and Project Management Expertise for Quality
July 24, 2002 Copyright (c) 2000-2002 Rex Black Page 11
Analyzing Quality Risks
Stakeholders use RPN to guide appropriate depth and breadth for testing
Set priority for testing each subcharacteristic with key stakeholders
Set priority for testing each quality risk with key stakeholders
Key stakeholders list possible failure modes, predict their effects on system, user, society, etc., assign severity, priority, and likelihood, then calculate risk priority number (RPN)
Functionality, Reliability, Usability, Efficiency, Maintainability, Performance (FRUEMP), then decompose into key subcharacteristics for your system
Functionality, states and transactions, capacity and volume, data quality, error handling and recovery, performance, standards and localization, usability, etc.
Start with categories, char-acteristics, or subsystems
Start with six main quality characteristics
Start with the classic quality risk categories
Failure Mode and Effect AnalysisISO 9126Informal
Target your testing investment depending on the priority of each risk
Rex Black Consulting Services, Inc.Technical and Project Management Expertise for Quality
July 24, 2002 Copyright (c) 2000-2002 Rex Black Page 12
Essential Techniques
Typically testersTypically developersAll stakeholdersWho
Profilers, coverage analyzers, harnesses, debuggers, data generators, test case generators
Usually unit, component, integration
Identify bugs in functions or interfaces
Test how system works
Structural
Tools
Phase
Why
What Test what system doesTest without running
Intrusive and non-intrusive GUI tools, load generators, performance tools, data generators
Simulators, code analyzers, spell- and grammar checkers, graphic/diagramming, spreadsheets
Usually integration, system, and acceptance
During specification and development
Identify bugs in system results and behavior
Identify bugs before they’re built
BehavioralStatic
Tools, data, and cases can be shared: encourage cross-pollination of techniques
Rex Black Consulting Services, Inc.Technical and Project Management Expertise for Quality
July 24, 2002 Copyright (c) 2000-2002 Rex Black Page 13
Automated or Manual?Tests well-suited for manual testing
Operations and maintenanceConfiguration and compatibilityError handling and recoveryLocalizationUsability Installation and setupDocumentation and help
Manual, automated, or combined
FunctionalUse cases (user scenarios)User interfaceDate and time handling
Tests well-suited for automated testing
Regression and confirmationMonkey (or random)Load, volume, and capacityPerformance and reliabilityStandards complianceStructural, especially API-based unit, component, integration Static complexity and code analysis
Inappropriate manual testing misleads people about test coverageInappropriate automation usually failsSome tests mix techniques
Rex Black Consulting Services, Inc.Technical and Project Management Expertise for Quality
July 24, 2002 Copyright (c) 2000-2002 Rex Black Page 14
Pervasive Testing
Concurrent: testing starts with requirements and happens throughout the lifecycle
Smart test investments take months to mature
Successful testing is not a few people in a dark lab at the end of the project...
…winning test strategies pervade software efforts!
Cross-functional: many people play an active role in testing
Testers aren’t the only ones who design, write, and run tests
Collaborative: testing has dependencies throughout the project team
Teams must deliver what’s committed on time
Rex Black Consulting Services, Inc.Technical and Project Management Expertise for Quality
July 24, 2002 Copyright (c) 2000-2002 Rex Black Page 15
Pervasive Testing: TimeBegin testing activities on the first day of the projectRequirements, specifications, and sometimes even source code receive independent evaluation, are inputs to test developmentTest case, tool, and data development being as soon as possiblePhased test execution, moving from structural to behavioral
Rex Black Consulting Services, Inc.Technical and Project Management Expertise for Quality
July 24, 2002 Copyright (c) 2000-2002 Rex Black Page 16
Pervasive Testing: TasksFor optimal return on the testing investment, the whole development team participates in test tasks
Salespeople and marketers define product requirementsCustomer support and operations provide usage profilesDevelopers define specs and perform structural (white-box) testsTesters implement test tools and perform behavioral (black-box) tests
Promote re-usable tool development Across teams: Between development and testAcross phases: From unit to acceptance testing
Integrate testing of hardware and software componentsUse simulators if necessary for electronic and mechanical elementsHardware affects the software’s behavior, and vice versa
Include test results in the project management processUse test progress and defect metrics in the management dashboard
Rex Black Consulting Services, Inc.Technical and Project Management Expertise for Quality
July 24, 2002 Copyright (c) 2000-2002 Rex Black Page 17
Pervasive Testing: TeamworkTesting, being an evaluation and assessment process, receives its test objects and systems from outside groups
Development: writes the software to be testedManagement: provides resources and budgetsRelease/configuration management: creates the software installation mediaOperations/system administration: installs and supports the testnetwork
These teams must deliver on schedule and as promised to the test team
Timely, correct delivery of items will happen when testing is seen as an important, value-added activity International Teamwork: UN Security Council
Rex Black Consulting Services, Inc.Technical and Project Management Expertise for Quality
July 24, 2002 Copyright (c) 2000-2002 Rex Black Page 18
Payoff SummaryInvesting in testing can significantly reduce quality costs
Base your testing budget on a solid business caseTesting must be focused on usage profiles to catch customer-critical bugs
Build high-fidelity test systems that mimic customer usageUsage profiles involve a number of quality risks
Understand and test those quality risks critical to your customersCertain test techniques work best for certain quality risks
Create behavioral and structural, manual and automated tests
Testing tasks and support must penetrate the entire lifecycle for optimal effectiveness
Test early, involve everyone, and facilitate testingMake testing pervasive
Be bullish…about investing in software testing!
Champion bull, Friendly, photo © by and used with permission of Double D Ranch
Rex Black Consulting Services, Inc.Technical and Project Management Expertise for Quality
July 24, 2002 Copyright (c) 2000-2002 Rex Black Page 19
Further ResourcesB. Beizer, Software Testing Techniques, 2nd Ed.
Van Nostrand Reinhold, 1990.B. Beizer, Black Box Testing. Wiley, 1995. R. Black, Critical Testing Processes. Addison-
Wesley, 2002.R. Black, “Investing in Testing,”
www.stickyminds.com R. Black, Managing the Testing Process, Second
Edition. Wiley, 2002.R. Black and G. Kubaczkowski, “Mission Made
Possible,” STQE, Volume 4, Issue 4.J. Bullock, “Calculating the Value of Testing,”
www.stickyminds.com.J. Campanella, ed., Principles of Quality Costs.
ASQ Quality Press, 1999.P. Crosby, Quality is Free. McGraw-Hill, 1979. M. Fewster and D. Graham, Software Test
Automation. Addison-Wesley, 1999.D. Freedman and G. Weinberg, Handbook of
Walkthroughs, Inspections, and Technical Reviews. Dorset House, 1990.
E. Hendrickson, “More Testing, Worse Quality,”www.qualitytree.com.
International Standards Organization, ISO/IEC 9126. 1991.
B. Hetzel, The Complete Guide to Software Testing. Wiley, 1988.
C. Jones, Estimating Software Costs. McGraw-Hill, 1995.
J. Juran, Juran on Planning for Quality. The Free Press, 1988.
S. Kan, Metrics and Models in Software Quality. Addison-Wesley, 1995.
C. Kaner, et al., Lessons Learned in Software Testing. Wiley, 2001.
C. Kaner, et al., Testing Computer Software, 2nd
Ed. Wiley, 1999.P. Neumann, Computer-Related Risks. Addison-
Wesley, 1995.D. Stamatis, Failure Mode and Effect Analysis.
ASQC Quality Press, 1995.K. Wiegers, Software Requirements. Microsoft
Press, 1999.