®

IBM Software Group

© 2004 IBM Corporation

The Testing Profession

Geoffrey BessinMarket Manager, Software QualityIBM Rational Software

IBM Software Group

2

Obligatory Dilbert

IBM Software Group

3

Companies want YOU!

IBM Software Group

4

Agenda

The Value of Testers to the Organization

IBM Software Group

5

The bad old days….Part One

Time to Delivery

Cost

Quality

Feature Set

Fundamental Project Determinants

Feature Set locked early

Cost spent early

Time to MarketQuality

CHOOSE ONE

IBM Software Group

6

The bad old days….Part Two

Software engineering is the only engineering discipline for which

defects are accepted

IBM Software Group

7

The bad old days….Part Three

“Iterative Development”

“Unit Testing”

“Process driven”

Should be ready

Ok, try now

Here, we’re serious this

timeUh, you’ve

got two days

IBM Software Group

8

The bad old days….Part Four

No formal Tester training Not treated as a profession

Self-fulfilling prophesy – novice testers, underappreciated value

Developers weren’t being trained to test either….

IBM Software Group

9

So what changed?

Gartner estimates that, on average, a failed application costs an enterprise approximately $100,000 per hour.

And, with the National Institute of Standards and Technology ("The Economic Impact of Inadequate Infrastructure for

Software Testing," May 2002) reporting an estimated $22.2 billion to $59.5 billion lost per year as a result of poor-

quality software, catastrophic failures are here to stay.

Approximately 25 percent of all applications deployed on

distributed systems are effectively tested using modern tools and

techniques. Through the second half of 2004, the number of tested deployed applications will remain close to 25

percent.

IBM Software Group

10

Agenda

The Value of Testers to the Organization

Testing in Various Development Processes

IBM Software Group

11

Legacy Development Processes

Code & fix Purely an engineering effort

No testers at all

Fully dependent on skills of the programmers

IBM Software Group

12

Modern Development Processes….

Waterfall First proposed in 1970 by W. W. Royce Development proceeds linearly through

phases of requirements analysis, design, implementation, testing (validation), integration and maintenance

Risk is considered mitigated via problem analysis

Testing is typically direct derivative of requirements

Considered discredited for use as a Real World process However, in practice, the process rarely

proceeds in a purely linear fashion

IBM Software Group

13

Modern Development Processes

V model Portrays several distinct testing levels /

illustrates how each level addresses a different stage of the lifecycle

Testing process also involves identifying what to test (test conditions) and how they'll be tested (designing test cases), building the tests, executing them and evaluating the results, checking completion criteria and reporting progress.

Says nothing about how or how much requirements and designs should be documented

Doesn’t define when tests of any level should be designed or built.

IBM Software Group

14

Modern Development Processes

Spiral/Iterative Defined by Barry Boehm Each cycle considers critical

stakeholder objectives and constraints, product and process alternatives, risk identification and resolution, stakeholder review, and commitment to proceed.

Level of effort, degree of detail within each cycle driven by risk considerations

Each phase starts with a design goal and ends with the client reviewing the progress thus far.

Using analysis and measurement as drivers of the enhancement process is one major difference between iterative enhancement and the current Agile Methods.

IBM Software Group

15

Modern Development Processes

Agile Distillation of Scrum/Feature Driven/XP/Crystal

family/Adaptive Software Development/DSDM

Considered “lightweight” process family

Have short iterative cycles, feature-based planning, constant feedback, change tolerance, team proximity, customer intimacy, and a focus on the overall team ecology

Requires

Uncertain or volatile requirements

Responsible and motivated developers

Customer who understands and will get involved.

Minimizes risk by ensuring that software engineers focus on smaller units of work.

Advocates developer testing, scenario-based testing, user testing http://agilemanifesto.org/

IBM Software Group

16

Standards, Not Processes

Capability Maturity Model (CMM) - http://www.sei.cmu.edu/cmm/ model for judging the maturity of the software processes of an organization

No rules for creating such an organization though Five levels of maturity (Initial, Repeatable, Defined, Managed, Optimizing) Recently replaced by CMMI (CMM Integrated)

ISO 9000 - http://www.iso.ch/iso/en/iso9000-14000/index.html Family of standards primarily concerned with "quality management“ Focused on what the organization does to fulfil:

the customer's quality requirements, and applicable regulatory requirements, while aiming to enhance customer satisfaction, and achieve continual improvement of its performance in pursuit of these

objectives

Six Sigma - http://www.isixsigma.com/ methodology that uses data and statistical analysis to measure and improve

a company's operational performance by identifying and eliminating "defects" in manufacturing and service-related processes

commonly defined as 3.4 defects per million opportunities

ITIL - http://www.itil.co.uk/ best practice for IT Service Management - series of books giving guidance on

the provision of quality IT services, and on the accommodation and environmental facilities needed to support IT

IBM Software Group

17

Clarification….

IBM Software Group

18

Agenda

The Value of Testers to the Organization

Testing in Various Development Processes

Testing Approaches

IBM Software Group

19

Various Black Box Testing Approaches

Function testing

Domain testing

Specification-based testing

Risk-based testing

Stress testing

Regression testing

User testing

Scenario testing

State-model based testing

High volume automated testing

Exploratory testing

All tests involve choices and tradeoffs in at leastthese five areas:

Coverage: What you’re testingRisks: Problems you are trying to findEvaluation: How you know a test passed or failedActivities: How you testTesters: Who is testing

How you think about these issues dictates what combinations of test techniques you will use.

Caner, Bach, Pettichord“Lessons Learned in Software Testing”

IBM Software Group

20

Agenda

The Value of Testers to the Organization

Testing in Various Development Processes

Testing Approaches

Being a Good Tester

IBM Software Group

21

A Good Test Engineer

Has a 'test to break' attitude, an ability to take the point of view of the customer, a strong desire for quality, and an attention to detail

Tact and diplomacy are useful in maintaining a cooperative relationship with developers, and an ability to communicate with both technical (developers) and non-technical (customers, management) people is useful

Previous software development experience can be helpful as it provides a deeper understanding of the software development process, gives the tester an appreciation for the developers' point of view, and reduce the learning curve in automated test tool programming

Judgment skills are needed to assess high-risk areas of an application on which to focus testing efforts when time is limited.

http://www.softwareqatest.com/qatfaq2.html

“Quality is never an accident; it is always the result of high intention, sincere effort, intelligent direction and skillful execution; it represents the wise choice of many alternatives."

— Willa A. Foster

IBM Software Group

22

Succeeding as a Tester and as a Test Organization Facilitate communication across the team

Raise team consciousness about quality, best practices

Possible because they are not insulated from the customer context

Clarify requirements through your understanding of the customer From customer point of view From user point of view From programmer’s point of view Mismatches between these views Identify assumptions

Understand team characteristics to determine suitability of approaches to testing Skills Team size Management Schedule Severity of failure Testability needs

“There is nothing as mysterious as something clearly seen.”

Robert Frost

IBM Software Group

23

Succeeding as a Tester, as a Test Organization

Plan, plan, plan Refactoring tests as you go along without a goal is just bad

refactoring Where is the finish line? Test design finds errors (Boris Beizer, Software Testing Techniques

(Thomson Computer Press, 1990)

Collaborate to help design for testability We can learn from hardware…. Simplifies automation efforts for regression testing

Know the dimensions of test case quality Information

Reduction of uncertainty. How much do you expect to learn from this test?

Power If two tests have the potential to expose the same type of

error, one test is more powerful if it is more likely to expose the error.

Credibility Failure of the test should motivate someone to fix it, not to

dismiss it. Feasibility

How hard is it to set up the test, run the test, and evaluate the result?

“In preparing for battle I have always found that plans are useless, but planning is indispensable.”

Dwight D. Eisenhower

IBM Software Group

24

Agenda

The Value of Testers to the Organization

Testing in Various Development Processes

Testing Approaches

Being a Good Tester

How IBM Rational Can Help

IBM Software Group

25

Quality Drivers in the IBM SWG Product Line

Rational PurifyPlus

Rational Test RealTime

Rational Rose XDE Visual Trace

WSAD tracing

Rational RobotRational XDE

Tester

Rational Performance

Tester

Tivoli TMTP

CodeCode TestTest DebugDebugDefineDefine Functional TestFunctional Test Load TestLoad Test MonitorMonitorDeployDeploy

Developer Tester OperationsAnalyst Architect

DesignDesign

RUP – ClearCase (CM) – ClearQuest (DCT) – RequisitePro (RM) – TestManager (TM) – ProjectConsole

Rational Rose XDE

WebSphere Business Integrator

IBM Software Group

26

CodeCode TestTest DebugDebugDefineDefine Functional TestFunctional Test Load TestLoad Test MonitorMonitorDeployDeploy

Developer Tester OperationsAnalyst Architect

DesignDesign

Quality-enabling activities Quality-enabling activities must start must start herehere and involve and involve

the entire teamthe entire team

IT Operations must be IT Operations must be tiedtied into the lifecycle into the lifecycle

Integrated, Optimized Quality Solutions

Data and metrics must be Data and metrics must be open, accessible, traceableopen, accessible, traceable

Continuously Ensure QualityUniting innovative solutions and best practices to prevent, detect, diagnose and remove defects all across the iterative software application development and deployment lifecycle

IBM Software Group

27

IBM Rational - Defining and Implementing Standards

Extends UML 2.0 to accommodate static and dynamic test specification

Enables sharing of profiling and testing information between tools, vendors

Establishes same test abstraction benefits as UML provides to developers

Consortium lead

Open source, integrated test, trace and monitoring infrastructure

Promotes testing across lifecycle by opening door for innovation

Avoids lock-in forced by vendors Already shipping in WSAD

Laying groundwork for process and data integration

Founding member and primary contributor

1st implementation!

IBM Software Group

28

The IBM Software Development Platform

Analyst Architect Developer Tester

Visually construct, transform,integrate

and generate code

Design, create,

and execute tests

Model, simulate, assemble,

and monitor business

processes

Modelapplications

and data

IndustryStandards

Training & Mentoring

Microsoft

IBM and3rd Party Servers

DB2LotusRationalTivoliWebSphere

Pervasiveand

Embedded

Partners

Follow a common process Track project status Manage requirements

Follow a common process Track project status Manage requirements

Manage changeand assets

Manage quality

Manage changeand assets

Manage qualityProject Manager

Deployer

Provision, configure, tune and

troubleshoot applications

Develop Iteratively

Focus on Architecture

Manage Changeand Assets

Continuously EnsureQuality

IBM Software Group

29

Agenda

The Value of Testers to the Organization

Testing in Various Development Processes

Testing Approaches

Being a Good Tester

How IBM Rational Can Help

Resources

IBM Software Group

30

Reading Suggestions

Better Software Magazine - http://www.stickyminds.com/BetterSoftware/magazine.asp

Software Test and Performance - http://www.stpmag.com/

ASQ Software Quality Professional - http://www.softwarecertifications.com/

Journal of Software Testing Professionals – http://www.testinginstitute.com/journal.php

Quality Techniques Newsletter - http://www.soft.com/News/QTN-Online/

Methods and Tools - http://www.martinig.ch/mt/index.html

Software Development Magazine - http://www.sdmagazine.com/

IBM Software Group

31

Organizations

SEI - http://www.sei.cmu.edu/

IEEE - http://shop.ieee.org/store/default.asp?tabtype=stand

American Society for Quality - http://www.asq-software.org/ Certification - http://www.asq.org/cert/types/csqe/index.html

Quality Assurance Institute - http://www.qaiusa.com/ Certification - http://www.softwarecertifications.com/

International Institute for Software Testing - http://www.testinginstitute.com/

IBM Software Group

32

IBM Software Group

33