software testing from art to engineering: a …...what is the difference? •art: – depends on...

Software Testing From Art to Engineering:

A viewpoint and an Interactive Discussion

Dr. Magdy HannaInternational Institute for Software Testing

[email protected]

http://www.iist.org

© 2005 International Institute for Software Testing Art-to-Eng-1

What is the Difference?

• Art:– Depends on individual skills– No precise method or technique– Results are not predictable– Results depend on the mood of the artist– Results are subjective– No guarantee of “good” or “bad” results– Not even a definition of “good” or “bad” results– Acceptance of results depends on individual taste and mood


What is the Difference?

• Engineering:– Methods and techniques– Well defined creation process– Measures to control quality at each step– Results are predictable (in almost all cases)– Results are verifiable


What Do Engineers Do?

• They define the problem• They validate their definition• They look for alternative solutions• They “blueprint” the solution• They validate their blueprint• They implement (execute) their blue print• They always have predictable results


Are you an Artist or an Engineer?


Why Should We behave as Engineers?

• Software systems have become very complex• Software Systems control so much of our lives• The cost of poor quality systems is too high• We need to change the image of the test profession


How Can we Do it?


Testing has been an engineering discipline from the beginning.

The Art of Software Testing, Glen Myers, 1979!!!



Define the Problem

Look for alternatives

Blueprint solution

Validate blueprint

Validate the definition

Implement blueprint

Engineering

Define Requirements

Decide on a modeling technique

Develop a model

Validate the model

Validate Requirements

Develop test Scenarios/test cases

Testing

Data Model

Process Model

ObjectModel

StateModel

DecisionTables

UserInterface

Model

Decision Trees

UseCases

Requirement Document in

Natural Language


Facts about Models

• As different blueprints show different aspects of the engineering design, different models show different aspects of the requirements.

• Models are not one size fits all.• Like blueprints, models are an excellent way to communicate

discuss, and negotiate requirements.• Models provide a “scientific” means of identifying test scenarios

and test cases.


Requirement

Context: Checking into a flight through airport self service terminals

If customer is a Gold or Silver elite member they could upgrade to first class if available unless they were not placed on the waiting list in which case they are asked to see a ticket agent. If customer is a platinum elite member they could also upgrade a companion.


Requirement

Context: Checking into a flight through airport self service terminals

If customer is a Gold or Silver elite member they could upgrade to first class if available unless they were not placed in the waiting list in which case they are asked to see a ticket agent. If customer is a Platinum elite member they could also upgrade a companion.


Develop The Model: Decision table

Customer is a Silver or Gold Elite

Customer is a Platinum Elite

First Class is available

Customer is on waiting list

Customer upgraded to first class

Companion upgraded to first class

Customer is asked to see ticket agent

Causes

Effects


More causes and more effects may come up during model validation


1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16

Customer is a Silver or Gold Elite Y N Y N N N Y Y Y N N N N Y Y Y

Customer is a Platinum Elite Y N N Y N N Y N N Y Y N Y N Y Y

First Class is available Y N N N Y N N Y N Y N Y Y Y N Y

Customer is on waiting list Y N N N N Y N N Y N Y Y Y Y Y N

Customer upgraded to first class N N N N N N Y? N N Y Y

Companion upgraded to first class N N N N N N Y? N N Y N

Boarding pass is printed Y Y Y Y Y Y Y Y Y Y Y

Customer is asked to see ticket agent N N N N Y N Y? Y N N N

Develop Test Scenarios and Test Cases

• Every column in the decision table represents a scenario that MUST be tested

• Start with the maximum possible and eliminate those that can not be tested

• Write test cases for each scenario by determining data and expected results


Requirement

• Customers may cancel reservations any time without cancellation penalty as long the reservation has not been ticketed.

• Once reservation is ticketed, cancellation penalties apply.

• Reservations are ticketed at midnight of the next day after they have been paid.

• Cancelled reservation may be re-instated if cancellation was done through reservation agents not at the web site.

• Only ticketed reservation may have seat assigned.


The Model: State Model


Make reservation

Pay reservation

New reservation

paid reservation

cancel reservation

Cancelledreservation

Cancel reservationOn-line

Cancel reservationby agent

Inactive reservation

Ticketed reservation

Mid-night of the next day

Active reservation

Pay reservation

Re-instatereservation

Re-instatereservation

Cancel reservationOn-line / Assess Penalty

Cancel reservationby agent / Assess Penalty

Assign seat

ERRORAssign seat

ERROR

Assign seat

ERROR

Assign seat

ERROR

ERROR


• Develop test cases to exercise every possible path on the diagram

• Make sure to perform negative testing on every state by performing illegal operations state.

• Testing is 100% complete when every transition line is tested


Requirement• If a ticket has to be re-issued, the new ticket shall have

a new ticket number. The old ticket must then be marked void at that given point of time, any reservation has only one active ticket.

How many tests would you write for this requirement?


The Model: Data Model

Passenger Reservation

Active Ticket

Voided Ticket



• Define 4 test cases for each relationship!!


Conclusion

• We can not afford to do ad hoc testing anymore• Tests must be designed upfront• Designing tests must use precise techniques• Mastering those techniques by every member of the test team is

a must• Test design must drive development effort (TDD)• Bottom line is :

Testing is an Engineering Discipline


Thank youMagdy Hanna, Ph.D.

International Institute for Software Testing

http://www.iist.org


software testing from art to engineering: a …...what is the difference? •art: – depends on...

Documents