Download - testing from basics
-
7/27/2019 testing from basics
1/25
Fundamentals of Testing
Speaker: Jerr y Gao Ph.D.
San Jose State Un iversity
email : jerrygao@email .sjsu.edu
URL : http://www.engr.sjsu.edu/gaojerry
-
7/27/2019 testing from basics
2/25
Topic: F undamentals of Testing
- Software Error s
- Software Test Planing
- Software Test Requi rements
- Software Test Design- Review for Software Testing
- Software Test Executi on
- Problem Tracking System
- Software Test Management
-Software Test Engineers Tasks
- Test Automati on and Tools
Jerry Gao Ph.D. 7/20002
Presentation Outl ine
Al l Rights Reserved
-
7/27/2019 testing from basics
3/25
Topic: F undamentals of Testing
What i s a software error?
One common defi ni tion of a software error is a mismatch between the program
and its specif ication.
Defi niti on #1:
A mismatch between the program and its specification is an errorin the program if and only if the specification exists and is correct.
Defi niti on #2:
A software error is present for when the program does not do what
its end user reasonability expects to do. (Myers, 1976)
Defi niti on #3:There can never be an absolute definition for bugs, nor an
absolu te determination of their exi stence. The extent to which a program has
bugs is measured by the extent to which it fail s to be usefu l . Thi s is a
fundamentally human measure. (Besizer, 1984)
Jerry Gao Ph.D. 7/2002
Software Errors
-
7/27/2019 testing from basics
4/25
Topic: F undamentals of Testing
- User inter face errors, such as output errors, incorrect user messages.
- Function error s - Defect hardware
- I ncorrect program version - Testing errors
- Requirements errors - Design errors
- Documentation err ors - Architecture error s
- Module in terf ace error s - Per formance error s
- Er ror handling - Boundary-related error s
- L ogic errors, such as calculation errors
- State-based behavior errors - Communi cation errors
- Program structure err ors, such as control -fl ow error s
Jerry Gao Ph.D. 7/2002
Categories of Software Errors
-
7/27/2019 testing from basics
5/25
Topic: F undamentals of Testing
L ike other activi ties in software engineer ing phases, i t i s impossible to have a
cost-effective software test process without a very good planing,
The major objective of software test planing:
- generate a well-defined software test plan.
What content should be included in a software test plan?
- Testing activi ties and schedule
- Testing tasks and assignments
- Selected test strategy and test models
- Test methods and cr i teri a
- Requir ed test tools and envir onment
- Problem tracking and reporting- Test cost estimation
Other needed items: qual i ty contr ol process and standards
Jerry Gao Ph.D. 7/2002
Software Test Planing
-
7/27/2019 testing from basics
6/25
Jerr y Gao Ph.D. 7/2002
Topic: F undamentals of Testing
Software Test Requi rements
Befor e star ting test design, we must identify our test objectives, focuses, and test i tems.
The major purpose is to help us understand what are the targets of software testing.
Th is step can be done based on:
- Requi rements specif ications
- I nputs fr om developers- F eedback f rom customers
Benef its are:
- I denti fy and rank the major focus of software testing
- Check the requi rements to see if they are corr ect, completed, and testable
- Enhance and update system requi rements to make sur e they are testable
- Support the decision on selecting or def ining test strategy
For example,
- for performance testing, we need clear requi rements on system per formance.
-
7/27/2019 testing from basics
7/25
Jerr y Gao Ph.D. 7/2002
Topic: F undamentals of Testing
Software Test Requi rements
The essenti als of testing r equi rements include:
- Specif ied testing methods
- Required test types am d test coverage cr i teri a
- Selected or requi red test tools
- Testing focuses and test i tems for each type of software testing
An example of performance testing requi rements:
Check the system performance to make sure it meet 99% system
reliability requirements
A typical example for requi red test i tems is:
Test item #I: Test the call waiting feature (REQ #j) during system testing based on the
given requirements specifications.
-
7/27/2019 testing from basics
8/25
Topic: F undamentals of Testing
Jerry Gao Ph.D. 7/2002
Software Test Design
Software test design is an impor tant task for software test engineers.
A good test engineer always know how to come out quali ty test cases and perf orm
eff ective tests to uncover as many as bugs in a very tight schedule.
What do you need to come out an eff ective test set ?
- Choose a good test model and an eff ective testing method
- Apply a well -defi ned test cri ter ia
- Generate a cost-eff ective test set based on the selected test cr i teria
- Write a good test case specif ication document
What i s a good test case?
- I t must have a high probabil i ty to discover a software error
- I t is designed to aim at a specif ic test requi rement- I t is generated by fol lowing an eff ective test method
- I t must be wel l documented and easil y tracked
- I t is easy to be per formed and simple to spot the expected results
- I t avoids the redundancy of test cases
-
7/27/2019 testing from basics
9/25
Topic: F undamentals of Testing
Jerry Gao Ph.D. 7/2002
Software Test Design
What content should be included in a test case?.
Test Case ID: Test I tem:
Wrote By: (tester name) Documented Date:
Test Type: Test Suite#:
Product Name: Release and Version No.:
Test case descr iption:
Operation procedure:
Pre-conditions: Post-conditions:
I nputs data and/or events: Expected output data and/or events:
Requi red test scripts:
-
7/27/2019 testing from basics
10/25
Test execution can be performed:
- using manual approach
- using a systematic approach
- using a semi-automatic approach
Basis activi ties in test execution are:
- Select a test case
- Set up the pre-condi tions for a test case
- Set up test data
- Run a test case foll owing i ts procedure
- Track the test operations and resul ts- Moni tor the post-conditions of a test case & expect the test resul ts
- Ver ify the test resul ts and report the problems if there is any
- Record each test execution
Jerr y Gao Ph.D. 7/2002
Topic: F undamentals of Testing
Software Test Execution
-
7/27/2019 testing from basics
11/25
What do you need to per form test execution?
- a test plan
- test design specif ication with test case specif ications
- a test sui te with documented test cases (optional)
- test suppor ting facil i ty, such as test dr ivers, test stubs, simulators
What are the outcome of an test execution:
- Text executi on r ecord and report
- Problem and bug reports
- Er ror logs and
With automatic test execution tools (or test control tools), we can do:
- automatic test r unner
- record and replay
Jerr y Gao Ph.D. 7/2002
Topic: F undamentals of Testing
Software Test Execution
-
7/27/2019 testing from basics
12/25
Problem Analysis and Report
Jerr y Gao Ph.D. 7/2002
Topic: F undamentals of Testing
When do we issue a problem?
Whenever a bug or problem is found, we need to wr i te down a problem report immediately.
What ar e the content of a problem report?
Problem ID cur rent software name, release no. and version no.
Test type Reported by Reported date Test case ID
Subsystem (or module name) Feature Name (or Subject)
Problem type (REQ, Design, Coding, ) Problem severi ty (Fatal, Major, M inor, ..)
Problem summary and detail ed descri ption
Cause anal ysis How to reproduce? Attachments
-
7/27/2019 testing from basics
13/25
Problem Analysis and Report
Jerr y Gao Ph.D. 7/2002
Topic: F undamentals of Testing
How to track, control , and manage issued problems?
- A systematic solut ion is needed to tr ack and maintain the reported problems in a
repository.
- Defi ne and implement a problem control and analysis process to control problem
tracking, reporting, analysis, fi xing and resoluti ons.
Character istics of a problem report:
- Simple and understandable - Traceable and numbered
- Reproducible - Non- judgmental
Problem analysis:-F inding the most ser ious consequences
-F inding the simplest and most general condi tions
-F inding alternative paths to the same problem
-F inding related problems
-
7/27/2019 testing from basics
14/25
Topic: F undamentals of Testing
Jerry Gao Ph.D. 7/2002
Software Test Review
What i s a review?
- A review is a ver i f ication activity to assur e a corr ect method has been used to
create a software product.
- Parti cipants in a review take ful l responsibili ty for resul ts.
There are two types of r eviews:
- Formal r eviews:
- use a well -defi ned review method (or technique)
- generate formal review resul ts
- I nformal reviews
- use a desk checking approach- conduct an i nformal r eview
- generate information review resul ts
-
7/27/2019 testing from basics
15/25
Topic: F undamentals of Testing
Jerry Gao Ph.D. 7/2002
Software Test Review
Products shoul d be reviewed dur ing software testing:
Test Plan Test Design Specif ication
Test Procedur e Specif ication
Test Report Problem Reports
What do reviews accompli sh?
- Reviews provide the primary mechanism for r el iably evaluati ng progress.
- Reviews train and educate the parti cipants and have a signi fi cant posit ive eff ect on
staff competence.
- Reviews give earl y feedback and prevent more ser ious problems fr om ar ising.
- Reviews bring individual capabili ty to light.
-
7/27/2019 testing from basics
16/25
Topic: F undamentals of Testing
Jerry Gao Ph.D. 7/2002
Test Management
Test management encompass:
- management of a test team
- management of a test pr ocess
- management of test projects
A test managers role:
- Play as a leadership role in :
- plani ng projects - setting up a dir ection -motivating people
- bui ld a team - manage engineers
- Play as a controller in :
- product evaluation - performance evaluati on
- changing to a new dir ection
- Play as a supporter in:
- assist and train engineers - train engineers
- enforce and control test methods, standards, and cr iter ia
- select and develop test tool s
-
7/27/2019 testing from basics
17/25
Topic: F undamentals of Testing
Jerry Gao Ph.D. 7/2002
Test Management
Test management functions:
- Management - Manage test projects
- M anage team members
- M anage test processes
- Motivation - Motivate quali ty work fr om team members
- Simulate for new ideas and creative solu tions
- Methodology - Contr ol of setting up test methodology, process, standards.
- Control of establi shing test cri ter ia
- Mechanization - Control the selection and development of test tools
- M echanism for the conf iguration management of test sui tes
- Contr ol of setting up an integrated test environment
- Measur ement - Measur e test cost, complexi ty and eff orts
- M easure engineer per formance
- Measur e test effectiveness
- Measure product quali ty
-
7/27/2019 testing from basics
18/25
Topic: F undamentals of Testing
Jerry Gao Ph.D. 7/2002
Test Management
Test management encompass:
- management of a test team
- management of a test pr ocess
- management of test projects
A test managers role:
- Play as a leadership role in :
- plani ng projects - setting up a dir ection -motivating people
- bui ld a team - manage engineers
- Play as a controller in :
- product evaluation - performance evaluati on
- changing to a new dir ection
- Play as a supporter in:
- assist and train engineers - train engineers
- enforce and control test methods, standards, and cr iter ia
- select and develop test tool s
-
7/27/2019 testing from basics
19/25
Topic: F undamentals of Testing
Jerry Gao Ph.D. 7/2002
Test Management
Questions for managers who claim to be managing software testing:
Management: Do you plans address testing?
Do you know who is responsible?
Have you publ ished your testing poli cy?
Motivation: Do you provide incentive for people do quali ty work?
Do you encour age people to take advantage of train ing
oppor tuniti es in testing methods?
Methodology: Ar e your testing methods procedurali zed and are people
trained in their use?Ar e you aware of new testing techniques and are you
worki ng to in troduce them?
-
7/27/2019 testing from basics
20/25
Topic: F undamentals of Testing
Jerry Gao Ph.D. 7/2002
Test Management
Questions for managers who claim to be managing software testing:
Mechanization: Do you suf fi cient hardware and equipment to support testing?
Have you provided appropri ate software testing tools and aids?
Do you evaluate automated testing aids on an ongoing basis?
Measurement: Do you track err ors, faul ts, and fai lu res?
Do you know what testing costs?
Do you quanti tatively measur e testing per formance?
-
7/27/2019 testing from basics
21/25
Topic: F undamentals of Testing
Jerry Gao Ph.D. 7/2002
Test Engineers Tasks
What does a test engineer do?
- Ensure that testing is per formed
- Ensure that testing is documented
- Ensure that testing methodology, techniques, standar ds are
establ ished and developed
The basic ski l ls for a qual if ied test engineer :
Controll ed - Organized individual, systematic plani ng
- Good planing on testing
Competent - Technical awareness of testing methods, tools, and cr iteri a
Cri tical - I nner determination to discover problems
Comprehensive - Total understanding of the given system and specif ications
- Pay good attention to the detai ls
Considerate - Abi l i ty to related to others and resolve confl icts
-
7/27/2019 testing from basics
22/25
Topic: F undamentals of Testing
Jerry Gao Ph.D. 7/2002
Basic Test Engineers Tasks
The basic tasks of a test engineer include:
- Prepare testing plans and organi ze testing activiti es
- Design test cases and document them using well -defined test methods
- Develop test pr ocedur es and test data
- Wri te problem repor ts and text executi on records
- Use testing tools and aids
- Review test designs, resul ts and problems
- Test maintenance changes
- Oversee acceptance tests
-
7/27/2019 testing from basics
23/25
Topic: F undamentals of Testing
Jerry Gao Ph.D. 7/2002
Test Automation and Tools
What is software test automation?
Software test automation refers to a process and activities that reduce manual testing
activi ties and efforts in a software development l if ecycle.
Why software test automation?
- Reduce software testing time, cost, and efforts- I ncrease the quali ty of software testing
- Apply well -defi ned test methods thr ough tools
- Reli eve the complicated and redundant work fr om test engineers
What do we need to automate software testing?
- L imited cost and schedule
- Select and intr oduce quali ty test tools
- Develop necessary eff ective and eff icient test tools and facil i ty
- Apply wel l-defined testing methods and coverage
- F orm an integrated test envi ronment supporti ng vari ous software testing activi ties
for products on a production line
-
7/27/2019 testing from basics
24/25
Topic: F undamentals of Testing
Jerry Gao Ph.D. 7/2002
Test Automation and Tools
Basic steps for test automation:
Level #1: Automatic test management
- test case & sui te management, and documentati on
- test scr ipt management
Level #2: Au tomatic test execution- black-box test control and execution
- white-box test control and execution
Level #3: Au tomatic test generation
- black-box test generation
- wh ite-box test generation
Level #4: Automatic test measur ement
- test coverage and metr ics measur ement
- test cost and complexi ty measur ement
-
7/27/2019 testing from basics
25/25
Topic: F undamentals of Testing
Jerry Gao Ph.D. 7/2002
Test Automation and Tools
Classif ication of software test tools:
- GUI record and replay tools
- Program specif ication -based test tools
- Test management and conf iguration management tools
- Test generation tools, such as random test tools
- Program-based test tools (or whi te-box test tools)
- Program test metr ics tools
- Program per formance test tools
- performance monitor ing tools
- performance evaluati on tools
- Program load test tools
- Protocol-based conf irmation tool tools
- Program regression test tools (such as program change analysis tool)