test vs. inspection part 1 tor stålhane. what we will cover part 1 – introduction – inspection...
TRANSCRIPT
![Page 1: Test vs. inspection Part 1 Tor Stålhane. What we will cover Part 1 – Introduction – Inspection processes – Testing processes Part 2 – Tests and inspections](https://reader036.vdocument.in/reader036/viewer/2022062518/56649de75503460f94ae12ff/html5/thumbnails/1.jpg)
Test vs. inspectionPart 1
Tor Stålhane
![Page 2: Test vs. inspection Part 1 Tor Stålhane. What we will cover Part 1 – Introduction – Inspection processes – Testing processes Part 2 – Tests and inspections](https://reader036.vdocument.in/reader036/viewer/2022062518/56649de75503460f94ae12ff/html5/thumbnails/2.jpg)
What we will cover
• Part 1 – Introduction – Inspection processes– Testing processes
• Part 2– Tests and inspections – some data– Inspection as a social process – two experiments
and some conclusions
![Page 3: Test vs. inspection Part 1 Tor Stålhane. What we will cover Part 1 – Introduction – Inspection processes – Testing processes Part 2 – Tests and inspections](https://reader036.vdocument.in/reader036/viewer/2022062518/56649de75503460f94ae12ff/html5/thumbnails/3.jpg)
Introduction
![Page 4: Test vs. inspection Part 1 Tor Stålhane. What we will cover Part 1 – Introduction – Inspection processes – Testing processes Part 2 – Tests and inspections](https://reader036.vdocument.in/reader036/viewer/2022062518/56649de75503460f94ae12ff/html5/thumbnails/4.jpg)
Adam’s data - 1Mean Time to Problem Occurrence – years
Product 1.6 5 16 50 160 500 1600 5000
1 0.7 1.2 2.1 5.0 10.3 17.8 28.8 34.2
2 0.7 1.5 3.2 4.3 9.7 18.2 28.0 34.3
3 0.4 1.4 2.8 6.5 8.7 18.0 28.5 33.7
4 0.1 0.3 2.0 4.4 11.9 18.7 28.5 34.2
5 0.7 1.4 2.9 4.4 9.4 18.4 28.5 34.2
6 0.3 0.8 2.1 5.0 11.5 20.1 28.2 32.0
7 0.6 1.4 2.7 4.5 9.9 18.5 28.5 34.0
8 1.1 1.4 2.7 6.5 11.1 18.4 27.1 31.9
9 0.0 0.5 1.9 5.6 12.8 20.4 27.6 31.2
![Page 5: Test vs. inspection Part 1 Tor Stålhane. What we will cover Part 1 – Introduction – Inspection processes – Testing processes Part 2 – Tests and inspections](https://reader036.vdocument.in/reader036/viewer/2022062518/56649de75503460f94ae12ff/html5/thumbnails/5.jpg)
Adams’ data – 2
The main information that you get from the table on the previous slide is that
• Some defects are important because they will happen quite often.
• Most defects are not important since they will happen seldom.
How can we tell the difference?
![Page 6: Test vs. inspection Part 1 Tor Stålhane. What we will cover Part 1 – Introduction – Inspection processes – Testing processes Part 2 – Tests and inspections](https://reader036.vdocument.in/reader036/viewer/2022062518/56649de75503460f94ae12ff/html5/thumbnails/6.jpg)
Testing alone is not the solution
As can be seen from the next slide, testing is not an acceptable solution alone. It will•Take too long time•Cost too muchWe can generate tests automatically, but would never the less have to use large resources to check the result – the oracle problem
![Page 7: Test vs. inspection Part 1 Tor Stålhane. What we will cover Part 1 – Introduction – Inspection processes – Testing processes Part 2 – Tests and inspections](https://reader036.vdocument.in/reader036/viewer/2022062518/56649de75503460f94ae12ff/html5/thumbnails/7.jpg)
A limit resultThe following relation holds under a rather wide
set of conditions:
The initial number of defects – N0 – must be estimated e.g. based on experience from earlier projects as number of defects per KLOC.
tN
eMTTFt
0ˆ
![Page 8: Test vs. inspection Part 1 Tor Stålhane. What we will cover Part 1 – Introduction – Inspection processes – Testing processes Part 2 – Tests and inspections](https://reader036.vdocument.in/reader036/viewer/2022062518/56649de75503460f94ae12ff/html5/thumbnails/8.jpg)
An example from telecom
![Page 9: Test vs. inspection Part 1 Tor Stålhane. What we will cover Part 1 – Introduction – Inspection processes – Testing processes Part 2 – Tests and inspections](https://reader036.vdocument.in/reader036/viewer/2022062518/56649de75503460f94ae12ff/html5/thumbnails/9.jpg)
Testing and inspection – the V model
![Page 10: Test vs. inspection Part 1 Tor Stålhane. What we will cover Part 1 – Introduction – Inspection processes – Testing processes Part 2 – Tests and inspections](https://reader036.vdocument.in/reader036/viewer/2022062518/56649de75503460f94ae12ff/html5/thumbnails/10.jpg)
Testing and inspection – 1 The important message here is that testing
cannot always be done. In the first, important phases, we have nothing
to execute and will thus always have to do some type of inspection.
This might be considered one of the weaknesses of traditional software engineering over Agile development.
![Page 11: Test vs. inspection Part 1 Tor Stålhane. What we will cover Part 1 – Introduction – Inspection processes – Testing processes Part 2 – Tests and inspections](https://reader036.vdocument.in/reader036/viewer/2022062518/56649de75503460f94ae12ff/html5/thumbnails/11.jpg)
Testing and inspection – 2
In order to understand the main differences between testing and inspection, we should consider Fits’ list.
Based on this, we will give a short discussion of the relative merits of testing and inspection.
![Page 12: Test vs. inspection Part 1 Tor Stålhane. What we will cover Part 1 – Introduction – Inspection processes – Testing processes Part 2 – Tests and inspections](https://reader036.vdocument.in/reader036/viewer/2022062518/56649de75503460f94ae12ff/html5/thumbnails/12.jpg)
Area ofcompetence
Man Machine
Understanding Good at handling variations inwritten material
Bad at handling variations inwritten material
Observe General observations,multifunctional
Specialized, good at observingquantitative data, bad atpattern recognition
Reasoning Inductive, slow, imprecise butgood at error correction
Deductive, fast, precise butbad error correction
Memory Innovative, several accessmechanisms
Copying, formal access
Informationhandling
Single channel, less than 10bits per second
Multi channel, severalMegabits per second
Consistency Unreliable, get tired, dependson learning
Consistent repetition of several
actions
Power Low level, maximum ca. 150watt
High level over long periodsof time
Speed Slow – seconds Fast
![Page 13: Test vs. inspection Part 1 Tor Stålhane. What we will cover Part 1 – Introduction – Inspection processes – Testing processes Part 2 – Tests and inspections](https://reader036.vdocument.in/reader036/viewer/2022062518/56649de75503460f94ae12ff/html5/thumbnails/13.jpg)
Man vs. machine – 1 Good when we need the ability to • Handle variation• Be innovative and inductive• Recognize and handle patterns
Not so good when we need the ability to• Do the same things over and over again in a
consistent manner• Handle large amount of data
![Page 14: Test vs. inspection Part 1 Tor Stålhane. What we will cover Part 1 – Introduction – Inspection processes – Testing processes Part 2 – Tests and inspections](https://reader036.vdocument.in/reader036/viewer/2022062518/56649de75503460f94ae12ff/html5/thumbnails/14.jpg)
Man vs. machine – 2 In order to do the best job possible we need
processes where we let each part• Do what they are best at:– Man is innovative– Machine handles large amounts of data
• Support the other with their specialties.– Machine supports man by making large amounts
of information available– Man support machine by providing it with
innovative input
![Page 15: Test vs. inspection Part 1 Tor Stålhane. What we will cover Part 1 – Introduction – Inspection processes – Testing processes Part 2 – Tests and inspections](https://reader036.vdocument.in/reader036/viewer/2022062518/56649de75503460f94ae12ff/html5/thumbnails/15.jpg)
General considerations - documents
Architecture, system, sub-system and component design plus pseudo code. Here we can only use inspections.
Man will use experience and knowledge to identify possible problems
Machine can support by identifying information – e.g. find all occurrences of a string.
![Page 16: Test vs. inspection Part 1 Tor Stålhane. What we will cover Part 1 – Introduction – Inspection processes – Testing processes Part 2 – Tests and inspections](https://reader036.vdocument.in/reader036/viewer/2022062518/56649de75503460f94ae12ff/html5/thumbnails/16.jpg)
General considerations – code (1)
For executable code, we can use inspection, testing or a combination of both.
The size and complexity – degree of dynamism – of the code will, to a large degree, decide our choice.
Other important factors are the degree of experience with
• The programming language• The algorithms used
![Page 17: Test vs. inspection Part 1 Tor Stålhane. What we will cover Part 1 – Introduction – Inspection processes – Testing processes Part 2 – Tests and inspections](https://reader036.vdocument.in/reader036/viewer/2022062518/56649de75503460f94ae12ff/html5/thumbnails/17.jpg)
General considerations – code (2)
Simple code• Start with inspection – all code• Design and run testsComplex code• Start with inspection – focus on algorithm and
logic• Decide test completeness criteria – we cannot
test everything• Design and run tests
![Page 18: Test vs. inspection Part 1 Tor Stålhane. What we will cover Part 1 – Introduction – Inspection processes – Testing processes Part 2 – Tests and inspections](https://reader036.vdocument.in/reader036/viewer/2022062518/56649de75503460f94ae12ff/html5/thumbnails/18.jpg)
Inspection processes
![Page 19: Test vs. inspection Part 1 Tor Stålhane. What we will cover Part 1 – Introduction – Inspection processes – Testing processes Part 2 – Tests and inspections](https://reader036.vdocument.in/reader036/viewer/2022062518/56649de75503460f94ae12ff/html5/thumbnails/19.jpg)
Inspections – 1 The term “inspection” is often used in a rather
imprecise manner. We will look at three types of inspection:
• Walkthrough• Informal inspection – also called informal
review• Formal inspection – also called formal review
or just inspectionThe first two types are usually project internal
while the last one is used as a final acceptance activity for a document.
![Page 20: Test vs. inspection Part 1 Tor Stålhane. What we will cover Part 1 – Introduction – Inspection processes – Testing processes Part 2 – Tests and inspections](https://reader036.vdocument.in/reader036/viewer/2022062518/56649de75503460f94ae12ff/html5/thumbnails/20.jpg)
Inspections – 2
For all types of inspections:• The quality of the results depends on the
experience and knowledge of the participants. “Garbage in – Garbage out”
• It might be a good idea to involve customer representatives.
![Page 21: Test vs. inspection Part 1 Tor Stålhane. What we will cover Part 1 – Introduction – Inspection processes – Testing processes Part 2 – Tests and inspections](https://reader036.vdocument.in/reader036/viewer/2022062518/56649de75503460f94ae12ff/html5/thumbnails/21.jpg)
The walkthrough process
Walkthrough is a simple process – mostly used for early decisions for an activity. The document owner:
1. Makes a rough sketch of the solution – architecture, algorithm etc.
2. Presents – explain – the sketch to whoever shows up.
3. Registers feedback – improvements.
![Page 22: Test vs. inspection Part 1 Tor Stålhane. What we will cover Part 1 – Introduction – Inspection processes – Testing processes Part 2 – Tests and inspections](https://reader036.vdocument.in/reader036/viewer/2022062518/56649de75503460f94ae12ff/html5/thumbnails/22.jpg)
Walkthrough – pros and consPros:• Easy and inexpensive. Needs no extra
preparation.• Collect ideas at an early stage of
development.Cons:• No commitment from the participants• May collect many loose or irrelevant ideas
![Page 23: Test vs. inspection Part 1 Tor Stålhane. What we will cover Part 1 – Introduction – Inspection processes – Testing processes Part 2 – Tests and inspections](https://reader036.vdocument.in/reader036/viewer/2022062518/56649de75503460f94ae12ff/html5/thumbnails/23.jpg)
The informal inspection process
Individualchecking
Planning
Product document
Changerequests
Rules,checklists,procedures
Loggingmeeting
![Page 24: Test vs. inspection Part 1 Tor Stålhane. What we will cover Part 1 – Introduction – Inspection processes – Testing processes Part 2 – Tests and inspections](https://reader036.vdocument.in/reader036/viewer/2022062518/56649de75503460f94ae12ff/html5/thumbnails/24.jpg)
Informal inspections – pros and consPros:• Is simple and inexpensive to perform.• Can be used at all stages of development• Usually has a good cost / benefit ratio• Needs a minimum of planningCons:• No participant commitment • No process improvement
![Page 25: Test vs. inspection Part 1 Tor Stålhane. What we will cover Part 1 – Introduction – Inspection processes – Testing processes Part 2 – Tests and inspections](https://reader036.vdocument.in/reader036/viewer/2022062518/56649de75503460f94ae12ff/html5/thumbnails/25.jpg)
The formal inspection process
The formal inspection process described below is – with small variations – the most commonly used. The version shown on the following slides stem from T. Gilb and D. Graham.
We recommend this process as the final acceptance process for all important documents
![Page 26: Test vs. inspection Part 1 Tor Stålhane. What we will cover Part 1 – Introduction – Inspection processes – Testing processes Part 2 – Tests and inspections](https://reader036.vdocument.in/reader036/viewer/2022062518/56649de75503460f94ae12ff/html5/thumbnails/26.jpg)
Formal inspection process overview
Walk-through
Kick-off Individualchecking
Edit and follow-up
Planning
Process improvements
Product document
Changerequests
Rules,checklists,procedures
Loggingmeeting
![Page 27: Test vs. inspection Part 1 Tor Stålhane. What we will cover Part 1 – Introduction – Inspection processes – Testing processes Part 2 – Tests and inspections](https://reader036.vdocument.in/reader036/viewer/2022062518/56649de75503460f94ae12ff/html5/thumbnails/27.jpg)
Distribution of resources
Activity Range %Typicalvalue %
Planning 3 – 5 4
Kick-off 4 – 7 6
Individual checking 20 – 30 25
Logging 20 – 30 25
Editing 15 – 30 20
Process brainstorming 15 – 30 16
Leader overhead, follow up, entry,exit
3 – 5 4
![Page 28: Test vs. inspection Part 1 Tor Stålhane. What we will cover Part 1 – Introduction – Inspection processes – Testing processes Part 2 – Tests and inspections](https://reader036.vdocument.in/reader036/viewer/2022062518/56649de75503460f94ae12ff/html5/thumbnails/28.jpg)
Initiating the inspection process
• The inspection process starts with a “request for inspection” from the author to the QA responsible.
• The QA responsible appoints an inspection leader.
• First step is always to check that the document is fit for inspection.
![Page 29: Test vs. inspection Part 1 Tor Stålhane. What we will cover Part 1 – Introduction – Inspection processes – Testing processes Part 2 – Tests and inspections](https://reader036.vdocument.in/reader036/viewer/2022062518/56649de75503460f94ae12ff/html5/thumbnails/29.jpg)
Planning
Important planning points are:• Who should participate in the inspections– Who is interested?– Who have time available for preparation and
meetings?– Who has the necessary knowledge concerning
application, language, tools, methods?
![Page 30: Test vs. inspection Part 1 Tor Stålhane. What we will cover Part 1 – Introduction – Inspection processes – Testing processes Part 2 – Tests and inspections](https://reader036.vdocument.in/reader036/viewer/2022062518/56649de75503460f94ae12ff/html5/thumbnails/30.jpg)
Kick-off
Important activities here are:• Distribution of necessary documents:– Documents that shall be inspected– Requirements– Applicable standards and checklists
• Assignment of roles and jobs• Setting targets for resources, deadlines etc.
![Page 31: Test vs. inspection Part 1 Tor Stålhane. What we will cover Part 1 – Introduction – Inspection processes – Testing processes Part 2 – Tests and inspections](https://reader036.vdocument.in/reader036/viewer/2022062518/56649de75503460f94ae12ff/html5/thumbnails/31.jpg)
Individual checking
This is the main activity of the inspection. Each participant read the document to look for
• Potential errors - inconsistencies with requirements or common application experience
• Lack of adherence to company standards or good workmanship
![Page 32: Test vs. inspection Part 1 Tor Stålhane. What we will cover Part 1 – Introduction – Inspection processes – Testing processes Part 2 – Tests and inspections](https://reader036.vdocument.in/reader036/viewer/2022062518/56649de75503460f94ae12ff/html5/thumbnails/32.jpg)
Logging meeting
The logging meeting has three purposes:• Log issues already discovered by inspection
participants• Discover new issues based on discussions
and new information that arises during the logging meeting.
• Identify possible improvement to the inspection or development process.
![Page 33: Test vs. inspection Part 1 Tor Stålhane. What we will cover Part 1 – Introduction – Inspection processes – Testing processes Part 2 – Tests and inspections](https://reader036.vdocument.in/reader036/viewer/2022062518/56649de75503460f94ae12ff/html5/thumbnails/33.jpg)
Improve the product - 1
The author receives the log from the inspection meeting. All items - issues - in the log are categorised as one of the following:
• Errors in the author’s document.• Errors in someone else’s document.• Misunderstandings in the inspection team.
![Page 34: Test vs. inspection Part 1 Tor Stålhane. What we will cover Part 1 – Introduction – Inspection processes – Testing processes Part 2 – Tests and inspections](https://reader036.vdocument.in/reader036/viewer/2022062518/56649de75503460f94ae12ff/html5/thumbnails/34.jpg)
Improve the product - 2
• Errors in own document:Make appropriate corrections
• Errors in someone else’s documents:Inform the owner of this document.
• Misunderstandings in the inspection team:Improve document to avoid further misunderstandings.
![Page 35: Test vs. inspection Part 1 Tor Stålhane. What we will cover Part 1 – Introduction – Inspection processes – Testing processes Part 2 – Tests and inspections](https://reader036.vdocument.in/reader036/viewer/2022062518/56649de75503460f94ae12ff/html5/thumbnails/35.jpg)
Checking the changesThis is the responsibility of the inspection
leader. He must assure that all issues raised in the log are disposed of in a satisfactory manner:
• The documents that have been inspected• Related documents - including standards
and checklists• Suggested process improvements
![Page 36: Test vs. inspection Part 1 Tor Stålhane. What we will cover Part 1 – Introduction – Inspection processes – Testing processes Part 2 – Tests and inspections](https://reader036.vdocument.in/reader036/viewer/2022062518/56649de75503460f94ae12ff/html5/thumbnails/36.jpg)
Formal inspection – pros and cons
Pros:• Can be used to formally accept documents• Includes process improvement Cons:• Is time consuming and expensive• Needs extensive planning in order to succeed
![Page 37: Test vs. inspection Part 1 Tor Stålhane. What we will cover Part 1 – Introduction – Inspection processes – Testing processes Part 2 – Tests and inspections](https://reader036.vdocument.in/reader036/viewer/2022062518/56649de75503460f94ae12ff/html5/thumbnails/37.jpg)
Testing processes
![Page 38: Test vs. inspection Part 1 Tor Stålhane. What we will cover Part 1 – Introduction – Inspection processes – Testing processes Part 2 – Tests and inspections](https://reader036.vdocument.in/reader036/viewer/2022062518/56649de75503460f94ae12ff/html5/thumbnails/38.jpg)
Testing We will look at three types of testing:• Unit testing – does the code behave as
intended. Usually done by the developer• Function verification testing – also called
systems test. Does the component or system provide the required functionality?
• System verification testing – also called acceptance test. Does the hardware and software work together to give the user the intended functionality?
![Page 39: Test vs. inspection Part 1 Tor Stålhane. What we will cover Part 1 – Introduction – Inspection processes – Testing processes Part 2 – Tests and inspections](https://reader036.vdocument.in/reader036/viewer/2022062518/56649de75503460f94ae12ff/html5/thumbnails/39.jpg)
The unit testing processUnit testing is done by the developer one or
more times during development. It is a rather informal process which mostly run as follows:
1.Implement (part of) a component.2.Define one or more tests to activate the code3.Check the results against expectations and
current understanding of the component
![Page 40: Test vs. inspection Part 1 Tor Stålhane. What we will cover Part 1 – Introduction – Inspection processes – Testing processes Part 2 – Tests and inspections](https://reader036.vdocument.in/reader036/viewer/2022062518/56649de75503460f94ae12ff/html5/thumbnails/40.jpg)
Unit testing – pros and cons
Pros:• Simple way to check that the code works.• Can be used together with coding in an
iterative manner.Cons:• Will only test the developer’s understanding
of the spec.• May need stubs or drivers in order to test
![Page 41: Test vs. inspection Part 1 Tor Stålhane. What we will cover Part 1 – Introduction – Inspection processes – Testing processes Part 2 – Tests and inspections](https://reader036.vdocument.in/reader036/viewer/2022062518/56649de75503460f94ae12ff/html5/thumbnails/41.jpg)
The system test processA systems test has the following steps:1. Based on the requirements, identify– Test for each requirement, including error handling– Initial state, expected result and final state
2. Identify dependencies between tests3. Identify acceptance criteria for test suite 4. Run tests and check results against – Acceptance criteria for each test– Acceptance criteria for the test suite
![Page 42: Test vs. inspection Part 1 Tor Stålhane. What we will cover Part 1 – Introduction – Inspection processes – Testing processes Part 2 – Tests and inspections](https://reader036.vdocument.in/reader036/viewer/2022062518/56649de75503460f94ae12ff/html5/thumbnails/42.jpg)
Systems test – pros and cons
Pros:• Tests system’s behavior against customer
requirements.Cons:• It is a black box test. If we find an error, the
systems test must be followed by extensive debugging
![Page 43: Test vs. inspection Part 1 Tor Stålhane. What we will cover Part 1 – Introduction – Inspection processes – Testing processes Part 2 – Tests and inspections](https://reader036.vdocument.in/reader036/viewer/2022062518/56649de75503460f94ae12ff/html5/thumbnails/43.jpg)
The acceptance test process
The acceptance test usually has three activities – all involving the customer or his representatives:
• Rerun the systems test at the customer’s site.• Use the system to solve a set of real-world
tasks.• Try to break the system – by stressing it or by
feeding it large amounts of illegal input
![Page 44: Test vs. inspection Part 1 Tor Stålhane. What we will cover Part 1 – Introduction – Inspection processes – Testing processes Part 2 – Tests and inspections](https://reader036.vdocument.in/reader036/viewer/2022062518/56649de75503460f94ae12ff/html5/thumbnails/44.jpg)
Acceptance test – pros and cons
Pros:• Creates confidence that the system will be
useful for the customer• Shows the system’s ability to operate in the
customer’s environmentCons:• Might force the system to handle input that it
was not designed for, thus creating an unfavorable impression.