tutorial: going green to make it better: a tool kit for ... · tutorial: going green to make it...

Dolphin Computer Access Going Green Tutorial February 2011

(c) 2011 Dolphin Computer Access of 68

Tutorial: Going green to make it better:

a tool kit for defining and delivering quality

Isabel Evans FBCS CITP Quality and Design Manager

Dolphin Computer Access

[email protected] [email protected]



CONTENTS 1. Quality viewpoints .......................................................................................................................... 4

Exercise: what is Quality for your stakeholders? ................................................................................ 7

2. Weaver Triangle .............................................................................................................................. 8

Exercise: Setting missions and goals with a Weaver Triangle........................................................... 10

3. ETVX .............................................................................................................................................. 11

Exercise: ETVX ................................................................................................................................... 15

4. Setting Acceptability Levels .......................................................................................................... 19

Exercise: prioritising quality in use ................................................................................................... 21

Exercise: setting acceptance criteria................................................................................................. 32

5. Setting test coverage against risk ................................................................................................. 34

Exercise: coverage and risk ............................................................................................................... 36

6. Documenting and using a quality plan .......................................................................................... 37

Exercise: quality plan ........................................................................................................................ 39

7. Document release process ............................................................................................................ 40

Exercise: Identify review types for typical documents in your project ............................................ 43

8. Root Cause Analysis ...................................................................................................................... 45

Exercise: root cause analysis ............................................................................................................. 46

Personal Action Plan ......................................................................................................................... 48

9. Appendix – templates ................................................................................................................... 49

Template 1: Quality viewpoint map ................................................................................................. 49

Template 2: Weaver Triangle ............................................................................................................ 50

Template 3: ETVX .............................................................................................................................. 51

Template 4: RACI ............................................................................................................................... 52

Template 5: Quality in Use Risks and Priority ................................................................................... 53

Template 6: Context of use ............................................................................................................... 54

Template 7: User story and SMART acceptance criteria .................................................................. 60

Template 8 Attribute tables and risks ............................................................................................... 61

Template 9: Methods and techniques menu .................................................................................... 62

Template 10: Quality plan ................................................................................................................. 63

Template 11: Quality report – progress ............................................................................................ 64

Template 12: Example review checklist ........................................................................................... 65

Template 13: Root cause analysis ..................................................................................................... 66



Slide 2

Objectives

• Practical "how to" day

– Defining project goals, quality levels,

acceptance criteria and risks; setting up a

spreadsheet quality plan, using the quality

plan to track progress to delivery.

• Templates and spreadsheets to take away

and tailor to your own projects

Agenda

• Quality viewpoints

• Weaver triangle – goals

• On-one-page processes - ETVX

• Setting acceptability levels

• Setting test coverage against risk

• Documenting and using a quality plan

• Document release process

• Root cause analysis



1. Quality viewpoints

Slide 5

Dolphin’s Vision

“It is our vision to deliver independence to

people with vision and print impairments all

over the World. With over 160 million visually

impaired people and 600 million dyslexic or

learning disabled people, that's no small

challenge. But we're determined to make a

difference and have been working hard on that

goal since Dolphin started in 1986.”

Noel Duffy

Managing Director

Dolphin Computer Access

It is our vision to deliver independence to people with vision and print impairments all over the World. With over 160 million visually impaired people and 600 million dyslexic or learning disabled people, that's no small challenge. But we're determined to make a difference and have been working hard on that goal since Dolphin started in 1986.

Slide 6

Dolphin Values

• “To deliver high quality, easy to use,

innovative solutions”

What do we need to achieve...?

and when?

and why?

and how we are going to do it?

Dolphin Values include: "To deliver high quality, easy to use, innovative solutions" We ask ourselves: “What do we need to achieve...? and when? and why? and how we are going to do it?”



Slide 7

What is Quality?

M

U

V

P

TQuality in Use

Internal Quality

VALUE

USER

PRODUCT

MANUFACTURING

TRANSCENDENT

What is quality? The graphic shows the different quality viewpoints: Quality in use (with value and user viewpoints) at the top, “above the line” Internal or engineering quality (with manufacturing and product viewpoints at the bottom “below the line) Product view Manufacturing view Value view User view Transcendent view



Slide 8

SENCOs

DealersSales

Support

Marketing

Product

owners

Product /

Service

Design &

Development

Internal

Testers

Teachers

Purchasers

System

Administrators

Peri’s

Alpha &

Beta

Testers

People

with VI / TI

Quality for Dolphin stakeholders?

M

U

V

P

TQuality in Use

Internal Quality

The graphic shows the different quality viewpoints for Dolphin: Quality in use (with value and user viewpoints) at the top, “above the line” Internal or engineering quality (with manufacturing and product viewpoints at the bottom “below the line) Example Customer stakeholders are shown:

Purchasers and dealers will have a value viewpoint (above the line)

People with VI/TI, teachers and “Peris” (Peripatetic assistants/teachers) will have a user viewpoint (above the line)

Special Needs Coordinators (SENCOS), Alpha & Beta testers will have a partly a user and partly a product viewpoint (on the line)

System administrators will have a product and manufacturing viewpoint (below the line).

Dolphin stakeholders (in blue) show a similar range of viewpoints:

Sales, marketing and product owners will have a Value and User view (above the line)

Support sit on or just below the line

Development and testers will have a product and manufacturing view (below the line).

This means everyone needs to have an input into acceptance criteria, requirements, user stories and testing... We’ll see that each viewpoint provides different aspects of quality. Each stage of testing is done by different people to measure different qualities in the software.



Slide 9

Exercise: what is Quality for your stakeholders?

Quality in Use

Value

User

Transcendent

Product

Manufacturing

Internal quality



2. Weaver Triangle

Slide 11

Setting missions and goals with a

Weaver triangle

• Building a Weaver Triangle

– Aims

– Specific aims

– Objectives

– Targets

– Indicators

Setting missions and goals with a Weaver triangle Building a Weaver Triangle

Aims Specific aims Objectives Targets Indicators



Slide 12

Setting missions and goals with a

Weaver triangle• Jam making

– Aims• To make really good jam this year

– Specific aims• Not need to buy jam for ourselves

• Use jam for Christmas presents

– Objectives• Make red currant jelly

• Make damson jam

• Make marrow & ginger jam

• Gift wrap jam

– Targets• By October 31st have 25 lb jam for own use and 15 lb for presents

• Spend no more than £15 on ingredients

• Use home grown fruit

– Indicators• Spouse eats jam voluntarily

• Mother praises jam

• Jam wins WI prize

Jam making Aims

To make really good jam this year Specific aims

Not need to buy jam for ourselves Use jam for Christmas presents

Objectives Make red currant jelly Make damson jam Make marrow & ginger jam Gift wrap jam

Targets By October 31st have 25 lb jam for own use and 15 lb for presents Spend no more than £15 on ingredients Use home grown fruit

Indicators Spouse eats jam voluntarily Mother praises jam Jam wins WI prize



Exercise: Setting missions and goals with a Weaver Triangle

Your project Aims

Specific aims Objectives Targets Indicators



3. ETVX

Slide 15

Entry, Quality Gates and Exit

EXIT – how do I know I have completed?

ENTRY – how do I know I am ready to start?

Quality Gate

Entry, quality gates and exit Exit – how do I know I have completed? Quality gate (red line) Entry – how do I know I am ready to start?



Slide 16

Do it

Check you’ve done it

Decide what is needed

Check that you have understood what to do and have not forgotten anything

Activity

Make sure everything is lined up ready to go

Make sure it meets the customer’s expectations

Send it

T

I

M

E

Slide shows a graphic with a flow through an activity with red lines for quality gates:

Decide what is needed

Check you have understood and not forgotten anything (quality gate)

Do it

Check you have done it (quality gate)

Make sure it meets the customer’s expectations (quality gate)

Make sure everything is lined up ready to go (quality gate)

Send it



Slide 17

Detailed design, build, review, unit test

Internal test

Acceptance Criteria and Bill of Materials; Release Plan; Quality Plan; Project Plan

User centred Design; Detailed Attribute Requirements & Architectural Design

Joint reviews

Product Definition

Production Test

Acceptance Test (market-readiness, alpha, beta)

Release Management

T

I

M

E

Example to show a product – same diagram – red lines for quality gates - but it now says:

Product definition

Acceptance Criteria and Bill of Materials; Release Plan; Quality Plan; Project Plan

User Centred Design; Detailed Attribute Requirements & Architectural Design

Joint reviews (quality gate)

Detailed design, build, review, unit test

Internal test (quality gate)

Acceptance Test (market-readiness, alpha, beta) (quality gate)

Production Test (quality gate)

Release Management



Slide 18

Example ETVX: jam making

Entry: have I got everything ready? Recipe; Ingredients;

Preserving pan? Clean jars? Wax circles, covers? Thermometer?

Saucers in freezer? Cooling rack, labels. Spouse informed...

Task: make jam and put in jars, labelled, as per recipe in Delia

Smith , store in rack

Validation: test jam for setting using cold saucers and

thermometer; test jam for taste by asking spouse to try it

Exit: jam made, bottled, cooled, labelled, on rack

Equipment clean and put away; recipe book away, kitchen tidy

Example ETVX – jam making

Entry: have I got ready? Recipe; Ingredients; Preserving pan? Clean jars? Wax circles, covers? Thermometer? Saucers in freezer? Cooling rack, labels. Spouse informed...

Task: make jam and put in jars, labelled, as per recipe in Delia Smith , store in rack

Validation: test jam for setting using cold saucers and thermometer; test jam for taste by asking spouse to try it

Exit: jam made, bottled, cooled, labelled, on rack; Equipment clean and put away; recipe book away, kitchen tidy



Exercise: ETVX • defining a process using ETVX

• defining menu choices in a process

• setting entry criteria

• choosing v&v coverage

• setting exit criteria

TASK:

Previous tasks

ENTRY

TASK

VALIDATION

EXIT

Next task



Slide 19

Discussion

• Who is responsible for what quality

activities:

• R Responsible

• A Accountable

• C Contributes

• I Informed

Who is responsible for what quality activities: R Responsible A Accountable C Contributes I Informed

WHAT Set acceptance

criteria

Run Risk Workshop

Write Quality Plan

Design Acceptance

tests

WHO

R

A

C

I



Slide 20

Example: improve communication

• Meetings

– How could we use these ideas to improve

face to face meetings?

– Emails?

– Phone calls?

Slide 21

Example: improve communicationMeetings Phone Calls Emails

Why?

Who?

Where?

What?

How?

When?

How long?

Slide shows empty table: Columns are labelled Meetings; Phone calls; Emails Rows are labelled Why? Who? Where? What? How? When? How Long



Slide 22


Meetings Phone Calls Emails

Why? Goal, objective Goal, objective Goal, objective

Who? RACI

Reps, small numbers

RACI

1-to-1, conference

RACI

TO: v CC:

Where? Room? Open? Room? Open?

What? Small group 1-to-1 and small group Broadcast

How? Agenda; Facilitator;

Decisions; Actions

with dates

P-K-P-M-F

Skype; Agenda;

Facilitator; Decisions;

Actions with date

P-K-P-M-F

Read;

Action by:

Comment:

FYI:

When? Regular; as needed;

with notice

Regular; as needed;

with notice

Regular; as needed;

How long? Start on time, finish on

time; 1 hour then a

break; 15 minute

stand ups

Start on time, finish on

time; 1 hour then a

break; 15 minute stand

ups

Less than 1 screen?

Attachments

Slide shows table with communication improvements: Meetings •Goal, objective •RACI •Reps, small numbers •Room? Open? •Small group •Agenda; Facilitator; Decisions; Actions with dates •P-K-P-M-F •Regular; as needed; with notice •Start on time, finish on time; 1 hour then a break; 15 minute stand ups Phone Calls •Goal, objective •RACI •1-to-1, conference •Room? Open? •1-to-1 and small group •Skype; Agenda; Facilitator; Decisions; Actions with date •P-K-P-M-F •Regular; as needed; with notice •Start on time, finish on time; 1 hour then a break; 15 minute stand ups Emails •Goal, objective •RACI; TO: v CC: •Broadcast •Read; Action by: Comment: FYI: •Regular; as needed; Less than 1 screen? Attachments



4. Setting Acceptability Levels

Slide 24

Total cost of quality

Comparing organisations

COF COQ

COF is Cost of Failure - COQ is Cost of Quality

Quality

Security

Measurement of results

Development best practices

High costs?

Failure

Compromised security

NO measurement of results

Development worst practices

High costs

OK

Low costs

The graphic shows the total cost of quality graph. On the left of the graph, the histogram shows a situation where all the costs are costs of failure. As the cost of quality (testing, better processes, etc.) increases, the cost of failure decreases. But in fact the total cost drops too. This trend continues to a low point where there are still some failures but the overall cost is at a minimum. The total cost then rises, while the number of failures and hence cost of failure tends to zero.



Slide 25

ISO 25000 - Quality in Use

Usability

Flexibility

Safety

Effectiveness

Efficiency

Satisfaction

Context conformity

Context extendibility

AccessibilityCommercial damage

Operator health and safety

Public health and safety

Environmental harmU VT

Three Quality in Use attributes: Usability, Flexibility, Safety Usability – Effectiveness; Efficiency; Satisfaction Flexibility - Context conformity; Context extendibility; Accessibility Safety - Commercial damage; Operator health and safety; Public health and safety; Environmental harm These support the user, value and transcendent views of quality



Slide 26

Usability

Flexibility

Safety

Usability

Flexibility

Safety

Usability

Flexibility

Safety

WebsiteAvionicsDesk top

Comparison by sorting – is X more important than Y?

?

?

?

Your project

Desk top might be prioritised usability, flexibility then safety Avionics might be prioritised safety, usability, flexibility Website might be prioritised flexibility, usability, safety What about your project?

Exercise: prioritising quality in use

Effectiveness

Efficiency

Satisfaction

Context conformity


Accessibility

Commercial damage



Environmental harm



Slide 27

Jam acceptance criteria...

• Edible

– Taste – sweetness

– Fruit flavour

– No taint

• Spreadable

– Not too thick, not too thin

– Cooked to setting point

• Colourful

– But not artificial

Jam acceptance criteria Edible

Taste – sweetness Fruit flavour No taint

Spreadable Not too thick, not too thin Cooked to setting point

Colourful But not artificial



Slide 28

Jam acceptance criteria (2)

• Jam making: by October 31st• Make red currant jelly: 15lb; 10lb in pretty jars for gift

wrap

• Make damson jam: 5lb; 5lb in pretty jars for gift wrap

• Make marrow & ginger jam: 5lb own use

• Gift wrap jam

• Spend no more than £15 on ingredients

• Use home grown fruit

• Contingency: Poor / little fruit – only make for own use

– Indicators• Spouse eats jam voluntarily - mandatory

• Mother praises jam - desirable

• Jam wins WI prize - aspirational

Jam acceptance criteria (2) Jam making: by October 31st

Make red currant jelly: 15lb; 10lb in pretty jars for gift wrap Make damson jam: 5lb; 5lb in pretty jars for gift wrap Make marrow & ginger jam: 5lb own use Gift wrap jam Spend no more than £15 on ingredients Use home grown fruit Contingency: Poor / little fruit – only make for own use

Indicators Spouse eats jam voluntarily - mandatory Mother praises jam - desirable Jam wins WI prize - aspirational



Slide 29

Jam acceptance criteria (3)

• Edible

– Taste – sweetness

– Fruit flavour

– No taint

• Spreadable

– Not too thick, not too thin

– Cooked to setting point

• Colourful

– But not artificial

% sugars

% fruit

% foreign materials

% viscosity

Temperature

measured with

thermometer

Amount colourings

Pantone number

Jam acceptance criteria (3) Edible: Taste – sweetness (% sugars) Fruit flavour (% fruit) No taint (% foreign materials) Spreadable: Not too thick, not too thin (% viscosity) Cooked to setting point (temperature measured by thermometer) Colourful: But not artificial (amount of colourings; Pantone number)



Slide 30

Requirements & Acceptance

Criteria – example Usability• “make it user friendly”

• “needs to be used without a training

course”

• SMART Learnability:

– a new user is able to understand and use the

application without prior training

– after 1 hour of use they are able to achieve a

work speed of average 15 task completions

per hour

Requirements & Acceptance Criteria – example Usability “make it user friendly” “needs to be used without a training course” SMART Learnability:

a new user is able to understand and use the application without prior training after 1 hour of use they are able to achieve a work speed of average 15 task completions per hour



Slide 31

Requirements & Acceptance

Criteria – example User Story• As a ....

• I want to ...

• Because ...

• Sometimes I also want to ...

• I need to be prevented from ...

Context of use

to personas/example group

to user stories

Requirements & Acceptance Criteria – example User Story As a .... I want to ... Because ... Sometimes I also want to ... I need to be prevented from ... Context of use to personas/example group to user stories



Slide 32

What is Quality for Engineers?

M

U

V

P

T

Usability Flexibility Safety

SecurityOperability

Functional

suitability

Compatibility

Maintainability

Portability

Reliability

Performance

Quality in Use

Internal Quality

The graphic shows the Quality in Use attributes of Usability, Flexibility and Safety above the line with the value and user view of quality. The internal quality attributes of Operability, functional suitability, portability, security, compatibility, performance. Maintainability and reliability are with the product view and manufacturing view below the line as Internal quality. The transcendent view of quality is larger and overlaps everything. Usability, Flexibility and Safety and how they break down into sub attributes: Usability – Effectiveness; Efficiency; Satisfaction Flexibility - Context conformity; Context extendibility; Accessibility Safety - Commercial damage; Operator health and safety; Public health and safety; Environmental harm Internal quality is made up of: Functional suitability, operability, portability, compatibility, maintainability, security, performance, reliability



Slide 33

Examples

Stakeholder question

• How long can you

continue to (run your

business / carry out your

tasks) if the IT system is

not available?

• What is the timescale

between receiving

timesheets and sending

out pay packets? What

happens during that

time?

Underlying attribute

• Reliability

• Performance – response

times and throughput

Identify which quality views and attributes are important For this business For this project For this group of customers

Identify questions that will help them select the attributes Do not ask them directly about the attributes... Stakeholder question and Underlying attribute How long can you continue to (run your business / carry out your tasks) if the IT system is not available? - Reliability What is the timescale between receiving timesheets and sending out pay packets? What happens during that time? - Performance – response times and throughput



Slide 34

Examples

Stakeholder question

• Do you expect there to be

a training course before

people use the system?

• Do you want different

people to be able to use

this product on different

platforms?

• How volatile is the

business? How often do

you change your products

and services?

Underlying attribute

• Operability – learnability

• Portability

• Maintainability

The graphics are blank are clicked to uncover the examples Do you expect there to be a training course before people use the system? - Operability – learnability Do you want different people to be able to use this product on different platforms? - Portability How volatile is the business? How often do you change your products and services? - Maintainability



Slide 35

Internal quality supports quality in use

Security

Usability

Operability

Performance

Flexibility Safety

Functional suitability

CompatibilityMaintainabilityPortability

Reliability

Quality in use (usability flexibility and safety) supported by the attributes of operability, functional suitability, performance, security, reliability, portability, maintainability, compatibility) Usability is supported by Operability and Functional Suitability, but also by Performance, Security, Reliability – customers are likely to be aware of these. However, Portability, Maintainability, Compatibility will also affect the usability for some people. Flexibility is supported by operability (the user being able to choose how they carry out tasks) but also by portability, maintainability, compatibility. The business is less likely to be aware of these as worth paying for. Safety is supported by reliability, security, functional suitability. But operability is key – e.g. Flight software, London ambulance. In fact overall safety needs a close attention to all the attributes.



Slide 36

Discussion• For one important attribute, identify

SMART acceptance criteria: HARD!

The ISO 25000 Internal or engineering quality attributes also break down into sub attributes – the developers will deliver these to meet the requirements for quality in use. Most people don’t need to understand them:

Functional suitability: appropriateness; accuracy

Reliability: maturity; availability; fault tolerance; recoverability

Performance efficiency: time behaviour; resource utilization

Operability: appropriateness; recognisability; learnability; ease of use; attractiveness; technical accessibility

Security: confidentiality; integrity; non-repudiation; accountability; authenticity

Compatibility: co-existence; interoperability

Maintainability: modularity; reusability; analyzability; changeability; modification stability; testability

Portability: adaptability; installability; replaceability



Exercise: setting acceptance criteria

• Write a user story for your project

• Add SMART acceptance criteria

As a .................................................................................................

I want to ........................................................................................

Because .........................................................................................

Sometimes I also want to .............................................................

I should be prevented from ............................................................

SMART acceptance criteria:



Slide 37


• Meetings



– Emails?

– Phone calls?

Meetings How could we use these ideas to improve meetings? Emails? Phone calls?



5. Setting test coverage against risk

Slide 39

Acceptancecriteria

impact if not delivered

impact if delivered wrong

likelihood of failure

worth to Dolphin risk score

AC1 High Low Low High High

AC2 Medium High None Medium Medium

AC3 Low High High Low High

It is worth defining the high risk activities,

acceptance criteria, features, and

attributes carefully

Value to company of this attribute Value to our (potential) customers Impact of wrong attributes

Not at the right level (too little) Not at the right level (too much; too costly) Missing

Likelihood of it being wrong Frequency it is used/complained about Table shows 3 acceptance criteria with scores none, low, medium, high for each of the impact, likelihood and worth scores, giving a total risk score of AC1 High AC2 Medium AC3 High It is worth defining the high risk activities, acceptance criteria, features, and attributes carefully



Slide 40

test coverage typeno testinstall - deinstallinstall - 5 happy paths - uninstallunit test checklistsupport test checklistproduct owner checklistalpha tester checklistbeta tester checklistmarketing demo test20 happy pathsbreak it - defect hunt - free rangetraining pathsuser guide pathsbreak it - defect hunt - boundaries break it - defect hunt - pairsbreak it - defect hunt – combinationsPower user test...etc

Amount of testing is associated

with level of risk for each attribute

for the situations in which that

attribute comes into affect

Acceptance Criteria

Win XPIE 7

Office 2003

WIN 7IE 8

Office2007

WIN 7IE 8

Office 2010

VISTAIE 8

Office 2010

VISTA FirefoxOffice 2007

AC1 H L H L H

AC2 M H M N M

AC3 L N L H L

Amount of testing is associated with level of risk for each attribute for the situations in which that attribute comes into affect Table shows acceptance criteria against platforms (e.g. Win XP IE 7 Office 2003; Win 7, IE 8 Office 2010). Each cell is red, amber, green depending on risk (red=high; amber=medium; yellow=low; green=none) List shows test coverage type: No risk: no test or install – deinstall or install - 5 happy paths – uninstall Low risk:

unit test checklist

support test checklist

product owner checklist

alpha tester checklist

beta tester checklist Low to medium risk: marketing demo test; 20 happy paths; break it - defect hunt - free range Medium risks: training paths; user guide paths; break it - defect hunt – boundaries; break it - defect hunt – pairs High risk: break it - defect hunt – combinations: Power user test ...etc



Exercise: coverage and risk • For one aspect of your project,

– Identify risks

– Assess risks

– Choose tests from the menu



6. Documenting and using a quality plan

Slide 42

Quality plan (part)

Acceptance Criteria

Win XPIE 7

Office 2003

WIN 7IE 8

Office 2007

WIN 7IE 8

Office 2010

VISTAIE 8

Office 2010

VISTA Firefox

Office 2007

AC1Power user test

(Dave)

beta tester checklist (Colin)

defect hunt –combinations

(Mike)

beta tester checklist (Aj)

defect hunt –combinations

(Mike)

AC2user guide paths

(Neill)Defect hunt (Colin & Aj)

20 happy paths (Mike)

No test20 happy

paths (Mike)

AC3Beta tester checklist

(Dave)

Install, 5 happy paths

(Mike)

Beta tester checklist (Mike)

Defect hunt (Colin & Aj)


The table showing acceptance criteria against platform with a risk level in each cell now shows a test coverage and tester in each cell with the risk colour. e.g. High risk: Power user test (Dave) Red Medium risk: 20 happy paths (Mike) Amber



Slide 43

Quality Report (part)

Acceptance Criteria

Win XPIE 7

Office 2003

WIN 7IE 8

Office 2007

WIN 7IE 8

Office 2010

VISTAIE 8

Office 2010

VISTA Firefox

Office 2007

1Blue

Power user test (Dave)

Amberbeta tester

checklist (Colin)

AmberBeta tester

checklist (Aj)

Amberdefect hunt –combinations

(Mike)

Amberdefect hunt –combinations

(Mike)

2Blue

user guide paths (Neill)

BlueDefect hunt (Colin & Aj)

Green20 happy

paths (Mike)No test

Green20 happy

paths (Mike)

3Blue

Beta tester checklist (Dave)

GreenInstall, 5

happy paths (Mike)

RedDefect hunt (Colin & Aj)

BlueBeta tester

checklist (Mike)

BlueBeta tester

checklist (Mike)

The same table as the previous slide but now the colour shows progress, as well as the type of test and the tester Blue – in progress Amber – some defects/delays Red – showstopper/blocked Green - ready for live Grey – no test



Slide 44

Green for GO

• Using the quality plan to track progress to

Green for GO (ready to use, ready to ship

to the customer)

• Jam:

– Unit test – by jam maker (setting point)

– Internal test – by spouse (taste)

– Acceptance test – by spouse (appearance)

– Release as Christmas present only if...

Using the quality plan to track progress to Green for GO (ready to use, ready to ship to the customer) Jam:

Unit test – by jam maker (setting point) Internal test – by spouse (taste) Acceptance test – by spouse (appearance) Release as Christmas present only if...

Exercise: quality plan • Documenting a quality plan in a spreadsheet to summarise the quality requirements



7. Document release process

Slide 46

Discussion

• What documents require formal reviews?

• Simple document release process

including techniques to enable fast

effective static testing

• Choosing static test methods

What documents require formal reviews? Simple document release process including techniques to enable fast effective static testing Choosing static test methods



Slide 47

Plan Kickoff Prepare Meet Rework

Re-review

and closure

Reviewers

raise

defects

Author

edits and

reviews

Review

team

discuss

Author

edits and

reviews

Author

distributes

Signed

off

decide

review

type

objective

reviewers

checklists

timetable

Send

invites

Distribute

document

checklists

Explain

roles

Read

critically

against

checklist

Discuss

Identify

defects

&

solutions

Agree

next step

Edit whole

document

for raised

and

similar

defects

and

correct

Has

everything

been

done? Is it

fit for

purpose?

Explanation of diagram 1: the process flow The review process has six steps: Plan: decide the type of review, the review objectives, timescale and participants, checklists and roles. Happens while the author is editing and reviewing the document to prepare it for the review process Kickoff: explain the review type, objectives, timescale and participants, checklists and roles. Author distributes the document and checklists Prepare: each reviewer examines the document critically and objectively, with the help of the checklist, to identify areas for improvement, defects and questions, and if required potential solutions Meet: the review team meets the author to discuss all the points raised, identify more points, and (sometimes) also to look for solutions Rework: the author improves the document, edits and reviews it Re-review and closure: the improved document is reviewed and a decision made to release or not. If it is ready for release it is signed off.



Slide 48

S1 Un-

reviewed

document

S12 Issued

document

S9Peer

reviewed

document

S2 Author

reviewing

document

S3 Being

re-worked

S11 Fit

for

purpose

S4

Ready for

peer

review

S10

Being

Inspected

S5 Being

buddy

reviewed

S6 Being

informally

reviewed

S7 Being

walked

through

S8 Being

technically

reviewed

Explanation of diagram 2: the State Diagram The state diagram shows the following transitions between states, each of which is caused by a particular trigger event. These are listed below: S1 Un-reviewed document to S2 Author reviewing document S2 Author reviewing document to S3 Being reworked S2 Author reviewing document to S4 Ready for peer review S2 Author reviewing document to S11 Fit for purpose S3 Being reworked to S2 Author reviewing document S4 Ready for peer review to S5 Being Buddy reviewed S4 Ready for Peer review to S6 being informally reviewed S4 Ready for Peer review to S7 being walked through S4 Ready for Peer review to S8 being technically reviewed S5 Being buddy reviewed to S9 peer review complete S6 Being informally reviewed to S9 peer review complete S7 being walked through to S9 peer review complete S8 being technically reviewed to S9 peer review complete S9 Peer review complete to S3 Being reworked S9 Peer review complete to S11 fit for purpose S9 Peer review complete to S10 being inspected S10 Being inspected to S2 being reworked S10 Being inspected to S11 Fit for purpose S11 Fit for purpose to S12 signed off for release



Slide 49

Example: jam making

• Review the recipe before it goes in the

book

– Author

– Tasting panel

• Review the recipe before using it

– Both cook and spouse do this

– Trial run with small quantity/practice

• Review the labels

– Spouse

Example: jam making Review the recipe before it goes in the book

Author Tasting panel

Review the recipe before using it Cook and spouse Trial run with small quantity/practice

Review the labels Spouse

Exercise: Identify review types for typical documents in your project



Slide 50


• Meetings



– Emails?

– Phone calls?

Meetings How could we use these ideas to improve meetings? Emails? Phone calls?



8. Root Cause Analysis

Slide 52

Root cause analysis to resolve and

prevent problems

• Feedback loop

• WHY did we make that mistake?

• HOW can we prevent it in future?

• Why did the invoice not print?

– No paper

• No low paper warning (software driver change)

• No reordering process (process change)

• No-one checked (RACI change)

Root cause analysis to resolve and prevent problems Feedback loop WHY did we make that mistake? HOW can we prevent it in future? Why did the invoice not print?

No paper No low paper warning (software driver change) No reordering process (process change) No-one checked (RACI change)



Exercise: root cause analysis

Choose a problem e.g. A bug

Why did it happen?

Why?

Why?

Why?

Why?

Why?

Why?



Slide 54

Quality

Vision

Flexibility

Processes

Resources

Continuous improvement

People

Graphic shows quality as affected by and affecting: Vision People Flexibility Continuous improvement Resources Processes



Personal Action Plan

As a result of the workshop I will:

This week

Next month

In the next six months:



9. Appendix – templates

Template 1: Quality viewpoint map

Quality in Use

Value

User

Transcendent

Product

Manufacturing

Internal quality



Template 2: Weaver Triangle

Your project Aims

Specific aims Objectives Targets Indicators



Template 3: ETVX

TASK:

Previous tasks

ENTRY

TASK

VALIDATION

EXIT

Next task



Template 4: RACI

Task Responsible Accountable Contributes Informed Deputies



Template 5: Quality in Use Risks and Priority

Importance Impact if not delivered

Impact if wrong

Likelihood of being wrong

Value (money)

Risk score

Priority

Effectiveness

Efficiency

Satisfaction

Context conformity


Accessibility

Commercial damage



Environmental harm



Template 6: Context of use Context Analysis Report Template based on http://www.usabilitynet.org/tools/context.htm

Name and version of product

Report completed by Date

Organisation

Objectives

1.1 USER TYPES Affects Usability

1.1.1 User types being considered

a) user types identified

b) user types for usability evaluation

1.1.2 Secondary or indirect users who:

a) interact with the product

b) are affected by its output

Each user type is now analysed by:

- Personal characteristics - Tasks - Environment where the product is used

Fill in the forms for EACH user type identified.



Identify the characteristics of a distinct type of user:

User Type H-M-L Affects Usability

1.2 SKILLS & KNOWLEDGE

1.2.1 Training and experience in the business processes and methods which the product supports

1.2.2 Experience in

a) using the product

b) using other products with similar main functions

c) using products with the same interface style or operating system

1.2.3 Training in

a) tasks supported by the products main functions

b) using the products main functions

c) using other products with similar main functions

d) using products with the same interface style or operating system

1.2.4 Qualifications

1.2.5 Relevant input skills

1.2.6 Linguistic ability

1.2.7 Background knowledge

1.3 PHYSICAL ATTRIBUTES

1.3.1 a) Age range

b) Typical age

1.3.2 Gender

1.3.3 Physical limitations and disabilities

1.4 MENTAL ATTRIBUTES

1.4.1 Intellectual abilities

a) distinctive abilities

b) specific mental disabilities

1.4.2 Motivations

a) attitude to job & task

b) attitude to the product

c) attitude to information technology

d) employees attitude to the employing organisation



User Type H-M-L Affects Usability

1.5 JOB CHARACTERISTICS

1.5.1 Job function

1.5.2 Job history

a) how long employed

b) how long in current job

1.5.3 Hours of work / operation

a) hours of work

b) hours using product

1.5.4 Job flexibility

1.6 LIST OF TASKS

a) tasks identified

b) tasks for usability evaluation



User against tasks (do a table for four typical tasks for each user type identified above)

Task name Affects Usability

Affects Usability

Affects Usability

Affects Usability

2 TASK CHARACTERISTICS

2.1 Task goal

2.2 Choice

2.3 Task output

2.4 Side effects

2.5 Task frequency

2.6 Task duration

2.7 Task flexibility

2.8 Physical & mental demands

a) Factors which make task demanding

b) How demanding in comparison with others

2.9 Task dependencies

2.10 Linked tasks

2.11 Safety

2.12 Criticality of the task output

2 TASK CHARACTERISTICS

2.1 Task goal



2.2 Choice

2.3 Task output

2.4 Side effects

2.5 Task frequency

2.6 Task duration

2.7 Task flexibility

2.8 Physical & mental demands

a) Factors which make task demanding

b) How demanding in comparison with others

2.9 Task dependencies

2.10 Linked tasks

2.11 Safety

2.12 Criticality of the task output



User type against Environment

Affects Usability

3 Organisational Environment

3.1 Structure

3.1.1 Group working

3.1.2 Assistance

3.1.3 Interruptions

3.1.4 Management structure

3.1.5 Communications structure

3.2 Attitudes & Culture

3.2.1 IT Policy

3.2.2 Organisational aims

3.2.3 Industrial relations

3.3 Worker/User Control

3.3.1 Performance monitoring

3.3.2 Performance feedback

3.3.3 Pacing

4 Technical Environment

4.1 Hardware

A) required to run the product

B) likely to be encountered when using the product

4.2 Software

A) required to run the product (e.g. operating system)

B) likely to be encountered when using the product

4.3 Reference materials

PHYSICAL ENVIRONMENT

5.1 Environmental Conditions

5.1.1 Atmospheric conditions

5.1.2 Auditory environment

5.1.3 Thermal environment

5.1.4 Visual environment

5.1.5 Environmental instability

5.2 Workplace Design

5.2.1 Space and furniture

5.2.2 User posture

5.2.3 Location

A) of the product

B) of the workplace

5.3 Health & Safety

5.3.1 Health hazards

5.3.2 Protective clothing and equipment



Template 7: User story and SMART acceptance criteria

As a .... I want to ... Because ... Sometimes I also want to ... I need to be prevented from ...

SMART Acceptance criteria:

Check: Specific

Measureable Achievable Realistic Testable



Template 8 Attribute tables and risks Attribute Sub attribute Importance Impact (not done) Impact (wrong) Likelihood of being wrong Value Risk score Priority

Functional suitability Appropriateness

Accuracy

Operability Appropriateness

Recognisability

Learnability

Ease of use

Attractiveness

Accessibility

Reliability Maturity

Availability

Fault tolerance

Recoverability

Performance/efficiency Time behaviour

Resource utilization

Security Confidentiality

Integrity

Non-repudiation

Accountability

Authenticity

Compatibility Co-existence

Inter-operability

Maintainability Modularity

Re-usability

Analyzability

Changeability

Stability

Testability

Portability Adaptability

Installability

Replaceability



Template 9: Methods and techniques menu Activity Review Test coverage Sign off for release

No risk Author review

No test Unit test – informal by developer

install - deinstall

install - 5 happy paths - uninstall

Author

Low risk Author review Informal review Buddy review

Happy path test Minimum Classification Tree

unit test checklist

support test checklist

product owner checklist

alpha tester checklist

beta tester checklist

Author Reviewer/ Tester

Medium risk Buddy review Walkthrough Technical review

Minimum Classification Tree

marketing demo test

20 happy paths

break it - defect hunt - free range

training paths

user guide paths

break it - defect hunt - boundaries

break it - defect hunt - pairs

Review / test lead Customer representative Support team lead Product owner

High risk Walkthrough Technical review Inspection

break it - defect hunt – combinations

Power user test

Decision table

State transition

Customer representative Support team lead Product owner Management team member Board member



Template 10: Quality plan

Platform

Acceptance Criteria

Win XP

IE 7

Office 2003

WIN 7

IE 8

Office 2007

WIN 7

IE 8

Office 2010

VISTA

IE 8

Office 2010

VISTA Firefox

Office 2007

AC1 High – red


Med-low beta tester checklist

(Colin)

High – red defect hunt –

combinations (Mike)

Med-low beta tester checklist

(Aj)

Med- high defect hunt –

combinations (Mike)

AC2 Medium


High - red Defect hunt (Colin &

Aj)

Medium 20 happy paths (Mike)

No risk No test

Medium 20 happy paths (Mike)

AC3 Med-low


No risk Install, 5 happy paths (Mike)

Medium Beta tester checklist

(Mike)

Med – high Defect hunt (Colin &

Aj)

Med-low Beta tester checklist

(Mike)



Template 11: Quality report – progress

Blue = progress as expected

Green = completed ready for live

Amber = Some defects identified

Red = Testing blocked OR show stopper defects

Grey = no test planned

Platform

Acceptance Criteria

Win XP

IE 7

Office 2003

WIN 7

IE 8

Office 2007

WIN 7

IE 8

Office 2010

VISTA

IE 8

Office 2010

VISTA Firefox

Office 2007

1 Blue


Amber beta tester checklist

(Colin)

Amber Beta tester checklist

(Aj)

Amber defect hunt –

combinations (Mike)

Amber defect hunt –

combinations (Mike)

2 Blue


Blue


Green


No test Green


3 Blue


Green

Install, 5 happy paths (Mike)

Red


Blue


Blue




Template 12: Example review checklist

One viewpoint –sales and marketing

Point Questions Review Action

1 For each requirement /feature… 1.1 Is a priority identified? (Must; Should; Could)

1.2 Is a risk level identified? (High, Medium, Low) e.g.

impact of not having this

impact of having it but reduced or with faults

likelihood of it going wrong

1.3 Is a value to the company identified? (e.g. sales in £)

1.4 Is a size identified? (large, medium, small)

1.5 Are ideal and latest required date identified?

1.6 Can USPs be identified?

2 User stories and examples 2.1 Is it possible to identify a customer, a reason and example use for each

requirement e.g. in a user story or example: “As a …. I want to …. Because …”

2.2 Are alternative flows identifiable? e.g. “I also sometime want to…”

2.3 Are (human) error conditions identified? e.g. “I should be prevented from…” “If X is not available then…”

2.4 Is a context of use established (who will use it, their attributes where relevant, their physical environment)

3 Product attributes and acceptability 3.4 Is the appropriate level of usability identified (to meet. context of use)

Is the appropriate level of flexibility identified (e.g. how the user can adapt the product/service to their own uses/environment/language? How quickly will the customer expect new features/services? Are there any implications for health, safety, the environment?

4 Which of the following are important to this product? 4.2 Reliability: How long can a person / organisation continue without the

support of the product? Is any down-time allowable – if so how much and when?

4.3 Operability: How quickly do people need to learn the product and is it appropriate to have training courses? How much support do people need to remember commands etc? How simple does the interface need to be? How many beginner to power users? What aesthetic values are important to the customers? What technical support to accessibility is required (what supported)?

4.4 Security: Is confidential data stored by the product? Is personal data stored by the product?

4.5 Compatibility: What other products will be used at the same time as or in conjunction with this product?

4.6 Portability: What different platforms might people use? Will people want to move from one platform to another and keep the same settings in this product? Who will be installing it and how? Will customers expect to install it from Website, DVD, PEN, over network?

4.7 Maintainability: How often will the way the product is used change, requiring enhancements to features? How often will new features be introduced? How quickly must the business respond to change?



Template 13: Root cause analysis

Choose a problem e.g. A bug

Why did it happen?

Why?

Why?

Why?

Why?

Why?

Why?

tutorial: going green to make it better: a tool kit for ... · tutorial: going green to make it...

Documents