qa 3 best practices

1

ITMPI005

Webinar: QA Three Best Practices

March 11, 20093:30 PM – 5:00 PM

2

Jorge Luis BoriaSr. VP International Process Improvement

Liveware [email protected]

Michael MilutisDirector of Marketing

Computer Aid, Inc. (CAI)[email protected]

3

About Presenter’s Firm

LIVEWARE is a business consultant company in the field of Software Engineering. Since its creation in 1983 it has specialized in the dissemination, application and development of Software Engineering, particularly the concepts of Total Quality and the strategic value of Information Systems. In the past five years it developed a tight relationship with the Software Engineering Institute and is currently the sponsor of five lead appraisers, two certified high maturity lead appraisers, and two CMMI instructors.

4

• CAI is a global IT outsourcing firm currently managing active engagements with over 100 Fortune 1,000 companies and government agencies around the world.

• CAI is a leader in IT Best Practices for legacy support and new development application management.

• CAI’s focus is directed toward practical implementations that track and measure the right activities in software activity management

• CAI consistently promises and delivers double digit productivity in its outsourcing and consulting engagements.

• CAI makes all of this possible through the use of:

• Standard processes

• Management by metrics

• SLA compliance management

• Detailed cost, resource, and time tracking

• Capacity management

• Standard estimation

• A unique, metrics based methodology along with a proprietary, real time data repository and management system (TRACER®).

About Computer Aid, Inc. (CAI)

5

• The Project Management Institute’s ISSIG group has accredited this webinar with PDUs

• Stay tuned! Your PDU code will be displayed at the conclusion of this webinar.

PDU CREDITS FOR THIS WEBINAR

6

NOW AVAILABLE!

ONLINE WEBINAR RECORDINGS

ANYTIME ACCESS!

WWW. ITMPI.ORG / LIBRARY

Quality Assurance Role

• Provide teams and management with visibility into the process being used and into the products being built through:

1. reviewing and auditing work products and team activities to verify compliance with procedures and to standards

continues in next slide

Quality Assurance Role

2. supplying teams and management with feedback on reviews and audits

3. assisting in the definition of plans, standards, and procedures

4. addressing non-compliance issues through vigilance and escalation

from previous slide

Structural Verification

Integration and System Functional Verification

Validation (User Acceptance)

Component Functional Verification

Non-functional Verification

TESTINGTESTING

VALIDATEVALIDATE

QAQA

VERIFYVERIFYBUILDBUILD

Coding

Requirements

High Level Design

Component Design

Specification

QA vs Testing

QA Competency

• At a minimum:– Ensure product and process meets quality

system requirements as defined by customer specifications.

11

Typical Approach to QA• Process Police

– Academic approach– Very literal interpretation of CMMI– Improvement infrastructure independent of

application areas– Focus on punishment– Compliance, not commitment– Push

12

Better Approach to QA• Process Mentor

– Proactive Approach– Business-oriented interpretation of CMMI– CMMI as good practices input– Focus on progress– Achievements, not compliance– Pull

13

Case Study 1 • Fast growing federal & state e-commerce

company– Ten-fold growth in two years– Went from one product in the first five years to

multiple ones based on initial product

• Y2K Success Story• Product showed no defects over the millennium • Defects identified were 30% date related, 70% non-Y2K• Developers found defects that had been dormant for years

14

Obstacles to Quality - 1• Original Sins

– Developer-owned code– Success of a non-documented process– Buddy-network of the twelve original– Peter’s Principle

• Explosive Growth– New products, new people

• Small attrition (+)• Explosive demand (+/-)• Difficulties in recruiting (-)• Overtime the norm (-)

15

Obstacles to Quality - 2• Matrix Organization

– Project management vs. Project Administration

– Flexible yet chaotic team composition– Project leadership power struggles– Readiness for change highly challenged by

volatility of all project components

• Requires strong discipline to succeed

16

UAT Execution(SDS)

Test ReportSys Test Execution(SDS)

Test Report

Acceptance

Context - 3UAT Test Planning and Preparation

System Test Planning and Preparation

Unit Test Planningand Preparation

Acceptance

Requirements(SDS)

Acceptance

Specifications(TSD)

Coding(SDS)

Unit Test Execution(SDS)

Hand OffDeveloped Components(SDS)`

17

First Process Change• Shared test cases across testing,

analysis and development teams– Immediate payback with early detection – Shorter integration phase

• New process required new skills– SQA to help developers through the

processes

• Requirements revealed as problem

18

UAT Execution(SDS)

Test ReportSys Test Execution(SDS)

Test Report

Acceptance

Context - 3UAT Test Planning and Preparation

System Test Planning and Preparation

Unit Test Planningand Preparation

Acceptance

Requirements(SDS)

Acceptance

Specifications(TSD)

Coding(SDS)

Unit Test Execution(SDS)

Hand OffDeveloped Components(SDS)`

Phase End Review

Phase End Review

Post Mortem Project Review

Peer Review

Peer Review

19

Progress Attained• Requirements problems fixed

– Peer reviewed and agreed upon by testers and developers alike

– Baselined, accessible to all and put under change control and as root of the traceability matrix (under testers control)

– QA now seen as supporting projects • Waiving unnecessary procedures • No longer “process police”

20

And Compliance?• Testing (SQC) as the enforcers (at each

transition)– Process a consequence of their need to do

good work– Validated by

• past success in Y2K and • need to use process deliverables

• Handovers became smooth

21

The Good News• Quality counts went up

– Number of builds between freeze and release reduced by 50%

– Test preparation time vs. execution time changed from 20%-80% to 70%-30%

• Moving 50% of the total effort out of the project’s critical path

– Defects caught by developers increased an order of magnitude

– Developers pushing involvement of testers early in review of requirements and design specifications

22

Success Factors – 1– QA engineers (proactive support)

• had process ownership• could talk the talk and walk the walk• perceived as rescuing the project from itself

– Testing engineers (compliance control)• technical authority from Y2K• good reasons to demand process commitment• responsibility over the traceability matrix• perceived as process-focused, not witch-hunters

23

Case 2: Texas 3 Step• Well established financial institution with links

to military market– Very large market share in their segment– Stove pipe structure across divisions

• Community of overachievers• Growth had reached a plateau • Very low attrition over the years• Opportunities for promotion based on retirement of elders• Small market of similar employers starting to pick the

managers

24


– Lackadaisical approach to quality– New QA role filled in with rookies

• Transition Point– Mandate to achieve maturity levels in short

periods – Correctly identified as a quality, not process

problem (to quality through processes)

25

Obstacles to Quality - 2• Lots of generals, few soldiers

– Readiness for change highly challenged by bureaucratic approach to process

– Command and control, no matrix organization– No funding for a separate SQA organization– No objectivity by new hires

• “Run of the mill” checklist approach• No authority within organization• Direct reports to development manager (budget)

26

Process Needs• Expectation was improved quality will lower

cycle time– Releasing important individual efforts– Investment of those efforts in quality possible– Expert managers could become quality mentors

• objectivity achieved through independence• role seen as cross-pollination and introducing experience

to the team

27

New Obstacle• Expert manager too expensive to

perform menial tasks– Filling in checklist– Supporting template use– Doing the paperwork

28

Solution• Three roles and two actors • Expert manager from afar as QA expert• Interns as QA specialist

– Role 1: QA-experts help the team define their process and sets up the specialist’s goals and activities

– Role 2: specialist does the leg work and QA-expert remains on call

– Role 3: at selected milestones, the QA-expert works with the QA-specialist looking into the outcomes and audits the work done by the specialist for the project

29

Texas 3 Step

overall project execution

selected milestones when to perform

QA audits and reviewsprep

(all involved)

repetitive(specialist)

audit(all QA)

30

Progress Attained• Experienced managers

– shared management with • mentoring and • auditing responsibilities • large social reward within the company

– delegated internal support to the projects• keeping objectivity• provided by relatively inexperienced personnel.

31

The Good News• Steep learning curve

– 100% process compliance in six months – Significantly shortened delivery dates (no

hard data)

32

Success Factors – 2• Experienced Managers

• authority from experience and reputation

• perceived as process focused, not witch-hunters

• knowledge on how to make things happen faster and better

• Internal QA resources • artifact responsibility

• talked the talk and walked the walk

• perceived as helping the project doing productive work

33

Case 3: Another Division– Well established financial institution with

links to military market– Very large market share in their segment– Stove pipe structure across divisions

– Community of overachievers– Growth had reached a plateau – Very low attrition over the years– Opportunities for promotion based on retirement of

elders– Small market of similar employers starting to pick the

managers

34


– Lackadaisical approach to quality– New QA role filled in with rookies

• Transition Point– Mandate to achieve maturity levels in short

periods – Correctly identified as a quality, not process

problem (to quality through processes)

35

Obstacles to Quality - 2• Lots of generals, few soldiers

– Readiness for change highly challenged by bureaucratic approach to process

– Command and control, no matrix organization– No funding for a separate SQA organization– No objectivity by new hires

• “Run of the mill” checklist approach• No authority within organization• Direct reports to development manager (budget)

36

Process Change• Expectation was improved quality will lower

cycle time– Releasing important individual efforts– Investment of those efforts in quality possible– Expert managers could become quality mentors

• objectivity achieved through independence• role seen as cross-pollination and introducing experience

to the team

37

New Approach• Expert managers seen as stopped in

their tracks by lack of open positions– neighboring companies picking their up and

coming managers– opportunity for a new parallel track

• money not the problem• lack of open positions IS the problem

38

Solution• New QA function

– Using experienced senior managers with responsibility over

• mentoring • auditing the work of up-and coming first line

managers• making “ it” happen

– Line managers promoted to QA– Up and coming getting the jobs

39

Progress Attained• Experienced managers

– shared management with • mentoring and • auditing responsibilities • large social reward within the company

– delegated internal support to the projects• keeping objectivity• provided by relatively inexperienced personnel.

40

The Good News• New role well positioned by CEO

– Perceived as promotion in a clogged career ladder– New QA group found it very easy to coordinate

work – Their advice was very sound

• the organization quickly closed ranks behind them• organization was first one in the company to reach

Maturity Level 2– achieved in only six months.

41

Success Factors – 3 • Experienced Senior Managers

• authority from experience and reputation• perceived as process focused, not witch-hunters• knowledge to make things happen faster and

better• “been there, done that”• process responsibility at Level 2• could talk the talk and walk the walk• perceived as helping the project do productive

work

42

Common CSF – 1 • Use of Experienced Personnel

– had authority from experience and reputation

– were perceived as process focused, not witch-hunters

– had the knowledge to make things happen faster and better

43

Common CSF – 2 • Focus on progress

– help the projects to help the process improvement program

– avoid the “Gotcha!” attitude– make thinks happen, treat project members

as adults

44

Questions?

Your PDU CODE: S010-ITMPI09015

4545

CAI Sponsors

The IT Metrics & Productivity Institute:

• Clearinghouse repository of best practices: WWW.ITMPI.ORG

• Weekly educational newsletter: WWW.ITMPI.ORG / SUBSCRIBE

• Weekly webinars hosted by industry leaders: WWW.ITMPI.ORG / WEBINARS

• ACCESS WEBINAR RECORDINGS ANYTIME AT WWW.ITMPI.ORG / LIBRARY

http://www.itmpi.org/




46

Software Best Practices Conferences Around the World

WWW.ITMPI.ORG / EVENTS

Mar. 10 Orlando, FL

Mar. 31 Albany, NY

Apr. 9 Baton Rouge, LA

Apr. 28 Tallahassee, FL

Apr. 30 Detroit, MI

May 14 Tampa, FL

May 19 Philadelphia, PA

June 2 Hartford, CT

2009 Dates and Locations

June 9 Houston, TX

June 11 New York, NY

Sep. 15 Toronto, ON

Sep. 22 Chicago, IL

Sep. 29 New York, NY

Oct. 6 Baltimore, MD

Oct. 20 Philadelphia, PA

Nov. 10 Princeton, NJ

Nov. 17 Ft. Lauderdale, FL


47

Jorge Luis BoriaSr. VP International Process Improvement

Liveware [email protected]

Michael MilutisDirector of Marketing

Computer Aid, Inc. (CAI)[email protected]

qa 3 best practices

Documents