Testing Professional Development

Framework

“Raising competency, or testing clones?”

Mike Jarred & Dr. Luke Avsejs

Slide 2 © IDBS 2012

Agenda

◦ Objectives, and constraints

◦ IDBS – Delivering Software to Regulated Environments

◦ What is the Testing PDF?

◦ Why did we do it?

◦ How have we implemented the PDF

◦ Some things we had to think about…

◦ Mutual Benefit

◦ The good, the bad and the beautiful…

◦ Closing thoughts – Q&A

Slide 3 © IDBS 2012

Objectives and Constraints

Objectives of this session

◦ It’s a ‘show and tell’ – our journey to date, not an attempt to sell a generic

approach

◦ Keep it interactive – feedback and thoughts from other practitioners in

the community

Garner feedback and improvement suggestions. This is under continuous

improvement, all ideas welcome

Constraint

◦ Our framework is not publically available, but check out the BCS SFIA

plus for a good starting point if interested

http://www.bcs.org/category/7849

Slide 4 © IDBS 2012

Delivering Software to Regulated

Environments

IDBS is a global provider of innovative enterprise data management, analytics

and modelling solutions

◦ Used by more than 200 pharmaceutical companies, major healthcare providers, global

leaders in academic study, and high tech companies

◦ Our products enable increased efficiency, reduced costs and improved productivity of

industrial R&D and clinical research

◦ Our platforms are enabling translational medicine – personalised medicine (right drug, right

patient, right time)

Our customers work in highly regulated environments

◦ GxP (GLP, GCP, GMP etc…) FDA regulated, 21 CFR Part 11, EU Annexe 11

Customers audit IDBS to ensure risk of non-compliance introduced by our

products is mitigated through robust software development practice

The Test Group are under particular scrutiny as we provide the safety net...

Slide 5 © IDBS 2012

IDBS Testing Group

IDBS operate a “Test Centre of Excellence”

Standardised Testing Methodology, incorporating company best practices and tools

Centralised resource pool providing risk mitigation across product portfolio ◦ 35 Team Members located globally - Guildford, London, Portland US, China

Structure

◦ I report to VP of Product Delivery, who is part of the executive leadership team reporting to the CEO.

◦ Director of Test for corporate test strategy, and developing group capability ◦ 2 - Test Managers responsible for tactical delivery across products ◦ 4 - Test Leads specialising in product lines ◦ Technical roles for cross product service lines, Test Automation, Security Testing, Test

Environments & DBA, Load & Performance Testing, Toolset Administration & Support

Primarily science domain knowledge, 10 PhDs

4 - IT career testers to augment domain expertise

Test follow a professional development framework to ensure necessary competencies / skills and certification (ISTQB) are available to support product development.

Slide 6 © IDBS 2012

So, what is the Testing PDF?

It’s a mechanism for specifying roles, career paths, competencies, skills,

skill levels, training and qualifications required to deliver testing services

to our projects

Competencies and skills are categorised as Technical (what we want

people to do) and Transferable (how we want people to be, inc. soft skills

/ behavioral attributes)

Framework facilitates a consistent & logical approach to staff

development through benchmarking of knowledge, experience,

training and qualifications

Under continuous improvement – updated regularly as new competencies

and skills are identified as the demands on testing change

Lets take a look at it in a little more detail…

Slide 7 © IDBS 2012

Career Path

Senior Tester ISTQB Advanced Technical Test Analyst

/ Test Analyst

Tester 1/2 ISTQB Foundation

Tester 0 ISTQB Foundation

Test Manager ISEB Practitioner / ISTQB Advanced /

PRINCE2 Practitioner

Test Lead / Test Team Lead ISTQB Advanced Test Manager / ISEB

Practitioner

Test DBA ISTQB Advanced Technical Test Analyst

/ Test Analyst

Automation Test Engineer ISTQB Advanced Technical Test Analyst

Career Level Project Experience Training

Senior Testing Experience should include participation of Test Strategy definition and tactical

implementation of Testing Processes on at least one large project and two medium

projects.

Writing Test Strategies

Writing Test Plans

Web Application Security Tester

Automation Test Lead ISTQB Advanced Technical Test

Analyst

Security Tester ISTQB Advanced Technical Test Analyst

/ CSTA

Load & Performance Tester ISTQB Advanced Technical Test Analyst

Slide 8 © IDBS 2012

Example Competency Skills Matrix

Competency Skill Tester 1 Tester 2 Senior Tester Example Evidence / Indicators

Environment &

Data

Administration /

Coordination

Able to define test environment

hardware, software and data

requirements

2 2 3

Test Environment Plan

Updated Project Issues Log

Able to support test team with

Oracle issues relating to

strategy and infrastructure

1 1 2

Able to maintain Oracle

instances 1 1 2

Able to give environment

management advice 1 2 2

Able to assist developers with

Oracle related problems. 1 1 1

Able to assist with database

enhancements 1 1 2

Programming with Java 1 1 3

Test Design &

Maintenance

Test Script Creation 3 3 4 Fully Accepted Test Script

Checklists

Test Coverage Report

Updated Regression Test Packs

Test Script Maintenance 3 3 4

Test Model Creation and

Implementation 1 2 2

Use of Oracle, SQL and

PL/SQL within test scripts 1 2 3

Skill Level (0-5)

Skill Level Description

1 - Basic Basic knowledge has been acquired through limited training. Very limited experience, through classroom and/or case studies, or on-the-job training. Regular direction and assistance

is required to perform duties

2 - Intermediate Broader base of training has been received. A few successful experiences have been completed.

3- Senior Extensive training has been received. Several successful experiences have been completed. Performs almost all actions associated with this skill without assistance - may

occasionally require assistance from a more senior individual for only the most difficult aspects of this skill.

4 - Expert Very extensive training has been received. Training may be provided by this individual within their organisation. Many repeated successful experiences have been completed in a

variety of complex settings. Performs actions associated with this skill without assistance. Recognised within the organisation as "the person to ask" when difficult problems arise.

5- Master Very extensive training has been received. Is sought after to train others. Many repeated successful experiences have been completed in the most complex settings. Performs

actions associated with this skill without assistance. Is sought after, from both within and outside their organisation, when the most difficult problems arise.

Slide 9 © IDBS 2012

So, why did we do this…?

Organisational Drivers

Demonstrate to customers how we holistically maintain a competent testing group (we are always asked under

audit)

Demonstrate internally that we are driving professionalism / increasing capability in test group

Enable clear objective setting and structured appraisal reviews

Provide clarity regarding competencies and skills needed in the test group

Demonstrate the Career Paths open to individuals in the Test Group – there was an org structure, but no

visibility of how to progress

Assisting in formalising competency based recruitment

Slide 10 © IDBS 2012

How have we applied the PDF?

Awareness & Training Implementation

Overview of PDF, benefits, risks and pilot

implementation plan provided to Exec team –

secured approval and sponsorship

Team members skill levels for competencies are validated through the appraisal mechanism. Evidence of skill levels required

Test Management contributed and reviewed PDF

content

Outputs from appraisal are linked to Personal

Development Plans

Test Group received PDF overview as part of

change management

Experience, skills and qualifications are continuously

reviewed as part of personal development meetings

Initial 6 month pilot to capture improvement

suggestions

Once a level on the PDF is established, further

opportunities for development will be provided where

possible as part of project delivery and departmental

initiatives

Slide 11 © IDBS 2012

Some things we had to think about…

Slide 12 © IDBS 2012

Mutual benefit

Benefits to Testers Benefits to Company

A more structured & professional approach to career development

Supports Continuing Professional Development (CPD)

Increased sales and retention of existing business through successful audits

Visibility to all staff of how to progress within

testing

Employee retention / recruitment advantage

Increase skill levels and broaden experience Consistent appraisals across the department, equitable &

quantitative scoring

Gain Industry recognized qualifications Increased awareness of training needs to budgeting and

planning.

Commitment from IDBS to their development Identification of skills gaps, succession planning, coaching

& mentoring

Opportunities to cross skill into a different

framework e.g. Development (once rolled out to

other groups)

Assurance that recruitment targets area of greatest value

Slide 13 © IDBS 2012

The Good, the Bad and the Beautiful…

Lots of positive feedback (within, and outside the Test

Group)

Training economies of scale

Competency based recruitment, and succession planning

Increasing team flexibility

Some scepticism over value of PDF and our ability to

support career aspirations

Evidencing skill levels has proven time consuming, but is a

one off activity – consensus is it’s worth investing in

Evidencing transferable skills is difficult

Heat map / Pie Charts (management information is beautiful)

Slide 14 © IDBS 2012

Example heat map & pie charts

COMPETENCY SKILL Job Role 1 Job Role 1 Job Role 2 Job Role 2

Competency 1

Competency Skill 1 2 2 3 4

Competency Skill 2 1 2 2 2

Competency Skill 3 1 1 1 2

Competency Skill 4 2 2 2 3

Competency Skill 5 3 1 2 4

Competency Skill 6 1 3 2 4

Competency Skill 7 1 1 2 4

Competency 2

Competency Skill 1 2 1 2 2

Competency Skill 2 2 2 3 4

Competency Skill 3 1 1 3 4

Competency Skill 4 0 1 2 3

52%

30%

18%

Test Team Competency

Attaining

Not Meeting

Exceeding

51%

25%

24%

Tester 2 Competency

Attaining

Not Meeting

Exceeding

Slide 15 © IDBS 2012

Closing thoughts from our team…

“I find that it opens the discussion about what your line manager

expects of you in your role, and allows me to identify areas where I

could improve prior to having that meeting. It also provides a good

backing for the appraisal, which is pretty invaluable, as you tend to

forget the things you’ve done throughout the year”

“It provides a structure in line with the company organisation to allow us

to see the different paths for professional development; giving a clear

yet open idea about career progression within the team”

“I think it is useful. The only downside is that the number of fields does

make it lengthy to complete, it seems some fields could be consolidated

or perhaps a few removed that don’t seem relevant to a testers role”

Slide 16 © IDBS 2012

Some things we had to think about (Q&A)