MAURIZIO MANCINI • DIRECTOR OF QA • YELLOW PAGES CANADA • @QAANDPROCESSGUY

How to build a QA Team that matters

in an

Agile or Incremental and Iterative (IID) world.

GIANCARLO BISCEGLIA • SENIOR QA MANAGER • YELLOW PAGES CANADA

Who are we?

• Head of QA for Yellow Pages Canada (YP)

• Also responsible for Software Process Deployment at YP

• 25 years of software engineering experience

Maurizio Mancini

Giancarlo Bisceglia

• Senior QA Manager for Yellow Pages Canada (YP)

• 15 years of software engineering experience

About Our Team

• YP has ~ 2300 employees and ~ 1 Billion in Revenue

• Our team of 100 QA are embedded into 15 development teams

• Our team is geographically disbursed in Montreal (4 locations) and in Toronto (2

locations)

• Our team works on projects using all types of software development process

(Agile, Incremental & Iterative (IID), Waterfall)

Agenda

• Yellow Pages Yesterday and Today

• QA “State of the Nation” from our perspective

• How to hire and build a QA Team that will succeed in any SDLC

• Agile Testing – What’s that about?

• Test Automation – Fundamental Building Block

• The Testing Pyramid – Where does QA fit in?

• Different Layers of Testing

• Where else does QA fit in an Agile or IID World?

Yellow Pages Canada

• 3 years ago no real QA Team existed

• A few scattered QA souls left to fend

for themselves

• A lot of consultants

• Almost all testing was manual, very

little automation

• A lot of “End to End” testing

• Most QA was done by BA’s and The

Business

Yesterday Today• Team of 100 QA Engineers, Test

Automation Developers, and QA

Analysts

• A full QA practice that uses common

tools and QA approaches

• On average our testing is 50-50

Automated/Manual

• QA Teams focus on System Testing and

BA/Product Owners focus on UAT

• QA teams are Embedded with the Dev

Teams

What do our Teams Test?

Releases in 2014

QA “State of the Nation”

• Over the last 30 to 40 years, the QA software industry has gone from being a

“nice to have”

“a must”

“developers will do it all” in the Agile world

QA “State of the Nation”Like all extremes, we believe that “pendulum of sensibility” is starting to settle in

the middle

What is the role of QA in today’s

Agile or IID world?

The Death of QA… One of the heated debates in the industry is that QA is dead with Agile becoming

the predominant software development process…

We like to say that Agile has finally brought balance to the QA world…

Agile has brought balance to the QA world

More developers than testers, too much testing,

not enough time for QA to test everything…

Agile has helped rebalance the inequities that existed when it came to testing

software….

Agile has also helped reinforce that you cannot just

throw the code over the fence!!

Agile has brought balance to the QA world

• Agile has helped refocus development teams on the importance of testing.

• Agile has re-asserted what Software Engineering 101 has said all along…

Unit Testing is a MUST!

TESTING PYRAMIDTraditional Goal

End to End GUI Tests

Functional Tests

Integration Tests

Unit Tests

GUI Tests

Functional Tests

Integration Tests

Unit Tests

Q

A

DEV

UNIT Testing is not the

Donald Knuth

He once warned in a letter to one of his fans,

"Beware of bugs in the above code; I have only proved it correct, not tried it."

Knuth has been called the "father of the analysis of algorithms"

‘To tweak what Knuth said,

"Beware of the above code. I have only tested that it works. I haven't actually tried it.“’

http://jjinux.blogspot.ca/2012/05/unit-tests-dont-find-bugs-death-of-qa.html

Shannon Behrens

UNIT Testing is great but you need other types of testing!

Where do we go from here?

Let’s build that QA Team!

HiringWhich Roles should you be hiring for the current world of Software Development

that includes Agile, IID and sometimes waterfall?

• QA Engineer

• Also known as QA Specialist, QA Developer, Engineer in Test

• Test Automation Engineer

• QA Analyst

Hiring

What should the mix be?

• QA Engineer (75%)

• Also known as QA Specialist, QA Developer, Engineer in Test

• Test Automation Engineer (10%)

• QA Analyst (15%)

Candidate Profiles

Early Career – Look for people that think they want to be a developer.

Mid Career – Look for someone that may have coded before and has

discovered the love of testing.

Late Career – Look for someone with a passion for testing and breaking

software.

QA Engineer

QA EngineerAlso known as a QA Developer, QA Specialist, or Engineer in Test

Key skills and qualities you should be looking for:

Coding and/or scripting – Experience in Record and Playback not good enough

API and Web services background – ex: REST and SOAP

SQL and database background

Networking background – Should have the basic knowledge of networking

Linux/UNIX and/or Server background

Passion for testing

Solid understanding of QA and how it differs from Dev

Understands that there is more to testing than “executing a test”

Someone that likes to get under the hood QA Mechanic

Test Automation Developer

Test Automation DeveloperThis will be the person that will put in place your test automation framework

• Helps to implement test automation best practices for QA Engineers

• Will support the team as they use the framework in their project

• Helps conduct QA Teams code reviews

Test Automation DeveloperKey skills and qualities you should be looking for:

This is a pure developer that also has a passion for testing

Object Oriented coding skills

Scripting language skills

Deep understanding of QA Automation Tools

API and Web services background

SQL and database background

Networking background

Linux/UNIX and/or Server background

Solid understanding of QA and how it differs from Dev

Understands that there is more to testing than “executing a test”

Someone that likes to get under the hood QA Mechanic

QA Analyst

QA AnalystThis is your manual tester

• This is the person that will also drive Exploratory Testing

• This resource can play other roles – More on this later

QA Analyst

API and Web services background

SQL and database background

Networking background

Solid understanding of QA and how

it differs from Dev

Understands that there is more to

testing than “executing a test”

Has a natural tendency to want to

understand how something works

Key skills you should be looking for:

Passion for testing

Not just a GUI tester, someone that likes

to get under the hood QA Mechanic

Drives Technology in his/her home

(Hacks home technology)

Examples:

• Uses Opensource firmware on their

home router

• Pushes usage of his/her smart phone

beyond everyday usage

• Able to clean a computer, infected with

a virus, without the help of a tool

Hiring Team

Hiring Team

• You should form a Hiring Team

• Formulate a hiring plan – Who you are looking for, what are the criteria

• Each member of the team focuses on a set of skills to look for

• Have a battery of questions that the team uses. For example:

Linux/UNIX

SQL

Networking

Database

Coding

Actually test something live with bugs

• This will ensure that you are hiring consistently and that you build that awesome

QA Team

The Interview• QA Engineers and Test Automation Developers should be interviewed by a

Developer too

• Questions must include coding questions

• Your questions must include “white boarding”

• Include multi-dimensional questions

• Multi-System

• Many Technologies

• System Architecture Questions – Are they able to visualize it?

• Goal is to reveal if the candidate has a QA Mechanic mentality

• Do they have a QA mindset? More on this later.

• Do they have a desire to break software?

• Are they technically sound in the technical skills that you are looking for?

• Do they have the passion for QA?

QA Mechanic

Team Structure

Embed QA with the Dev teams!!

Central QA but with an Embedded Focus

• Common set of Automation Tools

• Shared Automation Framework

• Common standards

• Common Hiring practices

• Able to shift a QA resource when that resource wants to advance their career

• Helps to avoid unproductive Agile Teams

Benefits of a Central QA Team

Benefits of an Embedded QA Team

• QA involved in the daily team interactions

• QA testing early and often

• QA helps to keep a process focus within the team

• Get the benefits of a true agile team

How do we approach Testing?

Agile Testing

Agile Testing

TESTING PYRAMIDTraditional Goal

End to End GUI Tests

Functional Tests

Integration Tests

Unit Tests

GUI Tests

Functional Tests

Integration Tests

Unit Tests

Q

A

DEV

UNIT Testing is great but you need other types of testing!

• Integration Testing

• Stubbing external dependencies to facilitate UI automation

• Exploratory Testing

• Performance Testing

• UAT using BDD’s

Unit and

Integration

Stubbed

Functional Testing

System

Integration

Performance UAT

Owner Dev QA Engineers QA Engineers

QA Analysts

QA Engineers Product Owner

QA Analyst

Type White Box Black Box Black Box Black Box Black Box

Environment None Local Deployment

or CI Server

“Prod-like”

QA

Environment

Performance

Environment

Staging or

Training

Environment

Complexity

Focus

Low Level

Correctness

Maintainable

Negative

inputs

Mid Level

Extreme cases

Negative tests

UI Permutations

Failure Tests

Mid Level

Positive Tests

Exploratory

tests

System

Integration

High Level

Scalability

Robustness

Reliability

High Level

Business

Flows

Layers of Testing

Test Automation

What tools we chose

• Many paid tools are based on open source libraries nowadays.

• Don’t think that because you pay for a tool, it’s best of breed.

• Beware of Scriptless tools. You still need people that can code reusable classes. Yes you get a library but how complete is it?

• Record and playback just encourages end to end testing.

• Automation project needs to be run like a dev project, including code reviews

• Invest in a custom framework which uses libraries or frameworks that can be swapped out.

Don’t tie yourself to one technology

Leverage Continuous Integration

Ideal for helping Product Owners and QA Analysts define business requirements in an English like syntax that can be automated.

Limitations of BDD’s• Good to use for top of pyramid tests, flows that don’t change often• Not a replacement for unit testing or automated functional tests• Best if it’s a layer you add on top of your existing automation

framework• Syntactically discourages code reuse• Test Steps are expensive to maintain

BDD Usage in AutomationBehavior – Driven Development

• Time for Automation is considered expendable

• Environment Deployment Automation

• Coding Standards for the software being developed

Test Automation Challenges

A Different Way of Thinking

DON’T IGNORE WHAT’S IN BETWEEN

TEST WEB UI, MOBILE APPS, API, DATABASE, REPORTS INDEPENDENTLY

Break up the Test Effort

• Most expensive testing you can do

• Too late in the Cycle

• Hard to coordinate and manage

• Don’t make this your primary method of testing

• Use it as a last resort

End to End Testing

• No more excuses, QA unblocks itself

• Think outside the box, to find ways to test partial implementations (UI not implemented)

• Test system components in isolation using stubs

• Test Early, Test Often

• Push for continuous integration

• Deploy software and maintain QA environments

• Work with DevOps on deployment Automation

The New QA Mindset

Renovating an existing QA Team for an

Agile or IID world!

Skills Assessment

Skills Assessment

What you need to improve

NOW

What you have

today

What you need

tomorrow

Now What?

Now What?

• Train Existing Staff – On the Job, Classroom, Self Study

• Hiring new people – QA Engineers, Test Automation Developers

• New Approaches to Testing – Testing Pyramid

• Embed your team(s) if you haven’t already

• Change in Roles and New Roles

How do you present your new approach to your existing staff?

Here is a sample presentation …

Quality is Free!

If Developers created perfect code then you

wouldn’t need any testing…

What costs is the incorrect

implementation of a software requirement…

QA is there to make developers and

development better.

We don’t ensure better software – we

enable developers to build better

software.James Whittaker

Is there a role for QA in an Agile World?

Yes!!!

TESTING PYRAMIDTraditional Goal

End to End GUI Tests

Functional Tests

Integration Tests

Unit Tests

GUI Tests

Functional Tests

Integration Tests

Unit Tests

Q

A

DEV

What is the difference between QA and QC?

Quality Assurance – Is focused on the development, implementation and

measurement of software processes in an organization.

Focused on Defect Prevention

Quality Control – Is focused on the comparison of a product against the

requirements and ensuring that the requirements have been met.

Reactive approach focused on defect detection and removal

LESS

MORE

So where does QA fit in this new way of

thinking?

QA people should do more QA and less QC!

QA Coaching Role

QA can play a vital role coaching the

Development Team on how to test…

…so that Developers can create better

Unit and Integration tests.

Automated Tests

QA can play a role in real automation…

Not Record and Playback…

But Manual Testing is still important…

Automated Tests are great but

look around, it is still humans

using the software…

Exploratory Testing is

essential… and QA is key in

this role!

Let’s not forget System Integration Testing

Most systems today integrate

with many other systems

SIT is essential…

and QA is key in this role!

User Community of TestersIf your systems and applications allow it , then why not create a

User Community of real testers?

Have QA manage your user

community of testers!

Employees

Early Adopters

There is plenty of testing to go around!

QA Engineers at the Core of QA Team Agile Testing

Key Takeaways

New QA Mindset Use of Automation must be at the Core of your QA

Key Takeaways

Thank you!

MAURIZIO MANCINI • DIRECTOR OF QA • YELLOW PAGES CANADA • @QAANDPROCESSGUY

GIANCARLO BISCEGLIA • SENIOR QA MANAGER • YELLOW PAGES CANADA