digital qa summit - karen n. johnson - testing explained

71
TESTING EXPLAINED AN OVERVIEW OF SOFTWARE TESTING Karen N. Johnson Digital QA Summit

Upload: karen-n-johnson

Post on 15-Jul-2015

436 views

Category:

Technology


1 download

TRANSCRIPT

TESTING EXPLAINED AN OVERVIEW OF SOFTWARE TESTING

Karen N. Johnson Digital QA Summit

Do you find yourself needing to “check” something in production immediately after go live? Do you

need to investigate how a page or a campaign looks in multiple browsers or on a mobile device?

Without realizing it, you may be performing the role of a software tester in addition to your primary

role. What is a software tester?  In this session Karen offers 1) a brief overview of the software

testing profession, 2) a summary of different types of testing, 3) a list of resources to learning more

about testing. If testing is part of what you or your team need to juggle to “make it happen” come

attend this session.

 

About this talk …

Testing Explained © Karen N. Johnson 2014

•  Software Test Consultant •  Published Author (Beautiful Testing) •  Co-founder of WREST: Workshop on Regulated Software Testing •  Website: www.karennicolejohnson.com or www.karennjohnson.com •  Twitter: @karennjohnson

Karen N. Johnson

Testing Explained © Karen N. Johnson 2014

Do you get asked…

Hey, can you “just” check this out?

Testing Explained © Karen N. Johnson 2014

Can you make sure all the parts to the campaign “look good?”

Testing Explained © Karen N. Johnson 2014

But do you feel unsure how to test?

Testing Explained © Karen N. Johnson 2014

to software testing Testing Explained © Karen N. Johnson 2014

Testing is more than checking things out.

Testing Explained © Karen N. Johnson 2014

Software testing is a profession with it’s own “world.”

Testing Explained © Karen N. Johnson 2014

When you already have a fulltime job and a profession?

But how do you immerse yourself in another world?

Testing Explained © Karen N. Johnson 2014

Here’s look at some of the essential topics in the software testing community.

Waterfall & Agile Development

Testing Explained © Karen N. Johnson 2014

¨  There are many different ways to develop and test software.

¨  In past years, the Waterfall method was the most common software development methodology.

¨  In current times, an Agile development process is now more common.

Waterfall Development

Testing Explained © Karen N. Johnson 2014

Agile Development

Testing Explained © Karen N. Johnson 2014

The Agile Manifesto ¨  We are uncovering better ways of developing software by

doing it and helping others do it. Through this work we have come to value: ¤  Individuals and interactions over Processes and tools ¤  Working software over Comprehensive documentation ¤  Customer collaboration over Contract negotiation ¤  Responding to change over Following a plan

Acronyms: we love them! ¨  ATDD: Acceptance Test-Driven Development ¨  BDD: Behavior-driven development ¨  TDD: Test-driven development

Testing Explained © Karen N. Johnson 2014

Checking vs. Testing

Testing Explained © Karen N. Johnson 2014

¨  Within the profession of testing, there is a long-time discussion known as “checking vs. testing.”

¨  Often less experienced people in testing and/or people who are not professional testers “test” in such a manner that they are “checking” software works.

¨  In these situations, people are not exploring, they are not looking for issues, they are checking to confirm software works versus looking to expose or discover issues.

Checking vs. Testing

Testing Explained © Karen N. Johnson 2014

¨  If you never read an article on testing, if you never read a blog post on checking vs. testing, learn this one concept:

Know when you are checking to confirm software is working vs. testing with the intention of discovering possible issues.

¨  Do not confuse the two approaches (checking vs. testing) as meaning the same thing.

¨  There is nothing wrong with “checking” software or “confirming” that software works but this is not the same as intentionally looking to discover defects.

Tested = Checked + Explored

Testing Explained © Karen N. Johnson 2014

In Elisabeth Hendrickson’s book, Explore It!, she states that neither checking nor

exploring is enough.

“You’re not done testing until you’ve checked that the software meets expectations

and you’ve explored whether there are additional risks. A comprehensive test

strategy incorporates both approaches.”

Exploratory vs. Scripted Testing

Testing Explained © Karen N. Johnson 2014

¨  Within the profession of testing, there is a long-time discussion on the use of scripted vs. exploratory testing.

¨  Let’s look at each format.

Scripted testing: a definition

Testing Explained © Karen N. Johnson 2014

Scripted testing follows a path that is written by the tester

themselves or someone else. The script includes test steps that are

documented. It is expected that the tester will follow the steps as

detailed. The tester’s job in a scripted environment is to follow

each step and report findings (if any).

Exploratory testing: a definition

Testing Explained © Karen N. Johnson 2014

James Bach’s definition: “Exploratory testing is simultaneous learning, test design, and test execution.” This style of testing requires that your brain be fully engaged at all times. Cem Kaner’s definition: “Exploratory software testing is a style of software testing that emphasizes the personal freedom and responsibility of the individual tester to continually optimize the value of her work by treating test-related learning, test design, test execution, and test result interpretation as mutually supportive activities that run in parallel throughout the project.”

Exploratory & Scripted Testing

Testing Explained © Karen N. Johnson 2014

¨  As it turns out, you don’t have to choose one form over another. You could use a blended approach.

Schools of Software Testing

Testing Explained © Karen N. Johnson 2014

¨  Around 2003, several people in the testing field constructed a way to explain different core approaches to software testing. The construct was referred to as the schools of testing.

¨  The schools of testing have been greatly debated over the years. Some people have been so opposed to the construct of the schools concept, they have refused to even discuss the construct or acknowledge the schools as a way of discussing differing views.

Schools of Software Testing

Testing Explained © Karen N. Johnson 2014

¨  In 2003, the schools were defined by Bret Pettichord as:

¤ Analytical School ¤  Factory School ¤ Quality Assurance ¤ Context-Driven School

Schools of Software Testing

Testing Explained © Karen N. Johnson 2014

¨  You don’t have to be involved in the debate.

¨  You don’t have to choose a school.

¨  But it can be helpful to know this “argument” exists in the testing profession.

¨  It can be helpful to know what school a tester identifies with.

The Seven Basic Principles of the Context-Driven School

1.  The value of any practice depends on its context.

2.  There are good practices in context, but there are no best practices.

3.  People, working together, are the most important part of any project’s context.

4.  Projects unfold over time in ways that are often not predictable.

5.  The product is a solution. If the problem isn’t solved, the product doesn’t work.

6.  Good software testing is a challenging intellectual process.

7.  Only through judgment and skill, exercised cooperatively throughout the entire

project, are we able to do the right things at the right times to effectively test

our products.

The Original Quadrants of Testing

Testing Explained © Karen N. Johnson 2014

From Brian Marick’s blog

The Original Quadrants of Testing

Testing Explained © Karen N. Johnson 2014

Support programming vs. critique product

¤  “When it comes to supporting programming, tests are mainly about preparing and reassuring.”

¤  “On the other side of the division, tests are about uncovering prior mistakes and omissions.”

Agile Testing Quadrants

Testing Explained © Karen N. Johnson 2014

From the book, Agile Testing by Lisa Crispin and Janet Gregory

The Quadrants are being reconsidered

Testing Explained © Karen N. Johnson 2014

¨  Gojko Adzic http://gojko.net/2013/10/21/lets-break-the-agile-testing-quadrants/

¨  Markus Gärtner http://www.shino.de/2012/07/30/the-testing-quadrants-we-got-it-wrong/

¨  Elisabeth Hendrickson http://www.slideshare.net/ehendrickson/the-thinking-tester-evolved?ref=http://www.shino.de/2012/07/30/the-testing-quadrants-we-got-it-wrong/

Certification & Standards

Testing Explained © Karen N. Johnson 2014

¨  There are several certification programs available. ¨  There are several boards such as IEEE and ISO, that

devise standards around testing. ¨  And there are testers who are in agreement with

certifications and/or standards. ¨  And there are testers who are strongly opposed.

Types of Testing

Testing Explained © Karen N. Johnson 2014

¨  There are numerous types of testing. Each type has a focal point and serves a different purpose. For example, user acceptance testing vs. performance testing.

¨  In some cases, a type of testing will be more likely to occur at different times during the development process. For example, unit testing vs. system testing.

Black Box Testing

Testing Explained © Karen N. Johnson 2014

¨  Black box testing is one of the most common terms used in testing.

¨  The basic premise is the person testing does not know how the code was written and does not have insight inside the “box” or the system.

¨  Black box testing is testing without a technical understanding of how a program was written. This is why the “box” (or the item being tested) is “black” (because we don’t know what is inside.)

White Box or Glass Box Testing

Testing Explained © Karen N. Johnson 2014

¨  The opposite of black box testing. White box testing means the tester has an inner understanding of how the program was written, this form of testing allows for more targeted technical testing.

¨  Much of the time, black box testing is the type of testing being used.

Testing Explained © Karen N. Johnson 2014

An example … Suppose I am testing the credit card entry part of an ecommerce website. Black box testing: •  Try using an invalid credit card White box testing: •  Review the code •  Learn after three invalid credit card entries, the account is locked •  Enter three invalid attempts check error handling and messaging With black box testing, I might try an invalid credit card once, see an error message and not continue to test because I don’t know about the three attempt condition.

Unit Testing

Testing Explained © Karen N. Johnson 2014

Martin Fowler’s description:

“The common properties of unit tests — small scope, done by the programmer

herself, and fast — mean that they can be run very frequently when

programming.”

Integration Testing

Testing Explained © Karen N. Johnson 2014

Paul Jorgensen’s definition:

Integration is the phase in software testing in which individual software modules are combined and tested as a group. It occurs after unit testing.

Lee Copeland’s definition:

In integration we assemble units together into subsystems and finally into systems. It is possible for units to function perfectly in isolation but to fail when integrated.

System Testing

Testing Explained © Karen N. Johnson 2014

Lee Copeland’s definition:

A system consists of all of the software (and possibly hardware, user manuals,

training materials, etc.) that make up the product delivered to the customer.

System testing focuses on defects that arise at this highest level of integration.

Typically system testing includes many types of testing: functionality, usability,

security, internationalization and localization, reliability and availability,

capacity, performance, backup and recovery, portability, and many more.

UAT or User Acceptance Testing

Testing Explained © Karen N. Johnson 2014

Lee Copeland’s definition:

Acceptance testing is defined as that testing, which when completed

successfully, will result in the customer accepting the software and giving us

their money.

Testing Explained © Karen N. Johnson 2014

An example … Suppose I am testing the purchasing process of an ecommerce website. Unit testing: Build a unit test for credit card expiration date. Integration testing: When the code for credit card processing is integrated with the code that handlings shipping information, I can build an integration test. At the system level, I might build testing from the start of product selection through the entire place an order, credit card and shipping process; a test that brings all the pieces together.

Regression Testing

Testing Explained © Karen N. Johnson 2014

¨  A common form of testing; used to retest to ensure changes in one part of a program have not introduced issues in another part of a program.

The “ilities” of testing ¨  Software testers commonly refer to a series of terms as the

“ilities” of testing.

¨  The “ilities” are often a challenge to test but are often the same items that customers are often most concerned about.

Testing Explained © Karen N. Johnson 2014

The “ilities” of testing ¨  Capability ¨  Reliability ¨  Usability ¨  Security ¨  Scalability ¨  Performance ¨  Installability

¨  Compatibility ¨  Supportability ¨  Testability ¨  Maintainability ¨  Portability ¨  Localizability

Testing Explained © Karen N. Johnson 2014 Source: James Bach, Rapid Software Testing

More types of testing ¨  Stress ¨  Multi-user ¨  Cookies ¨  Compatibility ¨  Developer ¨  Unit ¨  Integration ¨  System ¨  User Acceptance ¨  Installation ¨  API/web services

¨  Black box ¨  White or glass box ¨  Gray box ¨  Functional ¨  Automated ¨  Regression ¨  Security ¨  Data ¨  Exploratory ¨  Scripted ¨  Performance

Testing Explained © Karen N. Johnson 2014

Some “words” around testing

Testing Explained © Karen N. Johnson 2014

¨  Risk

¨  Heuristics

¨  Oracles

But if you only learn one question to ask, let the question be …

What if?

Testing Explained © Karen N. Johnson 2014

•  What if? Is a challenge question.

•  What if? Is the type of question that begs for investigation and

discovery.

•  What if? Does not require technical training or knowledge.

Testing Explained © Karen N. Johnson 2014

During storyboarding and

design is a great time to

use: What if?

Testing Explained © Karen N. Johnson 2014

Testing Explained © Karen N. Johnson 2014

Testers focus on where and how a story can breakdown.

Testing Explained © Karen N. Johnson 2014

Testing Explained © Karen N. Johnson 2014

“State transitions provide a

Bermuda-Triangle-like moment in time

where “things” can go wrong for a user

or system. And once a user or system

is in a bad state, they can be awkwardly

trapped or lost.”

Testing Explained © Karen N. Johnson 2014

State transition

“When a system must remember something about what has happened

before or when valid and invalid orders of operations exist, state-

transition diagrams are excellent tools to record this information.”

Source: Lee Copeland. A Practitioner's Guide to Software Test Design

Testing Explained © Karen N. Johnson 2014

Testing Explained © Karen N. Johnson 2014

Some final advice …

1. You will never find everything.

Testing Explained © Karen N. Johnson 2014

The number of combinations and possibilities is greater than what you can test.

There are more ways to test than there are hours in the day.

2. You will never have enough time.

Testing Explained © Karen N. Johnson 2014

If you have more ideas for testing than time, choose what you will test wisely and get help from your team.

3. You cannot do it all alone.

Testing Explained © Karen N. Johnson 2014

Build testing into the job regardless of who in your agency gets it done. Do not allow testing to be an optional activity.

4. Don’t allow testing to be an extra bill to the client.

Testing Explained © Karen N. Johnson 2014

Note-taking, mind mapping, voice recording: whatever it takes, don’t lose a great idea.

5. Capture your test ideas.

Testing Explained © Karen N. Johnson 2014

6. To find issues, you have to test beyond the simple path.

Testing Explained © Karen N. Johnson 2014

7. Ask questions. Especially ask: “what if?”

Testing Explained © Karen N. Johnson 2014

Email: [email protected] Twitter: @karennjohnson

Karen N. Johnson

Testing Explained © Karen N. Johnson 2014

Checking vs. Testing

Testing Explained © Karen N. Johnson 2014

¨  James Bach, Blog: Testing and Checking Refined, http://www.satisfice.com/blog/archives/856

¨  Michael Bolton, Blog: Testing vs. Checking

¨  http://www.developsense.com/blog/2009/08/testing-vs-checking/

¨  Elisabeth Hendrickson, Explore It!

¨  Ministry of Testing, http://www.ministryoftesting.com/2012/07/mindmaptesting-and-checking/

Exploratory vs. Scripted Testing

Testing Explained © Karen N. Johnson 2014

¨  James Bach, paper: Exploratory Testing Explained (10pp),http://www.satisfice.com/articles/et-article.pdf#page=1&zoom=auto,-55,792

¨  James Bach, video, Understanding exploratory testing (6 minute video), https://www.youtube.com/watch?v=I-ItEKt_N_s

¨  James Bach, Session-Based Test Management (1 page), http://www.satisfice.com/sbtm/

¨  James Bach, What is Exploratory Testing? And How it Differs from Scripted Testing

¨  http://www.satisfice.com/articles/what_is_et.shtml

¨  Cem Kaner, slide set (180pp) A Tutorial in Exploratory Testing, http://www.kaner.com/pdfs/QAIExploring.pdf

¨  Michael Bolton, Resources on Exploratory Testing, Metrics, and Other Stuff, http://www.developsense.com/resources.html

¨  Gregory Mooney, Exploratory vs. Scripted Testing: One or the Other, or Both?, http://blog.smartbear.com/testing/exploratory-vs-scripted-testing-one-or-the-other-or-both/

Schools of Testing

Testing Explained © Karen N. Johnson 2014

¨  Bret Pettichord, Schools of Software Testinghttps://www.prismnet.com/~wazmo/papers/four_schools.pdf

¨  Bret Pettichord, http://www.testingeducation.org/conference/wtst_pettichord_FSofST2.pdf

¨  Cem Kaner, Blog: Schools of software testing, http://kaner.com/?p=15

¨  Context-driven testing, http://context-driven-testing.com/

¨  James Bach, http://www.satisfice.com/blog/archives/134

Quadrants of Testing

Testing Explained © Karen N. Johnson 2014

¨  Brian Marick blog post: http://www.exampler.com/old-blog/2003/08/22/#agile-testing-project-2

¨  Lisa Crispin, Janet Gregory, Agile Testing (book) ¨  Gojko Adzic http://gojko.net/2013/10/21/lets-break-the-agile-testing-quadrants/ ¨  Markus Gärtner http://www.shino.de/2012/07/30/the-testing-quadrants-we-got-it-wrong/ ¨  Elisabeth Hendrickson http://www.slideshare.net/ehendrickson/the-thinking-tester-evolved?ref=http://www.shino.de/2012/07/30/the-testing-quadrants-we-got-it-wrong/

Rapid Testing ¨  Rapid testing is a method, a series of techniques and a class,

all in one taught by James Bach.

¨  James is arguably the most known tester across the globe.

¨  James posts the slides to his Rapid Software Testing class on his website. www.satisfice.com

Testing Explained © Karen N. Johnson 2014

BBST: the Black Box Software Testing ¨  Cem Kaner offers a series of classes taught through slides and

videos.

¨  Cem posts the materials to his classes:http://www.testingeducation.org/BBST/

Testing Explained © Karen N. Johnson 2014