gla testing presentation by test partners ltd v1

Software Testing

PresentationOctober 2017

Introduction

• Founded in 2001

• Test Lab in Central London

• Specialise in testing web-based applications and mobile applications

• Functionality, compatibility and accessibility

• 2000+ projects

• 500+ clients

Software Testing Presentation - October 2017


http://www.bt.com/

http://www.bt.com/

Today’s Topic

Point 8 of the local government digital service standard:

“Be able to test the end-to-end service in an environment similar to that of the

live version, including all common browsers and devices.”

http://localgovdigital.info/localgov-digital-makers/outputs/local-government-

digital-service-standard/standard/8/


http://localgovdigital.info/localgov-digital-makers/outputs/local-government-digital-service-standard/standard/8/

Today’s Topic

I will talk about:

• Comparison of the benefits and shortcomings of manual testing and

automated checking.

• Context-driven testing.

• Scripted and exploratory testing.

• Client-side test environments.

• Online services such as BrowserStack and Perfecto.

• Choosing the devices and browsers to test.

• Approaches to manual regression testing.

• Test data.


Today’s Topic

I will not talk about:

• Unit testing.

• Automated checking (with one small exception).

• Security and load testing.

• Infrastructure and test environments.

• Development approaches such as TDD and BDD.

• Accessibility testing (but perhaps another time).


Key Requirements

From the standard:

• Test not only during development but throughout the life of the service.

• As services will be part of a continual improvement review, it is important to

be able to test changes.

• It is important that the service is tested on the devices and internet browsers

that we know are in use within the user base.


Comparison of the benefits and shortcomings of manual testing and automated

checking


Criterion Manual Testing Automated Checking

Objective Investigate the application to verify expected behaviours, identify unexpected behaviours and identify any other information of interest to stakeholders.

Verify expected behaviours.


checking



Speed (initial test) Slow, but produces results immediately from the start.

Slower than manual testing due to scripting time. Typically no results for some time.


checking





Speed (subsequent tests)

Much the same as the initial test.

Much faster than the initial test.


checking





Speed (subsequent tests)

Much the same as the initial test.

Much faster than the initial test.

Ability to accommodate change

Exploratory testing easily accommodates change. Scripted tests need to be updated.

Poor - requires script maintenance.


checking



Thoroughness (test scenarios)

Typically includes both simple and complex scenarios.

Typically limited to simple "happy path" scenarios.


checking






Thoroughness (test data)

Very limited. Can check a huge number of data variations.


checking






Thoroughness (test data)

Very limited. Can check a huge number of data variations.

Performs "what if…" tests

Exploratory testing does but scripted tests typically do not.

No. Automated tools cannot ask questions or apply intelligence.


checking



Notices behaviour that was not the focus of the test.

It is an important feature of exploratory testing. People doing scripted tests could but typically do not.

No. Automated checks only verify the assertions they have been programmed to do.


checking



Notices behaviour that was not the focus of the test.

It is an important feature of exploratory testing. People doing scripted tests could but typically do not.

No. Automated checks only verify the assertions they have been programmed to do.

Ability to test without design documentation

No problem at all for exploratory testing. Scripted tests require design documentation.

Automated checks require design documentation.

Context-driven Testing

Your context includes factors such as:

• Objectives

• Timescale

• Available budget and other resources

• Risk

• Scope and nature of changes

• Regulatory environment

• Lifespan

• Constraints

• Organisational culture


Context-driven Testing

• Start by identifying your context.

• Consider how your context should influence your approach to testing.

• Be aware when your context changes. Do you need to change your testing?

• Be wary of “best practices” that are implemented without consideration –

these cause:

• Waste.

• Risk.

Sounds obvious but it rarely happens in practice. People tend to do what they

have always done.

http://context-driven-testing.com/


http://context-driven-testing.com/

Different Types of Testing

• Give you different things.

• Automated checking is not better than manual testing. It’s totally different.

• Exploratory testing is not necessarily better than scripted testing (although it

is for most purposes).

• Choose the types of testing that are most appropriate in your context.


Scripted and Exploratory Testing

• Scripted testing focuses on verifying the documented requirements.

• Cannot test without documented requirements.

• Vulnerable to intentional undocumented design changes.

• Exploratory testing focuses on what users can do to the system.

• Verifies expected behaviours.

• Identifies unexpected behaviours.

• Little or no design documentation required.

• Can accommodate incomplete functionality and undocumented design

changes.




Criterion Scripted Testing Exploratory Testing

Skill level required Minimal. Anyone can do it but requires skill to do well.





Speed Very slow. 2 to 3 times faster.






Good for finding bugs Not if your developers are any good.

Finds heaps of bugs.








Records test coverage Yes. Can do.









Provides detailed audit trail

Yes. Can do but adds overhead.









Provides audit trail Yes. Can do but adds overhead.

Fun level Tedious beyond words. Highly engaging.

Client-side Test Environments

If you want accurate, repeatable results:

• Test on dedicated “clean” test machines i.e. new installation of the operating

system and nothing else.

• Avoid emulators.

• Avoid virtual machines such as Parallels, VMWare and VirtualBox.

• Don’t use tools that run multiple browser versions in one browser.

• Check the provenance of all sources of browsers and plug-ins. Build a library

while new versions are still available.

If you value speed, cost and convenience over accuracy and repeatability, feel

free to ignore all the above.


Client-side Test Environments

SaaS services include:

• BrowserStack - https://www.browserstack.com/

• Perfecto Mobile - https://www.perfectomobile.com/

Advantages

• Cheap compared with buying devices.

• Easier than maintaining a software library.

• Probably ok for automated checking.

Disadvantages

• My personal experience was that they were terrible for manual testing.


https://www.browserstack.com/

https://www.perfectomobile.com/

Choosing Which Browsers and Devices to Test

The standard says “analyse current user behaviour to determine the device and

browser types in use”.

• OK up to a point. But if the website does not work well with that browser or

device, its browser or device usage statistics will be disproportionally low.

• Use other sources of statistics but:

• Check the demographics. Do they match yours?

• Check the methodology. Is it valid?

• Check the time period the statistics were collected over.

• A narrow time period may not be statistically valid.

• A wide time period will give emphasis to older browsers.

• I usually advise one week to one month.


Choosing Which Browsers and Devices to Test

There are plenty of free sources but the best data usually has to be paid for:

• http://marketshare.hitslink.com/

• https://mixpanel.com/trends/#report/ios_frag/from_date:0,to_date:0

• http://www.w3schools.com/browsers/browsers_stats.asp (techie-biased)

• https://www.fiksu.com/resources/ios_trackers

• http://opensignal.com/reports/2015/08/android-fragmentation/

• https://www.browserstack.com/test-on-the-right-mobile-devices

• http://gs.statcounter.com/

• https://www.appbrain.com/stats

• https://david-smith.org/iosversionstats/


http://marketshare.hitslink.com/

https://mixpanel.com/trends/#report/ios_frag/from_date:0,to_date:0

http://www.w3schools.com/browsers/browsers_stats.asp

https://www.fiksu.com/resources/ios_trackers

http://opensignal.com/reports/2015/08/android-fragmentation/

https://www.browserstack.com/test-on-the-right-mobile-devices

http://gs.statcounter.com/

https://www.appbrain.com/stats

https://david-smith.org/iosversionstats/

Manual Regression Testing

There is no “right” answer. Possible strategies include:

• Always do the same tests with the same test data.

• Always do the same tests with different test data.

• Do different tests each time.

• A mixture of these.

• You stand the best chance of finding a bug if you do something you have not

done before.

• You might have a core set of tests, but introduce as much variation as you

can.


Manual Regression Testing

More options:

• Execute all the same “small” tests that you initially did.

• Very time-consuming.

• Potentially wasteful.

• Execute a subset of the same “small” tests that you initially did.

• Saves time but is potentially risky.

• Write a few “mega” tests that touch all the important features and areas.

• Efficient. Frees up time to do more targeted testing.

• One bug can block all subsequent tests.

• Complex scenarios can make fault diagnosis difficult.


Test Data

When entering data in forms:

• Don’t just “bang on the keyboard”.

• Create data that helps you identify if a bug has occurred.

• Enter unique data wherever possible.

• Make the data self-describing.

• Keep a record of the data you enter, with time and date stamps.

Benefits

• Wherever or whenever you see a piece of data, you know where and when

you entered it.

• Corrupted and misplaced data is easily identified.


Test Data

Example of typical data used when registering accounts for 10 users of an

e-commerce system.

Use Excel’s concatenation function to automatically create more complex data.

Automatic data creation reduces probability of errors.


Date Time Title First Name Last Name Email Address 1 Address 2 Address 3 Town County

12 June 2017 09:37 Mr first1 last1 [email protected] Address1-1 Address2-1 Address3-1 Town1 County1

12 June 2017 10:12 Mrs first2 last2 [email protected] Address1-2 Address2-2 Address3-2 Town2 County2

12 June 2017 10:18 Miss first3 last3 [email protected] Address1-3 Address2-3 Address3-3 Town3 County3

13 June 2017 14:34 Ms first4 last4 [email protected] Address1-4 Address2-4 Address3-4 Town4 County4

13 June 2017 16:05 Dr first5 last5 [email protected] Address1-5 Address2-5 Address3-5 Town5 County5

13 June 2017 16:17 Mr first6 last6 [email protected] Address1-6 Address2-6 Address3-6 Town6 County6

13 June 2017 17:01 Mrs first7 last7 [email protected] Address1-7 Address2-7 Address3-7 Town7 County7

14 June 2017 10:33 Miss first8 last8 [email protected] Address1-8 Address2-8 Address3-8 Town8 County8

14 June 2017 12:55 Ms first9 last9 [email protected] Address1-9 Address2-9 Address3-9 Town9 County9

14 June 2017 13:42 Dr first10 last10 [email protected] Address1-10 Address2-10 Address3-10 Town10 County10

mailto:[email protected]










Steve Green

Managing Director

Test Partners Ltd

68-72 Queen Victoria Street, London EC4N 4SJ

020 3002 4176 (direct)

0800 612 2780 (switchboard)

07957 246 276 (mobile)

020 7692 5517 (fax)

Skype: testpartners

Twitter: testpartners

http://uk.linkedin.com/in/stevegreen2

[email protected]

www.testpartners.co.uk

Contact


http://uk.linkedin.com/in/stevegreen2


http://www.testpartners.co.uk/

gla testing presentation by test partners ltd v1

Data & Analytics