testing trends

Stuart Reid PhD, FBCSSTA Consulting Inc.

54, Seolleung-ro 100-gilGangnam-gu

Seoul

([email protected])

© STA Consulting Inc. 2015

Testing TrendsJune 2015

• Agile• Internet of Things• Risk-Based Testing & Standards• Crowd Testing (& Localization)• Offshore Testing• Gamification

Scope

• World Quality Report 2014-15– 93% use agile for some projects in some way, but…– 36% of testing is done in agile projects– 55% struggling with test automation– 61% feel their agile test approach is wrong– Majority still use test professionals– 35% don’t have enough testing skills in their teams

• Continuing time to market pressures suggest agile• Agile co-location needs are making offshoring less likely• Shortened test times increase demand for risk-based

testing

Agile & Testing – Still Some Way to Go

• Continuous Integration– fast feedback on check-in

• Customer Engagement– talk to the customer – all the time!

• 100% Unit Regression Testing– know if you break something

• Test-driven Development– assured 100% coverage

• Continuous deployment– try it for production in DevOps

Cherry Picking the Best Agile Practices

THESE TWO ARE ‘MUST-HAVES’

Evolution of the Internet of Things (IoT)

Connected People & Locations

Mobile2015

~10 Billion Phones

Connected People

WWW2000

~360 Million Users

Connected Everything

Internet of Things2020

~212 Billion ‘Things’(up to US$5 Trillion)

Internet of Things – A Simple Model

DATA COLLECTORS (INTERNET)

DATA/ANALYTICS (BIG DATA)

(M2M/P2P/M2P) APPS

STATIC

OR

MO

BILE

THINGS (SENSOR & ACTUATORS)

SENSOR & ACTUATOR NETWORKS (SANS)

Internet of Things – Data & Control





(M2M/P2P/M2P) APPS

STATIC

OR

MO

BILE

DATA

CON

TRO

L

Internet of Things – Certified Things & SANS





(M2M/P2P/M2P) APPS

STATIC

OR

MO

BILE

The testing of individual ‘Things’ and ‘SANS’ is the responsibility of the suppliers – they should meet defined interface standards

with corresponding certification.

Internet of Things – Fuzz Testing





(M2M/P2P/M2P) APPS

STATIC

OR

MO

BILE

Use high volume, automated Fuzz Testing based on mutated protocols and test

assertions specific to the data collector.

We must ensure that only the required (minimum) set of (valid) data is passed upwards

– and that data is secure.

Big Data – Test Data = Real Data?





(M2M/P2P/M2P) APPS

STATIC

OR

MO

BILE

Testers of Apps that are dependent on Big Data (especially real-time data) may find the creation of test

data sets impractical and be forced to use real data, potentially opening up problems with data privacy.

Internet of Things – Analytics & MBT





(M2M/P2P/M2P) APPS

STATIC

OR

MO

BILE

For complex Apps based on data analytics we will need to build system models as the basis of

model-based testing and require Specialist Testers to define testable risk profiles based on them.

MBT Tool

Risk-Based Model-Based Testing for Big Data

TESTEXECUTION& CHECKING

SpecialistTest Modeller

MODELRISK

PROFILE

TEST MODELCREATION

TESTCASES

(millions)

TESTGENERATION(high volume)

TESTMODEL

(& ORACLE)

SYSTEM REQUIREMENTS

RISK PROFILEGENERATION

Domain-SpecificTest Strategist

Data Scientist

• Risk-Based Testing IS about generating a far more cost-effective test strategy that saves money and maintains quality

• Risk-Based Testing IS NOT about prioritization and an excuse to not test some features

Risk-Based Testing (and Standards)

HIGH RISK ðMORE TESTING

HIGHRISK

MoreTesting

Less TestingLOWRISK

LOW RISK ðLESS TESTING

RISK-BASED TESTING è SAME QUALITY & LOWER COSTS

29119Standards

Internet of Things – Release Testing





(M2M/P2P/M2P) APPS

STATIC

OR

MO

BILE

A/B Testing may become the normal pre-release approach for non-critical applications, with users

unaware that they are part of a test

Internet of Things – End User Testing





(M2M/P2P/M2P) APPS

STATIC

OR

MO

BILE

For complex P2P and M2P Apps the User Testing will require such a variety of ‘test’ environments & users that it will be beyond the internal capabilities

of most test organizations…hence crowd testing

Crowd Testing

Client with App

Crowd Testers (with NDAs)

App with Testing Needs

App with Testing

Instructions

Test ResultsTest Report

Crowd TestingService

$

$$$$$ $$$$

Crowd Testing & Localization

Crowd TestingLocalization

Service

Korean Client with App Localized to

English

Korean Crowd Testers

English Crowd Testers

English Client with App Localized to

Korean

Why Localize to Korea?• 3rd largest market globally

for mobile app revenue• Biggest spenders on digital

games worldwide

Offshore Testing

time

% offshore testing

COST SAVINGS •REPETITIVE TASKS•AUTOMATION

INCREASINGPAY

PARITY

•BETTER COMMS•LOSS OF

ONSHORE SKILLS

20NOW

TIME TO MARKET

Gamification of Testing

Scoring• Measuring Progress• Against tasks or others• Special Challenge• Completion Bonus

Rewards• Hidden Treasures• Higher Coverage• Coverage Measures

Competition• League Tables• Peer Pressure

Teams• Encouragement• Sharing skills

Virtual Testing• Testing in Games and

Virtual Worlds• User Interface

Testing

Fast Feedback• Exploratory Testing• Validating Defects

TEST

• Agile• Internet of Things• Risk-Based Testing & Standards• Crowd Testing (& Localization)• Offshore Testing• Gamification

Conclusions

Thanks for listening J

Any Questions?

testing trends

Software

big data test data

data analytics

real data

realtime data

data privacy

creation of test data

internet of things iot

testing of individual