special interest group in software testing (sigist) presentation sydney... · 2017-11-29 ·...
TRANSCRIPT
Special Interest Group in Software Testing (SIGiST)
Tue 30 Nov
Sydney
Leanne Howard, Steve Toms
Copyright © ANZTB
Please note that this presentation will be available tomorrow on the ANZTB website on the downloads page -http://www.anztb.org/downloads.php#SydneySIGiST
Please be aware that ANZTB may be taking photos of this event for marketing and social media purposes. If you do not wish to be included in the photos please notify the photographer.Thank you.
Copyright © ANZTB
5:30pm Welcome
5:40pm The Digital Tester by Andrew Simister
6:15pm Networking & Refreshments
6:45pm Continuous Testing: Preparing for DevOps by Susmit Bhattacharya
7:20pm Close
Copyright © ANZTB
Copyright © ANZTB
ANZTB is a non-profit organisation and a member of the International Software Testing Qualification Board (ISTQB®).
ANZTB aims to support, improve and advance the software testing profession within Australia and New Zealand.
Australia New Zealand Testing Board
Copyright © ANZTB
Team ANZTBAssociate Members
• Anne Carter (Partner Program) – Adelaide
• Ronak Panchal (SIGiST Co-ordinator, Tools Admin) – Auckland
• Veronica Belcher – Brisbane
• Nathan Bligh (Tertiary Liaison) – Canberra
• Roan O’Connor – Perth
Honorary Members
• Chris Carter – Sydney
• David Hayman – Auckland
Board Members
• Ian Ross (Chair) – Christchurch
• Marie Walsh (Vice Chair , Marketing Coordinator) – Brisbane
• Nigel Saunders (Treasurer, Governance Officer) – Auckland
• Graeme Mackenzie (Accreditation Chair, ISTQB Voting Representative) – Wellington
• David Fuller – Sydney
• Stanley Johannes – Christchurch
• Leanne Howard (Social Media) – Sydney
• Sarah Benstead (Exam Chair) – Perth
• Steve Toms – Sydney (Training Provider Liaison)
Copyright © ANZTB
https://twitter.com/ANZTB
https://www.facebook.com/ANZTB/
Company: https://www.linkedin.com/company/anztbGroup: https://www.linkedin.com/groups/2260082
http://bit.ly/ANZTBFlickr
http://anztb.org
Tweet now #ANZTBSIGIST and tag @anztb
Copyright © ANZTB
ANZTB
SIGiSTs
Conference
Assist ISTQB international
working groups
ISTQB Partnership
Program
Training Provider
Accreditation
Support the development of
ISTQB syllabi
Create & run exams
Certification of software testers
What we offer
SIGIST
Copyright © ANZTB
Auckland
Wellington
Sydney
Melbourne
Christchurch
Brisbane
PerthAdelaide
Canberra
Please speak to your facilitator if you would like to get involved.
• SIGiSTs are sponsored by ANZTB• Discuss white papers, new trends in
Testing, share information and networking
• About 30 SIGiSTs in a year• Presentations on the ANZTB website
Copyright © ANZTB
When: 1 June 2018Where: Hyatt Hotel Canberra, Australia
This one-day event will feature local and international testing experts, discussing the latest advances in the profession.
For more details and registration visit www.anztb.org in the coming months
ANZTB Test 2018‘Tailoring Testing for Success’
Copyright © ANZTB
Certification Paths
Copyright © ANZTB
Additional Certifications Offered by ANZTB
ASTQB* Certified Mobile Tester (CMT)Digital disruption is happening now, do you have the right testing skills?
The ASTQB* Certified Mobile Tester qualification provides learning for any level of testing professional working on or intending to work in mobile projects.
You can find the ASTQB* Certified Mobile Tester syllabus and sample exam on the ANZTB website: http://anztb.org/downloads.php#ASTQB-Mobile
GTB* Certified Usability TesterDoes your product delight your customers? Do your customers find your product easy to learn and usable?
The GTB* Certified Usability Tester qualification is aimed at anyone involved in software testing who wishes to broaden their knowledge of usability and accessibility testing, or anyone who wishes to start a specialist career in usability testing.
You can find the GTB* Certified Usability Tester syllabus and sample exam on the ANZTB website: http://anztb.org/downloads.php#GTB-Usability
*GTB = German Testing Board*ASTQB = American Software Testing Qualifications Board
Copyright © ANZTB
Certified Testers
Over 15,500 certified testers in Australia and New Zealand.
0
2000
4000
6000
8000
10000
12000
14000
16000
18000
2004 2005 2006 2007 2008 2009 2010 2011 2012 2013 2014 2015 2016 2017
Tes
ters
Year
ISTQB Certified Testers in ANZ region
ISEB CT-FL CTFL-AT CTFL-MBT CTAL-TA CTAL-TM CTAL-TTA CTAL-ST CTEL-TM-MTT CTEL-TM-OTM CTEL-TM-STM
Copyright © ANZTB
ANZTB and ISTQB® Partner ProgramFor details visit: http://anztb.org/aboutus.php#PartnerProgram
Copyright © ANZTB
ANZTB Accredited Training Providers
Copyright © ANZTB
ISTQB® international working groupsANZTB participates in ISTQB® international working groups. For details visit http://www.istqb.org
Presentation 1The Digital Testerby Andrew Simister
Copyright © ANZTB
The Digital Tester
Andrew Simister
Director of Digital QA
Planit Testing
© Planit 17
• Upswing in Mobile and Digital
• Challenges with Evolving Platforms, OS and Devices
• Interesting Statistics
• The Challenge for The Digital Tester
• What’s the next Digital Disruption
© Planit 18
Presentation Overview
© Planit 19
Digital Ecosystem
7.5 BN
4.77 BN
2.5 BN
Access to a Mobile Phone Access to proper sanitationWorlds Population
© Planit 20
Source:
1. Market Guide for Mobile Application Testing Services, Gartner, 28 June 2017
2. Commowealth Bank ups technology spend to AU$1.2b in H1, ZDNet, 15 February 2017
Commonwealth Bank Case Study 2
53% of transactions are made via digital devices80% of logins are made via mobile devices
According to Gartner 1
By 2022, 70% of software interactions will occur on mobile devices.
© Planit 21
Digital Investment
• Global spending on digital transformation technologies will
reach US$1.2 trillion in 2017 (up 17.8% from 2016)
and is forecast at US$2 trillion by 2020 3
• In Australia, spending was AU$79 billion in 2014
and is forecast at AU$139 billion in 2020 3
Source:
3. Worldwide Digital Transformation Spending Guide, IDC, February 2017
Impacts on your Digital Portfolio
© Planit 22
Business Demands
Customer Demands
Evolving Platforms, OS and Devices
New Technologies
Digital Portfolio
© Planit 23
Challenges with Market Changes
• Fragmented market
How to ensure your applications are compatible with a large range of
devices, OS and browsers being used?
• Continuously changing OS, browser, devices
How to ensure your apps keep working while the environment changes
continuously?
• Need for speed
How do you respond quickly enough to the market changes to satisfy
customers and internal stakeholders?
© Planit 24
Fragmentation & Compatibility
• There are at least 3 current versions of iOS and 7 for Android OS
• OS market share: 41% Android, 35% Windows, 13% iOS, 5% OSX 4
• 52% of Australian smartphones access a ‘connected device’ (IoT) 5
Source:
4. StatCounter GlobalStats, July 2017
5. Worldwide Digital Transformation Spending Guide, IDC, February 2017
iPhone 7 Plus 1080 x 1920
iPhone 7 750 x 1334
iPhone 6 Plus/6S Plus 1080 x 1920
iPhone 6/6S 750 x 1334
iPhone 5 640 x 1136
iPod Touch 640 x 1136
iPad Pro 2048 x 2732
iPad 3rd & 4th Gen 1536 x 2048
iPad Air 1 & 2 1536 x 2048
iPad Mini 768 x 1024
iPad Mini 2 & 3 1536 x 2048
Google Nexus 6P 1440 x 2560
Google Nexus 5X 1080 x 1920
Google Pixel 1080 x 1920
Google Pixel XL 1440 x 2560
Samsung Galaxy Note 5 1440 x 2560
LG G5 1440 x 2560
One Plus 3 1080 x 1920
Samsung Galaxy S7 1440 x 2560
Samsung Galaxy S7 Edge 1440 x 2560
Google Nexus 7 (2013) 1200 x 1920
Google Nexus 9 1536 x 2048
Samsung Galaxy Tab 10 800 x 1280
Google Chromebook Pixel 2560 x 1700
Are you testing for these
devices and resolutions?
How do you keep up to date?
© Planit 25
New OS / Browser 2016/17
iOS 9 6 releases in 2016
iOS 10 4 updates/releases in 2017
Android 2 upgrades in 2016
Chrome 8 updates in 2016
Edge browser 15 updates in 2016
Safari Browser 8 updates in 2016
Windows 10 Mobile 8 updates in 2017
Windows 10 monthly updates
Android TV 2 updates in 2016
Apple TV - TVOS 3 updates in 2016
3 updates in 2017
New Devices 2016
Samsung 31 devices
Huawei 22 devices
LG 19 devices
Lenovo 26 devices
HTC 15 devices
Apple 3 devices
© Planit 26
Potential Impacts: Functionality + Usability
53% of users will
uninstall a mobile app
that crashes, freezes
or has errors 6
68% of users will
leave a website due
to an unintuitive
user interface 7
80% of users will
attempt to use a
problematic mobile
app 3 times or less 6
6. Failing to Meet Mobile App User Expectations, Dimensional Research, 2015
7. UX 101: What is User Experience?, Homestead, 2013
© Planit 27
Potential Impacts // Accessibility
15% of the world’s population
(1bn people) experience some
form of disability 8
The aging population is
expected to grow by
300% by 2050 9
8. World Report on Disability, World Bank/WHO, 2011
9. Why Web Accessibility Should Be a Priority Now: 3 Stats to Prove It, Siteimprove, November 2016
© Planit 28
Potential Impacts // Performance
53% of mobile site visits
are abandoned if it takes
over 3 secs to load 10
49% of users expect a
mobile app to respond
in 2 seconds or less 6
36% of users will stop
using a mobile app due
to heavy battery usage 6
6. Failing to Meet Mobile App User Expectations, Dimensional Research, 2015
10. The Need for Mobile Speed, DoubleClick by Google, September 2016“Performance Outweighs Functionality”
© Planit 29
Potential Impacts // Security
86% of websites have at least one
serious vulnerability 11
The average cost of
a data breach is
$4 million 12
11. 2015 Website Security Statistics Report, WhiteHat Security, May 2015
12. Cost of Data Breach Study, IBM Security, June 2016
When Digital Goes Wrong
Recent Survey shows:
© Planit 30
8% late for work
15% Lost work
15% Got lost
17% paid a higher price elsewhere
30% missed out on a sale
• 44% of users will tell friends about a bad online
experience
• 37% of users think less of a company's brand if
their mobile app has issues
• Digital Problems Send Customers to Competitors
© Planit 31
Immediate Impact on your Brand
Digital Feedback // Everyone is a Critic
Instant Feedback
© Planit 32
WhatsApp met with
backlash after giving users’
data to Facebook
29%of customers likely to
complain on Social Media
© Planit 33
Todays Digital Tester
Accessibility
Tools & Assistive
Technology
Security PerformanceGeo-location UsabilityFunctionality
Challenge
Devices, OS, Browser
understanding
Compatibility
Agile Customer Focus
UX Digital World Data Analytics Appropriate Quality
Characteristics
What is the next Digital Disruption?
© Planit 34
Use of DataArtificial Intelligence iOT
[ASK THE EXPERT]
Any Questions?
© Planit 35
Andrew Simister
Director of Digital QA
+61 449 734 206
AU: 1300 992 967
NZ: 0800 752 648
I N : +91 40 6635 9555
UK: +44 203 457 5020
Copyright © ANZTB
Networking and RefreshmentsWe will resume at 6:45pm
Presentation 2Continuous Testing: Preparing for DevOpsby Susmit Bhattacharya
Copyright © ANZTB
Susmit Bhattacharya, Solution Architect, Tricentis APAC
© 2017 by .
Continuous TestingPreparing for DevOps
Listening to horses talking in the early 1900s …
I’m worried. Will all these new mechanical muscles make
horses unnecessary?
No worries. It will make our lives easier. Even if this car thingy takes off, there will
be new jobs for horses we can't imagine.
0
50,000
100,000
150,000
200,000
250,000
0
10,000
20,000
30,000
40,000
50,000
1860s 1880s 1900s 1915 1940s 1960s 1980s
Mechanical Disruption
Since 1915, the number of horses
has fallen by
Human Population vs. Horse Population
Humans [k]
Horses [k] 85%
We didn’t do anything wrong,
but somehow, we lost.‹‹ Stephen Elop, CEO Nokia, 2013 ››
Core Conclusion
It’s no longer about mastering continuous improvement,
it’s about mastering continuous change.Stephen Perry
*Bottom Line. Whatever goes up must come down - to disrupt, or to be disrupted.
People in
the world.
People own a
mobile phone.
People have regular
access to a toilet.
7.4 4.5Billion Billion
Software Fuels Digital Disruption
5.5Billion
Driving Force
Digital Disruptors
Owns no vehicles. Owns no real estate.Everyone can be a taxi driver. Everyone can be a landlord.
World’s largest taxi company. World’s largest accommodation provider.
Their value is all digital.
Digitalization creates new ways of business operations.Technology enables people instead of people managing technology.
Disrupted Verticals
Transport Hospitality Music Television Shopping Mobiles
Education Insurance
Real Estate
Logistics Healthcare
EnergyFinance
ConstructionLegal
Critical
Physical Limit
Testing
Present
Perceived
Disruption
DigitalTesting
AgileTesting
ClassicalTesting
ContinuousTesting
DevOpsChasm
Maintenance TrapYears
MonthsMonthsWeeks
WeeksDays
DaysMinutes
Internet ofThings
The Age ofRobotics
PredictiveAnalytics
MachineLearning
Provide instant feedback
through uninterrupted testing.
ArtificialIntelligence
TechnicalComplexity
Rate ofChange
QuantumComputing
I failed my way to success.Thomas A. Edison
Fail Fast ›› Fail Cheap ›› Fail Forward
Dev Ops
Continuous
Release & DeploymentCollaborative
Development
Continuous
Testing
Continuous
Monitoring
Continuous
Integration
Continuous
Feedback
DevOps Movement
DevOps is no longer a
unicorn’s discipline.
TestDev OpsShift Left Shift Right
Continuous Testing – What is it?
Trying to do the old way harder-is not going to work.
Modern ArtHumans have creative mind
Continuous Testing – What is it?
Human mind is really Creative!!!!
*Bottom Line. Test automation has not kept its promises.
60-80%
Manual Testing
40-20%
Automated Testing
UI
API
Today
The reliance on
manual testing is the
top technical challenge
in app development. World Quality Report, 2015/16
Test automation
requires developers.
Testing Present
remaining manual tests(exploratory and manual spec-based)
automated UI tests
automated API tests
+
service virtualization(key enabler for high automation)
GUI
API
Model-Based
Test Automation (MBTA) for UI and APIAutomating the automation framework.Automate
Session-Based
Exploratory TestingMake your testing intellectually rich.Explore
Orchestrated
Service VirtualizationMake testing independent.
Stateful
Test Data Management
Manage
Computer-Aided
Test Portfolio DesignRedundancy-free test case portfolio with
maximum risk coverage and fewer tests.Optimize
Integrate
CT Integration (CI, CD)
Distributed ExecutionTarget StateCurrent State
Budgets won’t increase
60 - 80%Manual
Testing
40 - 20%Automated
Testing
+85%
Strategies for Testing
ris
k c
ove
rag
e
60%
80%
100%
40%
20%
0%
𝑴
𝑵=
0
de
fec
t ra
te
max
100%
test cases
20%0% 40% 60% 80%
critical defects
M N
𝟏 weight𝟏 𝟏 𝟏 𝟏 𝟏 𝟏… …[%]
Risk-Based Testing
60%
80%
100%
40%
20%
0%
ris
k c
ove
rag
e
100%20%0% 40% 60% 80%
test cases
critical defects
0
de
fec
t ra
te
max
M N
𝒘𝟏 𝒘𝟐 𝒘𝟑𝒘𝟒 𝒘𝒊 𝒘𝑵… … … … 𝒘𝒊 ≥ 𝒘𝒊+𝟏weight
Risk-Based Testing
60%
80%
100%
40%
20%
0%
IntuitiveTest Design
MethodicalTest Design
100%20%0% 40% 60% 80%Test Cases
Risk
Coverage
Risk-Based Testing
Achieve maximum risk coverage with
a minimal number of powerful test cases.
Automation Approaches
Record & ReplayLinear Framework
Sophisticated
Framework
Model-Based
Automation
1st Generation 2nd Generation 3rd Generation
Script-Based
10%Automation
30%Automation
90%Automation
System
Under Test
Test Case(s)
Model(s)
Technologydecoupling
Business
Data Sheet(s)
Busin
ess
decoupling
Data
Data
Model Based Test Automation (MBTA)
Set-up
Maintenance
Future Test Automation will focus on API testing
Task
Runtime
Timing
4x
6x
100+ x
FactorAPI
testing
25%
16%
<1%
progressive
UI test
automation
100%
100%
100%
regressive
UI vs API
Risks
Automated Testing
I’m an automated test case.
Usability Issue
Stability Issue
Reliability IssueScalability Issue
Coherence Issue
Understandability Issue
Testability Issue
Convenience Issue
Security Issue
Accessibility Issue
Performance Issue
Functional Issue
Automation is doing, what automation does.
I’m an automated test case.
Manual Testing
I’m the same test case executed manually.
Plan as you test!
Story-Based
Motivating
Credible
Exploratory Testing
Exploratory BranchingThe art of deciding on what to test next.
NewTestingIdea Exploratory Testing
Perfect counterpart to specification-based
automated & manual testing.
Test Cases vs. ScenariosPre-specified inputs vs. hypothetical situation
Specification
Based Testing
Exploratory Testing
Analyze Potential RisksProblem vs. No Problem
Monitor Known RisksPass Result vs. Fail Result
Agile Testing Law
Checked≫ Efficient Automated Testing ≪
Explored≫ Effective Exploratory Testing ≪
Tested≫ Productive Testing ≪
+
=
𝑅𝜇𝜈 −1
2𝑅𝑔𝜇𝜈 + Λ𝑔𝜇𝜈 =
8𝜋𝐺
𝑐4𝑇𝜇𝜈
Agile Equation Of Motion
Focus Area
Risk-/Value-based
Functional Structure
Exploratory Testing
Automate
M1
Explore
LIMITED
LIMITED
LIMITED
DEV ONLY
LIMITED
LIMITED
HAPPY PATHS
DEV ONLY
M2 M3 M4 M5
Testcase Design
CT Integration (CI, CD)
UI Automation: Script-based
UI Automation: Model-based
API Automation
Active Test Data Management
Test Driven Service Virtualization
Most enterprise
customers are on
level M2
Manage
Integrate
Optimize
CTMM - Continuous Testing Maturity Model
Efficiency (Risk Coverage/
Cost, Time)
1
M1 M2 M3 M4 M5
1.3
4
>10
> 20
Brittle script-based UI
automation
Exploratory T.
UI & API automation
(model based)
CI/T/D integration
+ Test Case Design
+ Active TDM
+ TD Service Virtualization
Comprehensive
adoption
Maturity Levels & Test Efficiency
Copyright © ANZTB
If you would like to present at a future SIGiST or at any Australian or New Zealand venues, please contact any of your local facilitators –
Leanne Howard ([email protected])David Fuller ([email protected])Steve Toms ([email protected])
or the SIGiST CoordinatorRonak Panchal ([email protected]).
Copyright © ANZTB
Please provide your feedback via a quick survey: link