where is test in devops?
TRANSCRIPT
DT4Session6/9/163:00PM
WhereIsTestinDevOps?
Presentedby:
NikhilKaul
SmartBearSoftware
Broughttoyouby:
350CorporateWay,Suite400,OrangePark,FL32073888---268---8770··[email protected]://www.techwell.com/
NikhilKaulSmartBearSoftwareNikhilKaulisproductmanager,testingproductsatSmartBearSoftware,theleaderinsoftwarequalitytoolsfortheconnectedworld.NikhilwaspreviouslyatDigité,anapplicationlifecyclemanagementsolutionsprovider,wherehegainedinsightintothesoftwaredevelopmentandtestingmarket.Nikhilisveryinvolvedindiscussionstakingplaceindevelopmentandtestingcommunities.HismostrecentthoughtscanbefoundinForbesandCIO.com.Inhisfreetime,Nikhilenjoysrunning,hiking,andreading.FollowhimonTwitter@kaulnikhil.
Where’s Test in DevOps?
2
Quick Intro
Nikhil Kaul
3
SmartBear’s technology stack
4
On our plate today
1
Industry trends
2
What these trends mean for testers
3
Practices that can help facilitate the transformation
5
The move away from Microsoft
Source: Gartner, Apple, Google, Microsoft, Nokia, Blackberry, a16z
6
Another 1 billion to come online
Source: ITU, a16z
7
The way apps are being built is changing
• Cross-platform dev becoming easier
— Microsoft buys Xamarin
— Windows 10
• Universal Windows Apps
• Write once & deploy it across platforms
— Apple making Swift open source
8
Web continues to get more sophisticated
Approval for plugins
Closing APIs exposing
local machines
Technology giving an
access
Responsive UI
Web Components CSS implementation varies
9
Web and mobile worlds colliding
‒ Cordova
‒ PhoneGap
‒ Open web tech
Mobile
‒ HTML
‒ JavaScript
‒ CSS
Web
React, React Native, Combined JSX, getUserMedia , Geolocation
10
Browsers & app stores are becoming locked
Approval time for plugins and apps increasing
Closing APIs exposing
local machines
Technology giving an
access
11
More to test in less time
Cross platform apps
Changes in web & mobile
High quality
Reusability critical
12
Testing in a DevOps world involves deciding
Test Reporting
Test Reduction
Test Design
Where to start
13
Testing in a DevOps world involves deciding
Test Reporting
Test Reduction
Test Design
Where to start
14
Start small for quick wins
1
Latest
3
Beta
2
Previous
Browser
App
80%
15
Testing in a DevOps world involves deciding
Test Reporting
Test Reduction
Test Design
Where to start
16
Test pyramid strategy can come in handy
Service
Unit
UI
17
Creating API tests helps prevent false defects
Web
Mobile
Tablet
1
2
3
4
18
Reuse those API tests across SDLC
• API functional test
• API load test
• GUI functional test
• API security test
• Monitoring
Time reduced
by 80%
19
The feedback from UI tests is more exhasutive
Browser: Chrome
End-to-end testing
x x x x
HTML5, Angular JS Network Service/API/Database
20
Maintaining brittle tests can be a pain
Text
Duplication
Maintainability
Magic Strings
21
Create one page object for each page
Home Page of Amazon
Result Search Page
Product Details Page
Confirmation Page
22
Abstraction solves some of the problems
Page Object Class 1
Test
Page Object Class 2
Abstract control identification
Test steps
Abstract control identification
Abstract control actions
Abstract control actions
23
The brittle code now looks like
Magic Strings Locating the control
Abstracting the set text action for the control
Using the abstracted set text action
24
Testing in a DevOps world involves
Test Reporting
Test Reduction
Test Design
Where to start
25
Use PairWise testing to cut down tests
I(0,1)
B (0,1)
U(0,1)
App
B I U
1 1
1 0
0 1
0 0
B I U
1 0
0 0
1 1
0 1
B I U
1 1
1 0
0 1
0 0
B I U
1 1 1
1 0 0
0 1 0
0 0 1
1
2
3
Final Set
26
Testing in a DevOps world involves
Test Reporting
Test Reduction
Test Design
Where to start
27
Numbers often don’t share the right picture
• 300 tests
• Each test has .5% failure rate
• Pass rate for each tests: 99.5%
• Pass rate for the suite: (99.5%)^300 = 22.23%
28
Use statistics for measuring test stability
Sample Size (N)
• 50
P
• 29/50 = .58
1-p
• 21/50
Desired confidence (α)
• 95% confidence
29
Takeaways
1
The role QA is becoming more important
2
Start small and build a modular test design
4
Reuse tests across software development lifecycle
3
API testing is key: Don’t forget about the GUI
Thank You!