testing trends: virtualization, mobility, cloud, etc.files.meetup.com/2625872/voke_denver squad...

© 2013 voke media, llc. All Rights Reserved. Reproduction and distribution are prohibited.

◉ Testing Trends:

Virtualization, Mobility, Cloud, etc.

Theresa Lanowitz, voke Founder

Denver, CO | 17 September, 2013


◉ Agenda

• About voke

• Global Trends

• QA Today

• Extreme Automation

- Service Virtualization

- Development Testing

- Mobility

• Embracing Extreme Automation

• Net/Net


๏ About voke


◉ About voke

• Independent industry analyst firm

• Founded in 2006 by Theresa Lanowitz

• Firm focuses on the application lifecycle and its transformation

including virtualization, cloud computing, embedded systems,

mobile and device software

• Provides strategic, independent, and impartial advice and market

observations through qualitative and quantitative research


๏Global Trends


◉ Current Reality


◉ NASDAQ – August 22, 2013

A “computer malfunction”

froze thousands of

securities listed on the

Nasdaq Stock Market for

three hours. Many of the

country’s most-traded

shares ground to a

standstill as brokers were

unable to execute orders.


◉ Sabre Holdings – September 11, 2012

American Airlines,

Southwest Airlines,

JetBlue Airways, and

Alaska Airlines lost

access to their

reservation systems

hosted by Sabre Holdings, a global distribution system.


◉ More Software Failures Daily

http://aws.amazon.com/

http://www.ingnycmarathon.org/index.htm


◉ Why Are Software Failures Making Headlines?

• Economy

- Lack of investment during Global Financial Crisis (GFC)

• Faster, faster, faster

- Faster is not equal to better

• Old technology

- Most organizations are not up to date on current software platforms

- Most organizations are not leveraging the power of modern solutions


YouTube reported a peak of more than 8

million concurrent livestreams of Felix

Baumgartner’s 24-mile fall to Earth.

◉ Red Bull Stratos & YouTube – October 14, 2012


๏QA Today


◉ QA Roadmap – Critical for Business Success

Chaos

Establish

QA Role

Dev and

QA Parity

IT & Biz

Alignment

Change

Agent


◉ QA Roadmap – Leadership and Innovation

Relationship

Leadership

Process

Leadership

Technology

Leadership

Professional services

Software vendors

Line of business / customer

advocacy

Center of Excellence (CoE)

Appropriate metrics

Brand assurance and advocacy

mALM

Lifecycle virtualization

Development testing

Continuous integration

Build managment

Requirements

Application performance

Application security


◉ Parity – A Basic Requirement


๏Extreme Automation


Solving classic computing problems across the

lifecycle through the use of modern tooling

technology.

Removing barriers and facilitating

communication, collaboration, and connectivity

of development, QA, and operations to support

the line of business and the insatiable demand

for quality software.

Source: voke – June 2013

◉ Extreme Automation – Definition


◉ Classic Challenge – Deliver All Three

Cost

Quality Schedule


◉ People, Process, Technology – Building Blocks

People Process Technology

• Internal skills

• Trusted partners

and providers

• Customer advocate

• Relationship

manager

• Process designation

• Communication

• Collaboration

• Optimized

• Testing platform


◉ Minimize Risk – Move to Extreme Automation

How Development Ensures

Working Software

Technology

Companies

Enterprise Companies

Continuous Integration 61% 47%

Code reviews 60% 64%

Automated unit tests 63% 38%

Manual unit tests 46% 56%

Automated functional testing 51% 40%

Manual functional testing 59% 64%

Performance testing 58% 42%

Source code analysis tools used 29% 11%

Code refactoring 39% 20%

Database refactoring 20% 9%

Design patterns 21% 16%

Domain-driven design 14% 4%

Source: voke Market SnapshotTM Report: Agile Realities – July 11, 2012


◉ Factors Driving Extreme Automation Need

1. Composite applications

2. Software supply chain

3. Consumerization of IT

4. Demand for quality

5. Mobile

6. Cloud

7. Competitive business differentiation through software

Source: voke Market Mover ArrayTM Report: Testing Platforms – June 5, 2012


๏Service Virtualization


Server Desktop Lifecycle (Service)

Market Adoption

Phase

Mature Trending Evolving

Technology

Phase

Stable and full

featured

Stable with

emerging

features

Stable and

evolving

Primary Users Data center Service

center

Dev, test, and

operations

Vendors VMware,

Microsoft, Citrix

Citrix,

Microsoft,

VMware

CA, Parasoft, HP,

Microsoft, IBM

◉ Evolution of Virtualization


Enabling development and test teams to

simulate and model their dependencies of

unavailable or limited services.

Removes constraints and wait times frequently

experienced by development and test teams to

access components, architectures, databases,

mainframes, etc.

Source: voke Market SnapshotTM Report: Service Virtualization – December 2012

◉ Service Virtualization – Definition


◉ Service Virtualization – Sample Use Cases

• Environment replication

• Accessibility

• Supply chain availability


◉ Access to Required Systems

33 Average number of

systems needed

for dev or test

18 Average number

of systems with

unrestricted

access for dev

or test



◉ Lab Access – Wait Time

96% Wait for

access

Wait Time to Access Lab Participant PercentagesNone – immediate access 4%

1 hour or less 7%

2-4 hours 14%

5-8 hours 11%

2-4 days 38%

1 week 11%

2-4 weeks 14%

10 weeks 1%



◉ Lab Access – Restrictions

69% Time

restrictions

Lab Access Restrictions Participant Percentages4 hours or less 31%

5-7 hours 6%

1-2 days 22%

1 week 6%

4-6 weeks 4%

No restrictions 31%



◉ Service Virtualization – Rationale for Adoption

Rationale for Adoption Participant PercentagesRestricted access to dependent

services, components, or systems72%

Scheduling constraints 62%

Parallel development 59%

More realistic performance testing 33%

Other 7%



◉ Service Virtualization – Use

Types of Virtualized Assets Participant PercentagesApplications 65%

Third-party services 56%

Third-party applications 53%

Mainframe 37%

Partner software 34%

Architecture 22%

Other 4%



Category Metric

Reduced Wait Times 64% -achieved a 50% – 100% reduction in

wait time

Greater Availability of

Services

75% - achieved a 50% or greater increase in

availability of services

Decreased Software Cycles 55% - achieved a 25% - 50% reduction in

overall software cycle

Increased Test Coverage 59% - achieved 50% - 90% increase in test

coverage

Shorter Test Cycle Times Average 58% decrease in test cycle length

Improved Time-to-Market Average 24% faster time-to-market

Fewer Production Defects Average 22% reduction of production

defects


◉ ROI Service Virtualization – Summary


๏Development Testing


◉ Development Testing – Definition

Complement traditional blackbox functional testing.

Transform the software testing activity from reactive to proactive by

moving testing upstream to identify and remediate defects earlier in

the lifecycle.


◉ Traditional Cost of Rework

Source: voke Strategic Brief: Cost of Rework Models for Agile and Non-Agile Projects– July 2012

Requirements Design Development TestCustomerSupport

Cost of Large Fix $7,500 $32,500 $107,500 $159,000 $365,000

Cost of Medium Fix $1,500 $7,500 $11,500 $18,000 $40,000

Cost of Small Fix $1,000 $1,500 $2,000 $3,500 $7,500

$0

$50,000

$100,000

$150,000

$200,000

$250,000

$300,000

$350,000

$400,000

$450,000

voke Cost of Rework Model


◉ Development Testing – Overview

Primary User Tool Types Function Vendors Developers Static code analysis Defect prevention

• Memory leaks

• Deadlocks

• Race conditions

Coverity

Parasoft

Developers Dynamic code analysis Detect dependencies

based on system execution

Coverity

Parasoft

Developers Unit testing Smallest testable part of an

application to ensure code

meets its design and

behaves as intended

Coverity

Microsoft

Parasoft

Developers

QA professionals

Application security testing Identify security

vulnerabilities through

scanning and penetration

testing

Coverity

HP

IBM

Parasoft

Development Managers Policy Management Define consistent coding

standards across teams

Coverity

Parasoft


◉ Development Testing – Benefits

• Test at every phase of the application lifecycle

• Collaborate and communicate with development to increase test

coverage

• Remediate defects earlier in the lifecycle to save time and money

• Reduce traditional testing cycle by focusing testing on areas that

require more thorough testing (increase code coverage)

• Reduce the number of defects entering production


๏Mobility


◉ Evolution of Complexity – Testing Challenges

Mainframe Client/Server Web Mobile

Co

mp

lex

ity


◉ Pillars of Testing – Adding Mobile

Does it work? Does it perform? Is it secure?

Functional Performance Security

Lo

ad

S

tress

En

du

ran

ce

Netw

ork

s

Pla

tform

s

…

Re

qu

irem

en

ts

Man

ua

l

Reg

ressio

n

Ex

plo

rato

ry

Netw

ork

s

Pla

tform

s

User E

xp

erie

nce

…

Sc

an

s

Pen

etra

tion

Sta

tic A

naly

sis

De

vic

e

Da

ta

Co

mp

lian

ce …

= Additive for mobile

Infra

stru

ctu

re


◉ Modern Quality Challenge – Mobility and You

• Quality

- Mobile app testing IS application testing

- Mobile strategy is a necessity not a luxury

- Mobile apps demand high quality

- Mobile apps are visible

- Mobile apps are linked to the brand

- Supply chain issues are immediate, visible, and carry consequences

• Tools and Technology

- Innovation of mobile testing is tied to the HP testing assets

- Mobile app testing solutions integrated with and extending existing testing

platforms have an advantage

- Mobility will ultimately be a feature of the broader testing platforms

- Lifecycle virtualization solutions enable mobile app testing


◉ Mobile Apps – Best Practices

• Mobile testing IS application testing

• Extend enterprise best practices to incorporate mobile

uniqueness

- Gestures

- Location

• Leverage existing skills in QA

- Platform tooling

- Automation

- Performance

- Security

• Include new team members with specific mobile skills

- Hardware

- Infrastructure

- OS

- Networking


◉ Mobile App Testing – Vendors

Vendor HP Asset Use Integration

Experitest YES HP, IBM, Microsoft, Open

Source

Keynote DeviceAnywhere YES HP, IBM

Mobile Labs YES HP

Neotys YES HP QTP/UFT – offers mobile

load/performance testing

Parasoft YES HP, IBM

Perfecto Mobile YES HP

SOASTA YES IBM, Microsoft


๏ Embracing Extreme Automation


◉ Extreme Automation – Benefits

• Goal – Extreme Automation

• Automation is critical

- Traditional applications

- Mobile

• Reduces risk associated with multi-platform delivery

- Mobile demand

• Reduces overall software cycle time

• Increases code coverage

• Assists in managing the dynamic nature of mobility

• Enhances visibility into the software supply chain and reduces

risk

• Enhances communication and collaboration throughout the entire

lifecycle


◉ Extreme Automation – Getting There

People Process Technology

Leverage

relationships with

vendors

Select pilot projects

for new technology

Get current and stay

current on existing

tools

Leverage

relationships with

professional service

providers

Form a CoE for new

technology

Evaluate new tools

offered by current

suppliers

Update internal skills

to focus on achieving

extreme automation

Leverage a CoE

model to go from pilot

project to department

/ enterprise use

Explore innovation

from other vendors


๏ Net/Net


◉ Net/Net

• Software and the brand are linked

• QA organization is a customer advocate

• Embrace people, process, AND technology

• Deliver on extreme automation - Reduce the risk of failure in production

• Service virtualization is real and proven - Time-to-market

- Defects entering production

- Reduced software cycle

- Reduced test cycle time

- Increased test coverage

• Move testing upstream with development testing - Increase code coverage

• Mobile testing IS application testing - Identify YOUR mobile strategy


◉ Learn More at www.vokeinc.com

• Become a Research client at www.vokeinc.com

- voke Market Snapshot: Service Virtualization

- voke Category Snapshot: Lifecycle Virtualization

- voke Market Snapshot: Virtual Lab Management

- voke Market Snapshot: Agile Realities

- voke Strategic Brief: Cost of Rework

- voke Market Mover Array: Testing Platforms 2012

- voke Market Mover Array: Application Lifecycle Management

• Follow voke on Twitter: @vokeinc

• Take our surveys

http://www.vokeinc.com/



Thank You!!!


๏ Q&A


◉ voke Contact

voke, inc.

2248 Meridian Boulevard

Suite H

Minden, NV 89423

P: +1.866.895.9045

W: www.vokeinc.com

T: @vokeinc


testing trends: virtualization, mobility, cloud, etc.files.meetup.com/2625872/voke_denver squad...

Documents