testing trends - focusing on the right test tools and … · 2018. 4. 4. · test architecture and...

© Copyright 2015 Vivit Worldwide

Testing Trends - Focusing on the Right Test Tools

and Industrializing Their Usage

March 11, 2015


Brought to you by

Vivit Testing, Quality and Application Lifecycle

Management Special Interest Group (TQA-SIG)

and the South Africa Chapter

Leaders: Bernard P. Szymczak Jr, Andreas Birk, Robert Linton,

Christopher J. Scharer, Damian Versaci

www.vivit-worldwide.org

http://www.vivit-worldwide.org/


Hosted by

Christopher J. Scharer

Vivit Director,

Vivit TQA-SIG Leader and

Iowa Chapter Leader


Today’s Presenter

Mihai Grigorescu

Accenture Global HP Alliance Testing Lead

Vivit South Africa Leader


Housekeeping

• This “LIVE” session is being recorded

• The recording will be available on BrightTALK

immediately after this session

• Q&A: Please type questions in the Questions

Box below the presentation screen

• Additional information available for you

behind the Attachment button and later on the

Vivit website


AGENDA

Intro – Accenture testing

Testing trends – what the analysts say

Test Architecture and how TMMi fits into the picture

Testing ecosystem, choosing the right tools sets

ALM industrialization

Q & A



SAP Test tools implementation and support partner

HP SaaS SAP Sol Man to QC integration partner



Testing Spend

• Worldwide Discrete (Standalone) Enterprise Application Testing Services market was $16.7B in 2013 and growing 15% annually to

$33.7B in 2018 (Fig.1)

• U.S. Discrete (Standalone) Enterprise Application Testing Services market was $8B in 2013 and growing 13% annually to $15B in

2018 (see IDC report for graph)

• Budget spend % per projects is continuously increasing (Fig.2)

Fig.1

Source: IDC, Doc #249925 / July 2014 - Worldwide and U.S.

Discrete Testing Services 2014–2018 Forecast

Fig.2

Source: HP/SOGETI, World Quality Report 2014-2015



Drivers for using 3rd party services

• While drivers are multi-faceted, 31% of all respondents

indicate that the primary driver for using 3rd party testing

services centers on driving down costs.

• 37% of public sector organizations indicate the primary

driver for using 3rd party testing services is support for

complex infrastructure testing, with the second most

important being driving down costs.

• The top three characteristics buyers indicated were most

significant when selecting a provider for application testing

services were: reasonable costs (35%), high level of

responsiveness/rapid deployment (20%) and cloud testing

experience (20%)

0 5 10 15 20 25 30 35

Other

Support for vertical solution testing (e.g., banking, telecom)

Use turn-key test solutions for performance test or test automation

Support for localization – country specific readiness testing

Leverage fully managed specialized application test frameworks

Leverage fully managed application lifecycle testing services

As insurance against potential liability resulting from insufficiently tested products by in-…

Standardize internal QA environment across projects

Help with test data management services

Automate regression testing (to support complex business processes across multiple …

Leverage third-party tool expertise, talent and skills

Standardize testing tools and processes

Test staff augmentation

Drive productivity gains (e.g. more time for staff to pursue other initiatives)

Support for complex infrastructure testing (compatibility, interoperability)

Drive down costs

Source: IDC, Doc #249925 / July 2014 - Worldwide and U.S. Discrete Testing Services 2014–2018

Forecast



Fig.2


Challenges and Outsourcing trends

• There is common trend of outsourcing testing services to specialized vendors, with a drop from 51% of companies usinginternal resources in 2012 to 30% in 2014 (Fig.1)

• In Agile environments, hybrids persist (i.e. Scrumfall), no real Agile environments, no specific methodologies nor adequatetools utilized, with difficulties in executing automation (Fig.2)

Fig.1




• There is an increase in app

delivery which will reach

approximately 120 releases

per annum per app by 2020

• Companies are already

planning for upgrading to

HP ALM 12.x

Source: HP, March 2014

Source: VIVIT webcast poll, June 2014



• The main testing drivers for companies are: reduced time to

market, reduced costs, better resource management

(Fig. 1)

• Companies are investing in TaaS (Testing as a Service),

centralization, as well as managed services and outsourcing

their testing capabilities (Fig. 2)

Conclusion: we believe it is paramount to ensure

that test tools are optimized / industrialized within any test

organization in order to support the main testing drivers and

their respective processes adequately.

Source: HP, SOGETI - World Quality Report, 2012-2013

Source: HP, SOGETI - World Quality Report, 2013 -2014



Conclusions – key take away’s

• Digital transformation / business is here. One’s QA methodologies and tools need to caterdigital/TaaS/mobile testing. Virtualization (network, services etc.), testing in the cloud are allelements to consider and invest in.

• Applications are more fluid, delivered faster, which implies, leaner development & testing.Agile will take even more precedence than before, consequently develop competencies andcentralized capabilities.

• More automation (non-traditional) + DevOps are required for faster, organized release toproduction and for bridging the gab between dev teams and operations.

• With increase in overall testing spend, investing (more) in tools adoption is the next logicalinherent step.

• Security testing is still undermined (Sony knows for sure), companies need to invest in properenterprise IT security governance frameworks and conduct more testing in this space.

• Investments in skills and tools on one side, as well as change management on the other arethe crucial elements in winning the testing battle.

• With commoditization of testing, one needs to invest into more creative cost efficient TaaSmodels, more agile service based, output driven models, cross-shared and cross-skilledresources vs. legacy, traditional waterfall processes, dedicated resources and restrictivelicensing models.


Test architecture and how TMMi fits into the picture

What a test architect IS What a test architect ISN’T

Testing expert: a test architect MUST come from a testing background, with experience in all

delivery models

Advisor: solution architecture, estimations, test planning, metrics definition (and

implementation), sales proposals

Fixer: get into challenged projects, produce short term/quick wins and long term improvement

plans

Testing evangelist: deliver papers, presentations, training, provide technical leadership and

strategic direction for their testing organization

Implementer: Setup of TCoE / Test Factory, execute Test Assessments (Internal, External &

TMMi), Enterprise Test Strategy (Definition/Setup), Test Optimization & Improvement, Non-

Functional Testing, Standard and Emerging Technologies / Test Tools

Test Manager

“Fire-fighter” for daily

operational issues

Delivery person (project

wise)

Great article on the topic here:

https://blogs.oracle.com/johnmorrison/entry/test_architect

What is a Test Architect?

https://blogs.oracle.com/johnmorrison/entry/test_architect



How TMMi fits in the picture of testing tools

Test Tools Maturity

Model

(based on TMMi)

Test Capability Models

(based on TMMi)

Test Maturity Model

integration

(TMMi)




• TMMi has 5 levels

• A TCoE will aim at a level 2 to 3, while test factories, between level 3 and 5

• Out of experience, we know that the perceived maturity level is higher than the actual one


POLL 1


POLL 2




• Accenture based

Test capabilities models

(based on TMMi)




Level 3

Level 4

Level 2

Level 5

Le

ve

l 1

Defect Management

Test Management

Requirement Management

Planning (release management)

Libraries and cross project asset sharing and reuse

Automation, Performance, Mobile, Security

Complete reporting capabilities

Executive Scorecards

Tool QA governance

Seamless integration to 3rd

party tools (CI)

Defect Management

Test Management


Planning (release management)

Automation, Performance

Basic Mobile, Security

Advanced reporting (Dashboards across programs)

Scorecards

Tool QA governance

Basic CI

Defect Management

Test Management


Advanced reporting (Project dashboards)

Project templates

KPI Scorecard

Some automation & performance

Defect Management

Test Management

Basic reportingAd-hoc usage

Basic defect management

Testing Tool Usage- Capability Maturity Level

Low High


Testing ecosystem and focusing on the right tools

• Define (Test Architect) what tools to use in

the test organization based on needs and

long term requirements

• Ensure that all testing tools are integrated

for a streamlined testing process

• Maximize tools ROI via:

a) Tools industrialization

b) Strategic partnerships with tools

vendors, for band discounts


POLL 3


POLL 4


POLL 5


POLL 6



HP ALM Industrialization

Test Tool Map

Tools Strategy

Tools Team

Upgrade versions

Upgrade editions

Accelerators

Integrations

Reporting

Quick Wins

Shift Left



Test Tool Map

• Set up a test tools roadmap vs. your value chain / Software Development Life Cycle model

• Ensure that all testing areas are covered (Manual, Automation, Performance, Mobile, Sec., TDM, TEM



Tools Strategy

Test tools strategy: The enterprise wide approach

which describes the solution architecture for the

deployment of all integrated test tools suites:

manual, automation, performance, mobile &

security, incl. additional tools such as requirements

accelerators, COTS and in-house grown (internal)

software utilized in the testing effort.

• Define and implement an overall test tool strategy for the next 3-5 years

• The test tool strategy needs to include capital and operating expenses (including software, hardware and

services costs) in order to provide the most efficient, cost effective solution for the organization



Tools Team• Set up a dedicated test tools team to handle all test technologies in the organization

• Establish a test tool service catalogue & use the tools as bridging mechanisms into other business areas



Upgrade versions

• Upgrade to the latest HP QC/ALM versions to make sure your organization gets the best value out of the

tool

• Make sure that your current QC version is still under support

What extended support entails:

Clients can still buy support beyond the standard EOL (End of Life)

product, e.g. 31 Jan 2015 in this case, however:

• It requires a CSP – Customer Support Plan, for max of 1 year @

150% of list price

• Availability of support resources: best effort (when available)

• Support type: no product fixes/enhancements requests etc.

ALM/QC 11.0 will be out of support in Jan 2015

(latest communication from HP):

Why consider upgrading editions:

• Avoidance of extended maintenance costs

• Continuation of vendor support after QC/ALM 11.0 end of support (Jan 2015)

• Decreased risks of troubleshooting issues and receiving product fixes/patches etc.

• License assignment per domain/business area (for all versions >11.5), enabling the possibility of producing a charge back model (to

business) for recuperating software license costs (best suitable for medium to large installations/test capabilities)

• Less disruption: no additional required upgrades (from older versions to more recent ones) in the next 3-4 years due to end of support.

• Access to newest ALM features & functionalities

HP ALM / QC Vendor Support Time-Table

Version 9.x 10.x 11.0 11.5 11.52 12.0 12.01

Support End Date Nov 2010 Aug 2013 Jan 2015 TBA TBA TBA TBA



Upgrade editions

• If not already on Premier, consider the upgrade in Editions, in order to maximize the ROI on the test tools



Accelerators• Utilize accelerators, either tools or other reusable assets in order to achieve maximum efficiency with the

organization test tools sets.

• SAP ERP QC Project Templates, Predictive Analytics, E-approve, Business Intelligence, Operations

accelerator, Requirements Management aiding tools, SAP/Automation accelerators etc.



Integrations• Ensure that there are proper integration flows between test tools and that redundant / duplicate work is

not executed.

• Examples: AGM <-> ALM, Jira <-> ALM, Sol Man <-> ALM, ALM <-> PPM/XS, TFS <->ALM etc.



Reporting• Define and implement an adequate test metrics framework

• Use ALM’s innate reporting capabilities (document generator, graphs and dashboards, KPI scorecards,

cross project reporting), Business Views, in conjunction with 3rd parties business intelligence apps;

publish QC/ALM graphs to Share Point Site (SPS)



Quick Wins• Use the test capability to produce quick wins list that can be used with immediate effect in order to

improve the test processes (via the tools)

• Example below: defect management process, consistent project template

• Training: ART (Adoption Readiness Tool, other practical hands-on classes.



Shift Left• Use ALM to drive Shift Left transformation into the organization

• Use simplified version of Shift Left (from a defect mgmt. perspective) – improve quality via defect root

cause analysis and improved management

Knowledge sharing – capability development (training, coaching)

Earlier defect detection, inspections, walkthroughs, JADs Methodologies + Tools

“Shift Left”Pushing quality upstream in the SDLC by enabling the testing capability with methodologies and tools


Q & A

[email protected]


Upcoming Vivit SIG Webinar

Best Practices in IT ArchitectureMarch 18, 2015 at 10:00 am EDT / 15:00 pm CET

Brought to you by the

Vivit Business Service Management Special Interest Group

For more information and registration go to:




Vivit Board of Directors election

Vote now on the 2015 Vivit Board of Directors election

and the proposed Bylaw changes:

http://www.vivit-worldwide.org/surveys/?id=459388

http://www.vivit-worldwide.org/surveys/?id=459388


HP Discover Las Vegas 2015

• June 2 – 4, 2015 at

The Venetian Resort

in Las Vegas.

• All members can

Register Now via the

Vivit link

www.hp.com/go/discover/vivit

to save $300.

http://www.hp.com/go/discover/vivit





Vivit’s Deep Dive Sessions

• Deep Dive Sessions will be offered on Monday,

June 1st from 1:00 – 5:00 pm. Find more

information on website under the training section.

http://www.vivit-worldwide.org/?page=VATDownloads


Thank you



testing trends - focusing on the right test tools and … · 2018. 4. 4. · test architecture and...

Documents