Continuous Quality in the Digital Age

Bob RandallVirgin Media

Gidi PridorPerfecto

A FEW HOUSEKEEPING ITEMS

1. Submit questions using the Q&A panel.

2. Slides and recording sent after the webinar.

3. Please complete survey after the presentation.

Continuous Quality in the Digital Age

Bob RandallVirgin Media

Gidi PridorPerfecto

4

554% would switch banks based simply their satisfaction with the bank’s mobile appCNBC Apr 2015

WHO ARE VIRGIN MEDIA?

Double Play : 19%

• 4.2m fixed telco customers• 3.1m Mobile customers

• 4.5m broadband customers • 4.9M TV & VOD customers

Internet & Data TV & VideoTelco & Mobile

1

Double Play : 19% Double Play : 19%

OUR TESTING SERVICE AT A GLANCE

UK

Indi

a

75 FTE 197 FTE

…We’re 72% offshore, in India…Our testing service is now

4 years old

We’re responsible for the Test and Quality across all IT projects delivered

Mobile Broadband

Digital TV Mobile AppsBusiness Engineering

Corporate

Access

Transformation

TMMi Level 5accreditation awarded in 2015

Innovation programmes delivered ensuring best in class tooling and methodology

We run 100k test cases per year and find on average, 13k P1-P4 defects

One of only 9 organisations worldwide to achieve this top tier accreditation

30

2

42

69

102

150

2012 2013 2014 2015

9393 projects already delivered as at Q3 2015

IT P

roje

cts

per a

nnum

We deliver 250% more projects now compared 2012

+70%Aspiration for mobile test automation

90% reduction in P1-P2s in production

increase in the volume of defects identified prior to ‘go-live’, with only 1 P1 and 1 P2 since 2013 in production

55%

We have Transformed our Digital Quality 3

Managing Documents in VM Cloud

Our customers are Going Mobile

Watching TV

Controlling Set Top Boxes

Browsing Our Websites

Managing Their Accounts

Finding Virgin Wi-Fi Hot Spots

What Next?

4

Customer Behaviours are Driving Change

• Mobile browsing doubled in a 12 month period from January 2013

• Mobile conversions track 10% greater than browse

• Conversions doubled within 18 months

• Growth of mobile is constant

• Smart devices are now the norm, mobile first services are the default

• Desktop based orders will be the minority by December this year

THE RISING IMPORTANCE OF MOBILE FOR VIRGIN MEDIA

5

Old Process, New Challenges

100% Manual Testing (hand-held devices)

Devices were procured or borrowed individually by each team

Devices were insecure, often lost/missing and unavailable when needed

No corporate standard for mobile quality & test coverage

All testing had to be conducted on-site, no option to offshore

Quality Efficiency

Overhead Control

• Limited coverage• Poor oversight

• Slow cycle• QA bottleneck• Scalability• Availability

• Logistics• Cost• Can’t Offshore• Can’t keep up

• Security• Governance• Standardization• Audit• Reliability

6

TRANSFORMATION

We Had To Do More For Less, A Lot Less…

Transform the way we approached mobile testing

• Scale & Speed

• Reduce Costs

• Improve Quality

CQContinuous Quality

7

Key Requirements for Transformation

Robust Automation Sustainable Device Agnostic Unattended

Central Cloud Platform

Remote testing - offshore

Avoid device mgmt. Governance

Flexibility to Scale Add users Add use cases Update devices

Enterprise-grade

StandardSecurity SLA Reliability

Future Readiness Easily integrated Any environment Any Framework ,

CI etc.

Investing in technology was the only realistic way.

SPEED

SCALE

EFFICIENCY

COVERAGE

8

Cloud Based Continuous Quality Lab

Proprietary & Confidential

Dev

/Tes

t E

nviro

nmen

ts (+

CI/C

D)

9

Use cases

Functional application

testing

Performance testing

Web responsive testing

Customer set-top box/home hub/mobile

device testing

Hosted testing services for Liberty

Global affiliates

Service Desk mobile fault

analysis support

India

Netherlands

UK

10

How has it changed what we do?

We have embraced Continuous Quality as a practice

Mobile and Responsive Web

Testing

• Centralized enterprise test lab

• Virtualised devices as a cloud service

• Maximized test coverage

• Accelerated using Automation

Mobile Application Security Testing

• centralised Security Testing capability

Networks Virtualisation and Testing

• Network virtualisation

11

2012 2013 2014 - 2015

Transforming to Stay Ahead

Standardising Processes and Methodologies Enabling Efficiency Transformation & Innovation

Standardisation & Industrialisation Delivery Quality Cost Effectiveness

HP LOAD RUNNER STANDARDISATION

HP QUALITY CENTRE STANDARDISATION

STANDARDISED TESTING ACROSS VIRGIN MEDIA

TRANSITIONED 239 SILOS INTO STRATEGIC TCOE

CENTRALISED AND STABILISED TEST ENVIRONMENTS SV

REMOTE HFC MODEM ACCESS

DATA MASKING

MONITORING AND ANALYSIS

ENABLING OFFSHORE MOBILITY TESTING

TEST ENVIRONMENT VIRTUALISATION

ENABLING OFFSHORE TV TESTING

EARLY ENGAGEMENT – STATIC TESTING

STRATEGIC TEST AUTOMATION

TASKTOP SYNC: DEV AND TEST COLLABORATION

SAUCELABS DIGITAL CLOUD EXECUTION PLATFORM

TEST INNOVATION AND TRANSFORMATION PROGRAMME

TEST ENVIRONMENTS DASHBOARD

HP SAAS HOSTED APPLICATION LIFECYCLE MANAGEMENTHP SAAS HOSTED PERFORMANCE TESTING SUITE

FORMALISED RISK BASED TESTING

NEW MEDIA PERFORMANCE TEST ENVIRONMENT BUILT

TMMI LEVEL 5 ACCREDITATION

QUALITY MANAGEMENT

NON FUNCTIONAL PROCESS IMPROVEMENTS

TASKTOP DATA: REAL TIME DATA REPORTING

NEOLOAD: CLOUD PERFORMANCE TESTING

Embracing technology and tooling has been key to our success

SECURITY TESTING CAPABILITY

AGILE READY TEST CAPABILITIES

INTEGRATED MOBILE TEST AUTOMATION

12

Results

Perfecto’s tight integration with Selenium has helped us develop our strategic mobile automation framework, which will be our primary mechanism for automating all mobile application testing

Perfecto has enabled us to extend our Responsive Test Automation platform to use real devices instead of emulators to get increased confidence that we’re testing the real customer experience

ResponsiveTesting

Framework

The on-premise capabilities of Perfecto have enabled us to be able to test Connected Home type services from India,

On premise

capabilities

By utilising Perfecto, we’ve been able to take a significant portion of our mobile applications testing to India

Perfecto has enabled us to conduct 1st level application support through our offshore application support team

13

Our current challenges

CHANGING DELIVERY MODELS – THE SHIFT TO BI-MODAL IT

• Transformation to a Digital Business is demanding a change in IT delivery approaches

• The increasing need for agility is driving a need for Bi-Modal IT, through embracing Agile delivery methods for faster paced programmes

• The demands on our Testing service are changing• Increased levels of Automation, broader technical skill

sets, DevOps and changing governance controls and processes are needed

MODE 1 – WATERFALL

MODE 2 – AGILE / ITERATIVE

14

How Perfecto Mobile contributes to solving the challenges ahead 1

5

Integration of Perfecto Mobile within our new DevOps reference architecture for

Continuous Delivery operations

Deployment of Perfecto Local Link to enable more flexible testing in scrum

team environments

Further deployment of Perfecto Local Link to enable “Location Based” testing

Integrated automation through Selenium scripting

Requirement Management ALM Defect Management

SoftwareConfiguration Management Build and Packaging

Automated Testing Data Analysis/CodeQuality

Continuous Integration Artifact Repository Continuous DeliveryOrchestrates

Virgin Media Continuous Delivery Framework

Quality Maturity Model

1

Initializing Definition Transitioning Integration Continuous Quality

2 3 4 5

ReleaseFrequency

TestCoverage

Automation %

Your TestProcess

Your Environment

2-3/year Quarterly Every 45-60 days Monthly When done

<30% coverage -50% coverage 50-90% coverage 90%+ coverage 100% coverage

+/- automation Regression testing at end of sprint

<33% automation Some automation with legacy

QA tools-50% automation Up to 75% automation >75% automation

Agile developmentProcess +Waterfall

QA

Separate QA and Dev ALM tools

Test cycles take less than a week

Centralized testing platform

Performance testingProduction monitoring

No performance testing

No monitoring in production

Applying desktop testing approach to mobile

Isolated test environments (no lab)

Using dev tools for automated testing

(Java, C#)

Automation framework

Performance testing several times per year

Build process runs automated tests (CI)

Optimized automation framework

Managed lab

16

Future 17

• Perfecto Mobile is a key component in our strategy to implement CI and CD, we see this capability evolving and maturing in 2016, alongside further automation development

• Device types are proliferating – as new mobile and wearable technologies become relevant we’ll use Perfecto Mobile to host and test against these new devices

• Security is going to be more and more important as we adopt mobile devices into every aspect of our lives. We have a responsibility to ensure our customers and their data is kept as secure as possible so we intend to use Perfecto Mobile to support our device security strategy

• Virgin Media is a TMMI Level 5 accredited Test Centre of Excellence and we want to exploit the good practice, technologies, processes and approaches we’ve developed to support the wider Liberty Global family of companies

Summary 18

Change is the only constantTransformation isn’t easyAutomate to keep upBuild Enterprise grade solutions

Any Questions?