testing in the cloud

Testing in the Cloud

Our experience of testing software that is “out there”

Introduction

- The Presenters- Paul Selway – paul_selway@redpathcg.com- Scott Rostal – scott.rostal@thrivent.com

- What is “The Cloud”?- What is different / the same for cloud testers?- Testing Philosophy / Test phases- What was easier / harder than expected?- Key success factors- Questions?

What is “The Cloud”?

• Internet (“The Cloud”) accessible solution

• Off-premise multi-tenant architecture

• Accessed via a browser (PC or Wireless)

• Delivered as Software as a Service (SaaS)

Common cloud companies are:

Google, Amazon, WorkDay, and Salesforce

What things are the same about testing cloud solutions?

Requirements coverage based testing Risk-based structured test approach Test Plan / Test Phases Test Cases / Test Data / Test Automation Defect Management / Functional Test Use of off-shore resources / Virtualization Politics {Quality / Schedule / Resources}

What is different about testing cloud solutions?

Shared multi-tenant test environmentsSecurity (Test User IDs / SSO)Integration of on/off premise systemsPerformance / volume testDefect isolationDocumentation of “Out Of the Box” reqsRelease Management

Project Philosophy

• Risk-based structured testing approach {std}

• “Exploratory Testing” to discover the out of the box cloud solution features {non-std}

• Data Migration – Moving legacy data off premise {Non-Std}

• Performance / volume test only our new infrastructure – not the cloud app {Non-Std}

Testing Phases

Overlapping Test Phases

Unit / CodeCoverage

Inspections& Reviews

Exploratory& System

UAT &Pilot

End-To-EndIntegration

A mix of waterfall (with heavy phase overlapping), and iterative delivery

Load &Performance

DataMigration

What was harder than expected?• Isolating the architectural components for a

performance test• Understanding that a cloud solution vendor

delivers new releases which you have to take!

• Accepting that you have a “black box” in your environment

• Determining who owns defects • Data Migration Mock Runs

System Integration Architecture

Third Party

Home Office

Front OfficeFinancial Planning

4

HO Notify

2

5

3

Field Portal

Generic Mainframe Processing

324

Downloads / Updates

DB7

TriggerSynched

Download Requestor

Request Family

Request by CIF

ID

Request Weekly

Calendar

Member Download

Cert. Value Download

Request In-Force Illustration

10

Computer Application

AM Tool

Outbound

notifications

Webmethods

Web

Ser

vices

API

Customer Data

Customer ReusableServices

Web ServicesData

Warehouse

2

Field platform Integration

6

Staging Server

Informatica

Updates from HO to AM Tool(Customers, Assignments, Contracts, Users, Agents, Alerts

Updates from AM Tool to HO(Activities for WAT)

Informatica Adapter for Salesforce

1

10

Mobile Server

11

Security

Security

Gateway10

5

1

7

4

9

54

3

9

7

8

21

6

8

8

9

6

3

Integrated Apps

1

Not Tested due to Sunsetting

Not Tested due to Existing Infrastructure

Manual Testing

Implicitly Performance Tested

Performance Testing

What was easier than expected?

• Getting the IT leaders to embrace change

• Overlapping Test phases

• End User Adoption

• Use of off-shored test resources– Functional– Performance– Data ETL

Key Success Factors

• Learn the “Out of the Box” cloud application features, security model, and configuration capabilities

• Understand the holistic system architecture

• Understand the migrated test data

• Establish tight communications with the vendor

Questions?

Contact Info

• Scott Rostal – scott.rostal@thrivent.comPerformance Lead, Thrivent Financial for Lutherans

• Paul Selway – Paul_Selway@redpathcg.comPresident, Red Path Consulting Group