blokland & mengerink - testing cloud services - eurostar 2012

Kees Blokland and Jeroen Mengerink, Polteq

Testing of Cloud Services

The Approach: From Risks to Test

Measures

www.eurostarconferences.com

@esconfs #esconfs

Testing of Cloud Services

The Approach: From Risks to Test

Measures

Kees Blokland

Jeroen Mengerink

Polteq Test Services BV

The Netherlands

searching, recording, accounting, paying, writing,

reviewing, tracking, calculating, developing, listening,

analyzing, transmitting, learning, controll ing,

purchasing, testing, alarming, changing, updating,

deleting, accessing, rejecting, correcting, studying,

booking, receiving, tracing, protecting, deciding,

managing, teaching, facilitating, identifying, copying,

removing, demonstrating, checking, showing,

selecting, subscribing, unsubscribing, sharing,

mailing, communicating, reading, playing, working,

meeting, gambl ing, shopping, storing, cross

checking, retrieving, configuring, sketching, saving,

accelerating, enhancing, creating, growing, checking

in, checking out, finding out, reaching, denying,

talking, designing, making, verifying, measuring

5

Email

Surf

Transfer

Develop and Test

Operate and Manage Store

6

Email

Surf

Transfer

Develop and Test


storage claim

80% unused

redundancy, proliferation limitations

environmentally unfriendly

7

Email

Surf

Transfer

Develop and Test


standard software bandwidth

internet technology

virtualization

SOA

8

searching, recording, accounting, paying, writing,

reviewing, tracking, calculating, developing, listening,

analyzing, transmitting, learning, controll ing,

purchasing, testing, alarming, changing, updating,

deleting, accessing, rejecting, correcting, studying,

booking, receiving, tracing, protecting, deciding,

managing, teaching, facilitating, identifying, copying,

removing, demonstrating, checking, showing,

selecting, subscribing, unsubscribing, sharing,

mailing, communicating, reading, playing, working,

meeting, gambl ing, shopping, storing, cross

checking, retrieving, configuring, sketching, saving,

accelerating, enhancing, creating, growing, checking

in, checking out, finding out, reaching, denying,

talking, designing, making, verifying, measuring

Email

Surf

Transfer

Develop and Test


US: National Institute of Standards and Technology http://www.nist.gov

Essential characteristics

On-demand service

Self service provisioning, pay-per-use

No human interaction

http://www.nist.gov



On-demand service

Broad network access

Standard mechanisms over networks

“Any” client

http://www.nist.gov



On-demand service


Resource pooling

Multi-tenant

Storage, processing, memory, virtual machines, …

Location independent

http://www.nist.gov



On-demand service


Resource pooling

Rapid elasticity

Rapid scale in and out

“Any quantity” at any time

http://www.nist.gov



On-demand service


Resource pooling

Rapid elasticity

Measured service

Controlled resource use

Transparency, pay-per-use

http://www.nist.gov



On-demand service


Resource pooling

Rapid elasticity

Measured service

Deployment models

– private cloud

– community cloud

– public cloud

– hybrid cloud

Service Models

Software as a Service

Platform as a Service

Infrastructure as a Service

http://www.nist.gov

16

What is “done” in the cloud?

>500 Private Hybride Community IaaS, PaaS, DaaS, SaaS

Taas

*aaS

Data Centre Data Management Business processes

Consumer Public SaaS

Surf and mail Apps Social media Dropbox Google services Spotify Picasa Games ……………

<500, SME Public *aaS

Mail Storage

Infrastructure

CRM

Finance

Business processes

Continuity

Privacy

Multi platform

Legislation

Cyber crime

Impact organisation

Standards

143

Continuïty

Privacy

Multi platform

Legislation

Cyber crime

Impact organisation

Standards Performance

Security

Availability & Continuity

Functionality

Manageability

Legislation & Regulations

Suppliers & Outsourcing Ris

ks

19

Other customers

YOUR

Operational Profile

YOUR

Operational Profile

PLUS

Performance

Security


Functionality

Manageability



ks

20

Everything over the web

The idea:

“it’s safe”

Home ground for

hackers

Performance

Security


Functionality

Manageability



ks

21

Bring Your Own Device

No free choice of

device.

Endless

possibilities.

Performance

Security


Functionality

Manageability



ks

22

Backup and recovery

Taken care of.

Who will support

me?

Performance

Security


Functionality

Manageability



ks

23

Updates, patches, fixes, …

Planned and

controlled

Do I have a

choice?

Performance

Security


Functionality

Manageability



ks

24

Where is my data?

And is that OK?

In house.

Somewhere…

Performance

Security


Functionality

Manageability



ks

25

Performance

Security


Functionality

Manageability



ks

26

Testing?

Check

Intake

Trial

Interview

Proof of concept

27

Testing!

Check

Intake

Trial

Interview

Proof of concept Testen Proef Intake

Interview Proof of concept

28

Performance Testing

Security Testing

Manageability Testing


Testing

Functional Testing

Migration Testing

Testing caused by


Testing in Production

Testing during Selection T

es

t Me

as

ure

s

Testen Proef Intake

Interview Proof of concept

29

Performance

Security


Functionality

Manageability



ks

Performance Testing

Security Testing



Testing

Functional Testing

Migration Testing

Testing caused by




es

t Me

as

ure

s

30

Performance

Security


Functionality

Manageability



ks

Performance Testing

Security Testing



Testing

Functional Testing

Migration Testing

Testing caused by




es

t Me

as

ure

s

31

Performance

Security


Functionality

Manageability



ks

Performance Testing

Security Testing



Testing

Functional Testing

Migration Testing

Testing caused by




es

t Me

as

ure

s

32

Performance Testing

Security Testing



Testing

Functional Testing

Migration Testing

Testing caused by




es

t Me

as

ure

s

Performance

Security


Functionality

Manageability



ks

Architecture

From “individual” risks

to

“individual” test measures

33

Selection

Implementation

Production

Performance Testing

Security Testing



Testing

Functional Testing

Migration Testing

Testing caused by




es

t Me

as

ure

s

34

Performance Testing

Security Testing



Testing

Functional Testing

Migration Testing

Testing caused by




es

t Me

as

ure

s

Performance

Security


Functionality

Manageability



ks

35

logo van Flair

17-12-2010

Polteq logo_RGB.png

R G B

35 30 96 Blauw

232 62 38 Rood

108 174 68 Groen

Selection Criteria

Performance Testing

Security Testing



Testing

Functional Testing

Migration Testing

Testing caused by




es

t Me

as

ure

s

36

logo van Flair

17-12-2010

Polteq logo_RGB.png

R G B

35 30 96 Blauw

232 62 38 Rood

108 174 68 Groen

Proof of Concept

Performance Testing

Security Testing



Testing

Functional Testing

Migration Testing

Testing caused by




es

t Me

as

ure

s

37

Performance Testing

Security Testing



Testing

Functional Testing

Migration Testing

Testing caused by




es

t Me

as

ure

s

Performance

Security


Functionality

Manageability



ks

38

logo van Flair

17-12-2010

Polteq logo_RGB.png

R G B

35 30 96 Blauw

232 62 38 Rood

108 174 68 Groen

Known measures

tuned and tweaked

New measures developed

Performance Testing

Security Testing



Testing

Functional Testing

Migration Testing

Testing caused by




es

t Me

as

ure

s

39

logo van Flair

17-12-2010

Polteq logo_RGB.png

R G B

35 30 96 Blauw

232 62 38 Rood

108 174 68 Groen

Load Testing

YOUR

Operational Profile

YOUR

Operational Profile

PLUS

ACTUAL MOMENT

Performance Testing

Security Testing



Testing

Functional Testing

Migration Testing

Testing caused by




es

t Me

as

ure

s

40

logo van Flair

17-12-2010

Polteq logo_RGB.png

R G B

35 30 96 Blauw

232 62 38 Rood

108 174 68 Groen

Online – Offline

Use case testing.

Global testing.

Performance Testing

Security Testing



Testing

Functional Testing

Migration Testing

Testing caused by




es

t Me

as

ure

s

41

logo van Flair

17-12-2010

Polteq logo_RGB.png

R G B

35 30 96 Blauw

232 62 38 Rood

108 174 68 Groen

Any device – any platform

Multiplatform

testing.

Multiplatform

testing.

Performance Testing

Security Testing



Testing

Functional Testing

Migration Testing

Testing caused by




es

t Me

as

ure

s

42

Internet Explorer 6

Internet Explorer 7

Internet Explorer 8

Firefox 3.5

Firefox 3.6

Firefox 4

Safari 4

Safari 5

Chrome11

Opera11

Windows XP

Windows Vista

Windows 7

Windows 2003 server

Windows 8

Windows CE

Linux

Unix

Mac OS Lion

Mac OS Snowleopard

iOS

Android

Operating Systems

Browsers

Multiplatform

Devices

Computer

Mobile

Tablet

PC

Macintosh

SUN

NOKIA …

Samsung … Windows Mobile

iPhone ..

Xxx …

MOTOROLA…

Blackberry…

ASUS..

Xxx …

5-18

43

logo van Flair

17-12-2010

Polteq logo_RGB.png

R G B

35 30 96 Blauw

232 62 38 Rood

108 174 68 Groen

Any device – any platform

Multiplatform

testing.

Multiplatform

testing.

Performance Testing

Security Testing



Testing

Functional Testing

Migration Testing

Testing caused by




es

t Me

as

ure

s

44

logo van Flair

17-12-2010

Polteq logo_RGB.png

R G B

35 30 96 Blauw

232 62 38 Rood

108 174 68 Groen

Legislation + Regulations

=

Test basis

Incidental testing.

Compliancy testing.

Performance Testing

Security Testing



Testing

Functional Testing

Migration Testing

Testing caused by




es

t Me

as

ure

s

45

Performance Testing

Security Testing



Testing

Functional Testing

Migration Testing

Testing caused by




es

t Me

as

ure

s

Performance

Security


Functionality

Manageability



ks

46

logo van Flair

17-12-2010

Polteq logo_RGB.png

R G B

35 30 96 Blauw

232 62 38 Rood

108 174 68 Groen

Continuous

End-to-End Test

Functionals

and

non-functionals

Performance Testing

Security Testing



Testing

Functional Testing

Migration Testing

Testing caused by




es

t Me

as

ure

s

Continuity

Privacy

Multi platform

Legislation

Cyber crime

Impact organisation

Standards

Check

Intake

Trial

Interview

Proof of concept

49

Performance Testing

Security Testing



Testing

Functional Testing

Migration Testing

Testing caused by




es

t Me

as

ure

s

Performance

Security


Functionality

Manageability



ks

Architecture

From “individual” risks

to

“individual” test measures

Test starts earlier

Test scope is widened

Test will never stop

50

Performance Testing

Security Testing



Testing

Functional Testing

Migration Testing

Testing caused by




es

t Me

as

ure

s

Performance

Security


Functionality

Manageability



ks

Questions?

51

Performance Testing

Security Testing



Testing

Functional Testing

Migration Testing

Testing caused by




es

t Me

as

ure

s

Performance

Security


Functionality

Manageability



ks

Thank you!