Enabling continuous quality assurance and optimization in future enterprise cloud service brokers

Call: FP7-ICT-2011-8 Grant agreement no: 318392 www.broker-cloud.eu

Verification & Testing: SAP HANA Case StudyAnthony J H Simons (University of Sheffield)

Wolfgang K R Schwach (SAP Karlsruhe)

Overview

• Verification and Testing Toolset• Service Specification• Validation and Verification• Test Generation and Grounding• Test Execution on SAP HANA Cloud App

– V&T tools: Anthony Simons, Uni. Sheffield– App & grounding: Wolfgang Schwach, SAP

19/04/23 Anthony J H Simons, University of Sheffield 2

Verification & Testing Tools

19/04/23 Anthony J H Simons, University of Sheffield 3

Specification

• Why?– Gentle pressure towards common standards– Provide basis for formal service certification

• What?– Describe service interfaces and behaviour– State-based model of service API– IOPE model of detailed behaviour– XML language Java metamodel

19/04/23 Anthony J H Simons, University of Sheffield 4

Holiday Booking Service

19/04/23 Anthony J H Simons, University of Sheffield 5

View Days

Book DaysDelete

Days

select/fail

bookDays/ok

select/highselect/low

back/oksave/ok

back/okdelete/ok

choose/onechoose/fourchoose/seven

choose/onechoose/fourchoose/seven

save/failsave/duplicatesave/overflow

XML Specification – FSM

19/04/23 Anthony J H Simons, University of Sheffield 6

XML Specification – IOPE

19/04/23 Anthony J H Simons, University of Sheffield 7

Validation & Verification

• Why?– Designers write incomplete specifications– Designers write incorrect specifications

• How?– Validation tool explores state-transition diagram

• Reports unhandled events, missing transitions

– Verification tool simulates all input/memory space• Reports blocking/nondeterministic operations• Blocking: input exists which triggers no operation• Nondeterminism: some input triggers many operations

19/04/23 Anthony J H Simons, University of Sheffield 8

Verification Fault Analysis

19/04/23 Anthony J H Simons, University of Sheffield 9

off-by-one indexing

error

Test Generation

• Why?– More thorough coverage than manual testing– Assures full compliance to the specification

• How?– Generation tool explores the whole specification– Creates all interleaved sequences up to depth k– Reduces test-suite size by optimising

• Removes infeasible sequences (blocked in the spec)• Removes redundant sequences (equivalence-classes)• Compresses executable suite (multi-objective tests)

19/04/23 Anthony J H Simons, University of Sheffield 10

Generated Test Suite

19/04/23 Anthony J H Simons, University of Sheffield 11

multi-objective

test

Test Grounding

• Why?– High-level test suites are technology-neutral– Platform-specific translations required

• How?– Platform providers offer bespoke groundings– Broker@Cloud framework provides hooks

• XML test suite parsed as a Java metamodel instance• Visitor Design Pattern is used to generate concrete tests• Bespoke grounding to e.g. Java, SOAP, REST, Selenium

19/04/23 Anthony J H Simons, University of Sheffield 12

Example Java Grounding

19/04/23 Anthony J H Simons, University of Sheffield 13

auto-gen JUnit test assertions

SAP HANA Case Study

• User may book up to 5 days vacation– blocks of days can be booked, selected, deleted– prevents overlapping blocks, more than 5 days

• App designed using the SAP UI5 toolkit– visually pleasing point-and-click interface– operations nominally match the earlier specification– but a rich-client web app API is hard to test

• Deployed on the SAP HANA cloud platform– temporarily available at URI: https://brokercloud-

d050960trial.dispatcher.hanatrial.ondemand.com/

19/04/23 Anthony J H Simons, University of Sheffield 14

App Running on HANA

19/04/23 Anthony J H Simons, University of Sheffield 15

Selenium Test Driver

• Selenium: a tool for web-testing– normally used to capture-and-record a user's point-

and-click gestures interactively

– can replay recorded test scripts, stored as HTML tables, to run web app regression tests

• Leverage to test a rich-client cloud app– given the Selenium DOM for the web app

– translate high-level test suite into Selenium test scripts

– use the Selenium engine to execute the test scripts

19/04/23 Anthony J H Simons, University of Sheffield 16

Selenium Grounding Rules

19/04/23 Anthony J H Simons, University of Sheffield 17

Executable Test Scripts

19/04/23 Anthony J H Simons, University of Sheffield 18

Executing the Tests

19/04/23 Anthony J H Simons, University of Sheffield 19

Subtle Faults Detected

• Functionality– service failed to notify duplicate holiday booking as

a separate error case– service failed to distinguish overflow and duplicate

holiday booking responses

• Security– service executed operations in extra states, when

the operations should be ignored– service revealed response information, when the

request should be ignored

19/04/23 Anthony J H Simons, University of Sheffield 20

Beyond State-of-the-Art

• Extends leading SXM theory for MBT– Adds reasoning about the full SXM model– Adds validation and full verification capability– Adds test optimisation capability

• Exceeds all industrial testing methods– Full coverage of the specification achieved– All positive and negative test cases created– Identifies top 10% uniquely discriminating tests

19/04/23 Anthony J H Simons, University of Sheffield 21

• AcknowledgementFP7-ICT-2011-8 grant agreement no: 318392

• Project websitewww.broker-cloud.eu

Questions

19/04/23 Anthony J H Simons, University of Sheffield 22

References

• Broker@Cloud D20.1 State of the art and research baseline, http:www.broker-cloud.eu, 2013.

• Broker@Cloud D20.3 Requirements for continuous quality assurance and optimisation in cloud brokerage, http:www.broker-cloud.eu, 2013.

• AJH Simons and RE Lefticaru, Specification, verification and testing of brokered software services in the cloud, IEEE ICST 2015, Graz (submitted, under review)

• AJH Simons and RE Lefticaru, Stream X-Machine based verification and testing of cloud services, 2nd Int. Workshop on Cloud Service Brokerage, ESOCC, Manchester, 2 Sept., CCIS, 2014.

• M Kiran, A Friesen, AJH Simons and WKR Schwach, Model-based testing in cloud brokerage scenarios, 1st Int. Workshop on Cloud Service Brokerage, ICSOC, Berlin, 2 Dec., LNCS, 2013.

19/04/23 Anthony J H Simons, University of Sheffield 23