ting ring with online ng - usi - faculty of informatics o re tion with high fidence ting ring with...

Towards ProTowards Pro active Adaptationactive Adaptationwith High Confidencewith High Confidencewith High Confidencewith High ConfidenceAugmenting Monitoring with Online TestingAugmenting Monitoring with Online Testing

Andreas MetzgerAndreas Metzger, Osama Sammodi, Klaus Pohl, Mark Rzepka, Osama Sammodi, Klaus Pohl, Mark Rzepka

Paluno (The Ruhr Institute for Software Technology)Paluno (The Ruhr Institute for Software Technology)University of DuisburgUniversity of Duisburg EssenEssenUniversity of DuisburgUniversity of Duisburg EssenEssenEssen, GermanyEssen, Germanywww paluno dewww paluno dewww.paluno.dewww.paluno.de

Osama Sammodi, SSE-Workshop, Westerholt, 04.12.2009

Funded by the European Community's 7th Framework Funded by the European Community's 7th Framework Programme FP7/2007Programme FP7/2007--2013 / Objective 1.22013 / Objective 1.2

www.swww.s--cubecube--network.eunetwork.eu

AgendaAgendaAgendaAgenda

•• Motivation and Problem StatementMotivation and Problem Statement

•• State of the ArtState of the Art

•• Solution Idea & Technical ChallengesSolution Idea & Technical Challenges•• Solution Idea & Technical ChallengesSolution Idea & Technical Challenges

•• Conclusion & Future WorkConclusion & Future WorkConclusion & Future WorkConclusion & Future Work

Osama Sammodi, SSE-Workshop, Westerholt, 04.12.2009A. MetzgerA. Metzger SEAMS@ICSE, Cape Town, May 2010SEAMS@ICSE, Cape Town, May 2010 22

Need for AdaptationNeed for AdaptationNeed for AdaptationNeed for Adaptation

Hi hl d i SBA t tHi hl d i SBA t t h fh f•• Highly dynamic SBA contexts:Highly dynamic SBA contexts: changes ofchanges of–– requirementsrequirements–– user typesuser types–– 33rdrd party servicesparty services

i idi id–– service providersservice providers–– ......N d dN d d d i i d id i i d i ii•• Need to respond toNeed to respond to deviations during rundeviations during run timetime–– no guarantees that 3no guarantees that 3rdrd party service fulfils its contract (SLA)party service fulfils its contract (SLA)h d t b h i f i f t t ( th I t t)h d t b h i f i f t t ( th I t t)–– hard to assess behavior of infrastructure (e.g., the Internet)hard to assess behavior of infrastructure (e.g., the Internet)during design timeduring design time

–– �…�…


�…�…

A. MetzgerA. Metzger SEAMS@ICSE, Cape Town, May 2010SEAMS@ICSE, Cape Town, May 2010 33

Reactive vs. ProReactive vs. Pro active Adaptationactive AdaptationReactive vs. ProReactive vs. Pro active Adaptationactive Adaptation

ii d id i•• ReactiveReactive adaptationadaptation–– observe running SBA instances (�“monitoring�”)observe running SBA instances (�“monitoring�”)–– trigger adaptation in case of failures, deviations,trigger adaptation in case of failures, deviations,changes, ...changes, ...

–– drawbacksdrawbacks: execution of faulty services, reduction of: execution of faulty services, reduction ofperformance, inconsistent endperformance, inconsistent end states, ...states, ...

•• ProPro activeactive adaptationadaptation–– adapt SBAadapt SBAadapt SBAadapt SBAinstances to preventinstances to preventthem from failingthem from failing


gg


ProPro active Adaptationactive AdaptationProPro active Adaptationactive Adaptation

•• IllustrationIllustrationSBASBA instanceinstance 11

SBASBA instance 2instance 2

SBASBA instanceinstance 11

..

..

..

Osama Sammodi, SSE-Workshop, Westerholt, 04.12.2009A. MetzgerA. Metzger SEAMS@ICSE, Cape Town, May 2010SEAMS@ICSE, Cape Town, May 2010 55SBASBA instance ninstance n

Avoiding Unnecessary AdaptationsAvoiding Unnecessary AdaptationsAvoiding Unnecessary AdaptationsAvoiding Unnecessary Adaptations

U d t ti bU d t ti b•• Unnecessary adaptations can beUnnecessary adaptations can be–– costlycostly (e.g., SLA negotiation; expensive alternative services)(e.g., SLA negotiation; expensive alternative services)

faultyfaulty ( bi di f �“b �” i )( bi di f �“b �” i )–– faultyfaulty (e.g., binding of a �“buggy�” service)(e.g., binding of a �“buggy�” service)

PProro active adaptation should be based onactive adaptation should be based onconfidenceconfidence that failure will occur in SBA instancesthat failure will occur in SBA instancesconfidenceconfidence that failure will occur in SBA instancesthat failure will occur in SBA instances

•• ButBut: difference: difference from traditional SW systems:from traditional SW systems:–– lack of controllack of control on 3on 3rdrd party servicesparty services–– functionality andfunctionality and QoSQoS can varycan varyb t i i tib t i i tibetween service invocationsbetween service invocations(even if input is identical)(even if input is identical)

–– Service implementation canService implementation can changechange


pp ggwithout noticewithout notice


State of the Art (1)State of the Art (1)State of the Art (1)State of the Art (1)•• Statistical evaluation / data mining ofStatistical evaluation / data mining of monitorinmonitoring datag data [Liang et al.[Liang et al.

2006 Fu et al 20072006 Fu et al 2007 SahooSahoo et al 2003]et al 2003]2006, Fu et al. 2007,2006, Fu et al. 2007, SahooSahoo et al. 2003]et al. 2003]–– requires large volume of monitoring datarequires large volume of monitoring data–– adaptation may invalidate past monitoring data and thus predictionadaptation may invalidate past monitoring data and thus prediction

•• StatisticalStatistical testintestingg [[PoorePoore et al. 1998, Trammell et al. 1995]et al. 1998, Trammell et al. 1995]prediction ofprediction of overalloverall system�’s reliabilitysystem�’s reliability (and not individual failures)(and not individual failures)–– prediction ofprediction of overalloverall system s reliabilitysystem s reliability (and not individual failures)(and not individual failures)

–– based on usage profile of the systembased on usage profile of the system–– requires large number of test casesrequires large number of test cases ((significant effort/costs due to invocationsignificant effort/costs due to invocation

f l )f l )of external services)of external services)

•• Reliability modelling andReliability modelling and analanalyysissis [[GoelGoel et al. 1985, Zhang et al. 2008,et al. 1985, Zhang et al. 2008,ggRoshandelRoshandel et al 2006]et al 2006]–– prediction ofprediction of overalloverall system�’s reliabilitysystem�’s reliability (and not individual failures)(and not individual failures)–– static analysis techniquestatic analysis technique (can complement monitoring & online testing)(can complement monitoring & online testing)


static analysis techniquestatic analysis technique (can complement monitoring & online testing)(can complement monitoring & online testing)


State of the Art (2)State of the Art (2)State of the Art (2)State of the Art (2)

J i i i & i ffJ i i i & i ff•• Joint monitoring & testing effortsJoint monitoring & testing efforts [www.secse[www.secse project.eu]project.eu]

–– Proposed applications:Proposed applications:1.1. Use monitoring data to mimic service behaviourUse monitoring data to mimic service behaviour2.2. Generate test cases to violate the SLAsGenerate test cases to violate the SLAs (search(search based technique usingbased technique using

monitoring data)monitoring data)

3.3. Analyse the monitoring data to infer invariantsAnalyse the monitoring data to infer invariants

•• Online testingOnline testing [Wang et al. 2004,[Wang et al. 2004, DeussenDeussen et al. 2003,et al. 2003, BaiBai et al. 2007]et al. 2007]

–– �“Definition�”:�“Definition�”: SBA is fed with dedicated test input in parallelSBA is fed with dedicated test input in parallelDefinition :Definition : SBA is fed with dedicated test input in parallelSBA is fed with dedicated test input in parallelto its normal use and operationto its normal use and operation (�“production system�”)(�“production system�”)

Current techniques not used for building confidenceCurrent techniques not used for building confidence


–– Current techniques not used for building confidenceCurrent techniques not used for building confidence99A. MetzgerA. Metzger SEAMS@ICSE, Cape Town, May 2010SEAMS@ICSE, Cape Town, May 2010

Solution IdeaSolution Idearesponseresponse confidenceconfidence

Solution IdeaSolution Idea

timetime95%95%

xxxxxxxxxxxx

xx computed confidencecomputed confidencexxxx xxxx xxxx xxxx xxxxxxxx

ThresholdThreshold(800 ms)(800 ms)

online testingonline testing datadata

monitored datamonitored data

timetime

online testingonline testing datadata


timetime

AssumptionsAssumptionsAssumptionsAssumptions

ilil i ii i ii1.1. FailureFailure in constituentin constituent serviceservicerequirements deviationrequirements deviation (end(end toto end quality violation)end quality violation)

–– not necessarily the case in all situationsnot necessarily the case in all situations•• e.g.; slower response of service B compensated bye.g.; slower response of service B compensated byfaster response of service Afaster response of service Af p ff p f

2.2. Test of services hasTest of services has no side effectsno side effects.. Test of services hasTest of services has no side effectsno side effects–– e.g., no books are delivered if testing online book servicee.g., no books are delivered if testing online book service

3.3. ServicesServices notify about changesnotify about changes that could invalidatethat could invalidatemonitoring/testing datamonitoring/testing data


monitoring/testing datamonitoring/testing data


Steps of the ApproachSteps of the ApproachSteps of the ApproachSteps of the Approach

Important for

1. Determine1. DetermineRepresentative DataRepresentative Data

Important forTrigger B!

2. Determine2. Determine

Two possibleTwo possiblekinds of triggers:kinds of triggers:

2. Determine2. DetermineCurrent ConfidenceCurrent Confidence

[not[not

A)A) FailureFailureobserved byobserved by

[sufficient][sufficient]

sufficientsufficient]]3. Execute3. ExecuteOnline TestsOnline Tests

monitoringmonitoring

B)B) Change ofChange ofserviceservice

4. Predict Failure4. Predict Failure 5. Decide on Pro5. Decide on Pro

serviceservice


OccurrenceOccurrence active Adaptationactive Adaptation


Technical ChallengesTechnical ChallengesTechnical ChallengesTechnical Challenges


2. Determine2. Determine2. Determine2. DetermineCurrent ConfidenceCurrent Confidence

[not[not


sufficientsufficient]]3. Execute3. ExecuteTestsTests

3. Execute3. ExecuteOnline TestsOnline Tests








[not[not








Relaxing the AssumptionsRelaxing the AssumptionsRelaxing the AssumptionsRelaxing the Assumptions

1.1. FailureFailure requirements deviationrequirements deviation•• SS Cube approach:Cube approach: RunRun time verificationtime verification [Bianculli et al. 2008 ,[Bianculli et al. 2008 ,

Gehlert et al. 2010]Gehlert et al. 2010]

•• Reduces effort for online testing:Reduces effort for online testing:I iti t t l ifI iti t t l if d i ti (d i ti ( St t A)St t A)–– Initiate steps only ifInitiate steps only if deviation (deviation ( Strategy A)Strategy A)

2.2. Tests haveTests have no side effectsno side effects•• Service invocation = query onlyService invocation = query only•• Dedicated test mode / interfaceDedicated test mode / interface

3.3. Notification about changesNotification about changes•• Novel service registriesNovel service registries

N SLA iN SLA i


•• New SLA constituentsNew SLA constituents


Conclusion and Future WorkConclusion and Future WorkConclusion and Future WorkConclusion and Future Work

•• Avoid unnecessary proAvoid unnecessary pro active adaptationsactive adaptations•• Avoid unnecessary proAvoid unnecessary pro active adaptationsactive adaptations•• Many data points needed for confident predictionMany data points needed for confident prediction

–– Thus: augment monitoring with online testingThus: augment monitoring with online testing

•• Interesting challenges remain to be addressedInteresting challenges remain to be addressed•• Interesting challenges remain to be addressedInteresting challenges remain to be addressed

P i i h h i f h F I f S iParticipate to the shaping of the Future Internet of ServicesApply as S Cube Associate Member

Application Deadline: May, 17

www s cube network eu


www.s cube network.eu

References (1)References (1)• [Bai et al. 2007] Bai, X., Chen, Y., Shao, Z.: Adaptive web services testing. In: 31st Annual International

Computer Software and Applications Conference (COMPSAC). (2007) 233�–236• [Bianculli et al. 2008] Bianculli, D. et al.. A Guided Tour through SAVVY WS: a Methodology for[ ] , g gy

Specifying and Validating Web Service Compositions. In Proceedings Advances in SoftwareEngineering 2008

• [Chan et al. 2007] Chan, W., Cheung, S., Leung, K.: A metamorphic testing approach for online testingof service oriented software applications. International Journal of Web Services Research 4 (2007)61�–81

• [Deussen et al. 2003] Deussen, P., Din, G., Schieferdecker, I.: A TTCN 3 based online test andvalidation platform for Internet services In: Proceedings of the 6th International Symposium onvalidation platform for Internet services. In: Proceedings of the 6th International Symposium onAutonomous Decentralized Systems (ISADS). (2003) 177�–184

• [Fu el al. 2007] Song Fu and Cheng Zhong Xu. Exploring event correlation for failure prediction incoalitions of clusters. Proceedings of the 2007 ACM/IEEE conference on Supercomputing, Novembercoalitions of clusters. Proceedings of the 007 ACM/I conference on Supercomputing, November10 16, 2007, Reno, Nevada

• [Gehlert et al. 2010] Gehlert, A. et al.: Exploiting Assumption Based Verification for the Adaptation ofService Based Applications. In Proceedings SOAP @ SAC 2010

• [Goel et al. 1985] Goel, A.L.: Software reliability models: Assumptions, limitations, and applicability.IEEE Trans. Softw. Eng. 11 (1985) 1411�–1423


References (2)References (2)

• [Liang et al. 2006] Y. Liang, Y. Zhang, A. Sivasubramaniam, M. Jette, and R. K. Sahoo. BlueGene/Lfailure analysis and prediction models. In Proceeding of IEEE International Conference on DependableSystems and Networks (DSN), 2006Systems and Networks (DSN), 2006

• [Poore et al. 1998] Poore, J., Trammell, C.: Engineering practices for statistical testing. Crosstalk: TheJournal of Defense Software Engineering 11 (1998) 24�–28

• [Roshandel et al 2006] Roshandel, R., Banerjee, S., Cheung, L., Medvidovic, N., Golubchik, L.:[ ] , , j , , g, , , , ,Estimating software component reliability by leveraging architectural models. In: ICSE. (2006) 853�–856

• [Sahoo et al. 2003] R. K. Sahoo, A. J. Oliner, I. Rish, and et al. Critical event prediction for proactivemanagement in large scale computer clusters. In Proceeding of ACM International Conference onKnowledge Discovery and Data Dining (SIGKDD), 2003

• [Trammell wt al. 1995] Trammell, C.: Quantifying the reliability of software: statistical testing basedon a usage model In: ISESS �’95: Proceedings of the 2nd IEEE Software Engineering Standardson a usage model. In: ISESS 95: Proceedings of the 2nd IEEE Software Engineering StandardsSymposium, Washington, DC, USA, IEEE Computer Society (1995) 208

• [Wang et al. 2004]Wang, Q., Quan, L., Ying, F.: Online testing of Web based applications. In:Proceedings of the 28th Annual International Computer Software and Applications Conferenceg p pp(COMPSAC). (2004) 166�–169

• [Zhang et al. 2008] Zhang, F., Zhou, X., Chen, J., Dong, Y.: A novel model for component basedsoftware reliability analysis. In: HASE �’08: Proceedings of the 2008 11th IEEE High Assurance Systems


Engineering Symposium, Washington, DC, USA, IEEE Computer Society (2008) 303�–309


End of SlidesEnd of Slides

Osama Sammodi, SSE-Workshop, Westerholt, 04.12.2009A. Metzger SEAMS@ICSE, Cape Town, May 2010 22

ting ring with online ng - usi - faculty of informatics o re tion with high fidence ting ring with...

Documents