istituto di scienza e tecnologie dell'informazione “a ... · the definition of an...

Istituto di Scienza e Tecnologie dell'Informazione “A. Faedo”Software Engineering Laboratory

SOA Test Governance:enabling service integration testing

across organization and technology borders

Keynote Talk @ WebTest2009: 1st Int. Wksp on Web Testing

April 4th, 2009

Co-located with ICST 2009, Denver, Colorado, USA

Antonia [email protected]

Andrea [email protected]

WebTest2009: SOA Test Governance ©2009 A.Bertolino, A. Polini 2

Agenda

SOASOA GovernanceSOA Testing Challenges SOA Test Governance (STG)STG Cases:

AuditionSOCT

Conclusions&discussion


SOA: a new paradigm

Service-orientation is claimed to be the “New Computing Paradigm for the Networked World”

(c.f.r. the theme of the First ICSOC in 2003)Revolution or Evolution?

Service-orientation (like Rome) “was not built in a day”: it is nota revolution in IT, rather it is the natural (and relatively slow) evolution from the confluence of several recent and less recentapproaches and technologies.

The WWW era in the 90’s heralded SOA byemphasizing:

shift from intra-organization to inter-organization integrationneed: “B2B” adventincreasingly dynamic nature of IT market: companies enter and exit the scene continuously and fastly


SOA promises

IT-based companies seem infatuated of the SO wave and in different ways and with differentexpectations are jumping on the bandwagon.The new paradigm brings in many attracting features, among which:

reuseinteroperabilityloose couplingstandardization flexibilityfaster time to deployment


SOA in numbers“SOA will be used in more than 50 percent of new mission-critical operational applications and business processes designed in 2007 and in more than 80 percent by 2010”

Press Release from Gartner Symposium/ITxpo2007: Emerging Trends

In its report “SOA Infrastructure Market Shares, Strategies, and Forecasts, 2008 to 2014”, Research And Markets predicts SOA worldwide market will grow froman estimated $2 billion in 2007 to more than $9 billion by2014

See at http://blogs.zdnet.com/service-oriented/?p=1104, or youcan buy the whole report from R&M (for € 3,531.00)

http://blogs.zdnet.com/service-oriented/?p=1104


Conceptual View

Service-orientation is a design paradigm comprised of service-orientation design principles. The logic units are services yielding distinct design characteristics that support the overall goals and vision of service-oriented computing.

[From Thomas Erl’s book: “SOA: Principles of Service Design”Copyright Prentice Hall/PearsonPTR]


ServiceA software service is a software component providing a functionality to other software componentsA service defines a precise interface to be used by clients in order to access it. Precisely, each service exposes one or more access points, defining the location where a provided functionality can be accessedThe interface and the access points can be dynamically retrieved and used by other software componentsService development is not related to the requirements of its future usersService is an abstract concept that is realized through the definition of an “including” infrastructure


Service Oriented ArchitectureSOA is a conceptual architecture supporting the idea and the characteristics of services.In a minimalistic scenario there exist a service provider and a service requestor or consumer.

Publishing a Web service involves to create the software artifact and make it accessible to potential consumers.

Service discovery is the processof finding a suitable service fora given task. Optionally a provider can explicitly register a service within a service registry or broker.


SOA DevelopmentTraditional SE emphasizes the importance of fully documentingsystem requirements and specifications, based on which thoroughanalyses of the system under development can be performed asearly as possible

SOA development in its very conception is drastically different: the vision is that SOA applications are composed on the fly at runtime using existing services

Therefore, SOA development is a collaborative effort fromseveral parties: service developers, service providers, service brokers, application integrators.

SOA engineering tasks are performed by separate organizations, none of which having complete information or control over the building pieces.


Need for SOA Governance

To achieve SOA promises we need to set the “Rules of the game” that the stakeholders haveto abide by

“Counterintuitive as it may seem, SOA requiresmore organizational discipline than previousdevelopment models. Your intuition might tellyou that flexibility results from fewer rules, not more, but that’s not the case”.

[Philippe J. Windley. SOA governance: Rules of the game. On line at http://www.infoworld.com, January 2006]


Defining SOA Governance


GovernanceThe rules, policies, practices and responsibilitiesby which any complex system of relations and interactions (a country, a company, a social community) is controlled and administered.

(as compared to “government” term) brings in notions of

societal participation and shared responsibility;moral qualification, orientation to the common welfare

(e.g., Wikipedia states: assuring, on behalf of thosegoverned, a worthy pattern of good while avoiding anundesirable pattern of bad)


Enterprise GovernanceGovernance applied to a business organizationaimed at:

providing strategic directionsensuring that company objectives are achievedascertaining that risks are appropriately managedverifying that the enterprise’s resources are usedresponsibly.

EG’s emphasis is on business goals

[International Federation of Accountants, Professional Accountants in Business Committee. “Enterprise governance: Getting the balance right.”http://www.ifac.org/Members/DownLoads/EnterpriseGovernance.pdf, 2004]


IT Governance

IT Governance is a subset of EG, of growingimportance in modern enterprises, thatspecifically focuses on IT assets and theirexploitation“ITG is the strategic alignment of IT with the business such that maximum business value isachieved through the development and maintenance of effective IT control and accountability, performance management and risk management”

[Phyl Webb, Carol Pollard, and Gail Ridley. Attempting to define IT governance: Wisdom or folly? Hawaii Int. Conf. on System Sciences, 8:194a, 2006]


SOA Governance

Different key components make SOA governance at design-time and at run-timeAt design time it manages and deploys servicesthroughout their lifecycle in order to ensurehigh levels of control and monitorabilityAt run-time policies and procedures are embedded into the service management infastructure

e. g. the service registry


Governance intra-organization

Assigns responsibilities, authorities and tasks(structural component of governance)Establishes measurement and enforcementmechanisms of assigned roles and responsibilities (dynamic or measurementcomponent of governance)


Governance intra-organization (ctd)Taken from [S. Chulani, C. Williams and A. Yaeli, Software DevelopmentGovernance and Its Concerns, SDG’08, May 12, 2008, Leipzig, Germany]


Inter-organization level

Governance is listed among the “grandchallenges” for Service-oriented computing in reference 1. below, because of the “cross-organizational nature of end-to-end business processes”Indeed, service compositions across differententerprises must establish ownership and governance responsibilities.

1. [M. Papazoglou, P. Traverso, S. Dustdar, F. Leymann, and B. Kramer. Service Oriented Computing Research Roadmap. In DagstuhlSeminar Proceedings 05462 (SOC), 2006.]


Similarity between SO and CB development

There are similarities between CB developmentand Service-oriented life-cycleSuch similarities can be traced back to the separation of roles and concerns between thosewho develop a piece of software (a componentor a service) and those who later use such pieceof software for invocation or for integration


Component/Service Developer

Public Comp. Repository

decides ( on business grounds)

develops Component Spec, Code and Validate

Component Documentation

For ext. use

Custom piece or reusable Component ?

augments

packages

Requirements

Maintenance

Code

Specification

Testing

Design


Component/Service UserSystem Architecture

Component AComponent C

Component B

Component D

Candidatecomponent


System Assembly

Discovers and selects

Analyzes and tests

Integrates

basedbased onon

in the in the contextcontext ofof


CommunicationBetween

Developer and User

Seen altogether


decides ( on business grounds)

develops Comp.Spec, Code and Validate

Component Documentation

For ext. use

Custom piece or reusable Comp.?

augments

packages

Candidatecomponent


System Assembly

Discovers and selects

Analyzes and tests

Integrates


Concerning testing

acceptancetest

systemtest

integrationtest

unittestcode

design

specs

reqs

The classical V-model presupposes that the software tester yields full control and observability

Testing in the traditional lifecycle relies on planning ahead based on development artifacts, and testing before release:


How is SOA testing different?

We get all “traditional” testing challenges (no discount!)

PLUS new ones:Global protocols of service interactionsAdaptability and dynamicityApplication “ilities”, e.g., availability, robustness, reliability, …, Distributedness, heterogeneity, mobility, …, aspects of the implementation first-classcitizen


Challenges in SOA testing

Reduced Control:control all and only through the provided interfaceno control over the services lifecycle (un-alertedevolution)

Reduced Observability:inter-organization distributednessno access to code neither to its environment

Reduced Information Runtime integration:

dynamic discovery and bindingcontext-awareness

Increased relevance of QoS characteristics


…and opportunities

Declarative and formal specification publiclyavailable both for functional and non functionalcharacteristicsNew “control” point in the platform to whichtesting activities can be associated

E.g., directory servicesOpen and standardized protocols and formats

SOAP, WSDL, XMLOrchestration provides a baseline for integration testingChoreography provide an open specification to whichcooperating services must conform

Need of trust among organizations


Need for testing research

From the EU report: “The service engineering area: An overview of its current status and a vision of its future” -July 2005ftp://ftp.cordis.europa.eu/pub/ist/docs/directorate_d/st-ds/sota_v1-0.pdf

“The area of testing services and their specificationshas until now received limited attention from the research community. ….”


Introducing SOA Test Governance

SOA Test Governance


Introducing SOA Test Governance

SOA Test Governance

STG is a subset of SOA governance thatspecifically concerns the establishment

and enforcement of policies, procedures, notations and tools that are required to

enable distributed testing of service integrations


STG: an artificial idea?Analysis of interacting BPEL web servicesX Fu, T Bultan, J Su – Proc. WWW 2004

…We assume that there is a virtual watcher which records the sequence of messages as they are sent by the peers. …

Model-based verification of web service compositionsH Foster, S Uchitel, J Magee, J Kramer – Proc. ASE 2003

… We need a specification that can closely resemble workflowspecification languages, and a tool to simulate from the specificationcreated in this language. …

Automatic conformance testing of web servicesR Heckel, L Mariani - Proc. FASE 2005

…we foresee the introduction of High-Quality Service Discovery agencies …. automatically tests a Web Service with respect to a provided model consisting of GT rules that specify the individual operations of the service.. …

……

All above and many others are assuming a STG


A 2007 press release from Gartner warns against risks of SOA project failure, which are both technical and organizational.

Gartner’s ‘hit list’ of the most common organisationalerrors

includes:1. Overlooking governance2. Thinking an SOA project should be organised just

like any other application development project3. Not anticipating service number explosions in a

maturing SOA4. Giving up on an integration competency center or

SOA center of excellence5. Outsourcing architects (or not having them at all)

[taken from http://www.gartner.com/it/page.jsp?id=508397]

Risks of Failure in SOA Projects


A 2007 press release from Gartner warns against risks of SOA project failure, which are both technical and organizational.

Gartner’s ‘hit list’ of the most common organisationalerrors

includes:1. Overlooking governance2. Thinking an SOA project should be organised just

like any other application development project3. Not anticipating service number explosions in a

maturing SOA4. Giving up on an integration competency center or

SOA center of excellence5. Outsourcing architects (or not having them at all)



“Through 2010, the biggest barriers to SOA adoption will be non-technical issues related

to inadequate governance”Gartner Research Vice Presidents, 2006


The ‘hit list’ of the most common technological errorsincludes:

1. Underestimating the technical complexity of a large-scale SOA

2. Bad selection of application infrastructure components (ESB, orchestration and adapters)

3. Insufficient validation of the SOA enabling technical infrastructure implementation (for example, no proof of concept and no stress tests)

4. SOA infrastructure, services and consumer applications are insufficiently instrumented for security/management/troubleshooting

5. Too-coarse/too-fine service granularity6. Insufficient/not up-to-date documentation




The ‘hit list’ of the most common technological errorsincludes:

1. Underestimating the technical complexity of a large-scale SOA

2. Bad selection of application infrastructure components (ESB, orchestration and adapters)

3. Insufficient validation of the SOA enabling technical infrastructure implementation (for example, no proof of concept and no stress tests)

4. SOA infrastructure, services and consumer applications are insufficiently instrumented for security/management/troubleshooting

5. Too-coarse/too-fine service granularity6. Insufficient/not up-to-date documentation


Risks of Failure in SOA Projects “testing is critical and at least 25

percent of the effort in a SOA project should be dedicated to this activity”

Gartner Vice President, 2005


Building a STG framework


STG Foundation


STG design


STG run-time


STG enforcement


STG stakeholders

Governance Boardthis is the organism that is deputed to define the rules. Typically will be constituted by experts fromorganizations willing to cooperate or interested in establishing an environment to be used forcooperation. At the same time the governance boardwill define the various roles and will assign specifictasks to the various roles.


Example: WS-I

Web Services Interoperability Organization (WS-I) is an open industry consortium chartered to promote Web Services interoperability across platforms, operating systems, and programming languages.

http://www.ws-i.org/

To ensure interoperability, Web services must comply with standards purposely created to enable communications across heterogeneous environments, both inside and across enterprise firewalls.

WS-I does not itself define standards for web services; rather, in cooperation with standards development organizations such as OASIS / W3C, it creates guidelines and tests for interoperability.


STG stakeholders

Service developer: develops a service, may be asconsequence of a request made by a service provider

governance within each given context could impose developmentstandards, implementation of specific interfaces and so on.

See, e.g., the role of service developer in SOCT case

Service provider: deploys the service over a server and makes it available to external organizations (for pay-per-use services it is the provider who gets money)

Also in this case the governance could impose rules on the way in which a service is provided, for instance requiring thatspecific information is publishedProvider can play the role of user when one of its services needto interact with a service deployed by another provider.


STG stakeholders

Service integrator: derive new functionality integratingexisting services.

is influenced by the governance that can define the rules and policies to be followed in the integrationcode for the composition is available, invoked services insteadare not generally under the control of the integrator

Service user: accesses and uses a serviceHas to abide by the established rules when using a service.


STG stakeholders

Monitor and enforcerThese are specific organizations that can usemechanisms within a SOA infrastructure to checkthat what is happening is in-line with what specifiedby the governance. The governance will have topreviously define which are the information ofrelevance for monitoring and which are the specificpowers of the “monitor and enforcer” organization

Certification Authorities (not shown in Figure)As standard architectures and policies are established, CAs will likely appear. The certification authority can provide formal“stamp” of approval


STG: two cases

STG in choreography testing: AuditionAntonia Bertolino and Andrea Polini. The auditionframework for testing web services interoperability. In Proc. EUROMICRO ’05

STG in orchestration testing: SOCTCesare Bartolini, Antonia Bertolino, and EdaMarchetti. Introducing service-oriented coverage testing. In ASE Wksp ARAMIS 2008


Audition testingThe Audition testing framework proposed toadd a testing phase before the service goes“on the scene”

The objective is to create a discovery service with trustable entries: registration isguaranteed only to services carefully tested

Development Admission Publication Live UsageInstallation

DevelopmentDevelopment TestingTestingAuditionAudition MonitoringMonitoring


STG behind Audition

On a pure SOA-based scenario the frameworkis not applicable. STG involves:

Service provider must be available and equipped foron-line testingRegistry takes on an enhanced roleThe testing service is an additional componentA choreography specifier publishes theirspecifications (according to a standardized format)Specific enforcement mechanisms must beimplemented


STG: two cases

STG in choreography testing: AuditionAntonia Bertolino and Andrea Polini. The auditionframework for testing web services interoperability. In Proc. EUROMICRO ’05

STG in orchestration testing: SOCTCesare Bartolini, Antonia Bertolino, and EdaMarchetti. Introducing service-oriented coverage testing. In ASE Wksp ARAMIS 2008


WS testing

Similarly to testing of Component-based systems, the testing process of service-oriented systems isdecomposed between two distinct stakeholders:

Service DeveloperFull access to code and specsDoes not know (future) use context

Service User/IntegratorOnly BB testingNeeds accompanyingdocumentationdesign

for test

Alice:Service Developer

Bob: Service User


WB testing for SOA?

Services need to ensure “implementationneutrality”, i.e., the internal implementationdetails of a service must be totally uninfluentialto its usage:

Services must be used as black boxes

“[…] to users and systems integrators, servicesare just interfaces. This hinders white-boxtesting methods based on code structure and data flow knowledge.[…]”

[ G. Canfora, M. Di Penta. “Testing Services and Service-centricSystems: Challenges and Opportunities”. IEEE IT Pro, 2006]


White-box testing of SOA?

It is a pity that service integrators cannot usewell-established and effective testing methodslike coverage measures.

we exploit the very feature of service-oriented technologyimplement coverage testing, realizing what we call

Service-oriented Coverage Testing (SOCT)


SOCT scenario

TCOV Provider

TestableServices 2: Invokes

testable services

3: Logs coverageinformation

Coverage ReportServices

4: Gets test reports

1: Instrumentsservices withTCOV probes


Bob: Service User


WB testingthrough TCov

Test roles in SOCT

The basic assumption behind SOCT is an STG agreedamong involved stakeholders:

Service Developer must be willing to incorporate the probes intothe servicesService Integrator performs WB testing, by analysing the log reports exported by the “testing Services”Appropriate “Testing Services” are deployed by (external or internal) service providers, Testing Houses, Regulatory bodies …


Bob: Service User


STG behind SOCT

Service developer must be willing to in strumentservice with the standardized probesTCov is a new stakeholder

Logs test runsProvides test reports

Service integrator interacts with TCov toobtain coverage measures


SOCT idea: Considerations

The framework presupposes a complexscenario, difficult to realizeNevertheless, all WS* technology assumesstandard processes and documentationCan be applied with varying thoroughness and at different levelsGovernance bodies must regulate the process


ConclusionsWe have introduced STG conceptWe have discussed challenges of WS testing and pointed at STG as a potential way to address themWe have outlined a generic minimal STG frameworkWe have overviewed two previous proposed approaches, in relation with STG:

AuditionSOCT

We believe STG is a sensible important concept forSOA testing


…to be doneSurvey literature and classify STG impliedapproachesRaise abstraction level and also look at process/organization issuesEmbrace and spread STG awarenessBuild concepts and processes for STGA consensus-based community effort


WEbTest 2009

SOA Test Governance:SOA Test Governance:enabling service integration testingenabling service integration testing

across organization and technology bordersacross organization and technology borders

istituto di scienza e tecnologie dell'informazione “a ... · the definition of an...

Documents