7/8 sep 2005nvo summer school 20051 part i: quick and dirty t he us n ational v irtual o bservatory...

7/8 Sep 2005NVO Summer School 2005 1

Part I: Quick and Dirty

THE US NATIONAL VIRTUAL OBSERVATORY

Web Service TechnologiesOr How the Magic Happens

Matthew J. GrahamCACR/Caltech


Overview

• Web services and SOA• RESTful services• Rich clients

• SOAP and WSDL• Attachments• Security• State• Asynchrony


Web services (oc)cult

• Invocations• Strange languages• Action at a distance• High priesthood


What is a web service?

• “A software system designed to support interoperable machine-to-machine interaction over a network. It has an interface described in a machine-processable format.” - W3C

• Not a new idea: RPC, RMI


Service Oriented Architecture

• An application architecture within which all functions are defined as independent services with well-defined invocable interfaces which can be called in defined sequences to form scientific processes.

• Not a new idea: CORBA, ICE• Principles:

– Service reusability– Service contract– Service loose coupling– Service abstraction

• Virtual Observatory = Service Oriented Astronomy

– Service composability– Service autonomy– Service statelessness– Service discoverability


How’s it done in the real world?

• “Things should be made as simple as possible, but no simpler” - Albert Einstein

• A web service is just a machine-readable web app and a web app is just a dynamic web site

• WWW is the largest, most distributed and scalable application on the planet HTTP and HTML (XML)


REST: The formal approach

• Architectural style not an implementation(http://www.ics.uci.edu/~fielding/pubs/dissertation/rest_arch_style.htm)

• Each resource has a URI• Exchange resource representations

(XML)• Uniform CRUD API:

HTTP protocol CRUD action Description

POST CREATE Create a new resource

GET RETRIEVE Retrieve a resource representation

PUT UPDATE Update a resource

DELETE DELETE Delete a resource


Accidentally RESTful

• HTTP GET/POST + XML (POX/HTTP)• Verbs allowed in URIs• Requires little new infrastructure - just HTTP and

XML processing technologies• Simple clients, e.g. wget or xsltproc• Commercially popular (85% of traffic, 6x faster):

– Amazon (http://www.amazon.com/gp/aws/landing.html)– Yahoo (http://developer.yahoo.net)– eBay (http://developer.ebay.com/rest)– Flickr (http://www.flickr.com/services/)– Del.icio.us (http://del.icio.us/doc/api)


What do RESTful services lack?

• Reliable messaging• Digital signatures• Message routing• Resource life cycle management• Asynchronous event notification


Is this a problem?

• Security – transport level (HTTPS): point-to-point– fast and well supported

• State– sessions

• Asynchrony – easy to implement


Client technologies: fat web pages

• XForms (http://www.w3.org/Markup)– MVC approach to web forms separates content

(XML model/instance) from presentation (XSLT/XHTML)

– Server-side implementation: • Orbeon (http://www.orbeon.org)• Chiba (http://chiba.sourceforge.net)

– Client-side implementation: • FormsPlayer (http://www.formsplayer.com)• FormFaces (http://www.formfaces.com) • Browsers (e.g. http://www.mozilla.org/projects/xforms)


AJAX (Asynchronous Javascript + XML)

• Uses browser’s XML support: DOM, XSLT• XMLHttpRequest• Google Maps, Google Suggest


The bleeding edge

• AFLAX (Asynchronous Flash + XML):– http://www.xamlon.com/software/

xamlonpro/flash/aflax.aspx– OpenLaszlo (http://openlaszlo.org)

• Mashups– On July 19, Google registered:

googlesolarsystem.*, googlegalaxy.*, and googleuniverse.*,


AJAX SOA


Part II: Standards

THE US NATIONAL VIRTUAL OBSERVATORY

Web Service TechnologiesOr How the Magic Happens

Matthew J. GrahamCACR/Caltech


Recap

• Client technologies: XForms, AJAX• RESTful approach:

– Distributed hypermedia model (WWW) is key architecture for loosely coupled distributed systems

– Each URL is a representation of some object, e.g. http://us-vo.org/conesearch?POS=123,34&SR=0.5

– Manipulate representation with CRUD interface– Application understands that resource access

method is semantically significant– Low REST vs. high REST


What is SOAP?

• Originally meant Simple Object Access Protocol

• An XML-based communication protocol and encoding format for exchanging structured information in a decentralized, distributed environment

• W3C specification (http://www.w3.org/TR/soap)


Anatomy of a SOAP message

• An envelope to encapsulate data which defines formatting conventions for describing the message contents and routing directions: header and body

• A message exchange pattern: request/response (RPC mechanism), fire-and-forget

• A transport or binding protocol• Data encoding rules for describing the

mapping of application-defined datatypes into an XML tag-based representation


SOAP example

Request:<soap:Envelope xmlns:xsi=http://www.w3.org/2001/XMLSchema-instance xmlns:xsd=http

://www.w3.org/2001/XMLSchema xmlns:soap="http://schemas.xmlsoap.org/soap/envelope/"> <soap:Body>

<ComovingLineOfSight xmlns="http://skyservice.pha.jhu.edu"> <z>float</z> <hubble>float</hubble> <omega>float</omega> <lambda>float</lambda>

</ComovingLineOfSight> </soap:Body>

</soap:Envelope>

Response:<soap:Envelope xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"

xmlns:xsd="http://www.w3.org/2001/XMLSchema" xmlns:soap="http://schemas.xmlsoap.org/soap/envelope/"> <soap:Body>

<ComovingLineOfSightResponse xmlns="http://skyservice.pha.jhu.edu">

<ComovingLineOfSightResult>float</ComovingLineOfSightResult>

</ComovingLineOfSightResponse></soap:Body>

</soap:Envelope>

http://www.w3.org/2001/XMLSchema-instance

http://www.w3.org/2001/XMLSchema-instance

http://www.w3.org/2001/XMLSchema



Why is SOAP better?

• Asynchrony• Routing• Reliable messaging (e.g. once-and-only

delivery, guaranteed or exact execution)• Send and receive complex datatypes to invoke

a particular method not just key-value pairs • Security • Binds to other protocols• Service description


What is WSDL?

• Web Services Description Language• An XML grammar for describing a web

service as a collection of endpoints capable of exchanging messages in a particular fashion

• W3C specification (http://www.w3.org/TR/wsdl)

• http://www.xmethods.net


Anatomy of a WSDL file

<definitions>

</definitions>

<message>* - model data exchanged<part></part>*

</message>

<binding>* - formatting and representation of SOAP

<operation>* message on the wire<input></input><output></output>

</operation></binding><service>* - identifies actual

endpoint for WS<port></port>*

</service>

<import>* - include other WSDLs<types> - define datatypes used in

<message><schema></schema>*

</types>

<porttype>* - a subset of operation supported for<operation>* an endpoint

<input></input> - define input and output parameters<output></output><fault></fault>*

</operation></porttype>


WSDL example<definitions xmlns:http=http://schemas.xmlsoap.org/wsdl/http/ xmlns:soap="http://schemas.xmlsoap.org/wsdl/soap/" xmlns:s=http://www.w3.org/2001/XMLSchema

xmlns:s0=http://skyservice.pha.jhu.edu xmlns:soapenc=http://schemas.xmlsoap.org/soap/encoding/

targetNamespace="http://skyservice.pha.jhu.edu" xmlns="http://schemas.xmlsoap.org/wsdl/"> <types><s:schema elementFormDefault="qualified" targetNamespace="http://skyservice.pha.jhu.edu"> <s:element name="ComovingLineOfSight">… <s:element minOccurs="1" maxOccurs="1" name="z" type="s:float" /> <s:element minOccurs="1" maxOccurs="1" name="hubble" type="s:float" /> <s:element minOccurs="1" maxOccurs="1" name="omega" type="s:float" /> <s:element minOccurs="1" maxOccurs="1" name="lambda" type="s:float" /> …</s:element> <s:element name="ComovingLineOfSightResponse”>… <s:element minOccurs="1" maxOccurs="1" name="ComovingLineOfSightResult" type="s:float" /> …</s:element> </s:schema></types> <message name="ComovingLineOfSightSoapIn"> <part name="parameters" element="s0:ComovingLineOfSight" /> </message> <message name="ComovingLineOfSightSoapOut"> <part name="parameters" element="s0:ComovingLineOfSightResponse" /> </message> <portType name="DistanceSoap"> <operation name="ComovingLineOfSight"> <documentation>Return the comoving line of sight distance...</documentation>

<input message="s0:ComovingLineOfSightSoapIn" /> <output message="s0:ComovingLineOfSightSoapOut" />

</operation> </portType> <service name="Distance">

<port name="DistanceSoap" binding="s0:DistanceSoap"><soap:address location="http://voservices.net/Cosmology/ws_v1_0/Distance.asmx" />

</port> </service></definitions>

http://schemas.xmlsoap.org/wsdl/http/



http://skyservice.pha.jhu.edu/

http://schemas.xmlsoap.org/soap/encoding/

http://schemas.xmlsoap.org/soap/encoding/


What about the binding?

<binding name="DistanceSoap" type="s0:DistanceSoap"><soap:binding

transport="http://schemas.xmlsoap.org/soap/http" style="document" />

<operation name="ComovingLineOfSight”><soap:operation

soapAction="http://skyservice.pha.jhu.edu/ComovingLineOfSight" style="document" />

<input><soap:body use="literal" />

</input><output>

<soap:body use="literal" /></output>

</operation></binding>


Binding attributes

• Style (representation on the wire)– rpc: the endpoint treats child elements in the

body as XML representation of method call (SOAP 1.1, sec. 7)

– document: the body can contain arbitrary XML

• Use (how data is serialized across the wire)– encoded: rules in a URL specified by

encodingStyle attribute– literal: rules specified by XML schema


WSDL binding flavours (I)

<message name=“Request”><part name=“x” type=“xs:int”/>

</message>

<message name=“empty”/>

<portType name=“foo”><operation name=“method”>

<input message=“Request”/><output message=“empty”/>

</operation></portType>

<message name=“Request”><part name=“x” type=“xs:int”/>

</message>





<types><schema>

<element name=“xElement” type=“xs:int”/>

</schema></types>

<message name=“Request”><part name=“x” element=“xElement”/>

</message>





RPC Document

Literal

Encoding


WSDL binding flavours (II)

<soap:envelope><soap:body>

<method><x xsi:type=“xs:int”>5</x>

</method></soap:body>

</soap:envelope>


<method><x>5</x>

</method></soap:body>

</soap:envelope>


<xElement>5</xElement></soap:body>

</soap:envelope>

RPC Document

Literal

Encoding


<types><schema>

<element name=“method”><complexType>

<sequence> <element name=“x” type=“xs:int”/></sequence>

</complexType></element>

</schema></types>

<message name=“Request”><part name=“parameters” element=“method”/>

</message>




</operation></portType

WSDL binding flavours (III)


<method><x>5</x>

</method>></soap:body>

</soap:envelope>

Document/literal wrapped


Which flavour to use?

• Doc style can pass entire transaction as an XML document (state)

• Doc style not constrained by RPC-oriented encoding

• Doc style can be validated at call time• Processing overhead in encoding payloads with

RPC• Doc style can use low memory parsers such as

SAX and StAX• RPC’s natural tendency to expose programming

language object structures doc/literal wrapped (95%)


Why not doc/literal wrapped?

• Overloaded operations:public void myMethod (int x, float y);public void myMethod (int x);

• Number of parameters:public void someOtherMethod(int x, float y);

• Data graphs:<complexType name=“Node”> <sequence>

<element name=“name” type=“string”/> <element name=“left” type=“Node”/> <element name=“right” type=“Node”/> </sequence></complexType>

RPC/encoding: <A> Literal:<A><name>A</name> <name>A</name><left href=“9999”/> <left><right href=“9999”/>

<name>B</name> </A> </left> <B id=“9999”> <right>

<name>B</name> <name>B</name>

</B> </right> </A>

A

Left Right

B

Left Right


Client Invocation Models

• Static: use generated stubs (wsdl2*)• Dynamic:

– no generated code– a proxy dynamically generates a class at runtime that

conforms to a particular interface, proxying all invocations to a single ‘generic’ method

– Examples: • Java : use javax.xml.rpc.Service.getPort() and createCall()• .NET : use RealProxy class (must extend ContextBound) or

Reflection.Emit

• Generic SOAP client: http://soapclient.com/soaptest.html


Interoperability

• Suitable for and capable of being implemented in a neutral manner on multiple operating systems and in multiple programming languages

• Not all web services are interoperable!• Web Services Interoperability

Organisation (http://www.ws-i.org)• WS-I Testing Tools

http://www.ws-i.org/


Attachments: opaque data

• “By value” – XML representationor– use xs:hexBinary or xs:base64Binary within the body– data expansion by a factor of ~1.33 - 4– anything within SOAP body gets parsed– processing costs of encoding/decoding

• “By reference”– attach pure binary data as external unparsed entity

outside SOAP message– use reference URI within the body


Reference solutions

• SwA (SOAP with Attachments)– Multipart MIME message: SOAP (0), data (1-n)– Use Content-Id as reference in body– Lack of length header on message sections– No recommendation just W3C Note

• DIME (Direct Internet Message Encapsulation)– Uses faster and more efficient binary encoding– No standard, disowned by Microsoft

• Both introduce a data structure outside realm of XML data model: no rules to specify how attachment content related to SOAP envelope so incompatible with WS-*


MTOM

• Message Transmission Optimization Mechanism

• Uses MIME - backwards compatible with SwA• Uses XOP:Include as reference mechanism

(XOP = XML Binary Optimized Packaging)• Conceptually binary data is base64-encoded in

SOAP XML document compatible with WS-*• Implementations:

– Axis2 (http://ws.apache.org/axis2)– WSE 3.0

(http://msdn.microsoft.com/library/default.asp?url=/library/en-us/dnwse/html/newwse3.asp)


Security

• Transport level (https)• Message level:

– End-to-end: allows for unlimited intermediaries– Data origin authentication– Different types of security tokens/credentials:

• unsigned (username/password)• binary (X.509 certificate) • XML (SAML token)

– Multiple credentials


WS-Security

• OASIS standard(http://www.oasis-open.org/committees/tc_home.php?wg_abbrev=wss))

• Security token validation (authentication):– validate authentication assertions made by principals

• Message integrity (signing):– verify message origin– validate encryption keys– confirm security token claims

• Message confidentiality (encryption)• Introduces extra XML into SOAP header


WSS Implementations

• Java:– WSS4J (http://ws.apache.org/wss4j)

• C#:– WSE 2.0

(http://msdn.microsoft.com/webservices/webservices/building/wse/default.aspx)

– WSRF.Net (http://www.cs.virginia.edu/~gsw2c/wsrf.net.html)

• Perl :– WS-Security will be supported by WSRF::Lite (but not yet)

(http://www.sve.man.ac.uk/Research/AtoZ/ILCT)

• Python:– pyGridWare (http://dsd.lbl.gov/gtg/projects/pyGridWare/)


State

• Stateless is good:– In case of failure, just restart without concern of

previous interactions (reliability)– New service instances can be created/destroyed in

response to system load (scalability)

• How to handle state?– Separate web service and state information (resource)– Identify resource with a unique key– Use message exchanges with the service to interact

with the resource (manipulate state)


WS-Resource

• An entity composed of a web service and a stateful resource

• The address is called an endpoint reference (WS-Addressing)

• ACID:– Updates made in all-or-nothing fashion (atomicity)– Consistent state even after failure (consistency)– Updates isolated within a given work unit (isolation)– Permanence of updates (durability)


WS-RF: the nuts and bolts

• WSDL for a stateful service:<definitions><import>*<types>

<xs:schema><xs:element

name=“StatefulResourceProperties”>…</xs:element><xs:schema>

<types><porttype wsdlpp:extends=“…”

wsrp:ResourceProperties=“tns:StatefulResourceProperties”>

• Implementations:– Java: GT4 (htttp://www.globus.org); Apache WSRF (http://ws.apache.

org/wsrf)– .NET: WSRF.Net (http://www.cs.virginia.edu/~gsw2c/wsrf.net.html)– Python: pyGridWare (http://dsd.lbl.gov/gtg/projects/pyGridWare/)– Perl: WSRF::Lite (http://www.sve.man.ac.uk/Research/AtoZ/ILCT)

http://ws.apache.org/wsrf




http://www.cs.virginia.edu/~gsw2c/wsrf.net.html





http://dsd.lbl.gov/gtg/projects/pyGridWare/







Asynchrony

• Real world is asynchronous• No current standards for asynchronous services but

most promising is OASIS ASAP (http://www.oasis-open.org/committees/tc_home.php?wg_abbrev=asap)

• Toolkits exist which facilitate asynchronous activities:– WS-RF (see above)– Axis2 (http://ws.apache.org/axis2)– JMS (http://java.sun.com/products/jms) /

Caffeine (http://caffeine.berlios.de/site/)– WSIF (http://www.apache.org/wsif)


Messaging operations

• WSDL defines four types of messaging operation that an endpoint can support:– One-way: endpoint receives a message– Request/response: endpoint receives a

message and sends a correlated message– Solicit/response: endpoint sends a

message and receives a correlated message– Notification: endpoint sends a message

• One-way/two-way transport behaviour


Patterns for asynchrony (I)

• Fire and Forget:

• Request/response (Transport timeout)

C S C S

C S


Patterns for asynchrony (II)

• Polling:

• Callback:

C S

C S


WS-Addressing

• No standard SOAP way to specify:– where a message is going– how to return a response– where to report an error

• WS-Addressing provides:– To– ReplyTo– FaultsTo– Anonymous– MessageId / RelatesTo– Standard for including service-specific

attributes


MEST (MESsage Transfer)

• Messaging:– No notion of client/server: just peers– Largely time independent: messages delivered

when peer is available– Messages can be duplicated and delivered to

multiple peers

• Messages and services are first class abstractions (no interfaces, data and operations)

• SSDL (http://www.ssdl.org)• Indigo: dual contracts are beyond WSDL


Conclusions

• Pick the right approach: do you need SOAP?

• Location of functionality: fat web pages• Think about interactions: WSDL• Use your favourite language and platform• Interoperability is a great idea• Be aware of emerging and converging

technologies and new paradigms

7/8 sep 2005nvo summer school 20051 part i: quick and dirty t he us n ational v irtual o bservatory...

Documents

nvo summer school

resource slide

implementation http

ebay http

yahoo http

openlaszlo http

planet http

orbeon http