advanced publish/subscribe communication services antonio carzaniga and alexander l. wolf university...

Advanced Publish/Subscribe Communication Services

Antonio Carzaniga and Alexander L. Wolf

University of Colorado at Boulder

Boulder, Colorado USA

{carzanig,alw}@cs.colorado.edu

About the Tutors

Antonio Carzaniga– PhD: Politecnico di Milano, Italy– Previously at CEFRIEL

Alexander L. Wolf– PhD: University of Massachusetts, Amherst USA– Previously at AT&T Bell Laboratories

General research interests– software engineering, distributed systems,

security, and networking– Developers of the Siena pub/sub system

Outline

What is publish/subscribe communication?Service models Implementation modelsSurvey of representative pub/sub systemsUsing publish/subscribe communicationAdvanced topics

Database triggers– Ingres

Message-oriented middleware– IBM MQSeries

Message-based software bus architecture– Field– Sun ToolTalk– DEC Fuse

Origins of Publish/Subscribe

mailboxmailboxmailboxmailbox

client 1client 1

client 2client 2

busbus

client 1client 1 client 2client 2

client 3client 3

clientclient

Common Characteristics

Asynchrony– send and forget

Reactive– computation driven by receipt of message

Location/identity abstraction– destination determined by receiver, not sender

Loose coupling– senders/receivers added without reconfiguration– one-to-many, many-to-one, many-to-many

Levels of Deployment

From machines to whole networks– Single machine

» call back, signal

– LAN» RPC

– WAN» client/server, overlay router network

Single machine/LAN solutions are matureWAN (e.g., the Internet) solutions still require

significant new research advances

Basic Approaches to Publish/Subscribe

Listener model– receivers register with senders– sender/receiver communication is point-to-point

Database trigger model– “senders” update tables– “receivers” (or internal code) notified of changes

Broker model– receivers register with common message broker– sender/receiver communication is mediated by

message broker through a middleware service

Publish/Subscribe Middleware Service

publish/subscribeservice

publish/subscribeservice

wantlow air faresto Europe

wantlow air faresto Europe

want to flyDecemberDEN-MXP

want to flyDecemberDEN-MXP

wantspecial offers

by United

wantspecial offers

by United

subscribe

notify

United offersDEN-MXPOctober

United offersDEN-MXPOctober

Alitalia offersDEN-MXPNov-Feb

Alitalia offersDEN-MXPNov-Feb

publish

subscriber

publisher

subscription

publication

notification

Sampling of Applications

GamingWorkflowAuctioning Intrusion detectionNetwork managementApplication management

File sharingSensor gridsNews distributionGroup communicationResource discoverySoftware component integration

General Goals and Objectives

Dynamic integration of heterogeneous computing and information services

Timely information dissemination

Support for loosely-coupled, distributed applications

Scalable deployment

Outline


Basic Elements of a Service Model

Data model– structure– types

Subscription model– filter language– scope– subscription modifications

General challenge– expressiveness versus scalability

Data Model

United DEN MXP 850 USD

Fare OfferFare Offer

getAirline

setAirline

getOrigin

setOrigin

getDestination

setDestination

getPrice

setPrice

getCurrency

setCurrency

Airline United

Origin DEN

Destination MXP

Price 850

Currency USD

string

list of values

set of named values

<fare-offer> <al>United</al> <orig>DEN</orig> <dest>MXP</dest> <price>850</price> <cur>USD</cur></fare-offer>

<fare-offer> <al>United</al> <orig>DEN</orig> <dest>MXP</dest> <price>850</price> <cur>USD</cur></fare-offer>

tree of named values

full type system

United DEN MXP 850 USDUnited DEN MXP 850 USD

string prefix, suffix, substring, regular expression

list value operators + indexingnamed values value operators + selectiontree value operators + structure walkfull type user-defined operators

Subscription Model: Filter Language

Operators are implied by the data model

Example: Siena Data Model

A publication is a set of typed attributes

string class = travel/airlines/offers date starts = Jun date expires = Aug string origin = DEN string destination = MXP string carrier = United float price = 850 string currency = USD

attribute ≡ (type,name,value)

Example: Siena Filter Language

A filter is a list of attribute constraints

string class >* travel/airlines/ date starts < Jul date expires > Jul string origin = DEN string destination = MXP

attribute constraint ≡ (type,name,operator,value)

Example: Siena Matching Rules

Matching is defined by covering relations

publication p filter f

p matches f (or f covers p)

airline = Unitedprice < 900price > 300

destination any

refundable = noprice = 850

destination = MXPairline = United

Subject Fare Offer

Airline United

Origin DEN

Destination MXP

Price 850

Currency USD

Scope/Expressiveness of Subscriptions

Channel– channel identifier– equality over channel

identifiers Subject (or topic)

– designated subject field– general filter applied to

designated field Content

– all message fields– general filter applied to

full message content

Subject Fare Offer

Airline United

Origin DEN

Destination MXP

Price 850

Currency USD Subject Fare Offer

Airline United

Origin DEN

Destination MXP

Price 850

Currency USD

joinChannel(37)

subscribe(“Fare”)

subscribe(“Origin=DEN,Price<900”)

channel37

224.11.22.35

224.11.22.33

224.11.22.34

partition

Multicast vs. Content-Based Pub/Sub

Multicast Service

joinGroup(IF,group)

leaveGroup(IF,group)

ip_send(datagram)

dest=224.11.22.33dest=224.11.22.33

dest=224.11.22.35dest=224.11.22.35

dest=224.11.22.34dest=224.11.22.34

dest=224.11.22.33dest=224.11.22.33

Content-Based Service

ifconfig(IF,predicate)

ifconfig(IF,false)

send(message)

resource=printerresource=printer

resource=printerlanguage=PS

resource=printerlanguage=PS

resource=printerlanguage=plc5

resource=printerlanguage=plc5

resource=printerstatus=ready

resource=printerstatus=ready

maker= HPage > 3

maker= HPage > 3

arbitrary subsets

Multicast vs. Content-Based Pub/Sub

Special case of content-based pub/sub– joining multicast group g is equivalent to defining

a receiver predicate that selects messages addressed to g

Multicast designed for streaming media– relatively stable set of groups and membership– simple channel-based interface

Content-based pub/sub is intended to support unpredictable interaction patterns– sophisticated content-based interface

Some Filter Language Issues

Negation operatorincreased expressivenessunclear utility

User-defined operatorincreased expressivenessunpredictable execution

Patternsincreased expressivenessunclear choice of semanticsunpredictable execution

set-of(f) ≡ pub \ set-of(f)set-of(f) ≡ pub \ set-of(f)

md5-hash(signature) = “7dc81ef”md5-hash(signature) = “7dc81ef”

(f1f2)* (f3|f4)(f1f2)* (f3|f4)

airline = Unitedprice < 900price > 300

destination any

Example: Siena Filter Pattern

A pattern correlates messages of interest

filter f2

f = f1 followed by f2

airline = Unitedprice < 1200price 900

destination any

filter f1

Subscription Modifications

Incremental style– effect of each subscription is “additive”– new filter represents logical relation among

subscriptions– service must provide an explicit unsubscribe

command– client must “remember” subscription history

Subscribe-and-forget style– effect of each subscription is complete– no need for unsubscribe nor history

Incremental Subscriptions

airline = Unitedprice < 400


airline = Uniteddestination = MXP


airline = United

destination = MXP

=

=



airline = Uniteddestination = MXP=








Subscribe and Forget





=

=


airline = Unitedprice < 900=

=

= f1f1 f2

f2f1f1 f2

f2

Outline


Basic Elements of Implementation Model

Architecture– centralized– static routing, federation, replication– store-and-forward routing– peer-to-peer versus client/server

Platform– multicast, TCP overlay, CORBA, active

distributed database, web servicesGateways

General Service Implementation

publish/subscribe service

subscribe publishnotify

Publications must meet subscriptions somewhere Service must decide whether publications match

subscriptions Service must notify subscribers

Centralized Implementation

publish/subscribe service


server/dispatcher/broker

Evaluating Centralized Implementation

Simple– every (client) application sends subscriptions

and publications to a single server– the server maintains and evaluates all

subscriptions locallyBut not very “scalable”

– one server handles every subscription and every message of every client

– the server becomes a network bottleneck– all clients must trust the same server (privacy)

Naïve Extension: Broadcast Replication


Publications are relayed to every server Subscriptions remain local to their servers

replicated serversreplicated servers

Evaluating Broadcast Replication

Simple– hopefully making use of network-level broadcast

Reduces computation– only local subscriptions– same number of publications

Still not very scalable– little flexibility– little privacy– increased network traffic

Another Simple Extension: Federation


Import/export statements Static, user-defined routes

4

1 3

2

x,y,a,b 1x,a,~b 3x,y,z 4

x,y,a,b 1x,a,~b 3x,y,z 4

……

……

……

……

conditional relayconditional relay

Evaluating Federation

Reduces traffic– only imported/exported traffic goes through

Improves privacyReduces computation

– local subscriptions plus import/exportStill not very scalable

– little flexibility– suboptimal network usage– potential errors due to misconfiguration

Advanced: Fully Distributed Architecture

subscribe

dynamic routingdynamic routing

……

……

……

…… ……

……

…………

……

Advanced: Fully Distributed Architecture

publishnotify

dynamic routingdynamic routing

……

……

……

…………

……

…… ……

……

subscribe

Dynamic Routing

Servers are interconnected through statically configured links

Servers function as store-and-forward publication dispatchers

Servers exchange subscription information according to a routing protocol

Servers forward publications according to – subscription information– forwarding protocol

Evaluating Fully Distributed Architecture

Increases reliability– exploiting multi-connected topologies

Adapts automatically– to changes in applications (subscriptions)– to changes in network (topology)

Reduces computation and communication– using optimized routing and forwarding

Requires complex protocols– to handle topological routing information– to handle content-based routing information

Example: Hierarchical Routing

simple extension of centralized serversimple extension of centralized server

1 3

2

“master” server connection

…1…1

1

A

2

C

B

4

5

3…1…1

…A…A

Evaluating Hierarchical Routing

Reduces traffic and computation– “leaf” nodes process only local traffic and traffic

that matches local subscriptions» no unnecessary traffic goes down the hierarchy

Still not very scalable– root server processes every subscription and

publication

Hierarchical Routing Improved

combine and simplify subscriptionscombine and simplify subscriptions

1

A

3

2

S1

B

S1 AS1 A

S1 1S1 1 S1



1

A

3

2

S2

B

S1 covers S2

S1=[airline=“UA”]S2=[airline=“UA”, dest=“MXP”] do not propagate S2

S1 AS1 A

S1 1S1 1S1 1S2 2

S1 1S2 2



1

A

3

2

S3

B

S1 covers S2

S1=[airline=“UA”]S2=[airline=“UA”, dest=“MXP”] do not propagate S2

S1 AS1 A

S1 1S2 2

S1 1S2 2S1 1S2 2S3 3

S1 1S2 2S3 3 S1 ,S2, and S3 are similar

S1=[airline=“UA”]S2=[airline=“UA”, dest=“MXP”]S3=[airline=“AZ”] propagate S*=[airline=any]

S* AS* A

S*

Evaluating Improved Hierarchical Routing

Reduce traffic and computation– only “new” subscriptions are propagated– servers store and process fewer, more generic

subscriptionsOpen to a variety of heuristic optimizations

– e.g., based on actual traffic profilesApplies to other architectures

– concept of “content-based subnet address”(see demonstration)

Other Considerations

Reliability– reliable transfer and storage of subscriptions– reliable transfer of publications

» assured delivery» causal ordering

Value-added services– support for temporarily disconnected clients– support for mobile clients– publication history service– authentication service

Outline


Survey

Standards– CORBA Notification

Service– Java Message Service

Commercial systems– FioranoMQ– IBM WebSphere MQ EB– TIBCO Rendezvous– IBM Gryphon

Caveat– not all features of the systems are presented

Research systems– Le Subscribe– Elvin– Hermes– Steam– JEDI– Siena

OMG’s CORBA Notification Service

Data model: predefined message object– header contains standard attribute/value pairs

and user-defined attribute/value pairs– body contains user-defined attribute/value pairs

and user-defined “any” fieldScope/expressiveness

– publishers publish to a topic channel– content-based filters applied to standard

attribute/value pairs and body’s user-defined attribute/value pairs

CORBA Notification Service (cont.)

Filter language– filters modeled as objects containing constraints– constraints correspond to a standard grammar– operators are SQL-like

Sun’s Java Message Service

Data model: predefined message object– header contains standard attribute/value pairs– properties are user-defined attribute/value pairs– body comes in five standard types

» stream, map, text, object, uninterpreted bytes

Scope/expressiveness– publishers publish to a topic channel– content-based filters applied to header attributes

and propertiesFilter language: SQL-like

Fiorano’s FioranoMQ

Data model– compliant with JMS 1.1– also supports XML documents as messages

Scope/expressiveness– compliant with JMS 1.1

Filter language– compliant with JMS 1.1– also supports XPath queries over XML

messages

Fiorano’s FioranoMQ (cont.)

Implementation model– “hub and spoke” (i.e., centralized broker)– centralized brokers can be federated using

“repeaters”– a repeater is a JMS client (statically) configured

to bridge between brokers

repeater

IBM’s WebSphere MQ Event Broker

Data model– headers and attribute/value pairs– also supports XML documents in message body

Scope/expressiveness– topic-based publication and subscription– “subscription point” property of a publisher

» allows subscribers to identify desired publisher set

– filters applied to topics and contentFilter language

– SQL and XPath

WebSphere MQ Event Broker (cont.)

Implementation model– broker and clients use message queues– brokers can be federated, with subscription

information shared among federation members» tree of brokers» collective of fully connected brokers (i.e., broadcast

replication)» tree of collectives

WebSphere MQ Event Broker (cont.)

Value-added services– compliance with JMS 1.0.2– publications can be retained for later retrieval

» implies database support

– brokers can be configured to apply application processing logic to publications» e.g., converting currency values

– publication ordering» sequence number or time stamp

TIBCO’s Rendezvous

Data model– subject field consisting of a list of strings– opaque body

Scope/expressiveness– filters applied to subject field

Filter language– indexed– string equality and limited regular expression

TIBCO’s Rendezvous

Implementation model– federated brokers– statically configured sharing among brokers

IBM’s Gryphon

Data model– attribute/value pairs

Scope/expressiveness– both subject- and content-based filtering

Filter language– SQL-like

Implementation model– fully connected, replicated brokers

Sampling of Research Systems

Le Subscribe (INRIA/NYU/Univ. of Toronto)– high-performance matching algorithm

Elvin (DSTC, Australia)– publisher-side filtering (“quenching”)

Hermes (Cambridge Univ., UK)– typed data model– routing based on per-type “rendezvous” points

» message unicast to rendezvous point, disseminated to tree of subscribers (c.f., IP multicast routing)

Sampling of Research Systems (cont.)

Steam (Trinity College, Dublin)– publish/subscribe in ad hoc networks– “proximity” filters, in addition to subject and

content filtersJEDI (Politecnico di Milano/CEFRIEL)

– publish/subscribe for mobile clients– uses explicit “moveOut” and “moveIn” functions

Siena (Univ. of Colorado, USA)– scalability through fully distributed architecture– filter optimization

Outline


Using Publish/Subscribe Systems

Siena– research prototype– mature implementation– focus on distributed architecture– advanced implementation features

JMS– industry “standard”– several implementations– focus on Java integration and smaller-scale

applications– rich interface

Siena

Fully functional publish/subscribe service– first release: 1998– available for anonymous download

http://www.cs.colorado.edu/serl/siena/Content-based subscription language

– SQL-like data model and operatorsJava and C++ implementationsBasic security through SSL Integrated with XMLArchitected as a distributed service

Sampling of Registered Users

.EDU– CMU, Columbia, Dartmouth, Indiana Univ., MIT,

Oregon, UCI, UMass, Imperial College, TU-Vienna, FU-Berlin, Univ. of Toronto, Univ. College London, Univ. of Bologna, Univ. of Oslo, Univ. of Manitoba, Univ. de Valladolid

.COM– Objsys, Tecknowledge, Lockheed Martin, MITRE

Corp., Boeing, Siemens AG Austria, Air France, Borland, Organix Consulting, Diverminds, DataVision Consulting Inc.

Hello World!

import siena.*;

public class HelloWorld {

public static void main(String [] args)

throws SienaException {

Siena siena = new ThinClient("tcp:host.domain.net:1234");

Notification n = new Notification();

n.putAttribute("greeting", "Hello World!");

siena.publish(n);

}

}

HelloWorld.java

Compiling and Running

> javac -classpath siena.jar HelloWorld.java

> java -classpath siena.jar

Exception in thread "main" siena.comm.PacketSenderException: Connection refused

atsiena.comm.TCPPacketSender.send(TCPPacketSender.java:80)

at siena.ThinClient.publish(ThinClient.java:384)at HelloWorld.main(HelloWorld.java:9)

We need to set up a server to connect to…

Setting up a Siena Network

First we start up a server to…

> java -classpath siena.jar \ siena.StartServer -port 1234

on host1

Then we connect other servers…

1

server address:“tcp:host1:1234”

> java -classpath siena.jar \ siena.StartServer -port 2222 \ -master tcp:host1:1234

on host2


on host3


on host4

2 3

4

Siena Connectors and Addresses

Siena communications– one-way, message oriented– receiver creates a Receiver object– sender creates a Sender object

» passing the address of the Receiver object to a SenderFactory

Address ::= connector-type:type-specific-part– available connector types

» tcp:host:port» udp:host:port» ka:host:port» ssl:host:port

Back to Our Example…

> java -classpath siena.jar:. HelloWorld

Nothing happens: there are no subscribers, so the notification goes nowhere

We need a subscriber to complete the HelloWorld example

Hello World Subscriber

import siena.*;

public class Subscriber implements Notifiable {

public void notify(Notification n) {

System.out.println(e);

}

public static void main(String [] args)

throws SienaException {

Siena siena = new ThinClient(args[0]);

Filter f = new Filter();

f.addConstraint("greeting", Op.PF, "Hello");

siena.subscribe(f, new Subscriber());

}

}

Subscriber.java

> java -classpath siena.jar:. Subscriber tcp:localhost:1234 > java -classpath siena.jar:. Subscriber tcp:localhost:1234

> java -classpath siena.jar:. Subscriber tcp:localhost:1234

event{ greeting="Hello World!"}

Hello World!

The Siena network is up and running

So, we first run the subscriber…

And then run the publisher…> java -classpath siena.jar:. HelloWorld > java -classpath siena.jar:. HelloWorld

>

Synchronous Subscriber

import siena.*;

public class SynchronousSubscriber {

public static void main(String [] args) throws Exception {

Siena siena = new ThinClient(args[0]);

Filter f = new Filter();

f.addConstraint("greeting", Op.PF, "Hello");

NotificationBuffer buf = new NotificationBuffer();

siena.subscribe(f, buf);

Notification n = buf.getNotification(-1); // wait forever

System.out.println(n);

siena.shutdown();

}

}

SynchronousSubscriber.java

HelloWorld in JMS [JMS Documentation]

Get a ConnectionFactory through JNDI Get a Destination (i.e., Topic) through JNDI Get a Connection from the ConnectionFactory Get a Session from the Connection Create a MessageProducer from Session and

Destination Create a Message from the Session Fill in message data Send the Message through the MessageProducer

JMS HelloWorld Publisher

import javax.naming.*; import javax.jms.*;public class JMSHelloWorld { public static void main(String [] args) throws Exception { Context context = new InitialContext(); ConnectionFactory cf; cf = (ConnectionFactory)context.lookup("ConnectionFactory"); Topic greetings = (Topic)context.lookup("Greetings"); Connection connection = cf.createConnection(); Session session = connection.createSession(false,Session.AUTO_ACKNOWLEDGE); MessageProducer sender = session.createProducer(greetings); TextMessage m = session.createTextMessage(); m.setText("Yo!"); m.setStringProperty("greeting", "Hello World!"); connection.start(); sender.send(m); }}

JMSHelloWorld.java

JMS (Synchronous) Subscriber

import javax.naming.*; import javax.jms.*;public class JMSSubscriber { public static void main(String [] args) throws Exception { Context context = new InitialContext(); ConnectionFactory cf; cf = (ConnectionFactory)context.lookup("ConnectionFactory"); Topic greetings = (Topic)context.lookup("Greetings"); Connection connection = cf.createConnection(); Session session = connection.createSession(false,Session.AUTO_ACKNOWLEDGE); MessageConsumer receiver = session.createConsumer(greetings); connection.start(); TextMessage m = (TextMessage)receiver.receive(5000); // 5 sec System.out.println(m.getText()); }}

JMSSubscriber.java

JMS (Asynchronous) Subscriber

import javax.naming.*; import javax.jms.*;class HelloListener implements MessageListener { public void onMessage(Message message) { System.out.println(message); }}

public class JMSAsyncSubscriber { public static void main(String [] args) { // // ...Initialization... // receiver.setMessageListener(new HelloListener()); connection.start(); }}

JMSAsyncSubscriber.java

Content-Based Subscriptions in JMS

import javax.naming.*; import javax.jms.*;public class JMSSubscriber { public static void main(String [] args) throws Exception { // // ...Initialization... // MessageConsumer receiver; receiver = session.createConsumer(greetings, "(greeting = ‘Hello’)"); connection.start(); TextMessage m = (TextMessage)receiver.receive(5000); // 5 sec System.out.println(m.getText()); }}

JMSSubscriber.java

Outline


Advanced Topics

Security

Client mobility

Benchmarking

Security of Publish/Subscribe Systems

Reliability

Access control (i.e., policies)

Authentication

Privacy

Reliability

Fault tolerance– heart-beat propagation of routing information– dynamic discovery of (optimal) paths within n-

connected network– generic, robust content-based routing

Fair resource allocation– flow control– accounting

Access Control Policies

Publish/subscribe firewalls– filters on outgoing publications

» confidentiality of intra-domain events» blocking and/or rewriting (anonymization)

– filters on incoming publications» mandatory restrictions on inter-domain traffic» additional filtering

– filters on outgoing subscriptions» confidentiality of subscriptions» blocking and/or rewriting (over-generalization)

Authentication

Value-added service– clients apply and verify MACs

Native implementation– “MAC” operator

[airline=“UA”, price<900, has_valid_mac(K)]

Privacy: “Strong” Security

Confidentiality of subscriptions with an untrusted infrastructure– clients want the service to filter publications on their

behalf without “knowing” what it is filtering» privacy of publications» privacy of subscriptions

Approximate solutions– encrypted computations– filter obfuscation– anonymization (hiding many subscribers behind a proxy)– over-generality in subscriptions

Mobility Goal

Combine the benefits of mobile applications with the communication services offered by advanced publish/subscribe services

Code mobility– movement of code among hosts

» weak mobility: code fragment is relocated by creating a fresh copy at the destination point

» strong mobility: movement of code while maintaining the state of execution

Kinds of Mobility

Host mobility– movement of mobile hosts

code codecode code

code

codecodecode

Mobility Introduces Problems

Client may lose some messages while disconnected

Part of client’s state is maintained by the service– subscriptions– routing information– messages in transit

S

C

S

C

m

S

C

Enabler: Message Persistence

m m

m mm

mm mm

mm

Persistence and Mobility

code code codecode

2) Disconnect(c,H)

Message Download Protocol

C

1) Save(c,H)

3) Move(c)

4) Connect(c,T)

6) Subscribe(f,c)

7) Download(c,H) Merge(c)

5) Save(c,T)

8) Download(T)

TH

C

Problem: race conditionsSolutions: scout message, delay

Evaluating Publish/Subscribe Systems

Very large parameter space– network topology– distribution of clients– rate of publication– rate of subscription– publication content

» distribution of attribute names, types, and values

– subscription content» distribution of constraint names, types, and values

We need a benchmark

A Publish/Subscribe Benchmark

Requirements– evaluate support for existing applications– evaluate performance with existing applications– evaluate performance for future applications

Benchmark sections– interface suitability: capability model– traces from existing applications– synthetic benchmark

Outline


Conclusion

Prospects for publish/subscribe– could it one day dominate network traffic?

» multiparty versus point-to-point communication

Open Problems and Future Directions– seamless integration with other services

» file transfer, remote procedure call, naming, …

– finding an appropriate network foundation» multicast, content-based networking

– communication ontologies

advanced publish/subscribe communication services antonio carzaniga and alexander l. wolf university...

Documents