apache camel - stéphane kay - april 2011
DESCRIPTION
Apache Camel - Stéphane Kay - April 2011TRANSCRIPT
Apache CamelStephane Kay
\\Ville de Lausanne\\
Scope
IS Integration
EIP
Routing
Mediation
Camel Container
Unit testing
Hands-on
Real Use Case
JMS / Active MQ
JBI / NMR
OSGI / ServiceMix
Scala
Table of contents
I. IS Integration heuristics◦ Common issues
◦ Paradigms
II. Camel’s bleat◦ Concepts responding to integration needs
◦ Getting started
III. What’s under the humps?◦ Camel architecture
◦ Distribution
IV. Camel in Action◦ Persistence, transactions, concurrency, failover
◦ Aggregation
V. A Real Use-Case◦ Description, issues & achievement
I. IS Integration Heuristics
Just do it ! Just coding simple things might be just not so simple …
public class FileCopier {
public static void main(String args[]) throws Exception {
File inboxDirectory = new File("data/inbox");
File outboxDirectory = new File("data/outbox");
outboxDirectory.mkdir();
File[] files = inboxDirectory.listFiles();
for (File source : files) {
if (source.isFile()) {
File dest = new File(
outboxDirectory.getPath()
+ File.separator
+ source.getName());
copyFile(source, dest);
}
}
}
private static void copyFile(File source, File dest)
throws IOException {
OutputStream out = new FileOutputStream(dest);
byte[] buffer = new byte[(int) source.length()];
FileInputStream in = new FileInputStream(source);
in.read(buffer);
try {
out.write(buffer);
} finally {
out.close();
in.close();
}
}
}
I. IS Integration heuristics
A Framework?
The bad :◦ Learning effort consumes time.
◦ Adds anarchical dependencies.
◦ Risk of coming with a new need / not implemented
The good :◦ Gain value after a training period
◦ Leverage your code to a higher abstraction
◦ Potentially a future standard ? Indicator : community commits & posts curve.
The plus-value◦ Not obvious
I. IS Integration heuristics
Separation of concerns
IS Integration involves
1. Endpoints
A common endpoint interface abstraction
2. Data Processing
Using Message / exchange abstraction and
conversion tool
3. Routing
A common vocabulary / pattern as DSL
foundation.
I. IS Integration heuristics
Message Endpoint pattern
Routes should refer to a common abstract
Message Endpoint rather than directly using their
technical interface
A good Endpoint abstraction is a URI description
I. IS Integration | endpoint
Uniformed Exchange
Message’s container during routing
II. Camel’s bleat | data
Data Transformation
Data Format Transformation
◦ Message body is transformed from one
format to another.
CSV -> XML
Data Type Transformation
◦ Java Type of message body is
transformed/casted into another.
java.lang.String -> java.jms.TextMessage
java.io.File -> java.lang.String
I. IS Integration | data
Use case : Filtering Route
◦ How can a component avoid receiving
uninteresting messages ?
◦ …coming from a different system
I. IS Integration | routing
EIP?
A Catalog of Design patterns for common IS integration issues
http://camel.apache.org/enterprise-integration-patterns.html
I. IS Integration | routing
EIPs Samples… Filter
◦ How can a component avoid receiving uninteresting messages?
Splitter
◦ How can we process a message if it contains multiple elements, each of which may have to be processed in a different way?
Router
◦ How can you decouple individual processing steps so that messages can be passed to different filters depending on a set of conditions?
Aggregator
◦ How do we combine the results of individual, but related messages so that they can be processed as a whole?
I. IS Integration | routing
Putting all needs together
Route definition attributes :◦ Endpoints Producer / Consumer(s)
◦ Processor(s) Uniformed exchange processing interface
◦ Predicate(s)
A route language should provide thosekey abstractions
Endpoint A = endpoint(“activemq:queue:quote”);
Endpoint B = endpoint(“mq:quote”);
Predicate isWidget = xpath(“/quote/product = „widget‟”);
from(A).filter(isWidget).to(B);
I. IS Integration heuristics
I. Review
IS Integration paradigms
◦ Endpoint abstraction -> URIs?
◦ Data Tx -> Functions // Java beans ?
◦ Routing -> EIPs
I. IS Integration heuristics
II. Camel’s bleat
Apache Camel is a Powerful Open
Source Integration Framework based
on known Enterprise Integration
Patterns
*
Mission Statement :
Making integration easier and more
accessible to developers
Endpoint URI
A camel endpoint URI consists of
three parts :ftp://riders.com/orders?username=rider&password=secret
Scheme
Context path Options
II. Camel’s bleat | endpoint
Java DSL : Builder pattern The Builder Pattern (GOF) is central to the Camel Java DSL
public class FileCopierWithCamel {
public static void main(String args[]) throws Exception {
// create CamelContext
CamelContext context = new DefaultCamelContext();
// add our route to the CamelContext
context.addRoutes(new RouteBuilder() {
public void configure() {
from("file:data/inbox?noop=true").to("file:data/outbox");
}
});
// start the route and let it do its work
context.start();
Thread.sleep(10000);
// stop the CamelContext
context.stop();
}
}
II. Camel’s bleat | routing
Data Format transformations
Marshal / unmarshal
from("activemq:QueueWithJavaObjects”)
.marshal().jaxb()
.to("mq:QueueWithXmlMessages");
from("activemq:queue:MY_QUEUE").unmarshal().zip().proce
ss(new UnZippedMessageProcessor());
http://camel.apache.org/data-format.html
II. Camel’s bleat | data
Message Translator EIP
Aka Adapter (GOF)
Using a Processorfrom("direct:start").process(new Processor() {
public void process(Exchange exchange) {
exchange.getIn().setBody(in.getBody(String.class) + " World!"); }
}).to("mock:result");
Using Bean bindingfrom("direct:start") .bean(MyBean.class, "hello(String,String)") .to("mock:result");
Using transform() DSLfrom("direct:start").transform(body().regexReplaceAll(“\n”,”<br/>”))
.to("mock:result");
II. Camel’s bleat | data
Getting started : IDE
Eclipse
Maven integration (m2eclipse)
◦ http://m2eclipse.sonatype.org/sites/m2e
◦ Check your eclipse.ini : locate your JDK
-vm C:/dev/java/jdk1.6.0_18/bin
Optional : Fuse Camel plugin for
Eclipse
◦ http://repo.fusesource.com/beta/eclipse/u
pdate/
II. Camel’s bleat | first ride
Getting started : Maven
Basic Maven configuration
II. Camel’s bleat | first ride
First Ride
Copy contents from one folder to
another, control final name / dir
JUnit
◦ Startup context
◦ Route definition : File component
◦ Processor / Exchange
◦ Simple Language (File Component)
Delegate JUnit test to Main class
II. Camel’s bleat | first ride
Second Ride
Add some features to JUnit Class
◦ Log
◦ Add failure if file already exist
◦ Predicate : Simple Language
◦ Error handling
Router EIP (Choice)
II. Camel’s bleat | first ride
Spring XML : content-based router
<camelContext>
<route>
<from uri="activemq:NewOrders"/>
<choice>
<when>
<xpath>/order/product = 'widget'</xpath>
<to uri="activemq:Orders.Widgets"/>
</when>
<otherwise>
<to uri="activemq:Orders.Gadgets"/>
</otherwise>
</choice>
</route>
</camelContext>
II. Camel’s bleat | spring
II. Review
Endpoint / Component URIs
Java DSL
How to start / process exchanges
Log, error handling / routing
II. Camel’s bleat
III. What’s under the humps?
High-level architecture
◦ Container & Services
Toolkit provided out-of-the box
Distribution
High-level view
Container responsibilities :
◦ Create and activate the necessary Endpoint instances using the available
Component implementations
III. What’s under the humps?
Camel Context Services Components
◦ Loads components on the fly either by autodiscovery on the classpath or when a new bundle is activated in an OSGi container.
Endpoints
◦ Contains the endpoints that have been created.
Routes
◦ Contains the routes that have been added.
Type converters
◦ Contains the loaded type converters. Camel has a mechanism that allows you to manually or automatically convert from one type to another.
Data formats
◦ Contains the loaded data formats.
Registry
◦ Contains a registry that allows you to look up beans. By default, this will be a JNDI registry. If you’re using Camel from Spring, this will be the Spring ApplicationContext. It can also be an OSGi registry if you use Camel in an OSGi container.
Languages
◦ Contains the loaded languages. Camel defines 3 different DSLs (Java, XML/Spring, and Scala) and allows you to use many different languages to create expressions.
III. What’s under the humps?
Out of the box : EIPs
50+ Routing Patterns based on EIPs http://camel.apache.org/enterprise-integration-patterns.html
III. What’s under the humps?
Out of the box : Components
70+ Endpoint impl. aka « Components »
All available via URI/scheme paradigm
http://camel.apache.org/components.html
III. What’s under the humps?
Out of the box : Data Format
18 Data Formats - uniformed usage :from(“file://input").marshal().zip().to("activemq:queue:MY_QUEUE");
http://camel.apache.org/data-format.html
III. What’s under the humps?
Out of the box : Languages
Predicates & Expressions
http://camel.apache.org/languages.html
III. What’s under the humps?
Camel distribution
http://camel.apache.org/download.html
Components sources
Examples
Maven Central (current v2.7)
◦ Snapshot Repo<repository>
<id>apache.snapshots</id>
<name>Apache Development Snapshot Repository</name>
<url>https://repository.apache.org/content/repositories/snapshots/</url>
<releases> <enabled>false</enabled> </releases>
<snapshots> <enabled>true</enabled> </snapshots>
</repository>
III. Review
Camel is an integration framework
Based on Enterprise Integration Patterns
Routing and mediation
Easy to use DSL to define routes
No heavy specification
No container dependency
Payload agnostic
Connectivity to a great wealth of
transports
IV. Camel in Action
Test support
Transactions
Exceptions
Concurrency
Persistence
Aggregation
Runtime
Camel Test Support
Provides context/registry support
MockEndpoint assertionsresultEndpoint.expectedBodiesReceived(expectedBody);
template.sendBodyAndHeader(expectedBody, "foo",
"bar");
resultEndpoint.assertIsSatisfied();
IV. Camel in Action| test
Using test support
CamelTestSupport Junit
@EndpointInject
@Produce
Filter Pattern
IV. Camel in Action| test
Transactions
Component level
◦ Polling strategy
public interface PollingConsumerPollStrategy {
boolean begin(Consumer consumer, Endpoint endpoint);
void commit(Consumer consumer, Endpoint endpoint, int polledMessages);
boolean rollback(Consumer consumer, Endpoint endpoint, int retryCounter, Exception cause) throws
Exception;
}
Routing level
◦ Transactional Client EIP http://camel.apache.org/transactional-client.html
◦ Transacted DSLfrom("jetty:http://localhost/myservice/order").transacted()
.to("bean:validateOrder").to("jms:queue:order");
IV. Camel in Action | tx
Propagation
Define transaction behaviour<bean id="PROPAGATION_REQUIRED" class="org.apache.camel.spring.spi.SpringTransactionPolicy">
<property name="transactionManager" ref="txManager"/>
<property name="propagationBehaviorName" value="PROPAGATION_REQUIRED"/>
</bean>
<bean id="PROPAGATION_REQUIRES_NEW" class="org.apache.camel.spring.spi.SpringTransactionPolicy"> <property name="transactionManager" ref="txManager"/> <propertyname="propagationBehaviorName" value="PROPAGATION_REQUIRES_NEW"/>
</bean>
Use transacted DSLfrom("direct:mixed") .transacted("PROPAGATION_REQUIRED“)
.setBody(constant("Tiger in Action"))
.beanRef("bookService")
.setBody(constant("Elephant in Action"))
.beanRef("bookService“).to("direct:mixed2");
IV. Camel in Action | tx
Exceptions
onException clause
new RouteBuilder() {
public void configure() {
onException(IOException.class).maximumRedeliveries(3);
from("direct:start") .process("processor1")
.process("processor2") // <--- throws a ConnectException
.to("mock:theEnd");
}
◦ Note : By default, a producer will not redeliver
HandleonException(ValidationException.class)
.handled(true).transform(body(constant("INVALID ORDER")));
from("jetty:http://localhost/myservice/order").transacted()
.to("bean:validateOrder").to("jms:queue:order");
IV. Camel in Action | errors
Dead Letter EIP
The Error Handler propagates error back to the client.
Dead Letter Channel will always moved failed exchanges to this queue.
useOriginalMessage is used for routing the original input message instead of the current message that potentially is modified during routing.
from("jms:queue:order:input").to("bean:validateOrder") .to("bean:transformOrder").to("bean:handleOrder");
errorHandler(deadLetterChannel("jms:queue:dead") .useOriginalMessage().mamimumRedeliveries(5)
.redeliverDelay(5000);
IV. Camel in Action | errors
Concurency
Dealing with data flows means mastering
concurency
◦ Seda Component :from("seda:bar?concurrentConsumers=5").to("file://output");
◦ Thread DSL
How do we avoid the messages that does not
expect a reply to block ?from("jms:queue:order") // do some sanity check validation
.to("bean:validateOrder") .to("mock:validate").threads(20)
// do some CPU heavy processing of the message (we simulate and delay)
.unmarshal(mySecureDataFormat).delay(1000)
.to("bean:handleOrder").to("mock:order");
IV. Camel in Action | toolkit
Sync to Async
toAsync Processor
from("direct:start").to("mock:a").toAsync("direct:bar", 5).to("mock:result");
from("direct:bar").to("mock:b").transform(constant("Bye World"));
How does this work?void process(Exchange exchange, AsyncCallback callback) throws Exception;
◦ With AsyncCallback interface :
void onTaskCompleted(Exchange exchange);
Http Async Producerpublic class JettyHttpProducer extends DefaultProducer implements AsyncProcessor
◦ Then invoke the callback when reply is ready on the Exchange.
◦ Routing with Spring DSL<route> <!-- we route from a direct endpoint -->
<from uri="direct:start"/> <!-- log the request -->
<to uri="log:+++ request +++"/> <!-- then doing a non blocking async request/reply to the http server with a pool of 10 threads -->
<to uri="jetty://http://0.0.0.0:9123/myapp" async="true" poolSize="10"/>
<!-- the reply from the server is logged -->
<to uri="log:+++ reply +++"/> <!-- and to our mock so we can assert we got all responses -->
<to ref="result"/>
</route>
IV. Camel in Action | toolkit
Persistence SQL, JDBC and JPA components provided out-of the box
File idempotent consumer with pluggable repository
from("file://inbox?idempotent=true").to("...");
<bean id="fileStore" class="org.apache.camel.processor.idempotent.FileIdempotentRepository">
<!-- the filename for the store -->
<property name="fileStore" value="target/fileidempotent/.filestore.dat"/>
<!-- the max filesize in bytes for the file. Camel will trunk and flush the cache if the file gets bigger -->
<property name="maxFileStoreSize" value="512000"/>
<!-- the number of elements in our store -->
<property name="cacheSize" value="250"/> </bean>
<camelContext xmlns="http://camel.apache.org/schema/spring">
<route>
<fromuri="file://target/fileidempotent/?idempotent=true&idempotentRepository=#fileStore&move=done/${file:name}"/>
<to uri="mock:result"/>
</route>
Stateful EIP : Aggregator
IV. Camel in Action | toolkit
Aggregator EIP
Combine a number of messages together into a
single message.
◦ http://camel.apache.org/aggregator2.html
Completion predicatesfrom("direct:start“).
.aggregate(header("id"), new BodyInAggregatingStrategy()).completionTimeout(3000)
.to("mock:aggregated");
from("direct:start“).aggregate(header("id"), new
BodyInAggregatingStrategy()).completionSize(3) .to("mock:aggregated");
from("direct:start“).aggregate(header("id"), new BodyInAggregatingStrategy())
.eagerCheckCompletion().completionPredicate(body().isEqualTo("END"))
.to("mock:aggregated");
from("direct:start“).aggregate(header("id"), new BodyInAggregatingStrategy()).
completionSize(header("mySize")) .to("mock:aggregated");
IV. Camel in Action | toolkit
Aggregator Strategy & Fail Over
Strategypublic class NumberAggregationStrategy implements AggregationStrategy {
public Exchange aggregate(Exchange oldExchange, Exchange newExchange) {
if (oldExchange == null) { return newExchange; }
Integer num1 = oldExchange.getIn().getBody(Integer.class);
Integer num2 = newExchange.getIn().getBody(Integer.class);
Integer num3 = (num1 != null ? num1 : 0) + (num2 != null ? num2 : 0);
oldExchange.getIn().setBody(num3); return oldExchange;
}
}
http://camel.apache.org/aggregate-example.html
IV. Camel in Action | toolkit
Running Camel
Known Deployment Options
◦ Standalone Java Application
◦ Java Web Start
◦ Spring Application
◦ Web Application
◦ Google App Engine
◦ J2EE Application
◦ JBI
◦ OSGi
IV. Camel in Action | runtime
Running inside App Server
Servlet component
camel-example-servlet-tomcat
CXF Component
camel-example-cxf-tomcat
Camel Web console
actually « Web Component »
IV. Riding Camel | runtime
IV. Review
Camel is not a server
Camel is lightweight and embeddable
Camel supports Enterprise Integration paradigms from the JEE/Spring world.
Camel is not an ESB though itimplements EIPs.
Camel doesn’t need but supports anymessaging infrastructure (JMS / MQ)
Camel supports messaging at JBI abstraction level
V. A real Use-Case
Multiple source FTP import to a centralized repository
◦ Collect files (0-5 x 2-20 Megs) from N different FTP endpoints, every
hours on a 24/7 basis
◦ End-user configuration
◦ Logs : technical + business
◦ Monitor failures
FTP
SAN
OPCON
Constraints
Big binary files : maybe JMS would not fit
Implementation should fit some common corporate approach to entreprise intergation.
Currently only SAN @ client. Do not need App Server yet.
Client is technical-oriented : would like hands-on.
Keep intact current logs / traces, but must be possible to connect on JDBC.
Would be interested in indexing meta –infos within Alfresco (JSR-170).
Central schedule/alerting infrastructure must be connected.
Nagios could be an option at mid-term.
Real issues
Multiple source endpoints pattern :
◦ not found (!!!).
FTP component is an active poller; no
easy way to stop polling after having
consuming all files.
No easy way to put into configuration
the context path of Endpoint URIs
Other applications…
Camel as an ETL : Excel sources
PeopleSoft to ActiveDirectory
Bidirectional Federal <-> City
Entreprises DB official data
Reconcilliation of mutations streams
Conclusions
Open-Source
◦ Help yourself through debug mode on sources
Maven-based development
◦ « Real » modularity
Real Standards
◦ EIPs
◦ Maven, Spring, Java6.
◦ JBI (Glassfish)
Gateway to the future
◦ OSGI (Fuse ESB)
Camel. What else?
More information needed?
Camel website:◦ http://camel.apache.org
Camel article:◦ http://architects.dzone.com/articles/apache-
camel-integration
FuseSource website:◦ http://fusesource.com
Camel in Action book:◦ http://manning.com/ibsen
Camel in Action Source code:◦ http://code.google.com/p/camelinaction/
Questions