fusesource camel-persistence-part1-webinar-charles-moulliard

Copyright © 2010 Progress So3ware Corpora6on and/or its subsidiaries or affiliates. All rights reserved. 1 A Progress So3ware Company

A Progress So3ware Company

Database Integra6on with Apache Camel Part I

Charles Moulliard, October 21th, 2011

twitter : cmoulliard | LinkedIn: charlesmoulliard http://cmoulliard.blogspot.com


When you joined today’s session …

Audio is broadcast from your computer

Submit your questions via the Chat Window

Contact today’s Host via the Chat Window


Speaker – Charles Moulliard

Engineer in Agronomy & Master in Zoology

17 years of experience in IT world development (J2EE, .NET) • Specialised in new technologies

web/Internet & integra6on

• Project manager in bank, financial, telco, insurance and transport world

Solu6on Architect at FuseSource CommiYer on Apache projects :

• ServiceMix, Apache Karaf (PMC) , Camel, Fuse Fabric, Scalate

Follow me on :

hYp://twiYer.com/cmoulliard

hYp://www.linkedin.com/in/charlesmoulliard

hYp://cmoulliard.blogspot.com

hYp://www.slideshare.net/cmoulliard


Agenda

Goal : Present Database Integra6on with Camel This first part is focused on some Camel Components and Transac6on • Star6ng from "simple" to more "elaborated" • Describe some "interes6ng" components

• Discuss Pros and Cons • Introduce you through transac6on

Summing-‐up

Explain what we will cover in the 2nd part


Table of contents

Components presenta6on JDBC Component

SQL Component

JPA Component

Transac6on Summing-‐up



Table of contents

Components presenta,on JDBC Component

SQL Component

JPA Component




Components

Name Type Description Camel Version

JDBC SQL Simple SQL query >= 1.5

SQL SQL Idem as JDBC but using spring-jdbc template >= 1.5

JPA SQL Use Java Persistence Api in combination with OpenJPA, EClipseLink, HibernateJPA, …

>= 1.6

Hibernate SQL ORM tool > 1.x

iBatis SQL Another ORM tool > =1.6

myIbatis SQL Ibatis ORM engine (left Apache community) >= 2.7

Bean - Integrated with Service/DAO layers > 1.x

We have a big family of SQL components allowing to design all the required architectures


Components

Name Type Description Camel Version

Db4o NoSql NoSQL Database >= 2.5

HawtDB NoSql FuseSource project, key/value embedded DB >= 2.3

MongoDb NoSql Scalable, high-performance, open source, document-oriented database

Cassandra NoSql highly scalable distributed database using as data model (Bigtable's ColumnFamily)

And also noSQL components


Table of contents


SQL Component

JPA Component




Component : JDBC

Defini6on : "The jdbc component enables you to access databases through JDBC, where SQL queries and opera6ons are sent in the message body. This component uses the standard JDBC API."

Format : jdbc:dataSourceName[?options]!

Very simple component as it only required a datasource and SQL query defined in the body of the message


Component : JDBC -‐ Setup

(Step 0) Configure your database engine, tables, access rights, ….

(Step 1) Define DataSource in Spring XML file

DataSource manager can be what you want : Apache Commons DBCP, C3P0,…

Id of the bean is the key used by camel-jdbc component to find


Component : JDBC – Camel -‐ SELECT

(Step 3) Build the "SQL Body" to fetch, insert, update or delete data from Database

setBody allows to define the SQL Select statement

Timer component triggers the DB every 20s


Component : JDBC – Camel – SELECT WHERE

Can we do something more dynamic ? Yes using setBody, Processor or a CamelBean

We will use an ID key provided as a value from a file to select info from Database

Hey ! See How camel with simple language simplifies your life.


Component : JDBC – Camel – Externalize query

With camel-‐proper6es, we can externalise the SQL query (or a part)

Hey ! See How camel + properties with simple language retrieves the key from file


Demo using camel-‐jdbc

Demo using camel-‐jdbc with PropertyPlace Holder


Component : JDBC – Pros & Cons

Pros • Few dependencies required, less jars files • Easy to use and setup

Cons: • This component can only be used to define producer endpoints.

• Bean, processor, setBody or template language (velocity, freemarker, scalate) must be used to build complex SQL query

• Not Transac6onal friendly – Transac6on must be managed "manually" (e.g. Add spring)


Table of contents


SQL Component

JPA Component




Component : SQL

Defini6on : • Allows you to work with databases using Spring JDBC.

• The difference between this component and JDBC component is that the SQL query is a property of the endpoint and it uses message payload as parameters passed to the query

Format : sql:query! Example : sql:select * from table where id=# order by name[?options]!


Component : SQL -‐ setup

Step (0) and (1) are iden6cal to component JDBC

Difference with JDBC component we must define the SqlComponent and inject reference of the Datasource Reference to the Datasource is provided

as a property and will allow camel to create it accordingly when endpoint will be called



Step (2) : Define your camel route including the query in the URI defini6on. Observe that in this case, we do not need to define an exchange containing a query Body

Remark : query can also be externalised using property placeholder !

Query is part of the URI defini6on



This seems really sta6c !

How can we define dynamically the parameters of the query The component provides a placeholder # that we will use to replace the values received from the Body (Object[]) File contains the parameter key

that we convert into Object[]

Symbol of Placeholder is # and not ? To avoid conflict with URI defini6on


Component : SQL – Dynamic Query

Here is another approach using a camel bean component where the query xxx is defined at run6me (and not as before at DSL compile 6me!)

Look how we simplify the camel route. SQL component has disappeared but in fact it is called within camel bean (see next slide) and SQL query created


Component : SQL – Dynamic Query

We use ProducerTemplate, @EndpointInject of Camel

Query + SQL component are defined here and a camel exchange is created by the template


Demo using camel-‐sql (spring JDBC template)


Component : SQL– Pros & Cons

Pros • Easy to use and setup, very flexible – SQL • Uses prepared statements

• Use PlaceHolder to replace symbol # with values • Depends on Spring JDBC Template which simplifies SQL management

• Can be included in Camel <transacted/> route

Cons: • This component can only be used to define producer endpoints

• The query is in the URI which can become long

• More dependencies than JDBC


Table of contents


SQL Component

JPA Component




Component : JPA

Defini6on : • Enables you to store and retrieve Java objects from persistent storage using Java Persistence Architecture (JPA),

• Allows to use Object/Rela6onal Mapping (ORM) products such as OpenJPA, Hibernate, TopLink, …

Format : jpa:[entityClassName][?options]


Component : JPA

We can produce and consume !

Sending to the JPA endpoint You can store a Java en6ty bean in a database. • The body of the In message is assumed to be an en6ty bean (that is, a

POJO with an @En6ty annota6on on it) or a collec6on or array of en6ty beans.

Consuming from the JPA endpoint removes (or updates) en6ty beans in the database. • This allows you to use a database table as a logical queue: consumers

take messages from the queue and then delete/update them to logically remove them from the queue.

• If you do not wish to delete the en6ty bean when it has been processed, you can specify consumeDelete=false on the URI. This will result in the en6ty being processed each poll.


Component : JPA -‐ setup

(Step 0) Configure your database, tables, … (Step 1) Define En6ty

Class is enrich with JPA @Annotations



(Step 2) Configure persistence file

A

B

C



(Step 3) Configure Spring File – En6tyManager/TxManager We use Spring Local JPA Container & Tx Manager and provide a ref to DS

According to your JPA Vendor, select OpenJPA, Hibernate, ….


Component : JPA -‐ Consume

(Step 4) Configure Camel Route, JPA component to Consume

We can consume directly from JPA endpoint – (different from JDBC, SQL components)


Component : JPA -‐ Consume

Using namedQuery

namedQuery is passed as parameter (query can also place directly in the URI) and defined here


Component : JPA -‐ Produce

(Step 4) Configure Camel Route to produce

Extract incident from Map and enrich it

Insert record in DB

Bindy is used to transform CSV records into Map of Incident


Demo using camel-‐jpa


Component : SQL– Pros & Cons

Pros • Allow to consume (insert) or produce (select) • It is a standard • Can do boilterplate CRUD well (eg Create, Update, Delete an en6ty)

• Fully support Transac6ons and <transacted/> route Cons:

• Less flexible than JDBC, SQL as it requires more config, more development 6me is required

• Higher learning curve, For simpler use-‐cases its "overkill"

• Only one en6ty can be processed


Table of contents


SQL Component

JPA Component

Transac,on Summing-‐up



Transac6on -‐ Introduc6on

Developing messaging architecture involves also that we handle errors, excep6ons, loss of messages, connec6on issues, …

What will happen to our Camel Exchanges when excep6ons occur during DB connec6on ? The response depends on the ErrorHandler/Excep6on strategy put in place and the component used

Component like (File, JPA, SQL, …) implements a fallback strategy and than it will be possible to move the messages to directory, DLQ, … (they are called Tx clients)

When exchanges are send back to the caller, they could be • Redelivered x 6mes (= DefaultErrorHandler)

• Move to a DeadLeYerQueue (=DeadLeYerErrorHandler)

• Discarded


Transac6ons – Introduc6on (next)

Is it enough ? No as we must also rollback info pushed into the DB. Tx is all or nothing strategy while RDMS rely on ACID principles

Solu6ons 1) Transac6ons can be designed using Spring in the applica6on which are

the beans/POJO that we call

Cons -‐ This is not integrated with your Camel routes and you must intercept excep6ons and handle them

2) A beYer approach is to use the Transac6on support offered by Camel

This is not a proprietary solu6on Camel uses Spring Transac6on Api to work with Transac6ons Transac6on Manager can be provided by

• J2EE Applica6on Server (Tomcat, Jboss, WebSphere, …), OSGI pla}orm (Apache ServiceMix), Standalone Applica6on (Spring)


Transac6ons -‐ Setup

To be honest, this is really simple ….. A transaction Manager is required

Camel Route must be defined as Transactional using <transacted/> DSL

Every time an exception occurs, camel will propagate it to the caller (which is a Tx client) and the file will be moved as transaction is marked as rollback


Demo using Transac6on


Summing up!


Table of contents


SQL Component

JPA Component




Summing up

Camel provides a collec6on of Persistence components • SIMPLE: JDBC, SQL

• ELABORATED – ORM : Hibernate, Iba6s, MyIba6s. • STANDARD BASED – JPA : OpenJPA, Hibernate-‐EJB (JPA)

Your choice will depend on constraints like : • Datamodel (simple vs complicated),

• Business, • Architecture (Container, Tx Manager, ….) • Database (new schema or integrate with exis6ng)

• Stored procedure needed ? • Sql Vendor Trick

All components are really simple to configure and can be deployed in any container : J2EE, ServiceMix, Standalone

Camel supports transac6ons in an transparent way


Next Webinars

See hYp://fusesource.com/resources/video-‐archived-‐webinars/

In the second part, we will cover more stuffs about transac6on, use JTA Tx manager of Fuse ESB (jms, jdbc) and cover persistence with Idempotent Processor

22nd of November 2011: “Transac6on Management with Apache ServiceMix and EIPs”

See you then!


Useful references

Camel • hYp://fusesource.com/products/enterprise-‐camel/#documenta6on

• hYp://fusesource.com/docs/router/2.7/component_ref/index.html

Transac6on with camel • hYp://fusesource.com/docs/router/2.7/transac6ons/index.html

Spring • hYp://sta6c.springsource.org/spring/docs/3.0.6.RELEASE/spring-‐

framework-‐reference/html/

• ORM -‐ hYp://sta6c.springsource.org/spring/docs/3.0.6.RELEASE/spring-‐framework-‐reference/html/orm.html

• Transac6on -‐ hYp://sta6c.springsource.org/spring/docs/3.0.6.RELEASE/spring-‐framework-‐reference/html/transac6on.html


Learn More at hYp://fusesource.com

fusesource camel-persistence-part1-webinar-charles-moulliard

Technology