exploring data integration capabilities of the wso2 platform
TRANSCRIPT
![Page 1: Exploring Data Integration Capabilities of the WSO2 Platform](https://reader036.vdocument.in/reader036/viewer/2022062420/55b35432bb61eb9c728b457d/html5/thumbnails/1.jpg)
Exploring Data Integration Capabilities of the WSO2 Platform
June 2015
Jagath AriyarathneTechnical Lead
Ravi UndupitiyaSenior Software Engineer
![Page 2: Exploring Data Integration Capabilities of the WSO2 Platform](https://reader036.vdocument.in/reader036/viewer/2022062420/55b35432bb61eb9c728b457d/html5/thumbnails/2.jpg)
About the Presenters๏ Jagath Ariyarathne - [email protected]
Jagath is a Technical Lead at WSO2. Prior to joining WSO2, he worked for DirectFN (Pvt) Ltd, where he was involved in the design and development of DirectFN Financial Data dissemination back-end system. His main focus was on real-time data distribution with low latency, high performance and bandwidth friendly solutions. He holds a BSc First Class degree in Computer Science and Engineering from University of Moratuwa, Sri Lanka.
๏ Ravi Undupitiya - [email protected]
Ravi is a senior software engineer in the integration technology team where he mainly focuses on the WSO2 Enterprise Service Bus. Prior to joining WSO2 he interned at the Intelligent Agents and Multimedia Research Group at the University of Southampton where he worked on a smart grid project which was aimed at helping people make smarter decisions when consuming energy. Additionally he also dived into the world of Natural Language Processing and carried out an evaluation of several text classification algorithms.
![Page 3: Exploring Data Integration Capabilities of the WSO2 Platform](https://reader036.vdocument.in/reader036/viewer/2022062420/55b35432bb61eb9c728b457d/html5/thumbnails/3.jpg)
*
Agenda๏ Background to Data Integration
๏ SOA based approach to Data Integration
๏ WSO2 Data Services Server
๏ WSO2 Enterprise Service Bus
๏Demo
๏ Security ๏ Transactions
![Page 4: Exploring Data Integration Capabilities of the WSO2 Platform](https://reader036.vdocument.in/reader036/viewer/2022062420/55b35432bb61eb9c728b457d/html5/thumbnails/4.jpg)
*
Why we need Data Integration?๏ Data is an important asset.
๏ Found in various sources and often siloed within applications.
๏We must leverage this data.
๏ Data consumers on various devices and platforms. ๏Security and privacy.
๏ Needs to be flexible for change.
๏ Legacy system support.
![Page 5: Exploring Data Integration Capabilities of the WSO2 Platform](https://reader036.vdocument.in/reader036/viewer/2022062420/55b35432bb61eb9c728b457d/html5/thumbnails/5.jpg)
*
Data Integration Solutions ๏ Ad-hoc approach to data integration not good!
๏ Unmaintainable and messy.
๏ Not flexible enough for changes.
๏ Need data source specific implementations
๏SOA approach better!
๏ Data exposed as a services.
๏ Applications consume the data through services.
.NET J2EE C++
Data Sources
.NET J2EE C++
Data Sources
![Page 6: Exploring Data Integration Capabilities of the WSO2 Platform](https://reader036.vdocument.in/reader036/viewer/2022062420/55b35432bb61eb9c728b457d/html5/thumbnails/6.jpg)
*
Data Services on WSO2 Platform๏ WSO2 Data Services Server (DSS)
๏ Allows rapid development of data services.
๏ Data services are decoupled from the infrastructure where it is stored.
๏ Wizard based UI to generate data service description language.
๏Data service description language
๏ Maps service requests to queries operating on data store.
๏ Maps query results to XML schema definitions.
๏ Supports validation, transformation, eventing etc…
![Page 7: Exploring Data Integration Capabilities of the WSO2 Platform](https://reader036.vdocument.in/reader036/viewer/2022062420/55b35432bb61eb9c728b457d/html5/thumbnails/7.jpg)
DSS Key Features - Any Data Store
![Page 8: Exploring Data Integration Capabilities of the WSO2 Platform](https://reader036.vdocument.in/reader036/viewer/2022062420/55b35432bb61eb9c728b457d/html5/thumbnails/8.jpg)
DSS Key Features - Data Federation
![Page 9: Exploring Data Integration Capabilities of the WSO2 Platform](https://reader036.vdocument.in/reader036/viewer/2022062420/55b35432bb61eb9c728b457d/html5/thumbnails/9.jpg)
DSS Key Features - Server Side Batch Processing
![Page 10: Exploring Data Integration Capabilities of the WSO2 Platform](https://reader036.vdocument.in/reader036/viewer/2022062420/55b35432bb61eb9c728b457d/html5/thumbnails/10.jpg)
DSS Key Features - Client Side Batch Processing
![Page 11: Exploring Data Integration Capabilities of the WSO2 Platform](https://reader036.vdocument.in/reader036/viewer/2022062420/55b35432bb61eb9c728b457d/html5/thumbnails/11.jpg)
DSS Key Features - Array/UDT/Complex Types
![Page 12: Exploring Data Integration Capabilities of the WSO2 Platform](https://reader036.vdocument.in/reader036/viewer/2022062420/55b35432bb61eb9c728b457d/html5/thumbnails/12.jpg)
DSS Key Features - Distributed Transactions
![Page 13: Exploring Data Integration Capabilities of the WSO2 Platform](https://reader036.vdocument.in/reader036/viewer/2022062420/55b35432bb61eb9c728b457d/html5/thumbnails/13.jpg)
DSS Key Features - Notifications๏Event sources, subscribers, topics๏Allows a subscriber to be notified of a specific activity in a data service
๏Event Triggers๏ Input Event Trigger๏Output Event Trigger
![Page 14: Exploring Data Integration Capabilities of the WSO2 Platform](https://reader036.vdocument.in/reader036/viewer/2022062420/55b35432bb61eb9c728b457d/html5/thumbnails/14.jpg)
DSS Key Features - REST Support
![Page 15: Exploring Data Integration Capabilities of the WSO2 Platform](https://reader036.vdocument.in/reader036/viewer/2022062420/55b35432bb61eb9c728b457d/html5/thumbnails/15.jpg)
DSS Key Features - Caching Support
![Page 16: Exploring Data Integration Capabilities of the WSO2 Platform](https://reader036.vdocument.in/reader036/viewer/2022062420/55b35432bb61eb9c728b457d/html5/thumbnails/16.jpg)
DSS Key Features - Security Support
![Page 17: Exploring Data Integration Capabilities of the WSO2 Platform](https://reader036.vdocument.in/reader036/viewer/2022062420/55b35432bb61eb9c728b457d/html5/thumbnails/17.jpg)
DSS Key Features - Transformation
![Page 18: Exploring Data Integration Capabilities of the WSO2 Platform](https://reader036.vdocument.in/reader036/viewer/2022062420/55b35432bb61eb9c728b457d/html5/thumbnails/18.jpg)
*
Demo 1๏Exposing MySQL data table as a service.
๏ SOAP
๏ RESTful
![Page 19: Exploring Data Integration Capabilities of the WSO2 Platform](https://reader036.vdocument.in/reader036/viewer/2022062420/55b35432bb61eb9c728b457d/html5/thumbnails/19.jpg)
*
Data Integration Solutions ๏ Ad-hoc approach to data integration not good!
๏ Unmaintainable and messy.
๏ Not flexible enough for changes.
๏ Need data source specific implementations
๏SOA approach better!
๏ Data exposed as a services.
๏ Applications consume the data through services.
.NET J2EE C++
Data Sources
.NET J2EE C++
Data Sources
![Page 20: Exploring Data Integration Capabilities of the WSO2 Platform](https://reader036.vdocument.in/reader036/viewer/2022062420/55b35432bb61eb9c728b457d/html5/thumbnails/20.jpg)
*
Layered Approach๏SOA alone is not enough. Point-to-Point service integration. ๏Enterprise Service Bus for a cleaner layered approach.
Point-to-Point Service Integration (Spaghetti Integration) ESB in Action
![Page 21: Exploring Data Integration Capabilities of the WSO2 Platform](https://reader036.vdocument.in/reader036/viewer/2022062420/55b35432bb61eb9c728b457d/html5/thumbnails/21.jpg)
What is an ESB? ๏An ESB is a middleware solution that enables interoperability, among heterogeneous environments using a service oriented model. ๏Stateless and Seamless Integration๏Standard protocols/formats like SOAP, REST, JSON ๏Transports like HTTP/S, JMS, TCP, VFS (and many more)
![Page 22: Exploring Data Integration Capabilities of the WSO2 Platform](https://reader036.vdocument.in/reader036/viewer/2022062420/55b35432bb61eb9c728b457d/html5/thumbnails/22.jpg)
The WSO2 ESB๏Lightweight and high performance๏Feature rich and standards compliant๏ SOAP and WS-* standards๏REST support๏Domain specific transport support (FIX, HL7)๏User friendly and highly extensible ๏Configuration driven! ๏100% free and open source
![Page 23: Exploring Data Integration Capabilities of the WSO2 Platform](https://reader036.vdocument.in/reader036/viewer/2022062420/55b35432bb61eb9c728b457d/html5/thumbnails/23.jpg)
WSO2 ESB๏Apache Synapse ๏A lightweight and open source ESB implementation from ASF๏Makes up the mediation engine of WSO2 ESB๏Multithreaded and asynchronous message processing๏Based on a number of well known open source projects๏Apache Axis2 ๏Apache HTTP Components
![Page 24: Exploring Data Integration Capabilities of the WSO2 Platform](https://reader036.vdocument.in/reader036/viewer/2022062420/55b35432bb61eb9c728b457d/html5/thumbnails/24.jpg)
WSO2 ESB Fundamentals๏ Routing๏ Filtering๏ Transformation๏ Protocol Switching๏ Load Balancing ๏ Quality of Service
![Page 25: Exploring Data Integration Capabilities of the WSO2 Platform](https://reader036.vdocument.in/reader036/viewer/2022062420/55b35432bb61eb9c728b457d/html5/thumbnails/25.jpg)
More Features๏100% EIP support๏Connectors๏Message Stores/Processors๏JSON Support
๏ JMX based monitoring and control๏ Statistics collection๏ Priority based message mediation๏ XSLT, XPath, XQuery, Smooks, JSONPath๏ Caching, Throttling๏ Scripting languages (JavaScript, Groovy)๏ JDBC connectivity๏ Drools๏ Clustering
![Page 26: Exploring Data Integration Capabilities of the WSO2 Platform](https://reader036.vdocument.in/reader036/viewer/2022062420/55b35432bb61eb9c728b457d/html5/thumbnails/26.jpg)
Functional ComponentsMediator Sequence Endpoints Local Entry Proxy Service REST API Message Stores/Processors Templates Connectors Tasks Priority Executors Registry
![Page 27: Exploring Data Integration Capabilities of the WSO2 Platform](https://reader036.vdocument.in/reader036/viewer/2022062420/55b35432bb61eb9c728b457d/html5/thumbnails/27.jpg)
Functional Components๏Each functional component serves a specific purpose
๏Functional components can be mixed and matched to implement various integration scenarios and patterns
๏Configuring WSO2 ESB for a given scenario requires: ๏ Identifying the right set of components๏Putting them together in the most optimal manner
![Page 28: Exploring Data Integration Capabilities of the WSO2 Platform](https://reader036.vdocument.in/reader036/viewer/2022062420/55b35432bb61eb9c728b457d/html5/thumbnails/28.jpg)
ESB and DSS
๏ Why not access data stores directly from ESB?
๏ ESB has rudimentary DB support๏DBLookup Mediator๏DBReport Mediator
๏ ESB’s function is to ensure low latency with high throughput for stateless transactions. Data intensive processing done on ESB could hinder ESB performance.
๏ Separate JVM/servers to handle data intensive tasks.
๏ Can deploy DSS features on top of ESB (thanks to WSO2 Carbon platform).
![Page 29: Exploring Data Integration Capabilities of the WSO2 Platform](https://reader036.vdocument.in/reader036/viewer/2022062420/55b35432bb61eb9c728b457d/html5/thumbnails/29.jpg)
Layered Approach
๏ Why access DSS through ESB?
๏ Perfectly ok to access DSS data services directly.
๏ Handling DSS invocations through ESB provides a layer of abstraction over data layer for integration.๏ Interception point๏ Service Chaining๏ Parallel processing๏ Load balancing ๏ Simple transformations๏Handling Multiple protocols
![Page 30: Exploring Data Integration Capabilities of the WSO2 Platform](https://reader036.vdocument.in/reader036/viewer/2022062420/55b35432bb61eb9c728b457d/html5/thumbnails/30.jpg)
๏ A merger causes an organization to consolidate their account databases.
๏ Leverage WSO2 DSS to expose data from multiple data sources as services.
๏Leverage WSO2 ESB to provide a secured uniform interface and aggregation.
Demo 2 Use Case
MySQL Database
Excel Data Sheet
![Page 31: Exploring Data Integration Capabilities of the WSO2 Platform](https://reader036.vdocument.in/reader036/viewer/2022062420/55b35432bb61eb9c728b457d/html5/thumbnails/31.jpg)
*
Demo 2Aggregate data from multiple WSO2 DSS Data Services using WSO2 ESB
Excel Data Service
RDBMS Data Service
Excel File
MySQL DB
Data Services ProxyClient Service
● Aggregate Mediator● Clone Mediator● Header Mediator
Request
Response
Multiple Sources
![Page 32: Exploring Data Integration Capabilities of the WSO2 Platform](https://reader036.vdocument.in/reader036/viewer/2022062420/55b35432bb61eb9c728b457d/html5/thumbnails/32.jpg)
Demo 3 Use Case
๏ A fund transfer between two accounts need to occur in a transacted manner.
๏ Leverage WSO2 DSS for distributed transaction using boxcarring feature.
๏ Leverage WSO2 ESB for service chaining.
![Page 33: Exploring Data Integration Capabilities of the WSO2 Platform](https://reader036.vdocument.in/reader036/viewer/2022062420/55b35432bb61eb9c728b457d/html5/thumbnails/33.jpg)
*
Demo 3Boxcarring sample with WSO2 DSS and ESB
begin_boxcar
execution 1
execution 2
………
execution 10
end_boxcar
begin_boxcar
execution 1
execution 2
abort_boxcar
execution 10
end_boxcar
![Page 34: Exploring Data Integration Capabilities of the WSO2 Platform](https://reader036.vdocument.in/reader036/viewer/2022062420/55b35432bb61eb9c728b457d/html5/thumbnails/34.jpg)
*
Demo 3 cont
Bank Data Service
Bank DB (Book Store’s)
Payment Services ProxyBook Store
Application
● Customer’s Account Number● Book Store’s Account Number● Payment Amount
Bank DB (Customer’s)
● Customer’s Account Number● Payment Amount
● Book Store’s Account Number● Payment Amount
begin_boxcar
end_boxcar
credit_bookstore
debit_customer
Service Chaining
![Page 35: Exploring Data Integration Capabilities of the WSO2 Platform](https://reader036.vdocument.in/reader036/viewer/2022062420/55b35432bb61eb9c728b457d/html5/thumbnails/35.jpg)
Where to begin..
๏ WSO2 ESB Documentation ๏ https://docs.wso2.org/display/ESB481/WSO2+Enterprise+Service+Bus+
Documentation
๏ WSO2 DSS Documentation๏ https://docs.wso2.com/display/DSS322/WSO2+Data+Services+Server+D
ocumentation
![Page 36: Exploring Data Integration Capabilities of the WSO2 Platform](https://reader036.vdocument.in/reader036/viewer/2022062420/55b35432bb61eb9c728b457d/html5/thumbnails/36.jpg)
Further Reading๏Master Data Management using WSO2 Platform ๏ http://wso2.com/library/webinars/2013/10/mdm-with-wso2-enterprise
-middleware-platform/
![Page 37: Exploring Data Integration Capabilities of the WSO2 Platform](https://reader036.vdocument.in/reader036/viewer/2022062420/55b35432bb61eb9c728b457d/html5/thumbnails/37.jpg)
Contact us !