www.synerzip.com data virtualization & information as a service (iaas) by anil allewar senior...

www.synerzip.comwww.synerzip.com

Data Virtualization&

Information As A Service (IaaS)

By Anil AllewarSenior Solutions Architect - Synerzip

1


About Me!!

Confidential 2

Anil Allewar

Senior Solutions Architect @ Synerzip

Technology Evangelist & speaker

Core interests: JEE, EAI, EII


• Use cases

Agenda

• What does it mean?

• Implementation Frameworks

• Demo

• Questions?

3

• Architecture explained


Why it makes sense?

Confidential 4


Use Cases

Data Warehouse

ETL

Financial Data

OLTP Data

ETL

3rd Party Data

Data Mart

ETL

Web Service 1

Web Service 2

Legacy Data

Custom Program

Excel files

Confidential 5


Traditional Data Integration

Enterprise Information System

ETL

Source System

Source System

ETL

Confidential 6

Business Applications


Problems with ETL

Confidential 7

More than 1 copy of data for staging

Intermediate data => Errors

Lead time to add new source

Domain knowledge for mapping

Batch Process => No real time data


Problems with DBMS consolidation

Confidential 8

Alternate approach => Single EIS (say RDBMS)

Extensive changes to existing apps

Might not satisfy everyone’s requiremets


• Use cases

Agenda



• Demo

• Questions?

9



Data Virtualization & Federation

Confidential 10

Single API to access data

Only metadata stored at virtualization layer

Real time access without

copying/moving dataFederate data across hetero/homogenous

sources


Data Virtualization

Confidential 11


• Use cases

Agenda



• Demo

• Questions?

12



Architecture

Confidential 13

UserApplication

Com

mon

Acc

ess

API

Connector 1

Connector 2

RUNTIME & QUERY ENGINE

VirtualDatabase

Translator 1

Translator 2


• Use cases

Agenda



• Demo

• Questions?

14



Vendors

• Commercial Products– Composite Software

• http://www.compositesw.com/data-virtualization/

– Denodo• http://www.denodo.com/en/product/overview.php?n=h

– IBM• http://www-03.ibm.com/software/products/en/ibminfofedeserv

– Informatica• http://www.informatica.com/us/data-virtualization/

– Red Hat• http://www.redhat.com/products/jbossenterprisemiddleware/data-virtualizati

on/

• Open Source– Jboss Teiid

• http://teiid.jboss.org/

Confidential 15

http://www.compositesw.com/data-virtualization/



http://www.denodo.com/en/product/overview.php?n=h



http://www-03.ibm.com/software/products/en/ibminfofedeserv



http://www.informatica.com/us/data-virtualization/

http://www.informatica.com/us/data-virtualization/

http://www.redhat.com/products/jbossenterprisemiddleware/data-virtualization/



http://teiid.jboss.org/

http://teiid.jboss.org/


Selected Platform – JBoss Teiid

Confidential 16

Open Source

Number of relational/NoSQL/ERP/CRM data stores

JEE standards

Add custom EIS support using JEE

components

Active & responsive community Synerzip contribution: Defect

discovery, root cause analysis, feature verification


Teiid Components

• Virtual Database– container for components used to integrate

data from multiple data sources• Source Models

– structure and characteristics of physical data sources

• View Models– structure and characteristics of abstract structures you want to expose

to your applications

• Teiid Designer– Eclipse based UI to dynamically discover data

source objects and apply data federation– Generate virtual database from 1 or more

sources

Confidential 17


Teiid Components

• Translator– Provides abstraction later between Teiid Query

Engine and source system– Convert Teiid SQL commands to source specific

execution commands– Convert result data from source system to Teiid

specific format• Resource Adapter

– Provides connectivity to the physical data source

– Integration provided through Java Connector Architecture (JCA) API

Confidential 18


Teiid – Supported EIS

• Amazon SimpleDB• Apache Accumulo• Apache SOLR• Cassandra• File• Google Spreadsheet• JPA• LDAP• Excel – as file• SalesForce

• JDBC– MS access, DB2, derby, excel-

odbc, greenplum, h2 , hive(for accessing Hadoop), oracle, teradata and most RDBMS

• MongoDB• Object• OData• OLAP• Web Services• SAP Netweaver Gateway

Confidential 19


Performance Characteristics

• Access same data using Oracle and Teiid drivers

– Retrieval times comparable when accessing tables having no Blobs

Confidential 20

0 0 0 1 13693

8,652

505,005

505,007

720,850

859,457

1,003,798

1,103,9180

5,000

10,000

15,000

20,000

25,000No. of rows Vs Time: No Blobs

Oracle-JDBCTeiid-JDBC

No. of rows

ms


Performance Characteristics

– Teiid slower when accessing Blob data• Can be tuned

Confidential 21

0 0 2 42 21,804 32,531 185,4540

5,000

10,000

15,000

20,000

25,000

30,000

No. of rows Vs Time: Blobs

Oracle-JDBCTeiid-JDBCm

s

No. of rows


• Use cases

Agenda



• Demo

• Questions?

22



Demo

Confidential 23

JDBC Client JD

BC API

RDBMS Resource Adapter

MongoDB Resource Adapter

TEIID RUNTIME & QUERY ENGINE

Federated VDB

mySQL Translator

MongoDB Translator mySQL


Demo-Steps• Pre-requisites

– mySQL server 5.5+ installed– MongoDB 2.4.x+ installed

• Steps– Load the mySql and MongoDB database with sample

data– Setup environment – JBoss, Eclipse– Create Teiid project in Eclipse using Teiid designer

• Import source model using JDBC• Create the virtual model and federate data from the

source model• Create a virtual database (VDB) and deploy to JBoss

– Access data using JDBC client or through browser using OData

Confidential 24


Demo – Scenario

Confidential 25

FederatedData


Demo – Connection Profile

Confidential 26


Demo – Source Model

Confidential 27


Demo - Source Model Generation

Confidential 28


Demo – Map Source To View

Confidential 29


Demo - Association

Confidential 30


Demo – Data Federation

Confidential 31


Demo – Source Code

• Source code–https://github.com/Synerzip/JBoss-Teiid

–Contains• Configuration files• Instructions• “How-to” videos• VDBs, source models and view models

Confidential 32

https://github.com/Synerzip/JBoss-Teiid

https://github.com/Synerzip/JBoss-Teiid


Conclusion

• Data Virtualization and Federation is a rapidly emerging technology that solves traditional BI/ETL problems.

• It provides lower time to market, distributes data across the enterprise as a service and provides real time access to enterprise data.

Confidential 33


• Use cases

Agenda



• Demo

• Questions?

34



Contact Me

• [email protected]

Confidential 35

mailto:[email protected]

www.synerzip.comwww.synerzip.com• 3

684

Questions?

www.synerzip.comHemant Elhence

[email protected]


Synerzip in a Nutshell1. Software product development partner for

small/mid-sized technology companies• Exclusive focus on small/mid-sized technology companies,

typically venture-backed companies in growth phase• By definition, all Synerzip work is the IP of its respective

clients• Deep experience in full SDLC – design, dev, QA/testing,

deployment2. Dedicated team of high caliber software

professionals for each client• Seamlessly extends client’s local team, offering full

transparency• Stable teams with very low turn-over• NOT just “staff augmentation”, but provide full mgmt

support3. Actually reduces risk of development/delivery

• Experienced team - uses appropriate level of engineering discipline

• Practices Agile development – responsive, yet disciplined4. Reduces cost – dual-shore team, 50% cost

advantage5. Offers long term flexibility – allows (facilitates)

taking offshore team captive – aka “BOT” option


Our Clients

http://www.itko.com/

http://www.yagnaiq.com/index.htm

http://www.clearorbit.com/index.php?sess_id=b0a39bdaf17f0f7c5cd34f8cc3d49d10

http://www.zafesoft.com/index.asp

http://www.hallsettlementsystems.com/Default.aspx

http://www.hallsettlementsystems.com/Default.aspx

http://www.fd2s.com/home.php

http://www.repliqa.com/

http://www.broadlane.com/index.html

http://www.neofirma.com/Default.aspx

http://www.sevayatra.com/index.html

http://www.fuelquest.com/

http://www.hartic.com/index.php

http://www.coretrace.com/

http://www.healthcarepayment.com/home.cfm

http://www.sourcecorp.com/Home.aspx


Call Us for a Free Consultation!

Hemant Elhence [email protected]

469.322.0349

Thanks!

www.synerzip.com data virtualization & information as a service (iaas) by anil allewar senior...

Documents

copyingmoving data

intermediate data

copy of data

real time data slide

eii slide

confidential15 slide

confidential4 slide

feature verification