1

Enabling Data as a Service with

JBoss Data Services

Prajod Vettiyattil Twitter: @prajods

Gnanaguru Sattanathan Twitter:@gnanagurus

Website: bushorn.com

2

What this session is about

v The why and what of data services v How data services work v Use cases v JBoss Data Services Platform

3

Why

4

Proliferation of data

SQL File Mainframe NoSQL Email

Content Manage

ment System

Customer portal

Employee portal ERP CRM Accounting Billing

ERP

Vendor Management

Partner Management Sales Finance Marketing

Data Consumers

Data Sources and Data Managers 5

Proliferation: so what ?

6

•  Multiplicity of connections –  High development cost –  Huge operational overhead –  Difficult and risky to change Data Sources/Managers

•  Dispersed data connectors •  Data duplication

–  Too much ETL –  Lines of Business copies data

•  Duplicated data aggregation •  Impossible to create “Single source of truth” •  Data ownership issues •  No comprehensive view

–  No data movement dashboards –  Location of data and its status

What

7

Data Services and DSP The basic view

•  DSP = Data Services Platform •  Abstracts the data managers/

sources

Data Managers

Data Consumers

Data Services Platform Data Service 1

Data Service 2

Data Service 3

Data Service 4

C1 C2 C3 C4 C5 C6

D1 D2 D3 D4 D5 D6

8

•  Presents the data as a service to the consumer

•  ETL++

Dashboard in a DSP

Data Dashboard

Data Connections

Data movement status

Errors

Error Corrections Failures Alerts

9

How it works

10

Features of a DSP •  Enables architecture principles

–  Separation of concerns –  Protected variations

•  Data adapters •  Data mapping tools and standards •  Data caching

–  Local and distributed •  Service search and reuse •  Data security and data usage audit •  Data access control •  Central channel for all data requirements •  Data dashboard •  Configurable performance and reliability 11

Use cases

12

Auto manufacturing supply chain: Requirements

•  Vehicle ownership experience •  Business Process Automation •  Disparate data sources •  Multiple data feeds

–  Parts catalog –  Prices

•  Dealer updates –  Parts consumed –  Parts replaced –  Part failure statistics

•  Customer feedback –  Post purchase –  Breakdown support –  Service Quality Dashboards

•  Integration solutions based on batch transfers –  Unreliable –  Not traceable

13

Auto manufacturing supply chain: Layer Diagram

14

Business Processes

Data Services Platform

Customer feedback Customer

Master Parts

Catalog Dealer feeds Dealer

Info

Business Activity Monitoring Customer Experience Dashboards

Breakdown reports

Parts supplier

feeds

Enterprise Service Bus

Enterprise Data Access Layer: Requirements

15

•  Golden copy / System of Record / Single source of truth

•  Shared services team for Enterprise Data Management

•  Data usage audit •  Data access control •  Reduce request load on Data Management team •  Reduce data maintenance costs

Enterprise Data Access Layer: Layer Diagram

16

Enterprise Data Consumers

Data Services Platform

Mainframe

Data Services

Customer Master

Partner Data

Content Management

System

Data Access Control

Data base drivers Auditing Data Aggregation

Partner Info

Employee Info

Virtual DB Metadata

Reporting risk for securities: Requirements

•  Internal and external reporting –  Risk and margin

•  Centralized risk capture and management •  Calculate risk from different customer activities •  Report consolidated data to comply with regulation

–  Dodd Frank –  Sarbanes Oxley Act (SOX)

•  Dashboards for higher management

17

Reporting risk for securities: Architecture without DSP

18

COTS Trading Systems

Enterprise Middleware Systems (MQ, ESB, FTP, File shares)

Risk Management Settlement Ref Data

Mgmt Custom built

Apps

Customer facing Apps Partner Apps Government

Systems

Price feeds

Accounting

Trade feeds

Reporting Applications

Payment Systems

Margin Mgmt Trade Matching

Order Mgmt Execution Mgmt

Liquidity Mgmt

Position Mgmt Order Book

Clearing Ref Data Feeds

Reporting risk for securities: Patterns in this requirement

•  Regulatory requirement for transparency –  Cannot be met by opaque internal systems

•  Data Sources –  Large number of them –  Internal and external

•  Reports are read heavy •  No real time data requirements

–  once a quarter or once a year •  No excuses for incorrect data in reports •  Non-discretionary spending

19

JBoss Data Services Platform

20

Architecture

21

•  The EDS platform –  v5 Runs on SOA-P

•  Teiid •  ModeShape

Data Adapters

Data interfaces (JCR API, Web service, JDBC, ODBC, OData,..)

Data virtualization Metadata repository

SAP Sybase Flat file SalesForce

Oracle XML

Data Sources

Data consumers (Custom Applications, COTS products, Business Processes, Business Services )

Cassandra

Mongo DB

Data Services Platform SOA Platform

•  Parts of the architecture –  Data interfaces –  Data adapters –  Data virtualization –  Metadata repository

Oracle DB

IBM DB2

MS SQL Server

MySQL PostgreSQL

Sybase

Greenplum

Teradata Netezza Ingres Mondrian

MetaMatrix

LDAP Salesforce

Delimited file

XML file

Web services

Apache Hive

MS Excel

MS Access

JBoss Messagi

ng

JBoss HornetQ

TIBCO IBM MQ

22

Data sources

Data Mapping •  Teiid Designer

–  Map actual data tables using transforms to virtual tables

–  MDD; use Data Models, not SQL –  Semantic mapping –  Virtual procedures

•  A set of SQL statements, similar to DB stored procedures

23

Data Standards •  JCR

–  Java Content Repository(JSR-283) •  OData

–  Open Data Protocol •  JDBC •  ODBC •  Others

–  S-RAMP –  An SOA repository spec, OASIS

–  Web Services –  REST –  JMS

24

Access control and Audit •  Teiid

–  passwords –  MembeshipDomains for authentication –  Data roles

•  Fine grained access and visibility control of tables –  CRUD level permissions for VDB –  LDAP integration

•  ModeShape –  LoginContext –  AuthenticationProvider –  Role to Action mapping

25

Teiid and ModeShape Data type Teiid ModeShape Approach Relational Hierarchical Metadata repository Not suitable Yes Content repository Not suitable Yes ACID transactions Yes Yes SQL queries Yes Yes(JCR-SQL) Flat file data source Yes Not suitable Relational DB data source Yes Not suitable

Schema Fixed Optional NoSQL data sources Not suitable Yes Stores data No Yes

26

Summary •  Data Services

–  Why –  What –  How

•  Use cases –  Auto Manufacturer –  Enterprise Data Access Layer –  Regulatory Reporting

•  JBoss DSP –  Data virtualization –  Teiid –  ModeShape

27

Questions

28

Our Open Source Middleware Group on LinkedIn http://tinyurl.com/be6e93q

Prajod Vettiyattil Twitter: @prajods

Gnanaguru Sattanathan Twitter: @gnanagurus

Website: bushorn.com