enabling data as a service with the jboss enterprise data services platform
Post on 05-Dec-2014
2.528 Views
Preview:
DESCRIPTION
TRANSCRIPT
1
Enabling Data as a Service with
JBoss Data Services
Prajod Vettiyattil
Twitter: @prajods
Gnanaguru Sattanathan
Twitter:@gnanagurus
Website: bushorn.com
2
What this session is about
The why and what of data services How data services work Use cases JBoss Data Services Platform
3
Why
4
Proliferation of data
SQL File Mainfra
me NoSQL Email
Content
Manage
ment
System
Custom
er portal Employee
portal ERP CRM Accounting Billing
ERP
Vendor
Management Partner
Management Sales Finance Marketing
Data Consumers
Data Sources and Data Managers 5
Proliferation: so what ?
6
• Multiplicity of connections – High development cost – Huge operational overhead – Difficult and risky to change Data Sources/Managers
• Dispersed data connectors • Data duplication
– Too much ETL – Lines of Business copies data
• Duplicated data aggregation • Impossible to create “Single source of truth” • Data ownership issues • No comprehensive view
– No data movement dashboards – Location of data and its status
What
7
Data Services and DSP The basic view
• DSP = Data Services Platform
• Abstracts the data
managers/sources
Data Managers
Data Consumers
Data Services Platform Data
Service 1
Data
Service 2
Data
Service 3
Data
Service 4
C1 C2 C3 C4 C5 C6
D1 D2 D3 D4 D5 D6
8
• Presents the data as a service to the
consumer
• ETL++
Dashboard in a DSP
Data Dashboard
Data
Connections
Data
movement
status
Errors
Error
Corrections Failures Alerts
9
How it works
10
Features of a DSP
• Enables architecture principles – Separation of concerns – Protected variations
• Data adapters • Data mapping tools and standards • Data caching
– Local and distributed
• Service search and reuse • Data security and data usage audit • Data access control • Central channel for all data requirements • Data dashboard • Configurable performance and reliability
11
Use cases
12
Auto manufacturing supply chain:
Requirements
• Vehicle ownership experience • Business Process Automation • Disparate data sources • Multiple data feeds
– Parts catalog – Prices
• Dealer updates – Parts consumed – Parts replaced – Part failure statistics
• Customer feedback – Post purchase – Breakdown support – Service Quality Dashboards
• Integration solutions based on batch transfers – Unreliable – Not traceable
13
Auto manufacturing supply chain:
Layer Diagram
14
Business Processes
Data Services Platform
Customer feedback
Customer Master
Parts Catalog
Dealer feeds
Dealer Info
Business Activity Monitoring Customer Experience Dashboards
Breakdown reports
Parts supplier
feeds
Enterprise Service Bus
Enterprise Data Access Layer: Requirements
15
• Golden copy / System of Record / Single source of truth
• Shared services team for Enterprise Data Management
• Data usage audit • Data access control • Reduce request load on Data Management team • Reduce data maintenance costs
Enterprise Data Access Layer:
Layer Diagram
16
Enterprise Data Consumers
Data Services Platform
Mainframe
Data Services
Customer Master
Partner Data
Content Management
System
Data Access Control
Data base drivers
Auditing Data Aggregation
Partner Info
Employee Info
Virtual DB Metadata
Reporting risk for securities:
Requirements
• Internal and external reporting – Risk and margin
• Centralized risk capture and management • Calculate risk from different customer activities • Report consolidated data to comply with regulation
– Dodd Frank – Sarbanes Oxley Act (SOX)
• Dashboards for higher management
17
Reporting risk for securities:
Architecture without DSP
18
COTS Trading Systems
Enterprise Middleware Systems (MQ, ESB, FTP, File shares)
Risk Management
Settlement Ref Data
Mgmt Custom built
Apps
Customer facing Apps
Partner Apps Government
Systems
Price feeds
Accounting
Trade feeds
Reporting Applications
Payment Systems
Margin Mgmt Trade
Matching
Order Mgmt Execution
Mgmt Liquidity
Mgmt Position Mgmt
Order Book
Clearing Ref Data
Feeds
Reporting risk for securities:
Patterns in this requirement
• Regulatory requirement for transparency – Cannot be met by opaque internal systems
• Data Sources – Large number of them – Internal and external
• Reports are read heavy • No real time data requirements
– once a quarter or once a year
• No excuses for incorrect data in reports • Non-discretionary spending
19
JBoss Data Services
Platform
20
Architecture
21
• The EDS platform – v5 Runs on SOA-P
• Teiid • ModeShape
Data Adapters
Data interfaces
(JCR API, Web service, JDBC, ODBC, OData,..)
Data virtualization Metadata
repository
SAP Sybase Flat file SalesFo
rce
Oracle XML
Data Sources
Data consumers
(Custom Applications, COTS products, Business Processes, Business Services )
Cassan
dra
Mongo
DB
Data Services Platform
SOA Platform
• Parts of the architecture – Data interfaces – Data adapters – Data virtualization – Metadata repository
Oracle
DB IBM
DB2
MS
SQL
Server
MySQL PostgreS
QL Sybase
Greenpl
um Teradata Netezza Ingres Mondria
n MetaMa
trix
LDAP Salesfor
ce
Delimite
d file XML
file
Web
services Apache
Hive
MS
Excel MS
Access
JBoss
Messagi
ng
JBoss
HornetQ
TIBCO IBM
MQ
22
Data sources
Data Mapping
• Teiid Designer – Map actual data tables using transforms to virtual
tables – MDD; use Data Models, not SQL – Semantic mapping – Virtual procedures
• A set of SQL statements, similar to DB stored procedures
23
Data Standards
• JCR – Java Content Repository(JSR-283)
• OData – Open Data Protocol
• JDBC • ODBC • Others
– S-RAMP – An SOA repository spec, OASIS
– Web Services – REST – JMS
24
Access control and Audit
• Teiid
– passwords
– MembeshipDomains for authentication
– Data roles
• Fine grained access and visibility control of tables
– CRUD level permissions for VDB
– LDAP integration
• ModeShape
– LoginContext
– AuthenticationProvider
– Role to Action mapping
25
Teiid and ModeShape Data type Teiid ModeShape
Approach Relational Hierarchical
Metadata repository Not suitable Yes
Content repository Not suitable Yes
ACID transactions Yes Yes
SQL queries Yes Yes(JCR-SQL)
Flat file data source Yes Not suitable
Relational DB data source Yes Not suitable
Schema Fixed Optional
NoSQL data sources Not suitable Yes
Stores data No Yes
26
Summary
• Data Services
– Why
– What
– How
• Use cases
– Auto Manufacturer
– Enterprise Data Access Layer
– Regulatory Reporting
• JBoss DSP
– Data virtualization
– Teiid
– ModeShape 27
Questions
28
Our Open Source Middleware Group on LinkedIn
http://tinyurl.com/be6e93q
Prajod Vettiyattil
Twitter: @prajods
Gnanaguru Sattanathan
Twitter: @gnanagurus
Website: bushorn.com
top related