dealing with common data requirements in your enterprise
TRANSCRIPT
![Page 1: Dealing with Common Data Requirements in Your Enterprise](https://reader031.vdocument.in/reader031/viewer/2022030306/586e80521a28aba0038b51a3/html5/thumbnails/1.jpg)
Dealing with Common Data Requirements in your Enterprise
Nipun SuwandaratnaSenior Solutions Engineer - WSO2
WSO2 Solution Architecture Best Practices Webinar Series - 2016
![Page 2: Dealing with Common Data Requirements in Your Enterprise](https://reader031.vdocument.in/reader031/viewer/2022030306/586e80521a28aba0038b51a3/html5/thumbnails/2.jpg)
Agenda● Organizational Data
● Common Data Challenges of Modern Organizations
● Integrating with Different Messaging Infrastructures
● Data Services
● Data Analytics & Visualization
● High Availability
● Q&A
![Page 3: Dealing with Common Data Requirements in Your Enterprise](https://reader031.vdocument.in/reader031/viewer/2022030306/586e80521a28aba0038b51a3/html5/thumbnails/3.jpg)
Organizations & Data
![Page 4: Dealing with Common Data Requirements in Your Enterprise](https://reader031.vdocument.in/reader031/viewer/2022030306/586e80521a28aba0038b51a3/html5/thumbnails/4.jpg)
Old School
DB
Sys
tem
s
Files
Organization
![Page 5: Dealing with Common Data Requirements in Your Enterprise](https://reader031.vdocument.in/reader031/viewer/2022030306/586e80521a28aba0038b51a3/html5/thumbnails/5.jpg)
Modern Data Ecosystem
External Systems
External Users
![Page 6: Dealing with Common Data Requirements in Your Enterprise](https://reader031.vdocument.in/reader031/viewer/2022030306/586e80521a28aba0038b51a3/html5/thumbnails/6.jpg)
Organizational Data
● Master dataEg: Customer data, employee records, Supplier details, Product related data etc.
● Transactional dataThe data that master data participates in… transactions, discounts on bills etc. (changes constantly)
● Meta-dataData about data
Image: thinkpublic/photopin cc
![Page 7: Dealing with Common Data Requirements in Your Enterprise](https://reader031.vdocument.in/reader031/viewer/2022030306/586e80521a28aba0038b51a3/html5/thumbnails/7.jpg)
Common Data Challenges Organizations Face● Work with multiple Data Transports and Data Formats
● Data Transformation and Validation
● Exposing data as services
● Secure and managed data access
● Federated data stores
● Data/Entity Aggregation
● Data Analytics
● Visualization of Data
![Page 8: Dealing with Common Data Requirements in Your Enterprise](https://reader031.vdocument.in/reader031/viewer/2022030306/586e80521a28aba0038b51a3/html5/thumbnails/8.jpg)
Data Transports & FormatsFormats of data, their storage and transport mechanisms vary among
different systems
● Transports: HTTP, HTTPS, FTPS, SFTP, TCP, UDP, WebSocket, POP, IMAP, SMTP, JMS, AMQP, MQTT
● Formats & protocols: JSON, XML, SOAP, WS-*, HTML, EDI, HL7,Text, JPEG, MP4, binary formats
![Page 9: Dealing with Common Data Requirements in Your Enterprise](https://reader031.vdocument.in/reader031/viewer/2022030306/586e80521a28aba0038b51a3/html5/thumbnails/9.jpg)
Integration
![Page 10: Dealing with Common Data Requirements in Your Enterprise](https://reader031.vdocument.in/reader031/viewer/2022030306/586e80521a28aba0038b51a3/html5/thumbnails/10.jpg)
Integrating with Messaging Infrastructures
![Page 11: Dealing with Common Data Requirements in Your Enterprise](https://reader031.vdocument.in/reader031/viewer/2022030306/586e80521a28aba0038b51a3/html5/thumbnails/11.jpg)
Message Transformation● Protocol and Format conversion and Message Translation
○ eg: SOAP to REST and XML to JSON and translate the output from one system to match the input format required by the other system
● Enrich Content ○ eg: Add or remove data fields; may require accessing a separate data source
● Wrap Content○ eg: Include additional message header fields or encryption source to query
required data● Data Validation
○ eg: Validate input data against a schema
![Page 12: Dealing with Common Data Requirements in Your Enterprise](https://reader031.vdocument.in/reader031/viewer/2022030306/586e80521a28aba0038b51a3/html5/thumbnails/12.jpg)
Enterprise Service Bus
![Page 13: Dealing with Common Data Requirements in Your Enterprise](https://reader031.vdocument.in/reader031/viewer/2022030306/586e80521a28aba0038b51a3/html5/thumbnails/13.jpg)
Message Transformation ExampleProtocol / Content-Type Conversion
![Page 14: Dealing with Common Data Requirements in Your Enterprise](https://reader031.vdocument.in/reader031/viewer/2022030306/586e80521a28aba0038b51a3/html5/thumbnails/14.jpg)
Data Services
![Page 15: Dealing with Common Data Requirements in Your Enterprise](https://reader031.vdocument.in/reader031/viewer/2022030306/586e80521a28aba0038b51a3/html5/thumbnails/15.jpg)
Exposing Data-As-ServicesWhy ?
● Decouple data from the infrastructure and the data sources and expose them through standard web services interfaces.
● Ability to incorporate multiple data sources/entities into a single data model (Data Federation)
![Page 16: Dealing with Common Data Requirements in Your Enterprise](https://reader031.vdocument.in/reader031/viewer/2022030306/586e80521a28aba0038b51a3/html5/thumbnails/16.jpg)
Secure & Managed Data Access
● Transport and Application level security
● Authentication, authorization, confidentiality, integrity and encryption - with HTTP(S)
Basic Auth, WS-Security, WS-Trust, WS-SecureConversation, WS-Policy,
WS-Policy Attachment and WS-SecurityPolicy
● Authorization deals with defining who can access what
● Role based access control
● Fine-grained authorization with XACML
● Throttling access to data
![Page 17: Dealing with Common Data Requirements in Your Enterprise](https://reader031.vdocument.in/reader031/viewer/2022030306/586e80521a28aba0038b51a3/html5/thumbnails/17.jpg)
Federated Data Stores● Expose data from multiple data sources through a single service
● Facilitates entity aggregation
![Page 18: Dealing with Common Data Requirements in Your Enterprise](https://reader031.vdocument.in/reader031/viewer/2022030306/586e80521a28aba0038b51a3/html5/thumbnails/18.jpg)
Data/Entity Aggregation
![Page 19: Dealing with Common Data Requirements in Your Enterprise](https://reader031.vdocument.in/reader031/viewer/2022030306/586e80521a28aba0038b51a3/html5/thumbnails/19.jpg)
WSO2 Data Services Features
Ref: http://wso2.com/products/data-services-server/
![Page 20: Dealing with Common Data Requirements in Your Enterprise](https://reader031.vdocument.in/reader031/viewer/2022030306/586e80521a28aba0038b51a3/html5/thumbnails/20.jpg)
Analytics
![Page 21: Dealing with Common Data Requirements in Your Enterprise](https://reader031.vdocument.in/reader031/viewer/2022030306/586e80521a28aba0038b51a3/html5/thumbnails/21.jpg)
Data Analytics● Batch Analytics
Analyze a set of data collected over a period of time. Suitable for high volumes of data.
● Real-Time AnalyticsContinuous processing of input data in real time. Suitable for critical systems where immediate actions is required e.g: Flight radar systems
● Interactive AnalyticsObtaining fast results on indexed data by executing ad-hoc queries
● Predictive AnalyticsPredict future events by analyzing historical and current data
![Page 22: Dealing with Common Data Requirements in Your Enterprise](https://reader031.vdocument.in/reader031/viewer/2022030306/586e80521a28aba0038b51a3/html5/thumbnails/22.jpg)
Big Data
What is Big data ?
“Big data is a term for data sets that are so large or complex that traditional
data processing applications are inadequate to deal with them” - Ref: Wikipedia
![Page 23: Dealing with Common Data Requirements in Your Enterprise](https://reader031.vdocument.in/reader031/viewer/2022030306/586e80521a28aba0038b51a3/html5/thumbnails/23.jpg)
Big Data AnalysisWhy ?
● Make informed Business decisions - make decisions based on patterns emerging from analyzing historic data
● Improve customer experience - discover customer preferences, purchasing patterns and present the most relevant data
● Process Improvements - identify areas of the business process that needs improvement
![Page 24: Dealing with Common Data Requirements in Your Enterprise](https://reader031.vdocument.in/reader031/viewer/2022030306/586e80521a28aba0038b51a3/html5/thumbnails/24.jpg)
Big Data Analysis ExampleBetter customer experience in airline seat reservation/allocation
img ref: http://staticcontent.transat.com/airtransat/infovoyageurs/content/EN/seating-plan-a310-300(1).png
![Page 25: Dealing with Common Data Requirements in Your Enterprise](https://reader031.vdocument.in/reader031/viewer/2022030306/586e80521a28aba0038b51a3/html5/thumbnails/25.jpg)
Real Time Analytics
● Identify most meaningful events within an event cloud
● Analyze the impact
● Acts on them in real time
![Page 26: Dealing with Common Data Requirements in Your Enterprise](https://reader031.vdocument.in/reader031/viewer/2022030306/586e80521a28aba0038b51a3/html5/thumbnails/26.jpg)
Real Time Analytics ExampleCity Transport Control System - Analyzing traffic, monitor movement of busses, generate alerts based on traffic, speed & route
![Page 27: Dealing with Common Data Requirements in Your Enterprise](https://reader031.vdocument.in/reader031/viewer/2022030306/586e80521a28aba0038b51a3/html5/thumbnails/27.jpg)
Predictive Analytics & Machine LearningApproaches:
● Machine LearningMachine learning is the science of getting computers to act without being explicitly programmed - http://online.stanford.edu/
● Other approaches such as statistical modeling
![Page 28: Dealing with Common Data Requirements in Your Enterprise](https://reader031.vdocument.in/reader031/viewer/2022030306/586e80521a28aba0038b51a3/html5/thumbnails/28.jpg)
Predictive Analytics Examplee-Commerce sites use predictive analytics to suggest the most relevant merchandize, increasing sales opportunity
![Page 29: Dealing with Common Data Requirements in Your Enterprise](https://reader031.vdocument.in/reader031/viewer/2022030306/586e80521a28aba0038b51a3/html5/thumbnails/29.jpg)
WSO2 Data Analytics
Ref:wso2.com
![Page 30: Dealing with Common Data Requirements in Your Enterprise](https://reader031.vdocument.in/reader031/viewer/2022030306/586e80521a28aba0038b51a3/html5/thumbnails/30.jpg)
Data Visualization
![Page 31: Dealing with Common Data Requirements in Your Enterprise](https://reader031.vdocument.in/reader031/viewer/2022030306/586e80521a28aba0038b51a3/html5/thumbnails/31.jpg)
Data Visualization Contd.
What is Data Visualization ?
● View data in a constructive and comprehensible format
● Facilitates interaction with data - drill into the data for visual
analysis
● Detect patterns (e.g: sales patterns) that may go un-noticed unless
data is properly visualized
![Page 32: Dealing with Common Data Requirements in Your Enterprise](https://reader031.vdocument.in/reader031/viewer/2022030306/586e80521a28aba0038b51a3/html5/thumbnails/32.jpg)
High Availability
![Page 33: Dealing with Common Data Requirements in Your Enterprise](https://reader031.vdocument.in/reader031/viewer/2022030306/586e80521a28aba0038b51a3/html5/thumbnails/33.jpg)
High Availability of Data