Dealing with Common Data Requirements in your Enterprise
Nipun SuwandaratnaSenior Solutions Engineer - WSO2
WSO2 Solution Architecture Best Practices Webinar Series - 2016
Agenda● Organizational Data
● Common Data Challenges of Modern Organizations
● Integrating with Different Messaging Infrastructures
● Data Services
● Data Analytics & Visualization
● High Availability
● Q&A
Organizations & Data
Old School
DB
Sys
tem
s
Files
Organization
Modern Data Ecosystem
External Systems
External Users
Organizational Data
● Master dataEg: Customer data, employee records, Supplier details, Product related data etc.
● Transactional dataThe data that master data participates in… transactions, discounts on bills etc. (changes constantly)
● Meta-dataData about data
Image: thinkpublic/photopin cc
Common Data Challenges Organizations Face● Work with multiple Data Transports and Data Formats
● Data Transformation and Validation
● Exposing data as services
● Secure and managed data access
● Federated data stores
● Data/Entity Aggregation
● Data Analytics
● Visualization of Data
Data Transports & FormatsFormats of data, their storage and transport mechanisms vary among
different systems
● Transports: HTTP, HTTPS, FTPS, SFTP, TCP, UDP, WebSocket, POP, IMAP, SMTP, JMS, AMQP, MQTT
● Formats & protocols: JSON, XML, SOAP, WS-*, HTML, EDI, HL7,Text, JPEG, MP4, binary formats
Integration
Integrating with Messaging Infrastructures
Message Transformation● Protocol and Format conversion and Message Translation
○ eg: SOAP to REST and XML to JSON and translate the output from one system to match the input format required by the other system
● Enrich Content ○ eg: Add or remove data fields; may require accessing a separate data source
● Wrap Content○ eg: Include additional message header fields or encryption source to query
required data● Data Validation
○ eg: Validate input data against a schema
Enterprise Service Bus
Message Transformation ExampleProtocol / Content-Type Conversion
Data Services
Exposing Data-As-ServicesWhy ?
● Decouple data from the infrastructure and the data sources and expose them through standard web services interfaces.
● Ability to incorporate multiple data sources/entities into a single data model (Data Federation)
Secure & Managed Data Access
● Transport and Application level security
● Authentication, authorization, confidentiality, integrity and encryption - with HTTP(S)
Basic Auth, WS-Security, WS-Trust, WS-SecureConversation, WS-Policy,
WS-Policy Attachment and WS-SecurityPolicy
● Authorization deals with defining who can access what
● Role based access control
● Fine-grained authorization with XACML
● Throttling access to data
Federated Data Stores● Expose data from multiple data sources through a single service
● Facilitates entity aggregation
Data/Entity Aggregation
WSO2 Data Services Features
Ref: http://wso2.com/products/data-services-server/
Analytics
Data Analytics● Batch Analytics
Analyze a set of data collected over a period of time. Suitable for high volumes of data.
● Real-Time AnalyticsContinuous processing of input data in real time. Suitable for critical systems where immediate actions is required e.g: Flight radar systems
● Interactive AnalyticsObtaining fast results on indexed data by executing ad-hoc queries
● Predictive AnalyticsPredict future events by analyzing historical and current data
Big Data
What is Big data ?
“Big data is a term for data sets that are so large or complex that traditional
data processing applications are inadequate to deal with them” - Ref: Wikipedia
Big Data AnalysisWhy ?
● Make informed Business decisions - make decisions based on patterns emerging from analyzing historic data
● Improve customer experience - discover customer preferences, purchasing patterns and present the most relevant data
● Process Improvements - identify areas of the business process that needs improvement
Big Data Analysis ExampleBetter customer experience in airline seat reservation/allocation
img ref: http://staticcontent.transat.com/airtransat/infovoyageurs/content/EN/seating-plan-a310-300(1).png
Real Time Analytics
● Identify most meaningful events within an event cloud
● Analyze the impact
● Acts on them in real time
Real Time Analytics ExampleCity Transport Control System - Analyzing traffic, monitor movement of busses, generate alerts based on traffic, speed & route
Predictive Analytics & Machine LearningApproaches:
● Machine LearningMachine learning is the science of getting computers to act without being explicitly programmed - http://online.stanford.edu/
● Other approaches such as statistical modeling
Predictive Analytics Examplee-Commerce sites use predictive analytics to suggest the most relevant merchandize, increasing sales opportunity
WSO2 Data Analytics
Ref:wso2.com
Data Visualization
Data Visualization Contd.
What is Data Visualization ?
● View data in a constructive and comprehensible format
● Facilitates interaction with data - drill into the data for visual
analysis
● Detect patterns (e.g: sales patterns) that may go un-noticed unless
data is properly visualized
High Availability
High Availability of Data