future-proofing your data strategy - simplicitybi · who is using it has changed: everyone how...
TRANSCRIPT
Copyright © 2017 R20/Consultancy B.V., The Hague, The Netherlands. All rights reserved. No part of this material may be reproduced, stored
in a retrieval system, or transmitted in any form or by any means, electronic, mechanical, photographic, or otherwise, without the explicit
written permission of the copyright owners.
Future-Proofing Your Data Strategy
Rick F. van der LansIndustry analyst
Email [email protected] Twitter @rick_vanderlanswww.r20.nl
Copyright © 2017 R20/Consultancy B.V., The Hague, The Netherlands 2
The Data Center in the Cellar
Copyright © 2017 R20/Consultancy B.V., The Hague, The Netherlands 3
Data for the Happy Few Only
Copyright © 2017 R20/Consultancy B.V., The Hague, The Netherlands 4
The First Data Coming Out of the Cellar
Copyright © 2017 R20/Consultancy B.V., The Hague, The Netherlands 5
So Many New Technologies, Ideas, Architectures, Techniques, and Opportunities
Copyright © 2017 R20/Consultancy B.V., The Hague, The Netherlands 6
Data
Management
Technology
Usa
ge
Ch
arac
teri
stic
s
Data governance Data protection
Master data
Analytics
Data lake
Data science
Self-Service
Customer
Facing app
Data brokerage
Micro services
Artificial Intelligence
Robotica
Big data
Fast data
Dark data
IoT
Streaming analytics
+AI
+AI
Cloud computing
VR + AR
Block chaining
Hadoop NoSQL
MPP
Copyright © 2017 R20/Consultancy B.V., The Hague, The Netherlands 7
Copyright © 2017 R20/Consultancy B.V., The Hague, The Netherlands 8
Don’t Always Believe What You Hear
Copyright © 2017 R20/Consultancy B.V., The Hague, The Netherlands 9
Don’t Always Believe What You See
Copyright © 2017 R20/Consultancy B.V., The Hague, The Netherlands 10
Don’t Always Believe What You Read
Big data: A revolution that will transform how we live, work and think
Companies are being destroyed and created around big data, …
Without big data, you are blind and deaf in the middle of a freeway.
Big data has arrived and is shaping IT today
The disruptive power of big data
Copyright © 2017 R20/Consultancy B.V., The Hague, The Netherlands 11
Data hasn’t changed,
it’s just more of the same
Copyright © 2017 R20/Consultancy B.V., The Hague, The Netherlands 12
Data usage has changed:Who is using it has changed: everyone
How they’re using it has changed: continuouslyWhen they’re using it has changed: instantaneously
Complexity of usage has changed: analyticsWhich data they need has changed: all
Copyright © 2017 R20/Consultancy B.V., The Hague, The Netherlands 13
ETL ETLETL
Sourcesystems
Data martsStagingarea
Analytics &reporting
Datawarehouse
The Classic Data Warehouse Architecture
Copyright © 2017 R20/Consultancy B.V., The Hague, The Netherlands 14
Data Warehouse and Data Usage
Developers
IT specialists
Business Users
Development Styles
Pre-programmed, auditable, governable,
formally tested
Self-service, investigative
Pre-programmed
Self-service, investigative
Report Types
Batch and online business reports
Customer-facing apps
Ad-hoc reports
Simple data retrieval
Ad-hoc reports
Data mining, statistics
Dark data analysis
Consumers
Business users
Legislators
External parties
Consumers
Business users
Business users
Business users
Data scientists
Business users and IT
Copyright © 2017 R20/Consultancy B.V., The Hague, The Netherlands 15
Phase 1: Data Warehouse and Data Usage
Developers
IT specialists
Business Users
Development Styles
Pre-programmed, auditable, governable,
formally tested
Self-service, investigative
Pre-programmed
Self-service, investigative
Report Types
Batch and online business reports
Customer-facing apps
Ad-hoc reports
Simple data retrieval
Ad-hoc reports
Data mining, statistics
Dark data analysis
Consumers
Business users
Legislators
External parties
Consumers
Business users
Business users
Business users
Data scientists
Business users and IT
Copyright © 2017 R20/Consultancy B.V., The Hague, The Netherlands 16
Phase 2: Data Warehouse and Data Usage
Developers
IT specialists
Business Users
Development Styles
Pre-programmed, auditable, governable,
formally tested
Self-service, investigative
Pre-programmed
Self-service, investigative
Report Types
Batch and online business reports
Customer-facing apps
Ad-hoc reports
Simple data retrieval
Ad-hoc reports
Data mining, statistics
Dark data analysis
Consumers
Business users
Legislators
External parties
Consumers
Business users
Business users
Business users
Data scientists
Business users and IT
Copyright © 2017 R20/Consultancy B.V., The Hague, The Netherlands 17
Phase 3: Data Warehouse and Data Usage
Developers
IT specialists
Business Users
Development Styles
Pre-programmed, auditable, governable,
formally tested
Self-service, investigative
Pre-programmed
Self-service, investigative
Report Types
Batch and online business reports
Customer-facing apps
Ad-hoc reports
Simple data retrieval
Ad-hoc reports
Data mining, statistics
Dark data analysis
Consumers
Business users
Legislators
External parties
Consumers
Business users
Business users
Business users
Data scientists
Business users and IT
Copyright © 2017 R20/Consultancy B.V., The Hague, The Netherlands 18
Copyright © 2017 R20/Consultancy B.V., The Hague, The Netherlands 19
Object Detection in Images
Copyright © 2017 R20/Consultancy B.V., The Hague, The Netherlands 20
Automatic Image Caption Generation
Copyright © 2017 R20/Consultancy B.V., The Hague, The Netherlands 21
Retail ExampleFully integrated face detection, real-time content targeting, and statistics gatheringFace tracking and “just-in-time” content deliveryGender recognitionAge estimationEmotion detection (A/B testing and feedback gathering)Clothes size and color detection
Copyright © 2017 R20/Consultancy B.V., The Hague, The Netherlands 22
Our Users are Changing
Copyright © 2017 R20/Consultancy B.V., The Hague, The Netherlands 23
ETL ETLETL
Sourcesystems
Data martsStagingarea
Analytics &reporting
Datawarehouse
The Classic Data Warehouse Architecture
Copyright © 2017 R20/Consultancy B.V., The Hague, The Netherlands 24
Programming logic to verify, transform, cleanse, integrate, interpret, and standardize source data from production systems
Backup and recovery mechanisms
Security mechanisms and policies to protect against unauthorized access and misuse of the data and reports
A priority scheme for developing and maintaining reports
Manual procedures initiated by calamities
The monitoring of reporting performance, scalability, availability and other non-functional aspects of the operational reporting environment
Various administrative procedures related to human and computer resources
Data governance rules
Master data management
…
Rigid Processes for Development, Operation, and Management
Copyright © 2017 R20/Consultancy B.V., The Hague, The Netherlands 25
What We Need is …
One universal architecture for all the data
• Transactional, external, fast (streaming), dark
One universal architecture for all formsof data usage
• From standard reporting via self-service todata science
Quick, easy, and lean access to all the data
Correct access to data
Copyright © 2017 R20/Consultancy B.V., The Hague, The Netherlands 26
The Logical Data Warehouse Architecture
SQLdatabases
streamingdatabases
socialmedia data
productionapplication
Hadoop,NoSQL
database
website
ESBmessaging
analytics& reporting
unstructureddata
mobileApp
legacydatabase
internalportal dashboard
cloudapplications
privatedata
Logical Data Warehouse Architecture
applications
Copyright © 2017 R20/Consultancy B.V., The Hague, The Netherlands 27
Another View of the Logical DWA
ETLETL
Sourcesystems
Stagingarea
Analytics &reportingData
warehouse
Socialmedia data
Open data
Spreadsheets
LogicalD
ata Wareh
ou
se Arch
itecture
Big data
Copyright © 2017 R20/Consultancy B.V., The Hague, The Netherlands 28
The Logical DWA is Metadata Driven
Analytics &reporting
LogicalD
ata Wareh
ou
se Arch
itecture
ETLETL
Sourcesystems
Stagingarea
Datawarehouse
Socialmedia data
Open data
Spreadsheets
Big data
Repository
Copyright © 2017 R20/Consultancy B.V., The Hague, The Netherlands 29
Data Virtualization Overview (1)
productionapplication website
analytics& reporting
mobileApp
internalportal dashboard
Data Virtualization Server
SQLdatabases
streamingdatabases
socialmedia data
Hadoop,NoSQL
database
ESBmessaging
unstructureddatalegacy
database
cloudapplications
privatedata
applications
Copyright © 2017 R20/Consultancy B.V., The Hague, The Netherlands 30
Data Virtualization Overview (2)
streamingdatabases
socialmedia data
productionapplication website
analytics& reporting
mobileApp
internalportal dashboard
privatedata
ODBC/SQL JDBC/SQL XML/SOAP REST/JSON XQuery MDX/DAX
JMS SQL SQL+ XSLT Hive Prop. Excel JSONCICS SOAP
applications
SQL statement
JMS message SQL statement SOAP messageData Virtualization Server
unstructureddataSQL
databasesHadoop,NoSQL
database
ESBmessaging
legacydatabase
cloudapplications
Copyright © 2017 R20/Consultancy B.V., The Hague, The Netherlands 31
Dat
a V
irtu
aliz
atio
n S
erve
r
Virtual table pointing to source
Data consumer
Importing Source Data
Source
Copyright © 2017 R20/Consultancy B.V., The Hague, The Netherlands 32
Dat
a V
irtu
aliz
atio
n S
erve
r
Virtual table pointing to source
Virtual table:May contain row selections, column selections, column concatenations, transformations, column and table name changes, groupings, aggregations, data cleansing, …
Data consumer
Defining Tranformations and Intregrations
Source
Copyright © 2017 R20/Consultancy B.V., The Hague, The Netherlands 33
Layers of Virtual Tables
Enterprise data layer
Data consumptionlayer
Data sourcelayer
Data V
irtualizatio
n Server
Copyright © 2017 R20/Consultancy B.V., The Hague, The Netherlands 34
High-Quality Data
Transactional data
External data
Fast data
Dark data
Source: Kumar Gauraw, May 2015; http://www.dataintegration.ninja/relationship-between-data-quality-and-master-data-management/
Copyright © 2017 R20/Consultancy B.V., The Hague, The Netherlands 35
High-Quality Data at Business Speed
Integrated with logical data warehouse architecture
Data governance from the start
Lean and agile data quality
Self-service data quality
AI-driven data preparation
Good is sometimes good enough
Lean and agile master data management
Copyright © 2017 R20/Consultancy B.V., The Hague, The Netherlands 36
Pairing Master Data and Data Virtualization
Data Virtualization Server
Master data Data source 1 Data source 2
productionapplication website
analytics& reporting
mobileApp
internalportal dashboard
Copyright © 2017 R20/Consultancy B.V., The Hague, The Netherlands 37
Pairing Master Data and Data Virtualization
Prototyping
• Quick views of what golden records may look like
Consolidating multiple masters into an Enterprise Master View
• Data virtualization can combine one or more of these individual MDM systems and present one common semantically consistent “golden record” combining the most appropriate data elements from each departmental system -- all in real time
Consolidating Multiple Master Domains
• Combining siloed MDM solutions
Enriched MDM / 360 View
Copyright © 2017 R20/Consultancy B.V., The Hague, The Netherlands 38
Consolidating Siloed MDM Solutions
Data Virtualization Server
Master dataCustomerFinance
Master dataCustomerMarketing
Master dataProduct
productionapplication website
analytics& reporting
mobileApp
internalportal dashboard
Copyright © 2017 R20/Consultancy B.V., The Hague, The Netherlands 39
Future-Proofing
Your Data Strategy:For everyone
For continuous useFor instantaneous use
For analytical useFor all users
With high-quality data
Copyright © 2017 R20/Consultancy B.V., The Hague, The Netherlands 40
Data first!
Copyright © 2017 R20/Consultancy B.V., The Hague, The Netherlands 41