lecture @dhbw: data warehouse part vi: is ...data has always been my main focus during my long-time...
TRANSCRIPT
A company of Daimler AG
LECTURE @DHBW: DATA WAREHOUSE
PART VI: IS THE DWH DEAD?ANDREAS BUCKENHOFER, DAIMLER TSS
ABOUT ME
https://de.linkedin.com/in/buckenhofer
https://twitter.com/ABuckenhofer
https://www.doag.org/de/themen/datenbank/in-memory/
http://wwwlehre.dhbw-stuttgart.de/~buckenhofer/
https://www.xing.com/profile/Andreas_Buckenhofer2
Andreas BuckenhoferSenior DB [email protected]
Since 2009 at Daimler TSS Department: Big Data Business Unit: Analytics
ANDREAS BUCKENHOFER, DAIMLER TSS GMBH
Data Warehouse / DHBWDaimler TSS 3
“Forming good abstractions and avoiding complexity is an essential part of a successful data architecture”
Data has always been my main focus during my long-time occupation in the area of data integration. I work for Daimler TSS as Database Professional and Data Architect with over 20 years of experience in Data Warehouse projects. I am working with Hadoop and NoSQL since 2013. I keep my knowledge up-to-date - and I learn new things, experiment, and program every day.
I share my knowledge in internal presentations or as a speaker at international conferences. I'm regularly giving a full lecture on Data Warehousing and a seminar on modern data architectures at Baden-Wuerttemberg Cooperative State University DHBW. I also gained international experience through a two-year project in Greater London and several business trips to Asia.
I’m responsible for In-Memory DB Computing at the independent German Oracle User Group (DOAG) and was honored by Oracle as ACE Associate. I hold current certifications such as "Certified Data Vault 2.0 Practitioner (CDVP2)", "Big Data Architect“, „Oracle Database 12c Administrator Certified Professional“, “IBM InfoSphere Change Data Capture Technical Professional”, etc.
DHBWDOAG
Contact/Connect
As a 100% Daimler subsidiary, we give
100 percent, always and never less.
We love IT and pull out all the stops to
aid Daimler's development with our
expertise on its journey into the future.
Our objective: We make Daimler the
most innovative and digital mobility
company.
NOT JUST AVERAGE: OUTSTANDING.
Daimler TSS
INTERNAL IT PARTNER FOR DAIMLER
+ Holistic solutions according to the Daimler guidelines
+ IT strategy
+ Security
+ Architecture
+ Developing and securing know-how
+ TSS is a partner who can be trusted with sensitive data
As subsidiary: maximum added value for Daimler
+ Market closeness
+ Independence
+ Flexibility (short decision making process,
ability to react quickly)
Daimler TSS 5
Daimler TSS
LOCATIONS
Data Warehouse / DHBW
Daimler TSS ChinaHub Beijing10 employees
Daimler TSS MalaysiaHub Kuala Lumpur42 employees
Daimler TSS IndiaHub Bangalore22 employees
Daimler TSS Germany
7 locations
1000 employees*
Ulm (Headquarters)
Stuttgart
Berlin
Karlsruhe
* as of August 2017
6
• After the end of this lecture you will be able to
• Is a DWH obsolete with Big Data requirements?
WHAT YOU WILL LEARN TODAY
Data Warehouse / DHBWDaimler TSS 7
LOGICAL STANDARD DATA WAREHOUSE ARCHITECTURE
Data Warehouse and Big Data / DHBWDaimler TSS 8
Data Warehouse
FrontendBackend
External data sources
Internal data sources
Staging Layer(Input Layer)
OLTP
OLTP
Core Warehouse
Layer(Storage
Layer)
Mart Layer(Output Layer)
(Reporting Layer)
Integration Layer
(Cleansing Layer)
Aggregation Layer
Metadata Management
Security
DWH Manager incl. Monitor
IS BIG DATA A DISRUPTIVE TECHNOLOGY TO REPLACE DWHS?
Data Warehouse and Big Data / DHBWDaimler TSS 9
Sources: https://www.linkedin.com/groups/45685/45685-6224210695295168512?trk=hp-feed-group-discussion&_mSplash=1https://speakerdeck.com/nehanarkhede/etl-is-dead-long-live-streamshttps://gcn.com/blogs/reality-check/2014/01/hadoop-vs-data-warehousing.aspx
Read the paper https://www.computerwoche.de/a/big-data-oder-data-warehouse,3092517
• Is big data a disruptive technology to replace DWHs?
• What are Inmon and Kimball saying about big data?
• Why are DWH-users dissatisfied with many DWH solutions?
• What are limits for classical DWHs?
• Which advantages does Hadoop promise?
• What are current challenges for Hadoop?
BIG DATA OR DATA WAREHOUSE?
Data Warehouse and Big Data / DHBWDaimler TSS 10
Inmon
Subject-oriented, integrated view of data not feasible
with Hadoop
KimballHadoop has high potential due to
flexibility, performance, and
cost reduction
WHAT ARE INMON AND KIMBALL SAYING ABOUT BIG DATA?
Data Warehouse and Big Data / DHBWDaimler TSS 11
• Neglected communication
• Insufficient data modeling
• Missing documentation
• Inadequate data quality
• High costs (maintenance, development)
WHY ARE DWH-USERS DISSATISFIED WITH MANY DWHSOLUTIONS?
Data Warehouse and Big Data / DHBWDaimler TSS 12
WHAT ARE LIMITS FOR CLASSICAL DWHS?
Data Warehouse and Big Data / DHBWDaimler TSS 13
DWH
Variety
Volume
Costs(License, HW)
Velocity
Veracity
WHAT ARE LIMITS FOR CLASSICAL DWHS?
Data Warehouse and Big Data / DHBWDaimler TSS 14
Source: http://www.ibmbigdatahub.com/sites/default/files/infographic_file/4Vs_Infographic_final.pdf
• Flexibility
• (linear) Scalability
• Fast (batches)
• Reduced costs
• Variety of tools for specific needs
• Resilient to failure
WHICH ADVANTAGES DOES HADOOP PROMISE?
Data Warehouse and Big Data / DHBWDaimler TSS 15
• Many specialized tools (tools zoo)
• Weak integration of tools
• Degree of maturity
• No Table Joins
• Limited support of Ad-hoc user queries
WHAT ARE CURRENT CHALLENGES FOR HADOOP?
Data Warehouse and Big Data / DHBWDaimler TSS 16
IS BIG DATA A DISRUPTIVE TECHNOLOGY TO REPLACE DWHS?
Data Warehouse and Big Data / DHBWDaimler TSS 17
Big Data / Hadoop has high potential to solve challenges arising from
• Variety + Velocity + Volume + Veracity
Classical DWH is a general architecture and solution
• DWH is mature - Big Data architecture just at the beginning
• DWH is an integrated solution compared to Hadoop or NoSQL with many specialized tools (one purpose/use cases: no general solution)
IS BIG DATA A DISRUPTIVE TECHNOLOGY TO REPLACE DWHS?
Data Warehouse and Big Data / DHBWDaimler TSS 18
Big Data / Hadoop has high potential to solve challenges arising from
• Variety + Velocity + Volume + Veracity
Classical DWH is a general architecture and solution
• DWH is mature - Big Data architecture just at the beginning
• DWH is an integrated solution compared to Hadoop or NoSQL with many specialized tools (one purpose/use cases: no general solution)
DWHs will exist in future
together with Big
Data solutions
Daimler TSS GmbHWilhelm-Runge-Straße 11, 89081 Ulm / Telefon +49 731 505-06 / Fax +49 731 505-65 99
[email protected] / Internet: www.daimler-tss.com/ Intranet-Portal-Code: @TSSDomicile and Court of Registry: Ulm / HRB-Nr.: 3844 / Management: Christoph Röger (CEO), Steffen Bäuerle
Data Warehouse / DHBWDaimler TSS 19
THANK YOU