Download - Data vault: What's Next
Data Vault Modeling
WHAT’S NEXT???
© Dan Linstedt 2009-2012
This was a presentation I gave at an Array Conference In the Netherlands, in 2009.
2
A bit about me…
• Author, Inventor, Speaker – and part time photographer…
• 25+ years in the IT industry• Worked in DoD, US Gov’t, Fortune 50,
and so on…
• Find out more about the Data Vault:o http://www.youtube.com/LearnDataVaulto http://LearnDataVault.com
• Full profile on http://www.LinkedIn.com/dlinstedt
04/10/2023LearnDataVault.com
Who’s Using It?
04/10/2023LearnDataVault.com 3
Featured Word from Our Customer
04/10/2023LearnDataVault.com 4
Jonathan Rice, BScING Real Estate/Development International/Operational Support Location HP A.05.071PO Box 90463, 2509 LL The Hague, The Netherlands
They are connecting MS SharePoint back-end to Data Vault Modeling EDW.
What’s Changing the Game?
• Solid State Disk (SSD)• Hosted Cloud Computing• Column Based Databases• Unstructured Information• Ontologies / Taxonomies• Mining Engines• Broad Based Web Services
• Business Demand for Immediate Answers (pushing real-time)
• Compliance and Auditability (pushing volume)
04/10/2023LearnDataVault.com 5
• Data Visualization• Flash & Silverlight Front-Ends• Analytic functions in
Database Engines• Business Rules Engines
Melding with ETL and Web Services
• DW Appliances
Current EDW + DV Architecture
04/10/2023LearnDataVault.com 6
Sales
Finance
Contracts
StagingDVEDW
StarSchemas
ErrorMarts
ReportCollections
Enterprise BI SolutionSOA
(real-time)
(batch)
(batch)
Business Rules Downstream!(the Lens Filter)
First Change: Business Data Vault
04/10/2023LearnDataVault.com 7
Sales
Finance
Contracts
DVEDW
StarSchemas
ErrorMarts
ReportCollections
Enterprise BI SolutionSOA
(real-time)
Batch
Business Rules Downstream!(the Lens Filter)
BDVEDW
StagingStaging
Next Change: Staging Removal
04/10/2023LearnDataVault.com 8
Sales
Finance
Contracts
DVEDW
StarSchemas
ErrorMarts
ReportCollections
Enterprise BI SolutionSOA
(real-time)
Business Rules Downstream!(the Lens Filter)
BDVEDW
Write Back
Next Change: Virtual Marts
04/10/2023LearnDataVault.com 9
Virtual Marts
& Dynamic Cubes
Virtual Marts
& Dynamic Cubes
Sales
Finance
Contracts
DVEDW
Enterprise BI SolutionSOA
(real-time)
Business Rules Downstream!(the Lens Filter)
BDVEDW
Write Back
04/10/2023LearnDataVault.com 10
Next Change: Unstructured Data
Unstructured Data Sets Ontologies/Taxonomies
• Email• Docs• Images• Movies• Sound
Unstructured Processing Engine
Raw Data Vault EDW
Joins through LINK Structures
On-DemandCubes
A Look at Ontologies
04/10/2023LearnDataVault.com 11
Hierarchies of Data:Synonyms, Homonyms, Antonyms, Related Terms, Definitions, Categorizations, Organizations, Views of the data world
Ontologies HOLD THE KEY to understanding/conceptual relevanceOntologies can PIVOT raw data in to many different results
04/10/2023LearnDataVault.com 12
Data Vault EDW• Stored• Analyzed / Scored
Plateau: Operational Data Warehouse
Virtual Marts
Real-TimeMiningEngine
Staging Area
Non-S.O.R.Historical Batch Data
SORReal-Time Data
Real-TimeCollector
Web Interface (usually)
OperationalSystems
OperationalAlerts
StrategicReports& OLAP
& W.SVCS
Operational Systems
UnstructuredSemi-Structured
Non-SORBatch Data
OperationalApplicationsMaster Data
OperationalMetadata
Management
Direct Edits
Direct Edits
• Flexible• Accountable• Compliant • Scalable
• Normalized• Dynamic• Granular• Historic
StrategicReports& OLAP
& W.SVCS
Dynamic Cubes
Data Vault(EDW)
Operational Data Vault
04/10/2023LearnDataVault.com 13
Common Data Access
Layer
Operatio
nal
Applicatio
n
Read Current
Read/Lock
Insert Changes
Update/Unlock
Why go Operational Data Vault?• Benefits Include
o Direct access to datao Removal of 90% of batch streams, replaced by Transactionso Better/faster alerting capabilitieso Direct control over changes to BI answer sets
• What are the risks?o The Data Vault EDW becomes an ODV, brings with it ALL
responsibilities of the Operational System
• What are the problems?o Row Lockingo Consistent Data Accesso Data Version Controlo Security of Data through access points
• Has it been done before?o Yes, Cendant Timeshare Resource Group did it in 2002!
04/10/2023LearnDataVault.com 14
04/10/2023LearnDataVault.com 15
Plateau: Dynamic Data Warehouse
MetaMiningEngineH H H
H
H
H
L
L
L
L
DL
DL
Dynamically Constructed LinksDynamically Created HubsDynamically Altered SatellitesDynamically changed ETL/ELTDynamically Altered QueriesAutomatically Updated Cubes= Self Morphing (guided) Data Vault
Impacts of Dynamic Data VaultsBusiness• New found knowledge• Evolving Data Warehouse models over time• Faster Reporting• Self-Maintaining Back-Ends
Technical• Guided Changes to Structures• Auto-Adapting Load Routines• Auto-Adapting Query Sets• Auto-Suggested Star Schemas
04/10/2023LearnDataVault.com 16
Why go Dynamic?• Benefits include:
o Faster Turn Around Timeo Auto Configuration of “mundane tasks”o Discovery of new relationships (could result in increased revenue)o Self Healing Structures
• How Long before we See It?o 5 to 7 years out
• How do we get there?o Meta-Miningo Use of Ontologies inside the EDW
• What’s driving us there?o Business Users want faster turn around timeo Business Users want ingestion of Unstructured Data Setso Business Users need more control over their systems
04/10/2023LearnDataVault.com 17
18
Where To Learn More• The Technical Modeling Book: http://LearnDataVault.com
• The Discussion Forums: & eventshttp://LinkedIn.com – Data Vault Discussions
• Contact me:http://DanLinstedt.com - web [email protected] - email
• World wide User Group (Free)http://dvusergroup.com
04/10/2023LearnDataVault.com
