1030 stephen brobst semantic data modeling
TRANSCRIPT
-
Semantic Data Modeling: The Key to Re-usable Data
Stephen Brobst
Chief Technology Officer
Teradata Corporation
617-422-0800
-
2 Copyright 2013. Stephen Brobst. Do not duplicate without written permission
Not just a collection
of subjects...
Activity Party Product Account
Single, Integrated System
...but also their
relationships
Party Product
Account Activity
Dont model subjects individually!
Model your entire
business!
Enterprise Information Management Data Modeling
-
3 Copyright 2013. Stephen Brobst. Do not duplicate without written permission
Functional Views
Sales Marketing Finance Rates/
Regulatory
Customer
Service Risk
Demographics Pricing
General Ledger
Promotions
Products Safety Engineering
Production
HR
Contracts
Works OK for OLTP, but causes
data chaos for BI applications.
-
4 Copyright 2013. Stephen Brobst. Do not duplicate without written permission
Business Intelligence Requires Data Integration
Product Data
Customer Data
Account Data
Transaction Data
G/L Data
Market Data
External Data
-
5 Copyright 2013. Stephen Brobst. Do not duplicate without written permission Copyright 2005, Stephen A. Brobst. All rights reserved.
Data Modeling Techniques
Key observation: Practitioners in the data warehousing industry frequently confuse construction of the semantic data model, logical data model, and physical data model.
A semantic data model (SDM) captures the business view of information for a specific knowledge worker community or analytic application.
A logical data model (LDM) captures the business relationships in the enterprise information independent of a specific analytic application or departmental view.
A physical data model (PDM) captures the implementation design of tables in the data warehouse.
-
6 Copyright 2013. Stephen Brobst. Do not duplicate without written permission
Data Model Deployment
Conceptual Data Model
Project A Project B Project C
Enterprise Data Standards
Subject Area A
Enterprise Logical Data Model(3NF)
xxxxxxx
xxxxxxxxxx
xxxxx
xxxxxxx
xxxxxxx
xxxxxxxxxx
xxxxx
xxxxxxx
xxxxxxx
xxxxxxxxxx
xxxxx
xxxxxxx
xxxxxxx
xxxxxxxxxx
xxxxx
xxxxxxx
xxxxxxx
xxxxxxxxxx
xxxxx
xxxxxxx
xxxxxxx
xxxxx
xx
Subject Area
A
Physical Model Realization
Design Meta
Data
Semantic Model Views
Subject Area
B
Subject Area
C
Single Physical Data Model
Subject Area B
-
7 Copyright 2013. Stephen Brobst. Do not duplicate without written permission Copyright 2005, Stephen A. Brobst. All rights reserved.
Semantic Data Modeling
Semantic data modeling is a logical data modeling technique; the semantic view of information does not necessarily need to be physicalized in the database.
There may be a different semantic data model for each department/applications that uses the data warehouse.
Dimensional modeling is a common technique for constructing the semantic data model for an analytic application, but is not the only viable approach.
-
8 Copyright 2013. Stephen Brobst. Do not duplicate without written permission
Dimensional
Physical Data Extensions
Different Semantic Model Designs are Appropriate for Different Types of Knowledge Workers
Normalized Generic Structures
Index choices & selective table denormalizations
Relational ADS Application
-
9 Copyright 2013. Stephen Brobst. Do not duplicate without written permission Copyright 2005, Stephen A. Brobst. All rights reserved.
Physical Data Model
Physical data model represents the tables constructed in the database.
Recommendations:
Use the (3NF) LDM as the starting point for the PDM with selective denormalization when appropriate for (primarily) performance reasons.
Overlay (dimensional) SDM on top of PDM using views and/or semantic metadata in your BI tool.
Design LDM first, then use application-specific business requirements to derive the SDMs and performance considerations to map into the PDM.
-
10 Copyright 2013. Stephen Brobst. Do not duplicate without written permission
Semantic Models Should be BI Tool Agnostic
MicroStrategy
Teradata OLAP Connector
Tableau
Tier 3 Access
Tier 2 Integrated
Tier 1 Acquisition
-
11 Copyright 2013. Stephen Brobst. Do not duplicate without written permission
A collection of data modeling assets that help make database design and development faster and easier for the access layer:
> Access layer provides path for data from the integrated data model to end user consumption.
> When this layer not well-designed, it can impact speed, security, and simplicity in developing and delivering reports, BI applications.
Re-usable building blocks provide flexibility and consistency to the development process:
> SMBBs include pre-built semantic models.
Focuses on a specific analytic need in a specific industry:
> For example, Communications Mobile Revenue Analytics.
SMBBs are to the semantic layer as iLDMs are to the integrated layer of a data warehouse implementation.
What is a Semantic Modeling Building Block (SMBB) Portfolio?
-
12 Copyright 2013. Stephen Brobst. Do not duplicate without written permission
Dimensional Model
Dimension Building Blocks
Dimension Building Blocks Support a Range of Analytical Needs
Fixed, Normalized Hierarchy Fixed, Flattened Hierarchy Variable Depth Hierarchy (Recursive)
-
13 Copyright 2013. Stephen Brobst. Do not duplicate without written permission
What are SMBBs? How are they related to an LDM?
Building from the Foundation for your Data Warehouse:
An LDM is like a blueprint for a house that you are building. It serves as the foundation for your integrated data warehouse.
The SMBBs are like room designs that meet specific homeowner needs. Different rooms need different designs based on their purpose. Similarly, for each new business application, new semantic models are needed.
SMBBs provide different designs (building blocks) for the modeler to choose from in building the semantic models.
These flexible, reusable building blocks can be used for other analytic needs.
-
14 Copyright 2013. Stephen Brobst. Do not duplicate without written permission
Q: Where does it all start? A: Business requirements drive the process!
Relationships between the Three Types of Data Models
The Logical Model is
used to drive
generalization and
support source data
leverage and reuse.
Logical Data Model Physical Data Model Semantic Data Models
Data access patterns
Support data re-use
The Semantic Model
captures data
access patterns that
must be supported
by the core physical
model.
The Physical Model
provides core
support for data
integration within
the information
architecture.
-
15 Copyright 2013. Stephen Brobst. Do not duplicate without written permission
Semantic Layer Benefits
Efficient table joins can be encouraged inside the SDM views.
Views are low maintenance objects.
Views do not consume database space.
Join indexes (JIs) and aggregate join indexes (AJIs) can be created based on the access paths embedded in the SDMs.
PDM is not compromised with new application requirements.
Protection of code assets.
-
16 Copyright 2013. Stephen Brobst. Do not duplicate without written permission Copyright 2005, Stephen A. Brobst. All rights reserved.
Conclusions
Critical to distinguish between logical data modeling, semantic data modeling, and physical data modeling.
Separate the implementation of the semantic model from the physical data model (PDM) deployment for maximum flexibility.
Selective use of PDM extensions to optimize performance.
Either ANSI standard views of the semantic metadata within your BI tool of choice can be used for creating a semantic data layer without sacrificing flexibility of the PDM.