data warehouse and data mining - wordpress.comsnowflake schema • snowflake schema – advantages...

71
Naeem Ahmed Department of Software Engineering Mehran Univeristy of Engineering and Technology Jamshoro Email: [email protected] Data Warehouse and Data Mining Lecture No. 06 Data Modeling

Upload: others

Post on 06-Apr-2020

19 views

Category:

Documents


1 download

TRANSCRIPT

Page 1: Data Warehouse and Data Mining - WordPress.comSnowflake Schema • Snowflake schema – Advantages – With a snowflake schema the size of the dimension tables will be reduced and

Naeem Ahmed

Department of Software Engineering Mehran Univeristy of Engineering and Technology Jamshoro

Email: [email protected]

Data Warehouse and Data Mining Lecture No. 06

Data Modeling

Page 2: Data Warehouse and Data Mining - WordPress.comSnowflake Schema • Snowflake schema – Advantages – With a snowflake schema the size of the dimension tables will be reduced and

DW Modeling •  Data Modeling

–  Conceptual Modeling: •  Multidimensional Entity Relationship (ME/R) Model •  Multidimensional UML (mUML)

–  Logical Modeling: •  Cubes, Dimensions, Hierarchies

–  Physical Modeling: •  Star, Snowflake, Array storage

Page 3: Data Warehouse and Data Mining - WordPress.comSnowflake Schema • Snowflake schema – Advantages – With a snowflake schema the size of the dimension tables will be reduced and

Logical Model •  Goal of the Logical Model

–  Confirm the subject areas –  Create ‘real’ facts and dimensions from the subjects

that have been identified –  Establish the needed granularity for dimensions

•  Logical structure of the multidimensional model –  Cubes: Sales, Purchase, Price, Inventory –  Dimensions: Product, Time, Geography, Client

Page 4: Data Warehouse and Data Mining - WordPress.comSnowflake Schema • Snowflake schema – Advantages – With a snowflake schema the size of the dimension tables will be reduced and

Logical Model

Page 5: Data Warehouse and Data Mining - WordPress.comSnowflake Schema • Snowflake schema – Advantages – With a snowflake schema the size of the dimension tables will be reduced and

Dimensions •  Dimensions are…

–  analysis purpose chosen entities, within the data model •  One dimension can be used to define more than one cube •  They are hierarchically organized

Page 6: Data Warehouse and Data Mining - WordPress.comSnowflake Schema • Snowflake schema – Advantages – With a snowflake schema the size of the dimension tables will be reduced and

Dimensions •  Dimension hierarchies are organized in

classification levels (e.g., Day, Month, ...) –  The dependencies between the classification levels are

described by the classification schema through functional dependencies

•  An attribute B is functionally dependent on an attribute A, denoted A ⟶ B, if for all a ∈ dom(A) there exists exactly one b ∈ dom(B) corresponding to it

Page 7: Data Warehouse and Data Mining - WordPress.comSnowflake Schema • Snowflake schema – Advantages – With a snowflake schema the size of the dimension tables will be reduced and

Dimensions •  Classification schemas

–  The classification schema of a dimension D is a semi- ordered set of classification levels ({D.K0, ..., D.Kk}, ⟶ )

–  With a smallest element D.K0, i.e. there is no classification level with smaller granularity

–  A fully-ordered set of classification levels is called a Path •  If classification schema of the time dimension is considered, then one

has the following paths –  T.Day ⟶ T.Week and T.Day ⟶ T.Month ⟶ T.Quarter ⟶ T.Year

•  Here T.Day is the smallest element

Page 8: Data Warehouse and Data Mining - WordPress.comSnowflake Schema • Snowflake schema – Advantages – With a snowflake schema the size of the dimension tables will be reduced and

Dimensions •  Classification hierarchies

–  Let D.K0 ⟶ ...⟶ D.Kk be a path in the classification schema of dimension D

–  A classification hierarchy concerning these path is a balanced tree which

•  Has as nodes dom(D.K0) U...U dom(D.Kk) U {ALL} •  And its edges respect the functional dependencies

Page 9: Data Warehouse and Data Mining - WordPress.comSnowflake Schema • Snowflake schema – Advantages – With a snowflake schema the size of the dimension tables will be reduced and

Dimensions •  Example: classification hierarchy from the path

product dimension

Page 10: Data Warehouse and Data Mining - WordPress.comSnowflake Schema • Snowflake schema – Advantages – With a snowflake schema the size of the dimension tables will be reduced and

Dimensions Store Dimension Product Dimension

District

Region

Total

Brand

Manufacturer

Total

Stores Products

Page 11: Data Warehouse and Data Mining - WordPress.comSnowflake Schema • Snowflake schema – Advantages – With a snowflake schema the size of the dimension tables will be reduced and

Cubes •  Cubes consist of data cells with one or more

measures –  If a cube schema S(G,M) consists of a granularity G=

(D1.K1, ..., Dn.Kn) and a set M=(M1, ..., Mm) representing the measure

–  A Cube (C) is a set of cube cells, C ⊆ dom(G) x dom(M)

Page 12: Data Warehouse and Data Mining - WordPress.comSnowflake Schema • Snowflake schema – Advantages – With a snowflake schema the size of the dimension tables will be reduced and

Cubes •  The coordinates of a cell are the classification

nodes from dom(G) corresponding to the cell –  Sales ((Article, Day, Store, Client) (Turnover)) –  …

Page 13: Data Warehouse and Data Mining - WordPress.comSnowflake Schema • Snowflake schema – Advantages – With a snowflake schema the size of the dimension tables will be reduced and

Cubes •  4 dimensions (supplier, city, quarter, product)

Page 14: Data Warehouse and Data Mining - WordPress.comSnowflake Schema • Snowflake schema – Advantages – With a snowflake schema the size of the dimension tables will be reduced and

Cubes •  One can now imagine n-dimensional cubes

–  n-D cube is called a base cuboid –  The top most cuboid, the 0-D, which holds the highest

level of summarization is called apex cuboid

- The full data cube is formed by the lattice of cuboids

Page 15: Data Warehouse and Data Mining - WordPress.comSnowflake Schema • Snowflake schema – Advantages – With a snowflake schema the size of the dimension tables will be reduced and

Cubes •  But things can get complicated pretty fast

Page 16: Data Warehouse and Data Mining - WordPress.comSnowflake Schema • Snowflake schema – Advantages – With a snowflake schema the size of the dimension tables will be reduced and

Basic Operations •  Basic operations of the multidimensional model on

the logical level –  Selection –  Projection –  Cube join –  Sum –  Aggregation

Page 17: Data Warehouse and Data Mining - WordPress.comSnowflake Schema • Snowflake schema – Advantages – With a snowflake schema the size of the dimension tables will be reduced and

Basic Operations •  Multidimensional Selection

–  The selection on a cube C((D1.K1,..., Dg.Kg), (M1, ..., Mm)) through a predicate P, is defined as σP(C) = {z Є C:P(z)}, if all variables in P are either:

•  Classification levels K , which functionally depend on a classification level in the granularity of K, i.e. Di.Ki ⟶ K

•  Measures from (M1, ..., Mm)

–  E.g. σP.Prod_group=“Video”(Sales)

Page 18: Data Warehouse and Data Mining - WordPress.comSnowflake Schema • Snowflake schema – Advantages – With a snowflake schema the size of the dimension tables will be reduced and

Basic Operations •  Multidimensional projection

–  The projection of a function of a measure F(M) of cube C is defined as

πF(M)(C) = { (g,F(m)) ∈ dom(G) x dom(F(M)): (g,m) ∈ C} –  E.g. , price projection πturnover, sold_items(Sales)

Page 19: Data Warehouse and Data Mining - WordPress.comSnowflake Schema • Snowflake schema – Advantages – With a snowflake schema the size of the dimension tables will be reduced and

Basic Operations •  Join operations between cubes is usual

–  E.g. if turnover would not be provided, it could be calculated with the help of the unit price from the price cube

•  2 cubes C1(G1, M1) and C2(G2, M2) can only be joined, if they have the same granularity (G1= G2 = G) –  C1⋈C2= C(G, M1∪ M2)

Page 20: Data Warehouse and Data Mining - WordPress.comSnowflake Schema • Snowflake schema – Advantages – With a snowflake schema the size of the dimension tables will be reduced and

Basic Operations •  When the granularities are different, but there is

still need to join the cubes, aggregation has to be performed

–  E.g. , Sales ⋈ Inventory: aggregate Sales((Day,Article, Store, Client)) to Sales((Month, Article, Store, Client))

Aggregation: A whole formed or calculated by the combination of many separate units or items – Total

Page 21: Data Warehouse and Data Mining - WordPress.comSnowflake Schema • Snowflake schema – Advantages – With a snowflake schema the size of the dimension tables will be reduced and

Basic Operations •  Aggregation: most important operation for OLAP

operations –  Aggregation functions

•  Build a single values from set of value, e.g. in SQL: SUM,AVG, Count, Min, Max

•  Example: SUM(P.Product_group, G.City, T.Month)(Sales)

Page 22: Data Warehouse and Data Mining - WordPress.comSnowflake Schema • Snowflake schema – Advantages – With a snowflake schema the size of the dimension tables will be reduced and

Change support •  Classification schema, cube schema, classification

hierarchy are all designed in the building phase and considered as fix –  Practice has proven otherwise –  DW grow old, too –  Changes are strongly connected to the time factor –  This lead to the time validity of these concepts

•  Reasons for schema modification –  New requirements –  Modification of the data source

Page 23: Data Warehouse and Data Mining - WordPress.comSnowflake Schema • Snowflake schema – Advantages – With a snowflake schema the size of the dimension tables will be reduced and

Classification Hierarchy •  E.g. Saturn sells a lot of electronics

–  Lets consider mobile phones •  They built their DW on 01.03.2003 •  A classification hierarchy of their data until 01.07.2008 could

look like this:

Page 24: Data Warehouse and Data Mining - WordPress.comSnowflake Schema • Snowflake schema – Advantages – With a snowflake schema the size of the dimension tables will be reduced and

Classification Hierarchy •  After 01.07.2008 3G becomes hip and affordable

and many phone makers start migrating towards 3G capable phones –  Lets say O2 makes its XDA 3G capable

Page 25: Data Warehouse and Data Mining - WordPress.comSnowflake Schema • Snowflake schema – Advantages – With a snowflake schema the size of the dimension tables will be reduced and

Classification Hierarchy •  After 01.04.2010 phone makers already develop

4G capable phones

Page 26: Data Warehouse and Data Mining - WordPress.comSnowflake Schema • Snowflake schema – Advantages – With a snowflake schema the size of the dimension tables will be reduced and

Classification Hierarchy •  It is important to trace the evolution of the data

–  It can explain which data was available at which moment in time

–  Such a versioning system of the classification hierarchy can be performed by constructing a validity matrix

•  When is something, valid? •  Use timestamps to mark it!

Page 27: Data Warehouse and Data Mining - WordPress.comSnowflake Schema • Snowflake schema – Advantages – With a snowflake schema the size of the dimension tables will be reduced and

Classification Hierarchy •  Annotated Change data

Page 28: Data Warehouse and Data Mining - WordPress.comSnowflake Schema • Snowflake schema – Advantages – With a snowflake schema the size of the dimension tables will be reduced and

Classification Hierarchy •  The tree can be stored as dimension metadata

–  The storage form is a validity matrix •  Rows are parent nodes •  Columns are child nodes

Page 29: Data Warehouse and Data Mining - WordPress.comSnowflake Schema • Snowflake schema – Advantages – With a snowflake schema the size of the dimension tables will be reduced and

Classification Hierarchy •  Deleting a node in a classification hierarchy

–  Should be performed only in exceptional cases •  It can lead to information loss

–  How to solve it? •  Soon GSM phones will not be produced anymore •  But one might have some more in warehouses, to be delivered •  Or one might want to query data since when GSM was sold •  Just mark the end validity date of the GSM branch in the validity

matrix

Page 30: Data Warehouse and Data Mining - WordPress.comSnowflake Schema • Snowflake schema – Advantages – With a snowflake schema the size of the dimension tables will be reduced and

Classification Hierarchy •  Query classification

–  Having the validity information we can support queries like as is versus as is

•  Regards all the data as if the only valid classification hierarchy is the present one

•  In the case of O2 XDA, it will be considered as it has always been a 3G phone

Page 31: Data Warehouse and Data Mining - WordPress.comSnowflake Schema • Snowflake schema – Advantages – With a snowflake schema the size of the dimension tables will be reduced and

Classification Hierarchy •  As is versus as was

–  Orders the classification hierarchy by the validity matrix information

•  O2 XDA was a GSM phone until 01.07.2008 and a 3G phone afterwards

Page 32: Data Warehouse and Data Mining - WordPress.comSnowflake Schema • Snowflake schema – Advantages – With a snowflake schema the size of the dimension tables will be reduced and

Classification Hierarchy •  As was versus as was

–  Past time hierarchies can be reproduced

–  E.g., query data with an older classification hierarchy

•  Like versus like –  Only data whose classification

hierarchy remained unmodified, is evaluated

–  E.g. the Nokia 3600 and the Black Berry

Page 33: Data Warehouse and Data Mining - WordPress.comSnowflake Schema • Snowflake schema – Advantages – With a snowflake schema the size of the dimension tables will be reduced and

Schema Modification •  Improper modification of a schema (deleting a

dimension) can lead to –  Data loss –  Inconsistencies

•  Data is incorrectly aggregated or adapted

•  Proper schema modification is complex but –  It brings flexibility for the end user

•  The possibility to ask “As Is vs. As Was” queries and so on

•  Alternatives –  Schema evolution –  Schema versioning

Page 34: Data Warehouse and Data Mining - WordPress.comSnowflake Schema • Snowflake schema – Advantages – With a snowflake schema the size of the dimension tables will be reduced and

Schema Modification •  Schema evolution

–  Modifications can be performed without data loss –  It involves schema modification and data adaptation to

the new schema –  This data adaptation process is called Instance

adaptation

Page 35: Data Warehouse and Data Mining - WordPress.comSnowflake Schema • Snowflake schema – Advantages – With a snowflake schema the size of the dimension tables will be reduced and

Schema Modification •  Schema evolution

–  Advantage •  Faster to execute queries in DW with many schema

modifications

–  Disadvantages •  It limits the end user flexibility to query based on the past

schemas •  Only actual schema based queries are supported

Page 36: Data Warehouse and Data Mining - WordPress.comSnowflake Schema • Snowflake schema – Advantages – With a snowflake schema the size of the dimension tables will be reduced and

Schema Modification •  Schema versioning

–  Also no data loss –  All the data corresponding to all the schemas are always

available –  After a schema modification the data is held in their

belonging schema •  Old data - old schema

•  New data - new schema

Page 37: Data Warehouse and Data Mining - WordPress.comSnowflake Schema • Snowflake schema – Advantages – With a snowflake schema the size of the dimension tables will be reduced and

Schema Modification •  Schema versioning

–  Advantages •  Allows higher flexibility, e.g.,“As Is vs.As Was”, etc. queries

–  Disadvantages •  Adaptation of the data to the queried schema is done on the

spot •  This results in longer query run time

Page 38: Data Warehouse and Data Mining - WordPress.comSnowflake Schema • Snowflake schema – Advantages – With a snowflake schema the size of the dimension tables will be reduced and

Physical Model •  Defining the physical structures

–  Setting up the database environment –  Performance tuning strategies

•  Indexing •  Partitioning •  Materialization

•  Goal –  Define the actual storage architecture –  Decide on how the data is to be accessed and how it is

arranged

Page 39: Data Warehouse and Data Mining - WordPress.comSnowflake Schema • Snowflake schema – Advantages – With a snowflake schema the size of the dimension tables will be reduced and

Physical Model •  Physical implementation of the multidimensional

paradigm model can be: –  Relational

•  Snowflake-schema •  Star-schema •  Fast constellation

–  Multidimensional •  Matrixes

Page 40: Data Warehouse and Data Mining - WordPress.comSnowflake Schema • Snowflake schema – Advantages – With a snowflake schema the size of the dimension tables will be reduced and

Physical Model •  Relational model, goals:

–  As low loss of semantically knowledge as possible e.g., classification hierarchies

–  The translation from multidimensional queries must be efficient

–  The RDBMS should be able to run the translated queries efficiently

–  The maintenance of the present tables should be easy and fast e.g., when loading new data

Page 41: Data Warehouse and Data Mining - WordPress.comSnowflake Schema • Snowflake schema – Advantages – With a snowflake schema the size of the dimension tables will be reduced and

Relational Model •  Going from multidimensional to relational

–  Representations for cubes, dimensions, classification hierarchies and attributes

–  Implementation of cubes without the classification hierarchies is easy

•  A table can be seen as a cube •  A column of a table can be considered as a dimension mapping •  A tuple in the table represents a cell in the cube •  If one interprets only a part of the columns as dimensions, he/

she can use the rest as measures •  The resulting table is called a fact table

Page 42: Data Warehouse and Data Mining - WordPress.comSnowflake Schema • Snowflake schema – Advantages – With a snowflake schema the size of the dimension tables will be reduced and

Relational Model

Page 43: Data Warehouse and Data Mining - WordPress.comSnowflake Schema • Snowflake schema – Advantages – With a snowflake schema the size of the dimension tables will be reduced and

Relational Model •  Snowflake-schema

–  Simple idea: use a table for each classification level •  This table includes the ID of the classification level and other

attributes •  2 neighbor classification levels are connected by 1:n

connections e.g., from n Days to 1 Month •  The measures of a cube are maintained in a fact table •  Besides measures, there are also the foreign key IDs for the

smallest classification levels

Page 44: Data Warehouse and Data Mining - WordPress.comSnowflake Schema • Snowflake schema – Advantages – With a snowflake schema the size of the dimension tables will be reduced and

Relational Model •  Snowflake?

–  The facts/measures are in the center –  The dimensions spread out in each direction and

branch out with their granularity

Page 45: Data Warehouse and Data Mining - WordPress.comSnowflake Schema • Snowflake schema – Advantages – With a snowflake schema the size of the dimension tables will be reduced and

Snowflake Example

Page 46: Data Warehouse and Data Mining - WordPress.comSnowflake Schema • Snowflake schema – Advantages – With a snowflake schema the size of the dimension tables will be reduced and

Snowflake Example Advantage: Best performance when queries involve aggregation Disadvantage: Complicated maintenance and metadata, explosion in the number of tables in the database

Page 47: Data Warehouse and Data Mining - WordPress.comSnowflake Schema • Snowflake schema – Advantages – With a snowflake schema the size of the dimension tables will be reduced and

Snowflake Schema •  Snowflake schema – Advantages

–  With a snowflake schema the size of the dimension tables will be reduced and queries will run faster

•  If a dimension is very sparse (most measures corresponding to the dimension have no data)

•  And/or a dimension has long list of attributes which may be queried

•  Snowflake schema – Disadvantages –  Fact tables are responsible for 90% of the storage requirements

•  Thus, normalizing the dimensions usually lead to insignificant improvements –  Normalization of the dimension tables can reduce the performance of

the DW because it leads to a large number of tables •  E.g., when connecting dimensions with coarse granularity these tables are joined with

each other during queries •  A query which connects Product category with Year and Country is clearly not

performant (10 tables need to be connected)

Page 48: Data Warehouse and Data Mining - WordPress.comSnowflake Schema • Snowflake schema – Advantages – With a snowflake schema the size of the dimension tables will be reduced and

Relational Model •  Star schema

–  Basic idea: use a denormalized schema for all the dimensions

•  A star schema can be obtained from the snowflake schema through the denormalization of the tables belonging to a dimension

Database normalization is the process of organizing the fields and tables of a relational database to minimize redundancy and dependency

Normalization usually involves dividing large tables into smaller (and less redundant) tables and defining relationships between them

A de-normalization is the process of attempting to optimize the read performance of a database by adding redundant data or by grouping data

Page 49: Data Warehouse and Data Mining - WordPress.comSnowflake Schema • Snowflake schema – Advantages – With a snowflake schema the size of the dimension tables will be reduced and

Star schema Example

Benefits: Easy to understand, easy to define hierarchies, reduces # of physical joins, low maintenance, very simple metadata

Drawbacks: Summary data in the fact table yields poorer performance for summary levels, huge dimension tables a problem

Page 50: Data Warehouse and Data Mining - WordPress.comSnowflake Schema • Snowflake schema – Advantages – With a snowflake schema the size of the dimension tables will be reduced and

Star Schema •  Advantages

–  Improves query performance for often-used data –  Less tables and simple structure –  Efficient query processing with regard to dimensions

•  Disadvantages –  In some cases, high overhead of redundant data

Page 51: Data Warehouse and Data Mining - WordPress.comSnowflake Schema • Snowflake schema – Advantages – With a snowflake schema the size of the dimension tables will be reduced and

Star Schema The biggest drawback: dimension tables must carry a level indicator for every record and every query must use it. In the example, without the level constraint, keys for all stores in the NORTH region, including aggregates for region and district will be pulled from the fact table, resulting in error. Solution: FACT CONSTELLATION

Example: Select A.STORE_KEY, A.PERIOD_KEY, A.dollars from Fact_Table A

where A.STORE_KEY in (select STORE_KEY from Store_Dimension B where region = “North” and Level = 2)

Level is needed whenever aggregates are stored with detail facts.

PERIOD KEY

Store Dimension Time Dimension

Product Dimension

STORE KEYPRODUCT KEYPERIOD KEY

DollarsUnitsPrice

Period DescYearQuarterMonthDayCurrent FlagResolutionSequence

Fact Table

PRODUCT KEY

Store DescriptionCityStateDistrict IDDistrict Desc.Region_IDRegion Desc.Regional Mgr.Level

Product Desc.BrandColorSizeManufacturerLevel

STORE KEY

Page 52: Data Warehouse and Data Mining - WordPress.comSnowflake Schema • Snowflake schema – Advantages – With a snowflake schema the size of the dimension tables will be reduced and

Fact Constellation Schema •  FACT Constellation Schema describes a logical

database structure of Data Warehouse or Data Mart •  It can design with collection of de-normalized FACT,

Shared and Conformed Dimension tables •  FACT Constellation Schema is an extended and

decomposed STAR Schema •  In Fact Constellations, aggregate tables are created

separately from the detail, therefore, it is impossible to pick up –  Example, Store detail when querying the District Fact Table

Page 53: Data Warehouse and Data Mining - WordPress.comSnowflake Schema • Snowflake schema – Advantages – With a snowflake schema the size of the dimension tables will be reduced and

Fact Constellation Schema •  Fact Constellation is a good alternative to the Star, but when

dimensions have very high cardinality, the sub-selects in the dimension tables can be a source of delay

•  An alternative is to normalize the dimension tables by attribute level, with each smaller dimension table pointing to an appropriate aggregated fact table, the “Snowflake Schema”

•  Advantage: No need for the “Level” indicator in the dimension tables, since no aggregated data is stored with lower-level detail

•  Disadvantage: Dimension tables are still very large in some cases, which can slow performance; front-end must be able to detect existence of aggregate facts, which requires more extensive metadata

Page 54: Data Warehouse and Data Mining - WordPress.comSnowflake Schema • Snowflake schema – Advantages – With a snowflake schema the size of the dimension tables will be reduced and

Fact Constellation Example

Dollars Units Price

District Fact Table

District_ID PRODUCT_KEY PERIOD_KEY

Dollars Units Price

Region Fact Table

Region_ID PRODUCT_KEY PERIOD_KEY

PERIOD KEY

Store Dimension Time Dimension

Product Dimension

STORE KEYPRODUCT KEYPERIOD KEY

DollarsUnitsPrice

Period DescYearQuarterMonthDayCurrent FlagSequence

Fact Table

PRODUCT KEY

Store DescriptionCityStateDistrict IDDistrict Desc.Region_IDRegion Desc.Regional Mgr.

Product Desc.BrandColorSizeManufacturer

STORE KEY

Page 55: Data Warehouse and Data Mining - WordPress.comSnowflake Schema • Snowflake schema – Advantages – With a snowflake schema the size of the dimension tables will be reduced and

Snowflake vs. Star •  Snowflake

–  The structure of the classifications are expressed in table schemas

–  The fact table and dimension tables are normalized

•  Star –  The entire classification

is expressed in just one table

–  The fact table is normalized while in the dimension table the normalization is broken

•  This leads to redundancy of information in the dimension tables

Page 56: Data Warehouse and Data Mining - WordPress.comSnowflake Schema • Snowflake schema – Advantages – With a snowflake schema the size of the dimension tables will be reduced and

Snowflake vs. Star •  Snowflake •  Star

Page 57: Data Warehouse and Data Mining - WordPress.comSnowflake Schema • Snowflake schema – Advantages – With a snowflake schema the size of the dimension tables will be reduced and

Snowflake vs. Star Attributes Star Schema Snowflake Schema Ease of maintenance / change

Has redundant data and hence less easy to maintain/change

No redundancy and hence more easy to maintain and change

Ease of Use Less complex queries and easy to understand

More complex queries and hence less easy to understand

Query Performance Less no. of foreign keys and hence lesser query execution time

More foreign keys-and hence more query execution time

Type of Datawarehouse Good for datamarts with simple relationships (1:1 or 1:many)

Good to use for datawarehouse core to simplify complex relationships (many:many)

Joins Fewer Joins Higher number of Joins Dimension table Contains only single

dimension table for each dimension

It may have more than one dimension table for each dimension

When to use When dimension table contains less number of rows, go for Star schema.

When dimension table is relatively big in size, snowflaking is better as it reduces space.

Normalization/ De-Normalization

Both Dimension and Fact Tables are in De-Normalized form

Dimension Tables are in Normalized form but Fact Table is still in De-Normalized form

Data model Top down approach Bottom up approach !

Page 58: Data Warehouse and Data Mining - WordPress.comSnowflake Schema • Snowflake schema – Advantages – With a snowflake schema the size of the dimension tables will be reduced and

Snowflake to Star •  When should one go from Snowflake to star?

–  Heuristics-based decision •  When typical queries relate to coarser granularity (like product

category) •  When the volume of data in the dimension tables is relatively

low compared to the fact table –  In this case a star schema leads to negligible overhead through

redundancy, but performance is improved •  When modifications on the classifications are rare compared to

insertion of fact data –  In this case these modifications controlled through the data load

process of the ETL reducing the risk of data anomalies

Page 59: Data Warehouse and Data Mining - WordPress.comSnowflake Schema • Snowflake schema – Advantages – With a snowflake schema the size of the dimension tables will be reduced and

Which one is winner? –  It depends on the necessity

•  Fast query processing or efficient space usage –  However, most of the time a mixed form is used

•  The Starflake schema: some dimensions stay normalized corresponding to the snowflake schema, while others are denormalized according to the star schema

–  Snowflake schema: The decision on how to deal with the dimensions is influenced by

–  Frequency of the modifications: if the dimensions change often, normalization leads to better results

–  Amount of dimension elements: the bigger the dimension tables, the more space normalization saves

–  Number of classification levels in a dimension: more classification levels introduce more redundancy in the star schema

–  Materialization of aggregates for the dimension levels: if the aggregates are materialized, a normalization of the dimension can bring better response time

Snowflake or Star?

Page 60: Data Warehouse and Data Mining - WordPress.comSnowflake Schema • Snowflake schema – Advantages – With a snowflake schema the size of the dimension tables will be reduced and

More Schemas •  Galaxies

–  In pratice we usually have more measures described by different dimensions

•  Thus, more fact tables

Page 61: Data Warehouse and Data Mining - WordPress.comSnowflake Schema • Snowflake schema – Advantages – With a snowflake schema the size of the dimension tables will be reduced and

More Schemas •  Fact constellations

–  Pre-calculated aggregates

•  Factless fact tables –  Fact tables do not have non-key data

•  Can be used for event tracking or to inventory the set of possible occurrences

•  Factless fact table does not have any measures •  For example, consider a record of student attendance in

classes. In this case, the fact table would consist of 3 dimensions: the student dimension, the time dimension, and the class dimension.

Page 62: Data Warehouse and Data Mining - WordPress.comSnowflake Schema • Snowflake schema – Advantages – With a snowflake schema the size of the dimension tables will be reduced and

More Schemas •  Factless fact tables

•  This factless fact table would look like the following:

Page 63: Data Warehouse and Data Mining - WordPress.comSnowflake Schema • Snowflake schema – Advantages – With a snowflake schema the size of the dimension tables will be reduced and

Relational Model •  Relational model – disadvantages

–  The representation of the multidimensional data can be implemented relationally with a finite set of transformation steps, however:

•  Multidimensional queries have to be first translated to the relational representation

•  A direct interaction with the relational data model is not fit for the end user

Page 64: Data Warehouse and Data Mining - WordPress.comSnowflake Schema • Snowflake schema – Advantages – With a snowflake schema the size of the dimension tables will be reduced and

Multidimensional Model •  The basic data structure for multidimensional data

storage is the array •  The elementary data structures are the cubes and

the dimensions –  C=((D1, ..., Dn), (M1, ..., Mm))

•  The storage is intuitive as arrays of arrays, physically linearized

Page 65: Data Warehouse and Data Mining - WordPress.comSnowflake Schema • Snowflake schema – Advantages – With a snowflake schema the size of the dimension tables will be reduced and

Multidimensional Model •  Linearization example: 2D cube |D1| = 5, |D2| = 4,

cube cells = 20 –  Query: Jackets sold in March? –  Measure stored in cube cell D1[4], D2[3] –  The 2D cube is physically stored as a linear array, so D1[4], D2[3] becomes array cell 14

•  (Index(D2) – 1) * |D1| + Index(D1) •  Linearized Index=2*5+4=14

Page 66: Data Warehouse and Data Mining - WordPress.comSnowflake Schema • Snowflake schema – Advantages – With a snowflake schema the size of the dimension tables will be reduced and

Linearization •  Generalization:

–  Given a cube C=((D1, D2, ..., Dn), (M1:Type1, M2:Type2, ..., Mm:Typem)), the index of a cube cell z with coordinates (x1, x2, ..., xn) can be linearized as follows:

•  Index(z) = x1 + (x2 - 1) * |D1| + (x3 - 1) * |D1| * |D2| + ... +(xn -1)*|D1|*...*|Dn-1| = 1+∑i=1 n ((xi -1)*∏ j=1 i-1 |Di|)

Page 67: Data Warehouse and Data Mining - WordPress.comSnowflake Schema • Snowflake schema – Advantages – With a snowflake schema the size of the dimension tables will be reduced and

Problems in Array-Storage •  Influence of the order of the dimensions in the

cube definition –  In the cube the cells of D2 are ordered one under the other e.g., sales of all pants involves a column in the cube –  After linearization, the information is spread among more data blocks/pages –  If one considers a data block can hold 5 cells, a query

over all products sold in January can be answered with just 1 block read, but a query of all sold pants, involves reading 4 blocks

Page 68: Data Warehouse and Data Mining - WordPress.comSnowflake Schema • Snowflake schema – Advantages – With a snowflake schema the size of the dimension tables will be reduced and

Problems in Array-Storage •  Solution: use caching techniques

–  But...caching and swapping is performed also by the operating system

–  MDBMS has to manage its caches such that the OS doesn’t perform any damaging swaps

•  Storage of dense cubes –  If cubes are dense, array storage is more efficient. However,

operations suffer due to the large cubes –  Solution: store dense cubes not linear but on 2 levels

•  The first contains indexes and the second the data cells stored in blocks

•  Optimization procedures like indexes (trees, bitmaps), physical partitioning, and compression (run-length- encoding) can be used

Page 69: Data Warehouse and Data Mining - WordPress.comSnowflake Schema • Snowflake schema – Advantages – With a snowflake schema the size of the dimension tables will be reduced and

Problems in Array-Storage •  Storage of sparse cubes

–  All the cells of a cube, including empty ones, have to be stored

–  Sparseness leads to data being stored in many physical blocks or pages

•  The query speed is affected by the large number of block accesses on the secondary memory

–  Solution: •  Do not store empty blocks or pages but adapt the index

structure •  2 level data structure: upper layer holds all possible

combinations of the sparse dimensions, lower layer holds dense dimensions

Page 70: Data Warehouse and Data Mining - WordPress.comSnowflake Schema • Snowflake schema – Advantages – With a snowflake schema the size of the dimension tables will be reduced and

Problems in Array-Storage •  2 level cube storage

Page 71: Data Warehouse and Data Mining - WordPress.comSnowflake Schema • Snowflake schema – Advantages – With a snowflake schema the size of the dimension tables will be reduced and