data warehouse - sharif university of...

Data Warehouse

Ali Kamandi

Sharif University of Technology

Spring 2007

kamandi@ce.sharif.edu

2

Part 1:

Data Warehouse Concepts

3

Data Management

� A critical success factor: IT applications cannot be done without using data.

� The Difficulties of managing Data: • The amount of data increases exponentially with time

• Data are scattered throughout organization

• An ever-increasing amount of external data needs

• Data security, quality, and integrity are critical

4

Data Life Cycle

5

Data Sources

� Internal Data Sources: data about people,

products, services, and processes.

� Personal Data: IS users or other corporate

employees may document their own expertise

by creating personal data.

� External Data Sources: Data from commercial

databases to sensors and satellites.

6

Data, Data everywhere yet ...

� I can’t find the data I need

� data is scattered over the network

� many versions, subtle differences

� I can’t get the data I need

� need an expert to get the data

� I can’t understand the data I found

� available data poorly documented

� I can’t use the data I found

� results are unexpected

� data needs to be transformed from one

form to other

7

What is a Data Warehouse?

A single, complete and

consistent store of data

obtained from a variety of

different sources made

available to end users in a

what they can understand and

use in a business context.

[Barry Devlin]

8

Which are ourlowest/highest margin

customers ?

Which are ourlowest/highest margin

customers ?

Who are my customers and what products are they buying?

Who are my customers and what products are they buying?

Which customersare most likely to go to the competition ?

Which customersare most likely to go to the competition ?

What impact will new products/services

have on revenue and margins?

What impact will new products/services

have on revenue and margins?

What product prom--otions have the biggest

impact on revenue?

What product prom--otions have the biggest

impact on revenue?

What is the most effective distribution

channel?

What is the most effective distribution

channel?

Why Data Warehousing?

9

Decision Support

� Used to manage and control business

� Data is historical or point-in-time

� Optimized for inquiry rather than update

� Use of the system is loosely defined and can

be ad-hoc

� Used by managers and end-users to

understand the business and make judgements

10

Evolution of Decision Support

� 60’s: Batch reports

� hard to find and analyze information

� inflexible and expensive, reprogram every request

� 70’s: Terminal based DSS and EIS

� 80’s: Desktop data access and analysis tools

� query tools, spreadsheets, GUIs

� easy to use, but access only operational db

� 90’s: Data warehousing with integrated OLAP

engines and tools

11

Definition of data warehousing

According to W.H.Inmon

A data warehouse is a subject-oriented,

integrated, time-variant and non-volatile

collection of data in support of management’s

decision making process.

12

Characteristics of a Data Warehouse Subject-Oriented. Data are organized by subject and contain

information relevant for decision support only .

Consistency. Data in different operational databases may be encoded differently . In the data warehouse, though, they will be coded in a consistent manner.

Time variant. The data are kept for many years so that they can be used for trends, forecasting, and comparisons over time.

Non-volatile. Data are not updated once entered into the warehouse.

Multidimensional. Typically the data warehouse uses a multidimensional structure .

Web-based. Today’s data warehouse are designed to provide an efficient computing environment for web-based applications.

� often a copy of operational data

� with value-added data (e.g., summaries, history)

13

Why a Warehouse?

� Two Approaches:

� Query-Driven (Lazy)

� Warehouse (Eager)

Source Source

?

14

Query-Driven Approach

Client Client

Wrapper Wrapper Wrapper

Mediator

Source Source Source

15

Warehouse Architecture

Client Client

Warehouse

Source Source Source

Query & Analysis

Integration

Metadata

16

Advantages of Warehousing

� High query performance

� Queries not visible outside warehouse

� Local processing at sources unaffected

� Can operate when sources unavailable

� Can query data not stored in a DBMS

� Extra information at warehouse

� Modify, summarize (store aggregates)

� Add historical information

17

Advantages of Query-Driven

� No need to copy data

� less storage

� no need to purchase data

� More up-to-date data

� Only query interface needed at sources

18

OLTP vs. OLAP

� OLTP: On Line Transaction Processing

� Describes processing at operational sites

� OLAP: On Line Analytical Processing

� Describes processing at warehouse

19

OLTP vs. OLAP

� Mostly updates

� Many small transactions

� Mb-Tb of data

� Raw data

� Clerical users

� Up-to-date data

� Consistency,

recoverability critical

� Mostly reads

� Queries long, complex

� Gb-Tb of data

� Summarized, consolidated

data

� Decision-makers, analysts

as users

OLTP OLAP

20

OLTP vs. OLAP OLTP OLAP

users clerk, IT professional knowledge worker

function day to day operations decision support

DB design application-oriented subject-oriented

data current, up-to-date detailed, flat relational isolated

historical, summarized, multidimensional integrated, consolidated

usage repetitive ad-hoc

access read/write index/hash on prim. key

lots of scans

unit of work short, simple transaction complex query

# records accessed tens millions

#users thousands hundreds

DB size 100MB-GB 100GB-TB

metric transaction throughput query throughput, response

21

Data Marts

Data Mart: A small data warehouse designed for a

strategic business unit ( SBU) or a department

The advantage of data marts include: low cost

(Prices under $100,000 versus $1million or more for

data warehouses); significantly shorter lead time for

implementation (often less than 90 days), local rather

than central control (conferring power on the using

group), More rapid response and more easily

understood and navigated than an enterprise wide

data warehouse .

22

Part 2:

Data Warehouse Design

23

Building a Data Warehouse

24

Relational and Multidimensional Database

� Relational databases store data in two –

dimensional tables. Multidimensional

databases typically store data in arrays, which

consist of at least three business dimension.

25

Relational data model

� based on a single structure of data values in a two

dimensional table

CUSTOMER ORDER

………

…Lyn002

…Robert001

…Cus_nameCus_id

…

03 Dec 02

02 Dec 02

Ord_date

………

…Lyn02

…00201

…Cus_idOrd_no

26

A Sample Data CubeTotal annual sales

of TV in U.S.A.Date

Pro

duct

Cou

ntr

y

sum

sumTV

VCRPC

1Qtr 2Qtr 3Qtr 4Qtr

U.S.A

Canada

Mexico

sum

27

Cuboids Corresponding to the Cubeall

product date country

product,date product,country date, country

product, date, country

0-D(apex) cuboid

1-D cuboids

2-D cuboids

3-D(base) cuboid

• Cuboids show the data at different degrees of summarization.

• Given a set of dimensions, we can construct a lattice of cuboids, each showing the data at a different level of summarization, or group by. The lattice of cuboids is then referred to as a data cube.

28

Multidimensional Data Model

� Composed of one fact table and a set of dimension tables.

� Dimensional table: each dimension table has a simple table (non-composite) primary key that corresponds exactly to one of the components of the composite key in the fact table.

� A multidimensional data model is typically organized around a central theme, like sales, for instance.

29

Conceptual Modeling of Data Warehouses

Modeling data warehouses: dimensions & measures

� Star schema: A fact table in the middle connected to a set

of dimension tables

� Snowflake schema: A refinement of star schema where

some dimensional hierarchy is normalized into a set of

smaller dimension tables, forming a shape similar to

snowflake

� Fact constellations: Multiple fact tables share dimension

tables, viewed as a collection of stars, therefore called

galaxy schema or fact constellation

30

Example of Star Schematime_key

day

day_of_the_week

month

quarter

year

time

location_key

street

city

province_or_street

country

location

Sales Fact Table

time_key

item_key

branch_key

location_key

units_sold

dollars_sold

avg_sales

Measures

item_key

item_name

brand

type

supplier_type

item

branch_key

branch_name

branch_type

branch

31

Example of Snowflake Schematime_key

day

day_of_the_week

month

quarter

year

time

location_key

street

city_key

location

Sales Fact Table

time_key

item_key

branch_key

location_key

units_sold

dollars_sold

avg_sales

Measures

item_key

item_name

brand

type

supplier_key

item

branch_key

branch_name

branch_type

branch

supplier_key

supplier_type

supplier

city_key

city

province_or_street

country

city

32

Example of Fact Constellation

time_key

day

day_of_the_week

month

quarter

year

time

location_key

street

city

province_or_street

country

location

Sales Fact Table

time_key

item_key

branch_key

location_key

units_sold

dollars_sold

avg_sales

Measures

item_key

item_name

brand

type

supplier_type

item

branch_key

branch_name

branch_type

branch

Shipping Fact Table

time_key

item_key

shipper_key

from_location

to_location

dollars_cost

units_shipped

shipper_key

shipper_name

location_key

shipper_type

shipper

33

Typical OLAP Operations� Roll up (drill-up): summarize data

� by climbing up hierarchy or by dimension reduction

� Drill down (roll down): reverse of roll-up

� from higher level summary to lower level summary or detailed data,

or introducing new dimensions

� Slice and dice:

� project and select

� Pivot (rotate):

� reorient the cube, visualization, 3D to series of 2D planes.

� Other operations

34

Operations

� Rollup: summarize data

� e.g., given sales data, summarize sales for last

year by product category and region

� Drill down: get more details

� e.g., given summarized sales as above, find

breakup of sales by city within each region, or

within the Andhra region

35

More Cube Operations

� Slice and dice: select and project

� e.g.: Sales of soft-drinks in Andhra over the last

quarter

� Pivot: change the view of data

36

Cube

sale prodId storeId amt

p1 c1 12

p2 c1 11

p1 c3 50

p2 c2 8

c1 c2 c3

p1 12 50

p2 11 8

Fact table view:Multi-dimensional cube:

dimensions = 2

37

3-D Cube

sale prodId storeId date amt

p1 c1 1 12

p2 c1 1 11

p1 c3 1 50

p2 c2 1 8

p1 c1 2 44

p1 c2 2 4

day 2c1 c2 c3

p1 44 4

p2 c1 c2 c3

p1 12 50

p2 11 8

day 1

dimensions = 3

Multi-dimensional cube:Fact table view:

38

Another Example

sale prodId storeId date amt

p1 c1 1 12

p2 c1 1 11

p1 c3 1 50

p2 c2 1 8

p1 c1 2 44

p1 c2 2 4

• Add up amounts by day, product• In SQL: SELECT date, sum(amt) FROM SALE

GROUP BY date, prodId

sale prodId date amt

p1 1 62

p2 1 19

p1 2 48

drill-down

rollup

39

Pivoting

sale prodId storeId date amt

p1 c1 1 12

p2 c1 1 11

p1 c3 1 50

p2 c2 1 8

p1 c1 2 44

p1 c2 2 4

day 2c1 c2 c3

p1 44 4

p2 c1 c2 c3

p1 12 50

p2 11 8

day 1

Multi-dimensional cube:Fact table view:

c1 c2 c3

p1 56 4 50

p2 11 8

40

Design a Warehouse?

� Design data warehouse

� Design data marts

� Design representation (Star schema, …)

� gathering data

� Which data is needed?

� Where does it come from?

� cleansing, integrating, ...

� querying, reporting, analysis

� data mining

� monitoring, administering warehouse

41

Data Gathering

� Periodic snapshots

� Database triggers

� Log shipping

� Data shipping (replication service)

� …

42

Integration

� Data Cleaning

� Data Loading

� Derived DataClient Client

Warehouse

Source Source Source

Query & Analysis

Integration

Metadata

43

Data Cleaning

� Migration (e.g., yen � dollars)

� Fusion (e.g., mail list, customer merging)

billing DB

service DB

customer1(Joe)

customer2(Joe)

merged_customer(Joe)

44

Loading Data

� Incremental vs. refresh

� Off-line vs. on-line

� Frequency of loading

� At night, 1x a week/month, continuously

� Parallel/Partitioned load

45

Derived Data

� Derived Warehouse Data

� When to update derived data?

46

Part 3:

Data Mining

47

Data Mining Concepts

Data mining: The process of searching for

valuable business information in a large

database, data warehouse, or data mart.

48

Data Mining Application

Retailing and sales

Banking

Manufacturing and production

Insurance

Police work

Health care

Marketing

49

Text Mining

The application of data mining to non-

structured or less-structured text files.

50

Web Mining

The application of data mining techniques to

discover actionable and meaningful patterns form

web resources.

Web mining is used in the following areas:

information filtering, surveillance, mining of web-

access logs for analyzing usage.

51

Some basic operations

� Predictive:

� Regression

� Classification

� Descriptive:

� Clustering / similarity matching

� Association rules and variants

52

Classification

� Given old data about customers and

payments, predict new applicant’s loan

eligibility.

Age

Salary

Profession

Location

Customer type

Previous customers Classifier Decision rules

Salary > 5 L

Prof. = Exec

New applicant’s data

Good/

bad

53

Classification methods

Goal: Predict class Ci = f(x1, x2, .. Xn)

� Regression: (linear or any other polynomial)

� a*x1 + b*x2 + c = Ci.

� Nearest neighbor

� Decision tree classifier

� Neural networks

54

� Tree where internal nodes are simple

decision rules on one or more attributes and

leaf nodes are predicted class labels.

Decision trees

Salary < 1 M

Prof = teacher

Good

Age < 30

BadBadGood

55

Neural network

� Set of nodes connected by directed weighted

edges

Hidden nodes

Output nodes

x1

x2

x3

x1

x2

x3

w1

w2

w3

y

n

i

ii

ey

xwo

−

=

+

=

= ∑

1

1)(

)(1

σ

σ

Basic NN unitA more typical NN

56

Bayesian learning

� Assume a probability model on generation of data.

� Apply Bayes theorem to find most likely class as:

)(

)()|(max)|(max :class predicted

dp

cpcdpdcpc

jj

cj

c jj

==

57

Clustering

� Unsupervised learning when old data with class

labels not available e.g. when introducing a new

product.

� Group/cluster existing customers based on time

series of payment history such that similar

customers in same cluster.

58

What is association rule mining?

Hmmm, which items are frequently

purchased together by my customers?

milkcereal

breadmilk

bread

butter

milk bread

sugar eggs

Customer 1

Market Analyst

Customer 2

sugareggs

Customer n

Customer 3

Shopping Baskets 100100Basket 4

001011Basket 3

100111Basket 2

010011Basket 1

eggscerea

l

butte

r

suga

r

brea

d

milk

59

What is association rule mining? (cont.)

100100Basket 4

001011Basket 3

100111Basket 2

010011Basket 1

eggscerealbuttersugarbreadmilk

211233count

Support (milk)=3

Support (bread)=3

Support (sugar)=2

……

Support (milk U bread)=3

Support (milk U sugar)=1

……

Support (milk U bread U sugar)=1

……

Support (milk U bread U sugar U butter U cereal U eggs)=0

Confidence (A → B)=Support (A U B)/Support (A)

As Confidence (milk → bread) =

= Support (milk U bread)/Support (milk) = 3/3 = 100%,

Then milk → bread

If Confidence (A → B) >= min_conf, Then A → B

60

How DM improve your business?

Strategy 1: Placing milk

and bread within close

proximity may further

encourage the sale of

these items together

within single visits to

the store.

61

How DM improve your business?

Strategy 2: Placing milk and bread at opposite ends of the store may entice customers who purchase such items to pick up other items along the way.

62

How DM improve your business?

Strategy 3:Put these two

items into a package

at reduced price.

63

Stage in the evolution of knowledge discovery

Proactive , integrative ;

multiple business partners

Neural computing

advanced al models,

complex optimization,

web services

What is the best plan to

follow? how did we

perform compared to

metrics?

Advanced intelligent

systems; complete

integration(2000-2004)

Prospective , proactive

information delivery

Advanced algorithms,

multiprocessor computers,

massive databases

What’s likely to happen to

the tBoston unit’s sales

next month ? Why?

Intelligent data mining

(late 1990s)

Retrospective , proactive

data delivery at multiple

level

OLAP, multidimensional

databases, data

warehouses

What were the sales in

region A by product , by

salesperson?

Data warehousing and

decision support (early

1990s)

Retrospective , dynamic

data delivery at record

level

Relational databases

(RDBMS), structured

query language (SQL)

What were unit sales in

new England last March ?Data access (1980s)

Retrospective , static data

delivery

Computers ,tapes , disks What was my total

revenue in the last 5

years?

Data collection(1980s)

Business question enabling technologies characteristic Evolutionary stage