Data Warehouse Fundamentals

Rabie A. Ramadan, PhD

2

2

What did you do in Your Assignment ?

For an airlines company, how can strategic information increase the number of frequent flyers? Discuss giving specific details.

You are a Senior Analyst in the IT department of a company manufacturing automobile parts. The marketing heads are complaining about the poor response by IT in providing strategic information. Draft a proposal to them explaining the reasons for the problems and why a data warehouse would be the only viable solution.

3

What did you do in the Project ? Egypt Election System

• Governorates’ database system • Multiple databases on Multiple Servers

• Summarization System • Meta data

• Data Warehouse Server

• Web page with query based system

4http://www.inf.unibz.it/dis/teaching/DWDM/index.html

5

Definitions & Motivations

Why Data Mining? Explosive Growth of Data: from terabytes to petabytes Data Collections and Data Availability

• Crawlers, database systems, Web, etc.

Sources• Business: Web, e-commerce, transactions, etc.

• Science: Remote sensing, bioinformatics, etc.

• Society and everyone: news, YouTube, etc.

6

Why Data Mining?

Problem: We are drowning in data, but starving for knowledge!

Solution: Use Data Mining tools for Automated Analysis of massive data sets

7

What is Data Mining?

Data mining (knowledge discovery from data)• Extraction of interesting (non-trivial, implicit,

previously unknown and potentially useful) patterns or knowledge from huge amount of data

8

What is Data Mining?

Alternative names• Knowledge discovery (mining) in databases (KDD),

• knowledge extraction,

• data/pattern analysis,

• data archeology,

• Data dredging,

• information harvesting,

• business intelligence,

• etc.

9

Knowledge Discovery (KDD) Process

10

Knowledge Discovery (KDD) Process

11

Typical Architecture of a Data Mining System

12

Confluence of Multiple Disciplines

13

Why Confluence of Multiple Disciplines?

Tremendous amount of data• Scalable algorithms to handle terabytes of data (e.g., Flickr

had 5 billion images in September, 2010 [http://blog.flickr.net/en/2010/09/19/5000000000/])

High dimensionality of data• Data can have tens of thousands of features (e,g., DNA

microarray)

14

Why Confluence of Multiple Disciplines?

15

Different Views of Data Mining Data View

• Kinds of data to be mined Knowledge view

• Kinds of knowledge to be discovered Method view

• Kinds of techniques utilized Application view

• Kinds of applications

16

Data to Mined

In principle, data mining should be applicable to any data repository

We will have examples about:• Relational databases• Data warehouses• Transactional databases• Advanced database systems

17

Relational Databases

18

Data Warehouses

19

Transactional Databases

20

Advanced Database Systems(1)

21

Advanced Database Systems(2)

22

Knowledge to be Discovered

23

Characterization and Discrimination

24

Characterization and Discrimination (1)

25

Class Activity

• Differentiate between Data Mining and Data warehousing?

Data warehousing is merely extracting data from different sources, cleaning the data and storing it in the warehouse. Where as data mining aims to examine or explore the data using queries

What are the Different problems that “Data mining” can solve? Data mining can be used in a variety of fields/industries like marketing,

advertising of goods, products, services, AI, government intelligence. How does the data mining and data warehousing work

together? Data warehousing can be used for analyzing the business needs by storing

data in a meaningful form. Using Data mining, one can forecast the business needs. Data warehouse can act as a source of this forecasting. 

26

Frequent Patterns, Associations, Correlations

27

Classification and Prediction

28

Cluster Analysis

29

Outlier Analysis

30

Evolution Analysis

31

Techniques Utilized

32

Applications Adapted

33

Major Challenges in Data Mining

34

Summary