ims 6217: data warehousing / business intelligence 1 dr. lawrence west, management dept., university...

IMS 6217: Data Warehousing / Business Intelligence

1Dr. Lawrence West, Management Dept., University of Central [email protected]

Database Performance Part 1—Topics

• Doing vs. Deciding—OLTP vs. OLAP

• Data Warehouses

– Fact tables, Dimension tables, Granularity

– DW in an integrated Business Intelligence system

• Design Steps

• Designing Fact Tables

• Designing Dimension Tables

– The Time dimension

• Fact Table Exercises

• The AdventureWorks DW



"With uncertainty present…"

With the introduction of uncertainty—the fact of ignorance and necessity of acting upon opinion rather than knowledge—into this Eden-like situation, its character is completely changed. With uncertainty absent, man's energies are devoted altogether to doing things; it is doubtful whether intelligence itself would exist in such a situation; in a world so built that perfect knowledge was theoretically possible, it seems likely that all organic readjustments would become mechanical, all organisms automata. With uncertainty present, doing things, the actual execution of activity, becomes in a real sense a secondary part of life; the primary problem or function is deciding what to do and how to do it. The two most important characteristics of social organization brought about by the fact of uncertainty have already been noticed. In the first place, goods are produced for a market, on the basis of an entirely impersonal prediction of wants, not for the satisfaction of the wants of the producers themselves. The producer takes the responsibility of forecasting the consumers' wants. In the second place, the work of forecasting and at the same time a large part of the technological direction and control of production are still further concentrated upon a very narrow class of the producers, and we meet with a new economic functionary, the entrepreneur. Frank H. Knight

University of Chicago 1921



Doing vs. Deciding

• Organizations do many things

– List thirty transactions that your project organization executes or does

– Start with the Top-Ten list from Projects 2 & 3

• Managers decide things

– List thirty decisions that your project organization makes

– Identify where in the organizational hierarchy the decision lies

– What is the consequence/importance of the decision?

– What information influences each decision?



Doing vs. Deciding / OLTP vs OLAP

• Are systems designed to support the execution of events suitable for the making of decisions?

• Event/transaction support requires – High throughput– High reliability– Accuracy– DB structures tuned for storage & performance

• Online Transaction Processing (OLTP) systems support events– Provide data or information to support transactions– Record acts → New data



OLTP vs. OLAP—Let Me Count the Ways…

• Online Analytical Processing (OLAP) or Business Intelligence (BI) systems are oriented at decision making and analysis

• What are the problems with using our OLTP databases to support managerial decision making?

?



The Data Warehouse

• The DW is a separate storage structure

• Designed to optimize query execution

– Not storage efficiency

– Not transaction throughput

• Expected to be loaded during down times

• Supports "readability"

• May sacrifice details for summaries

• Data and structures anticipate user needs

– Recurring decisions

– Flexible exploration



Steps and Components

• Source Systems—provide raw data to the DW

• Integration Services—Provide transformation and loading services from source data to DW

• Data Warehouse—Customized data store for Business Intelligence

• Analysis Services—Tools for data mining and reporting

• Reporting Services—Our old friend acting on an enhanced data store



Our Approach

• This Week

– Discuss DW storage strategies

– Discuss data to be stored

• Internal data from OLTP systems

• External data

– Design exercises

• Next Week

– DW loading strategies

– DW tools—Analysis Services



Storage Strategies

• The DW stores transformed data that

– May be accessed directly to support analysis

– Supports actions of the Analysis Services to provide enhanced and efficient analysis

• Multiple Strategies

• We will look at the widely used approach using

– Fact tables,

– Dimension tables,

– Arranged in a Star Schema or Snowflake Schema (or both)



Fact Tables Contain Facts (duhhhh) of Interest

• No PK designated for fact table

• Natural PK is TimeKeyOrdered, ProductKey, CustomerKey

– This defines the granularity of the data

• CategoryKey FD on ProductKey

• SalesTerrKey, SalesRepKey FD on CustomerKey

• UnitsSold, TotalDiscounts

– Summed from source data

– Additive

• SalesPrice is not additive

• ValueSold is derivable and additive

SALES

TimeKeyOrderedTimeKeyShippedTimeKeyPmntRcvdProductKeyCategoryKeyCustomerKeySalesTerrKeySalesRepKeyUnitsSoldSalesPriceValueSoldTotalDiscounts



Star Schema & Dimension Tables

• Dimension Tables represent concepts (entities) used to group data in the fact tables

• Also contain descriptive attributes of the entity represented by the dimension table

• Simplest way for nontechnical users to picture the data

• Relate to FKs in the fact tables

SALES


DimDate

TimeKey

DimCustomer

CustomerKey

DimProduct

ProductKey

DimCategory

CategoryKey

DimSalesRep

SalesRepKey

DimSalesTerr

SalesTerrKey



Snowflake Schema & Dimension Tables

• Fewer direct links from dimension tables to fact table

• Dimension tables relate to each other

• Natural hierarchical relationships in data are preserved

– Implications for drilldown reports

• Increases complexity of data retrieval for nontechnical users

DimDate

TimeKey

DimCustomer

CustomerKey

DimCustomer

ProductKey

DimCategory

CategoryKey

DimSalesRep

SalesRepKey

DimGeography

GeographyKey

SALES

TimeKeyOrderedTimeKeyShippedTimeKeyPmntRcvdProductKeyCustomerKeySalesRepKeyUnitsSoldSalesPriceValueSoldTotalDiscounts

DimSalesTerr

SalesTerrKey



Granularity

• The granularity of the fact tables is a critical

• There are alternative levels of granularity

– Finer granularity → more detail, more recordsUse SalesDate instead of Month

– Coaser granularity → less detail, fewer recordsUse SalesMonth instead of SalesDate

• Finer granularity can be aggregated in the DW to find the coarser granularity values

• Coarse granularity cannot be decomposed

• Granularity decisions are made for each of the FKs from the dimension tables

SALES

SalesRecIDFDOWeekProductIDCategoryIDCustomerIDSalesTerrIDSalesRepIDUnitsSoldSalesPriceValueSoldTotalDiscounts



Design Steps

• It is impractical to design a one-source DW as the first deliverable

• Identify initial scope of DW

– Problem Statement

– Business Requirements

• Build DW Data Model

– Business Processes to address requirements

– Level of Detail

– Fact Tables (what we are measuring)

– Dimension Tables (how we look at the data)



Design Steps (cont.)

• Design Integration Services

• Design Analysis Services

• Design Reports

• Deploy and Manage DW

• Add additional business requirements

– Repeat process for new requirements

– Add additional dimensions to the DW



Fact Tables (Part 2)

• Identifying Fact Tables and their facts is an art

• No obvious mapping from OLTP tables to Fact or Dimension Tables

• The same DB table can contribute to multiple fact tables

• Requires analysis to discover central concepts that will become fact tables

– Decision maker interviews

– Reporting requirements

SALES




Fact Tables (Part 2—cont.)

• Look for a logical concept or event which measures of interest are about

– A sale (invoice)

– An order (purchase order)

– An enrollment (college DB)

• The concept/event should support the requirements

• The event is likely to be based on an OLTP table

– Not every OLTP table will become a fact table

• This concept/event will form the foundation for a fact table

SALES




Fact Tables--Measures

• Measures are the facts to be recorded for each row in the fact table

• Measures are often additive

– UnitsSold, TotalDiscounts, ValueSold

• Some are not additive

– SalesPrice

• Sometimes nonadditive measures are transformed into additive measures

– ValueSold = (UnitsSold * SalesPrice) - TotalDiscounts

SALES




Fact Tables—Measures (cont.)

• Measures may come from several sources—often not just values from a single OLTP source table

• Other candidates in our example

– COGS

– CurrentInterestRate – CompetitorPrice

– GrossMargin – NetMargin

– ShippingCost – ShippingWeight

SALES




Fact Tables--Dimensions

• Dimensions are ways of looking at the data

– Users may indicate they look at {fact table subject}"by" {dimension name}

– Sales by week

– Sales by customer

– Sales by product category

• Dimensions lead us to Dimension Tables

– Descriptive attributes about the dimension

– Foreign key to the fact table



Dimension Tables

• Dimension tables are often basedon an OLTP entity

• Denormalized to include descriptiveattributes from other tables

– Product might include

• SupplierName • CategoryName

• SubCategoryName • SupplierCountry

• In Snowflake dimension tables related hierarchical information may be retained in the hierarchical tables



Dimension Tables—Primary Keys

• Dimension tables should alwaysbe given an artificial identity PK—even if there is a suitable OLTP table PK

• If tables are ever loaded from multiple sources the natural PK may become invalid

– E.g., merging sales data from two business units with different databases

• Retain the business PK as an attribute in the dimension table

• Possibly include source system identifier for the row

DimCustomer

CustomerKeyCustomerIDSourceSystemLastNameFirstName :



Dimension Tables—Time

• Time is a hugely common"by" dimension

• Decide on time granularity

– Daily, Weekly, Hourly?

• You might consider two timedimensions

– Daily for grossest categorization

– Hour for additional precision

DimDate

TimeKeyCalendarYearCalendarQuarterCalendarMonthNameCalendarNumberOfMonthDayNumberOfMonthDayNumberOfYearDayNumberOfWeekFiscalYearFiscalQuarterManufacturingYearManufacturingQuarterManufacturingMonthSeasonNameHolidayFlagThanksgivingWeekendFlagPreChristmasFlag

DimTimeOfDay

HourNumTimeOfDayName



Dimensions—Time (cont.)

• The time dimension tablemaps from the measured timeattribute associated with thefact table record to variouslabels and aggregationsassociated with that value

• Facilitates summarizing byvarious aggregates with asingle time dimension measure

• TimeKey PK is often a datetime data type to the date level of precision

DimDate

TimeKeyCalendarYearCalendarQuarterCalendarMonthNameCalendarNumberOfMonthDayNumberOfMonthDayNumberOfYearDayNumberOfWeekFiscalYearFiscalQuarterManufacturingYearManufacturingQuarterManufacturingMonthSeasonNameHolidayFlagThanksgivingWeekendFlagPreChristmasFlag



Fact Tables--Granularity

• In the olden days granularity decisions were made at the DW DB design stage

• Granularity decisions traded off

– Number of records and computational overhead associated with more detailed granularity

– Lack of precision with coarser granularity

• Modern computational power supports finer granularity

• Analysis services provides support for fast computation over large data sets

• Just don't go overboard



Fact Table Exercise #1

• Are there any fact tables beyond the one illustrated on Slide 11 for the NorthWind DB?

• Are there additional facts that you might add to this table?

• Are there additional dimension tables you might add?



Fact Table Exercise #2

• Expand entities around the core of our University ERD

– See next slide

• Consider two business goals

– Understand real credit hour revenue

– Understand classroom utilization

• Identify and design Fact and Dimension Tables

DeptCodeCourseNoNameCreditHrsLabHrs

COURSE

SectionIDDeptCode <AK>CourseNo <AK>SecNo <AK>Term <AK>Year <AK>Room <FK1>DaysTimeInstructorID <FK2>

SECTION

SectionID <FK1>StudentID <FK2>Grade <FK3>

ENROLLMENT

StudentIDLastNameFirstName :

STUDENT

HasHas

Has

GradeGradePts :

GRADE

Has



Fact Table Exercise #2 (Cont.)

Student

StudentIDLname :Resident Y/N

Enrollment

SectionIDStudentIDGrade

Payment

PaymentIDStudentIDPaymentDatePaymentAmt

Section

SectionIDDeptCodeCourseNumSecNumTermYearDaysTimeInstructorIDCapacityBldgIDRoomNum

Instructor

EmployeeIDLastNameFirstNameDeptIDCurrentSalary

SalaryHistory

EmployeeIDStartDateEndDateSalaryStatus

Course

DeptCodeCourseNumCrHrTitleDescriptionLecHrLabHrDeptID

Department

DeprtIDNameChairIDDeptIDOfficeBldgOfficeRm

Textbook

ISBNTitleYear :WholsalePriceListPrice

SectionBook

ISBNSectionIDQtyOrderedQtySold

Room

BldgIDRoomNumTypeCapacity

Building

BldgIDNameFloorsSqFt

FeeSchedule

AcadYearInStateUGCrHrOutStateUGCrHrInStateGrCrHrOutStateGrCrHrInStateDCrHrOutStateDCrHrHealthFeeActivityFee

BrightFutureStat

StudentIDAcadYearPercentCovered



External Data

• What external data might you want to have in a sales-oriented DW?



Next Time

• Transformations to load the DW from the source OLTP (and other) data sources

– Automated support

– Do it yourself

• Analysis Services—putting our DW to work

ims 6217: data warehousing / business intelligence 1 dr. lawrence west, management dept., university...

Documents

lawrence west

university of central

fact of uncertainty

dimension tables

uncertainty present

time dimension fact

adventureworks dw slide

knight university of