dw architecture€¦ · about me senior bi consultant at insight mcitp – bi developer 2008 blog...
TRANSCRIPT
DW Architecture
More than just Facts & Dimensions
Martin Schoombee
About Me (NOT)
I have NOT presented at a SQL Saturday
I have NOT published any books
I have DO NOT have a Ferrari
I DO NOT have a Boston accent
4/29/2012 | DW Architecture: More than just Facts & Dimensions 2 |
About Me
Senior BI Consultant at Insight
MCITP – BI Developer 2008
Blog – www.did-i-say-that.com
Twitter – @sqlmartin
SpeakerRate – Martin Schoombee
4/29/2012 | DW Architecture: More than just Facts & Dimensions 3 |
The Facts of Life
4/29/2012 | DW Architecture: More than just Facts & Dimensions 4 |
The Facts of BI
Requirements will be vague (at best)
Change is inevitable
Growth will be exponential
4/29/2012 | 5 | DW Architecture: More than just Facts & Dimensions
The Key to Success
Do the things that nobody has asked you to
do…
4/29/2012 | DW Architecture: More than just Facts & Dimensions 6 |
Agenda
4/29/2012 | DW Architecture: More than just Facts & Dimensions 7 |
Approach
ETL
Database
Config
SQL Server Configuration
4/29/2012 | DW Architecture: More than just Facts & Dimensions 8 |
Approach
ETL
Database
Config
SQL Server Configuration
File Autogrowth Settings
MB, NOT %
Fail-safe Only
Log Files
Separate Drives
RAID 1+0 Ideal
4/29/2012 | DW Architecture: More than just Facts & Dimensions 9 |
SQL Server Configuration
Tempdb
¼ to ½ of Processor Cores (Paul Randal)
Autogrowth: MB, NOT %
MAXDOP
Less than 8
Never More than Physical Processors
Monitor CX-Packet Wait-Types
Test with Typical Workload
4/29/2012 | DW Architecture: More than just Facts & Dimensions 10 |
Database Areas
4/29/2012 | DW Architecture: More than just Facts & Dimensions 11 |
Approach
ETL
Database
Config
Database Areas
Admin
Staging
Reference
Data Mart
4/29/2012 | DW Architecture: More than just Facts & Dimensions 12 |
Admin Area
Separate Database? Yes
Why?
Flexibility; Backup Strategy; Recovery
Contents
Metadata
Logs
ETL Parameters
4/29/2012 | DW Architecture: More than just Facts & Dimensions 13 |
Staging Area
Separate Database? Yes
Why?
Flexibility
No Data Backups Required
4/29/2012 | DW Architecture: More than just Facts & Dimensions 14 |
Reference Area
“Sandbox” Area for Advanced Users
Separate Database? Yes
Why?
Flexibility; Backup Strategy; Recovery
Contents:
Maintained Tables
Generic Lookup Table
Tally Table
4/29/2012 | DW Architecture: More than just Facts & Dimensions 15 |
Data Mart Area
Separate Database? Yes
Contents
Facts
Dimensions
Snapshots
Helper/Bridge Tables
Use Schema’s for Logical Separation
4/29/2012 | DW Architecture: More than just Facts & Dimensions 16 |
ETL Process
4/29/2012 | DW Architecture: More than just Facts & Dimensions 17 |
Approach
ETL
Database
Config
ETL Process
“Engine Room” of Data Warehouse
Spend 80% of Your Time on ETL
Extract Data Quickly
Parameter-Driven
Sliding Window Approach
Automate Fully
4/29/2012 | DW Architecture: More than just Facts & Dimensions 18 |
ETL Process: SSIS
Choose Configuration/Logging Options Early
Create Templates
Use Queries Against Data Sources
Oracle/Teradata: Attunity Connectors
SCD Component: DO NOT USE IT
Experiment with Buffer Sizes
Parallel Processing
4/29/2012 | DW Architecture: More than just Facts & Dimensions 19 |
ETL Process: Staging Tables
Utilize Schema’s
Drop & Recreate Indexes
Mirror Source System Names
Map Source Objects 1:1
Varchar for All Source Fields
The Use of Identity Fields
Persist Lookup Values
Experiment with Computed Columns
4/29/2012 | DW Architecture: More than just Facts & Dimensions 20 |
ETL Process: Considerations
Stored Procs vs. SSIS Components
Consistency
Naming Conventions
Framework
Operational Data Store (ODS)
4/29/2012 | DW Architecture: More than just Facts & Dimensions 21 |
Approach
4/29/2012 | DW Architecture: More than just Facts & Dimensions 22 |
Approach
ETL
Database
Config
Data Mart Objects
Date Dimension
Surrogate Key: <yyyymmdd>
Comprehensive
Dimensions
Numeric vs. Varchar for Codes etc.
Type 1,2 for the Same Subject
Facts
Measures: Compatible Data Types
4/29/2012 | DW Architecture: More than just Facts & Dimensions 23 |
Reporting Layer
Cubes
Qualify Necessity
Calculated Measures
Reporting Services
Create Templates
Admin Reports
Use Stored Procedures
Excel
4/29/2012 | DW Architecture: More than just Facts & Dimensions 24 |
Measure of Success
Automation
Manual Interventions
Failures
Performance
Reports/Output
Data Manipulation
Intuitive Interface
End-User Adoption
4/29/2012 | DW Architecture: More than just Facts & Dimensions 25 |
The Key to Success
Do the things that nobody has asked you to
do…
Anticipate, Prepare, Deliver, Repeat
4/29/2012 | DW Architecture: More than just Facts & Dimensions 26 |
Martin Schoombee
Senior BI Consultant at Insight
MCITP – BI Developer 2008
Blog – www.did-i-say-that.com
Twitter – @sqlmartin
SpeakerRate – Martin Schoombee
4/29/2012 | DW Architecture: More than just Facts & Dimensions 27 |