next gen bi and datawarehouse solutions ross lo forte
Post on 15-Jun-2015
363 Views
Preview:
TRANSCRIPT
Next Generation BI and Data Warehouse Solutions
Ross LoForteSQL Technology ArchitectMicrosoft Technology Centers
Data Quality
Real-Time DW and Streaming Data
Advanced Analytics
MPP
MDM
Secure and Robust
Key TrendsMPP
(Parallel Data
Warehouse)
Master Data
Services
Database Security
StreamInsight
(Streaming Data)
Data Quality (Zoomix)
Data Warehouse Industry TrendsMicrosoft has steadily invested in the most important data warehouse
technologies
Column Store
Column Store
(Project Apollo)
Microsoft’s on-going investments in Data Warehousing
Heterogeneous Connectivity & Workloads
Data Integrity & Quality
Compliance & Security
Data Warehouse Scale
Data Warehouse Management
2005 2008 Futures
PB Warehouses>64 Core ProcessingScale out through MPP
Perf. Management ToolsBI Resource GovernanceImproved Predictability
Mixed workload supportContinuous Loading
Master Data Management(Stratature Integration)Integrated DQ Services (Zoomix)
Rights Management
10s of TB WarehousesParallel partitioningData compressionNew Reference
Architectures
Policy Based Admin.DB Resource
Governance
High Perf. Connectors(Oracle, Teradata, SAP BW)
Data Profiling
Policy based auditing
Multi TB WarehousesEnterprise scalabilityDW Reference
Architectures
Unified manageability
Enterprise class ETL tool
Data Cleansing(Fuzzy lookup/matching)
Data Protection & Tracing
SQL Server Top Achievements
Category MetricLargest single database 70 TBLargest table 20 TB
Biggest total data 1 application
88 PB
Highest database transactions per second 1 db (from Perfmon)
130,000
Fastest I/O subsystem in production (SQLIO 64k buffer)
18 GB/sec
Fastest “real time” cube 5 sec latency
Data load for 1TB 20 minutesLargest single cube 12 TB
Microsoft Data Warehousing solutions
Tier 1 offerings
Tier 1 Services and Support
Scalable and reliable platform for Data
Warehousing on any hardware
Reference Architectures offering best price
performance for data warehousing
Scalable and reliable platform for Data
Warehousing on any hardware
Appliance for high end Data Warehousing requiring highest
scalability, performance or complexity
Ideal for data marts or small to mid-sized EDWs
Ideal for data marts or small to mid-sized DWs
with scan centric workloads
Ideal for large data marts or mid-sized EDWs
Offers flexibility in hardware and architecture
Software only Reference Architectures (Software and Hardware)
Software onlyDW Appliance
(Fully integrated Software and Hardware)
Scale-Up DW Scale-Up DW Scale-Up DW Scale-Out DW with MPP
10s of TB 4 – 80 TB 10s of TB 10s - 100s of TB
$28.8K/Proc$9.9K/Svr + $162/CAL
$107K - $683K (2 – 8 Procs; includes
Hardware)$57.5K/Proc only $38.3K/Proc
Microsoft Data Warehousing solutions
Integrated ETL and Reporting toolsSimplified managementPredictable responseLower storage costsIntegrated Master Data Management tool
Tier 1 offerings
Scalable and reliable platform for Data
Warehousing on any hardware
Ideal for data marts or small to mid-sized EDWs
Software only
Scale-Up DW
10s of TB
$28.8K/Proc$9.9K/Svr + $162/CAL
Microsoft Data Warehousing solutions
All features and benefits of SQL Server 2008 R2 Enterprise Ability to scale up to 256 logical processorsAbility to scale memory beyond 2TBContinuous loading using StreamInsight
Tier 1 offerings
Scalable and reliable platform for Data
Warehousing on any hardware
Ideal for large data marts or mid-sized EDWs
Software only
Scale-Up DW
10s of TB
$57.5K/Proc only
Microsoft Data Warehousing solutions
Balanced solution for scan-centric workloadsBest price-to-performance ratioFeatures 12 reference architectures validated by MicrosoftAbility to scale up to 80 terabytes
Tier 1 offerings
Reference Architectures offering best price
performance for data warehousing
Ideal for data marts or small to mid-sized DWs
with scan centric workloads
Reference Architectures (Software and Hardware)
Scale-Up DW
4 – 80 TB
$107K - $683K (2 – 8 Procs; includes
Hardware)
Some Data Warehouses Today
Big SANBig SMP ServerConnected together
• Server can consume 32 GB/Sec of IO, but SAN can only deliver 12 GB/Sec
• Queries are slow− Despite significant investment in both Server and Storage
010101010101010101011101101011101010110101010101010101010110111101010101101101011010110101
010101010101010101011101101011101010110101010101010101010110111101010101101101011010110101
010101010101010101011101101011101010110101010101010101010110111101010101101101011010110101
010101010101010101011101101011101010110101010101010101010110111101010101101101011010110101
010101010101010101011101101011101010110101010101010101010110111101010101101101011010110101
010101010101010101011101101011101010110101010101010101010110111101010101101101011010110101
010101010101010101011101101011101010110101010101010101010110111101010101101101011010110101
010101010101010101011101101011101010110101010101010101010110111101010101101101011010110101
010101010101010101011101101011101010110101010101010101010110111101010101101101011010110101
010101010101010101011101101011101010110101010101010101010110111101010101101101011010110101
010101010101010101011101101011101010110101010101010101010110111101010101101101011010110101
010101010101010101011101101011101010110101010101010101010110111101010101101101011010110101
010101010101010101011101101011101010110101010101010101010110111101010101101101011010110101
010101010101010101011101101011101010110101010101010101010110111101010101101101011010110101
010101010101010101011101101011101010110101010101010101010110111101010101101101011010110101
010101010101010101011101101011101010110101010101010101010110111101010101101101011010110101
010101010101010101011101101011101010110101010101010101010110111101010101101101011010110101
010101010101010101011101101011101010110101010101010101010110111101010101101101011010110101
Challenges of traditional Data Warehouse
CPU
IO Channe
l
CPU Constraint
Sequential IO
capacity of
storage System
010101010101010101011101101011101010110101010101010101010110111101010101101101011010110101
010101010101010101011101101011101010110101010101010101010110111101010101101101011010110101
010101010101010101011101101011101010110101010101010101010110111101010101101101011010110101
010101010101010101011101101011101010110101010101010101010110111101010101101101011010110101
010101010101010101011101101011101010110101010101010101010110111101010101101101011010110101
010101010101010101011101101011101010110101010101010101010110111101010101101101011010110101
010101010101010101011101101011101010110101010101010101010110111101010101101101011010110101
010101010101010101011101101011101010110101010101010101010110111101010101101101011010110101
010101010101010101011101101011101010110101010101010101010110111101010101101101011010110101
010101010101010101011101101011101010110101010101010101010110111101010101101101011010110101
010101010101010101011101101011101010110101010101010101010110111101010101101101011010110101
010101010101010101011101101011101010110101010101010101010110111101010101101101011010110101
010101010101010101011101101011101010110101010101010101010110111101010101101101011010110101
010101010101010101011101101011101010110101010101010101010110111101010101101101011010110101
010101010101010101011101101011101010110101010101010101010110111101010101101101011010110101
010101010101010101011101101011101010110101010101010101010110111101010101101101011010110101
010101010101010101011101101011101010110101010101010101010110111101010101101101011010110101
010101010101010101011101101011101010110101010101010101010110111101010101101101011010110101
Sequential IO
capacity of
storage System
CPU
Storage System Constraint
IO Channe
l
010101010101010101011101101011101010110101010101010101010110111101010101101101011010110101
010101010101010101011101101011101010110101010101010101010110111101010101101101011010110101
010101010101010101011101101011101010110101010101010101010110111101010101101101011010110101
010101010101010101011101101011101010110101010101010101010110111101010101101101011010110101
010101010101010101011101101011101010110101010101010101010110111101010101101101011010110101
010101010101010101011101101011101010110101010101010101010110111101010101101101011010110101
010101010101010101011101101011101010110101010101010101010110111101010101101101011010110101
010101010101010101011101101011101010110101010101010101010110111101010101101101011010110101
010101010101010101011101101011101010110101010101010101010110111101010101101101011010110101
010101010101010101011101101011101010110101010101010101010110111101010101101101011010110101
010101010101010101011101101011101010110101010101010101010110111101010101101101011010110101
010101010101010101011101101011101010110101010101010101010110111101010101101101011010110101
010101010101010101011101101011101010110101010101010101010110111101010101101101011010110101
010101010101010101011101101011101010110101010101010101010110111101010101101101011010110101
010101010101010101011101101011101010110101010101010101010110111101010101101101011010110101
010101010101010101011101101011101010110101010101010101010110111101010101101101011010110101
010101010101010101011101101011101010110101010101010101010110111101010101101101011010110101
010101010101010101011101101011101010110101010101010101010110111101010101101101011010110101
IO Chann
el
Sequential IO
capacity of storage
System
IO Channel constraint
CPU
What is Fast Track Data Warehouse?• A method for designing a cost-effective, balanced
system for Data Warehouse workloads • Reference hardware configurations developed in
conjunction with hardware partners using this method
• Best practices for data layout, loading and management
Relational Database Only – Not SSAS, IS, RS
Fast Track SQL DW Architecture vs. Traditional DW
SQL 2008 Data Warehouse4 Processor 16 Core Server
Shared Network Bandwidth
Enterprise Shared SAN Storage
Dedicated Network Bandwidth
Traditional SQL DWArchitectureShared Infrastructure
Fast Track SQL DW ArchitectureDedicated DW InfrastructureArchitecture modeled after DW Appliances 1TB – 80TB Pre-Tested
Dedicated Low Cost SAN Arrays 1 for every 4 CPU Cores HP MSA2312
OLTP Applications
Benefits:-More System Predictability Thus User Experience-Pretested Configurations Lowers TCO-Balanced CPU to I/O Channel Optimized for DW-Modular Building Block Approach-Scale Out or Up within limits of Server and SAN
Reference architectures boost performance and reduce risk
HP Fast Track data warehouse configurations scale from SMB to Enterprise
• Prescriptive guidance and optimized methodology for data warehouse query workloads with large sequential data reads
• Balanced hardware approach ideal for data marts or small to mid-sized DW with scan-centric workloads
• Supports 1 to 80TB Data Warehouse at leading price/performance
• Configurations, tested performance guidance and best practices for deploying/operating/managing
• Packaged and custom support
Basic6 – 12TBDL38x w/
MSA P2000
Mainstream12 – 24TBDL585 w/
MSA P2000
Mainstream16 – 32 TB DL580 w/
MSA P2000
Premium24 – 80 TBDL980 w/
MSA P2000
Entry1-5TBDL370
w/D2700 DAS
DemoFast Track Data Warehouse
Microsoft Data Warehousing solutions
Enterprise Data Warehouse Appliance offeringHigh Scalability and performanceFlexibility and choiceIntegrated with Microsoft BI
Tier 1 offerings
Appliance for high end Data Warehousing requiring highest
scalability, performance or complexity
Offers flexibility in hardware and architecture
DW Appliance(Fully integrated
Software and Hardware)
Scale-Out DW with MPP
10s - 100s of TB
$38.3K/Proc
• All hardware from a single vendor• Orderable at the rack level• Vendor will:
− Assemble appliances− Image appliances with OS, SQL
Server, and PDW software• Appliance installed in 1 – 2 days• Support:
− Microsoft provides first call support− Hardware partner provides onsite
break/fix support
Parallel Data WarehouseAn appliance experience
Control Rack Data Rack
Compute Nodes Storage Nodes
Spare Compute
Node
Du
al
Fib
er
Ch
an
nel
SQL
SQL
SQL
SQL
SQL
SQL
SQL
SQLDu
al
Infi
nib
an
d
Control Nodes
Active /
Passive
Landing Zone
Backup node
SQL
Management Node
SQL
SQL
DemoParallel Data Warehouse
Admin Console – Home Page
• Menu options listed left to right by PDW activity and status.
Admin Console – Appliance State
• Appliance State tab lists the state of all active nodes within the appliance.
Admin Console – Dashboard Customizations• Can optionally include up to 38 available
performance counters.
…
Admin Console – Dashboard
• The Dashboard tab provides near real-time performance counters.
23
Distributed Data Warehouse Architecture
• Each business unit has own Data Marts− More responsive to business needs− Fits budget realities
• Hub provides centralized data governance etc.• Node-to-node data movement
− Parallel over Infiniband− >500GB per min− Parallel Database Export (PDE)
Delivered through a Familiar Interface• Self-Service access
& insight• Data exploration
& analysis• Predictive analysis• Data visualization• Contextual
visualization
The Microsoft BI Solution Stack
BUSINESS COLLABORATION PLATFORM
DATA INFRASTRUCTURE & BUSINESS INTELLIGENCE PLATFORM
BUSINESS USER EXPERIENCE
Business Productivity Infrastructure• Dashboards &
Scorecards• Excel Services• Web based forms
& workflow• Collaboration• Search• Content
Management• LOB data integration• PowerPivot for
SharePoint
The Microsoft BI Solution Stack
BUSINESS COLLABORATION PLATFORM
DATA INFRASTRUCTURE & BUSINESS INTELLIGENCE PLATFORM
BUSINESS USER EXPERIENCE
Data Infrastructure & BI Platform• Analysis Services• Reporting Services• Master Data Services• Integration Services• Data Mining• Data Warehousing
BUSINESS COLLABORATION PLATFORM
DATA INFRASTRUCTURE & BUSINESS INTELLIGENCE PLATFORM
BUSINESS USER EXPERIENCE
The Microsoft BI Solution Stack
Use Reports to Drive Decisions
• Create and share reports• Maintain a single version of truth with your Excel
Workbooks• Drive decision based on facts
Use Dashboards to Drive Decisions
• Visual displays of information needed to achieve one or more objectives
• Single-Screen display of information
• Answer fundamental questions
• Alerts the user to issues or problems
• Span Operational, Performance, Personal
• Align strategies and organizational goals
• Measure and manage Key Performance Indicators (KPI)
• Modeled after the business, not the data
PowerPivot for Excel PowerPivot for SharePoint
Use PowerPivot to Drive Self-Services
29
Microsoft Business Decision Appliance
• Rich insight: Empower users to easily create PowerPivot workbooks from real-time business data for faster, more accurate insights
• Reduced complexity: Overcome cost and complexity of BI; shift IT resources from running ad-hoc reports to innovation initiatives
• Easy manageability: Custom code for management dashboard and scripted data source integration ease deployment and simplify administration
SKUs Components
BDA Server Dual Intel X5650 Processor with 96GB (1U)
Storage 8 x internal 300 GB SAS disks
Software Windows Server 2008 R2 EE, SQL Server 2008 R2 EE, SharePoint 2010 EE, PowerPivot
Infrastructure None (install in existing rack)
Services Software technical support
End-to-end, pre-configured stack quickly enables BI for Excel power users
Complete Data Warehouse Solution
Flexibility and Choice Massive Scalability at a Low Cost
Microsoft Data Warehouse VisionMake SQL Server the fastest and most affordable
database for customers of all sizes
Simplified Data Warehouse Management
© 2009 Microsoft Corporation. All rights reserved. Microsoft, Windows, Windows Vista and other product names are or may be registered trademarks and/or trademarks in the U.S. and/or other countries.The information herein is for informational purposes only and represents the current view of Microsoft Corporation as of the date of this presentation. Because Microsoft must respond to changing market conditions,
it should not be interpreted to be a commitment on the part of Microsoft, and Microsoft cannot guarantee the accuracy of any information provided after the date of this presentation. MICROSOFT MAKES NO WARRANTIES, EXPRESS, IMPLIED OR STATUTORY, AS TO THE INFORMATION IN THIS PRESENTATION.
top related