sqlcat: designing scalable and complex analysis services cubes
TRANSCRIPT
SQLCAT: Designing scalable and complex Analysis Service CubesLessons learned from the Field
Denny Lee, Thomas KejserMicrosoft SQL Customer Advisory Team
BIA499A • SQLCAT: Designing scalable and complex Analysis Service Cubes: Lessons learned from the Field
Are you sitting in front of the correct Denny? They look very much alike.
Designing scalable and complex Analysis Service Cubes: Lessons learned from the Field – Room 615-617
Getting SQL Service Broker Up and Running – Room 609
BIA499A • SQLCAT: Designing scalable and complex Analysis Service Cubes: Lessons learned from the Field
SQL Server Customer Advisory Team (SQLCAT)
• Works on the largest, most complex SQL projects worldwide• MySpace - 4.4 million concurrent users at peak time, 8 billion friend
relationships, 34 billion e-mails, 1 PetaByte store, scale-out using SSB and SOA http://www.microsoft.com/casestudies/Case_Study_Detail.aspx?casestudyid=4000004532
• Bwin – Most popular European online gaming site – 30000 database trans / second, motto: “Failure is not an option”; 100 TB total storage http://www.microsoft.com/casestudies/Case_Study_Detail.aspx?casestudyid=4000004138http://www.microsoft.com/casestudies/Case_Study_Detail.aspx?casestudyid=4000001470
• Korea Telecom - Largest telco in Korea serves 26 million customers; 3 TB Data Warehousehttp://www.microsoft.com/casestudies/Case_Study_Detail.aspx?casestudyid=4000001993
• Drives product requirements back into SQL Server from our customers and ISVs
• Shares deep technical content with SQL Server community• SQLCAT.com & http://blogs.msdn.com/mssqlisv
BIA499A • SQLCAT: Designing scalable and complex Analysis Service Cubes: Lessons learned from the Field
SQL Server Design Win Program• Target the most challenging and innovative SQL Server
applications
• 10+ TB DW, 3k/tran/s OLTP, Large 500GB+ Cubes, Competitive migrations, Complex deployments, Server Consolidation (1000+)
• Invest in large scale, referenceable SQL Server projects across the world
• Provide SQLCAT technical & project experience• Conduct architecture and design reviews covering performance,
operation, scalability and availability• Offer use of HW lab in Redmond with direct access to SQL Server
development team
• Work with Marketing Team Developing PR
BIA499A • SQLCAT: Designing scalable and complex Analysis Service Cubes: Lessons learned from the Field
Upgrade Reliability Assurance
Regular status updates within one month of completing each playback test cycle
Complimentary testing for qualified customer against every new version of SQL Server
Opportunity to request improvements directly from the product group for potential upgrade issues
Preparing for a smooth transition to the next version of SQL Server ...
Status Reports Customer Workload
Microsoft Playback Lab
Two-way NDA and Participation Agreement to ensure security and
privacy
For more information, contact [email protected]
BIA499A • SQLCAT: Designing scalable and complex Analysis Service Cubes: Lessons learned from the Field
SQLCAT and SQL CSS Invite You…
ROOM 611
• To the SQL Server Clinic where the most experienced SQL Server experts in the world will be waiting to talk with you.
• Bring your toughest Questions / Challenges to the experts who have seen it all• Architect and Design your future applications with experts who have done it before
with some of the largest, most complex systems in the world• Or just stop in to say hello!
Session Objectives and Takeaways
Session Objective(s): • Provide the primary themes that revolve around
successful enterprise Analysis Services implementations• Show the themes in relation to how SQLCAT customers
address these issues
Themes
• Get that hardware balance right (yes, we have to talk about IOPS)
• Repeat after me: partitioning, partitioning, partitioning!
• Multi-user query concurrency– how to handle it• Keeping it simple• Locking – how it works – and how to work around it• What? Did you say ROLAP?
SQLCAT Case Studies
Generalized Learnings from the Field
BIA499A • SQLCAT: Designing scalable and complex Analysis Service Cubes: Lessons learned from the Field
The Importance of I/O
• Enterprise = lots of data = lots of threads on the storage engine
• Disk Partition Alignment Best Practices• Scale Out Using Read-Only
• 64K vs. 32K block size• Scale Out Query performance
• Key Point:• Need to understand System Engineering concepts and
principles
BIA499A • SQLCAT: Designing scalable and complex Analysis Service Cubes: Lessons learned from the Field
The Importance of I/ODisk Partition Alignment
Disk Partition Alignment Best Practices
6 Disks - Not Aligned6 Disks - Aligned
8 Disks - Not Aligned8 Disks - Aligned
0
5
10
15
20
25
30
35
40
45
50
Latency (ms)
Duration (s)
31
21 24
16
48.03
38.7138.65
31.45
Latency (ms) Duration (s)
*Lower is better
BIA499A • SQLCAT: Designing scalable and complex Analysis Service Cubes: Lessons learned from the Field
The Importance of I/OSSAS I/O pattern
Typically Low block sizes on scans – 32KIOPS are wide and shallow (lots of threads, short queue)Latency is key, low latency = less context switchingTypical NAND device speedup: 5-10x on SE heavy queries
0 32 64 96 128 160 192 224 2560
50000
100000
150000
200000
250000
0
1
2
3
4
5
8K IOPS on FusionIO drive
IOPSLatency (ms)
Queue Depth (total outstanding kept constant at 256)
BIA499A • SQLCAT: Designing scalable and complex Analysis Service Cubes: Lessons learned from the Field
Repeat after me: partitioning, partitioning, partitioning!
• To get performance on multi-core boxes,• Partition, partition, partition
• Query each partition, results in separate threads• Distinct count
• Non-overlapping, continuous • The importance of I/O is directly related to
storage engine queries
BIA499A • SQLCAT: Designing scalable and complex Analysis Service Cubes: Lessons learned from the Field
Repeat after me: partitioning,… Distinct Count performance
0:00:00
0:00:15
0:00:30
0:00:45
0:01:00
0:01:15
0:01:30
0:01:45
0:02:00
0:02:15
0:02:30
0:02:45
0:03:00
0:03:15
0:03:300
2
4
6
8
10
12
Physical Disk > Avg Disk Queue Length
Time x User (4) Time (23)Single UserTime x User (10) Time x Browser (4)
• Six different partitioning schemes• Server has 4 processors• Lower / Shorter Duration Avg
Disk Queue Length
Type Ranking
Time x User (4) 2.13
Time 3.72
Single 4.64
User 2.25
Time x User (10) 2.97
Time x Browser 5.37
BIA499A • SQLCAT: Designing scalable and complex Analysis Service Cubes: Lessons learned from the Field
Multi-user query concurrencyGeneral rules
• Heap/Virtual Memory config settings• MemoryHeapType = 2• HeapTypeForObjects = 0
• Long running queries block processing• It’s just time for a separation…
• Querying and processing servers• Also importance of scale-out querying
BIA499A • SQLCAT: Designing scalable and complex Analysis Service Cubes: Lessons learned from the Field
Multi-user query concurrencyScale Out Analysis Services: Read Only
AS Processing ServerAS
Que
ry La
yer
AS database
AS Query 1
AS Query 2
AS Query 3
AS Query 4
AS database
Fast Disk (e.g. SAN)
SE Query
Clients
Clients
Clients
NLB
SEQ
SynchronizeUser Queries
User Queries
SE Query
SE Query
SE Query
1
23
4
BIA499A • SQLCAT: Designing scalable and complex Analysis Service Cubes: Lessons learned from the Field
Multi-user query concurrencyQuery performance
One Two Three Four0
50
100
150
200
250
300
238.5634875
150.3148394
101.9169981
FE Heavy (80 cc users)
One Two Three Four0
20
40
60
80
100
120
88.177996003996
25.1254835164835
102.022311407161
28.3542539550375
SE Heavy
50 CC Users
60 CC Users
BIA499A • SQLCAT: Designing scalable and complex Analysis Service Cubes: Lessons learned from the Field
Keeping it simple!
Careful with calculation scripts• Don’t: One cube with calculation script thousands
of lines long• Cell-by-cell mode WILL kill you• Example from bank: speedup 1000x by rewriting
cell-by-cell to block styleWhen it’s large, careful with large features• Keep Parent/Child small (tens of thousands)• Keep M-M small (low millions max)
BIA499A • SQLCAT: Designing scalable and complex Analysis Service Cubes: Lessons learned from the Field
What? Did you say ROLAP?!!
-67158236
-2085251220
-458579892
-903461230
482449446
-1958890532
1101641660
-954727192
663180516
-808192897
2023856459
0 100000 200000 300000 400000 500000 600000 700000
MOLAP
ROLAP1757523554
-420451054
-1602608104
721132422
-889102219
990819614'
0
200000
400000
600000
800000
1000000
1200000
MOLAP
ROLAP
Case Study: AdCenter
How does AdCenter handle two 3.2TB (yes TB) cubes?
BIA499A • SQLCAT: Designing scalable and complex Analysis Service Cubes: Lessons learned from the Field
The Importance of I/O
EMC DMX V-Max to handle the I/O• V-Max are dedicated to the application• Hundreds of disks and spindles devoted to this process• Short-stroking the disks to ensure fastest I/O• Works closely with EMC directly (present at EMC World
regularly)• Testing with EMC EFDs (Enterprise Flash Drives)
• Systems Engineering team dedicated to DW process• Work closely with various vendors (EMC, HDS, etc.)
• References• Accelerating Microsoft adCenter
with Microsoft SQL Server 2008 Analysis Services.• REAL PRACTICES: Performance Scaling Microsoft adCenter with
Microsoft SQL Server 2008 Analysis Services on EMC VMAX
BIA499A • SQLCAT: Designing scalable and complex Analysis Service Cubes: Lessons learned from the Field
It’s all about I/OView of adCenter Production
Storage Area Network
OLAP Processing Server
Windows Server 2003 x64 SP2SQL Server Enterprise Edition
32 GB RAM, 8 Xeon procs (16 cores)
Staging Data Warehouse
Windows Server 2003 x64 SP2SQL Server Integration Services
Network Load Balancing
Data Feeds
HBA BHBA A
Windows Server 2003 x64 SP2SQL Server Analysis Services
64GB RAM, 8 Xeon procs (16 cores)
OLAP Standby Server
SAN Fabric A
SAN Fabric B
HBA BHBA A HBA BHBA A
Host Bus Adapters: 400 MB/sec each
HBA BHBA A HBA BHBA A HBA BHBA A
adCenter Production Environment
Windows Server 2003 x64 SP2SQL Server Analysis Services
64GB RAM, 8 Xeon procs (16 cores)
OLAP LUNStandby OLAP LUN
19200 Max Reads9600 Max Writes
DW LUN
180 300GB 10K Drives
RAID 119200 Max Reads9600 Max Writes
180 300GB 10K Drives
RAID 12560 Max Reads2560 Max Writes
32 300GB 10K Drives
RAID 1
BIA499A • SQLCAT: Designing scalable and complex Analysis Service Cubes: Lessons learned from the Field
Repeat after me: partitioning, …
Facts Database
1 Partition per Day31 Partitions, 1 Month of Data
ALTER PARTITION FUNCTION PerDay () SPLIT RANGE(CAST(CONVERT(varchar, GETDATE()+1, 112) AS int))
1 2 3 4 5 6 7 8 9 10 11
12 13 14 15 16 17 18 19 20 21 22 23
0
0 11 17 1 14 18 2 5 22 3 12 21
4 13 19 6 15 23 7 209 8 10 16
...WHERE [date] = CAST(CONVERT(varchar, GETDATE(), 112) AS int) AND [hour] IN (0, 11, 17)
8 Evenly Distributed Partitions per Day3120 Partitions, 13 Months of Data
8 Parallel Partition Processing Jobs
Current Day Partition Set
Current Day Partition
Cube
Case Study: E&D
Immense amounts of Xbox Live service usage, purchase, and achievements data
BIA499A • SQLCAT: Designing scalable and complex Analysis Service Cubes: Lessons learned from the Field
Partitioning Strategy Evenly distributed, continuous, and non-overlapping
Rollups
June 2008
Mar 2010
...
Usage
-9223372036854775808 TO -8070015239897208167
-8070015239897208166 TO -6917692104582506052
8069877141170462663 TO 9223372036854775807
13 more...
• Monthly partitions• Data further subdivided into 16
partitions each month (1 per core)• Split based on SHA1 hash of userid
• Provides even distribution and non-overlapping data spread
BIA499A • SQLCAT: Designing scalable and complex Analysis Service Cubes: Lessons learned from the Field
The Importance of I/OThe effect of design and hardware
Day Week Month Quarter 7 months0
500
1000
1500
2000
2500
3000
Usage Cube Distinct Count Performance
Amount of Data
Run
Tme
(sec
onds
)
select [Measures].[Online Unique Users] on 0,[Date].[Date].[Date] on 1from Usagewhere [Date].[Calendar Month].[Calendar Month].&[2009]&[1]
BIA499A • SQLCAT: Designing scalable and complex Analysis Service Cubes: Lessons learned from the Field
Multi-User Query ConcurrencyUsing SSD to allow each server to handle more simultaneous queries
Run 6, One at a time Run 6, All at once Run 12, All at once0
500
1000
1500
2000
2500
3000
Distinct Count Parallel Querying - Total Runtime
Dev HDDDev SSDSQLCat SSD
Access Pattern
Runti
me
(sec
onds
)
Case Study: Yahoo!
Yes, I said 12TB cube
BIA499A • SQLCAT: Designing scalable and complex Analysis Service Cubes: Lessons learned from the Field
Oracle 10gCDF
Ad Optimization Platform Architecture
SSAS Cube Builder
NAS
SSAS Query Servers
HW NLB
Users
Partition 1
Partition 2
Partition N
Partition 1
Partition 2
Partition N
1.2TB/dayFile1
File2
FileN
50GB/hr
12TB cube
Case Study – Large Investment Banks
BIA499A • SQLCAT: Designing scalable and complex Analysis Service Cubes: Lessons learned from the Field
Real time Cube
Locking in Analysis Services – The Truth you have to deal with
Process Commit
Long Running Query
Perceived Runtime from User
TimeOut
New Perceived Runtime
BIA499A • SQLCAT: Designing scalable and complex Analysis Service Cubes: Lessons learned from the Field
Relational locking – same story, different engineImportant limitations in real time part of cube
• SWITCH/MERGE must take SCH-M lock (but held only a few ms)
• Can cause blocking queues
• If bulking into partitioned table – may cause a sort operations (use small batch sizes)
• Alternative Solution: partition real time part into a seperate table. Keep stale data in partitioned tabled.
Table2000
2001
2002LCK_S
LCK_SCH_M LCK_S
LCK_SLCK_S
LCK_S
LCK_S
LCK_S
LCK_S
LCK_S
LCK_S
Long Running Query
BIA499A • SQLCAT: Designing scalable and complex Analysis Service Cubes: Lessons learned from the Field
MOLAP Switching In Action
Basic idea:• Use MOLAP for historical data• Process latest MOLAP partitions oftenTypical Latencies in minutesConcerns:• How to partition cube for fast, frequent MOLAP
processing• Handling the processing lock when you need to
update the data
High Level Cube PartitioningPartitions by BOTH time and region
Full processing can be done on ”active” partitions
• ”Roll with the daylight”
• Isolates changes and reloads
Requirement: Must be able to fully recalculate risk
2010-01-01
NY London Bombay Tokyo
2010-01-02
NY London Bombay Tokyo
2010-01-03
NY London Bombay Tokyo
Cube FlippingRecall: Server level lock required to process
Alternative Solution:
• Two copies of the cube, taking turns processing
• ”Flip” between them
Two ways to flip:
• Use ASLB from CodePlex
• Write small Excel Plug-in
Excel Plug-in may be used when user holds session state
• Session state automatically refreshes when connectionstring is changed
Process
Query
(Query)
(Process)
BIA499A • SQLCAT: Designing scalable and complex Analysis Service Cubes: Lessons learned from the Field
ROLAP + MOLAP in Action
Basic ideas:
• Use MOLAP for historical/stale data
• Use ROLAP for frequently updated data
Typical latencies achievable are in seconds
Concerns/Challenges:
• Must maintain coherent ROLAP cache
• Use Real Time OLAP=true in connectionstring
• When MOLAP changes – must have minimum impact on others
• Consider archiving strategy
• Locking inside SQL Server must be handled (concurrent load). Consider RCSI
BIA499A • SQLCAT: Designing scalable and complex Analysis Service Cubes: Lessons learned from the Field
Real time Cube
Active Book
MOLAP(Market
Snapshot)
ROLAP
Active Book
MOLAP(Market
Snapshot)
ROLAP
Trader
Active Book
MOLAP(Market
Snapshot)
ROLAP
Active Book
MOLAP(Market
Snapshot)
ROLAP
x 25Trader
| SQL Server Customer Advisory Team
Real Time risk runs (RCSI mode)
Real time Cube
Active Book_X
MOLAP(Market
Snapshot)
ROLAP
Market Snapshots (MOLAP Processing)
ROLAP_Book_X
Snapshot_Xx 25
x 25Trickle feed ETL
| SQL Server Customer Advisory Team
Real Time risk runs (RCSI mode)
Market Snapshots (MOLAP Processing)
ROLAP_Book_X
Snapshot_Xx 25
x 25
History
Day1
Empty
Day2
Day3
6 months
…
Merge +
Agg
SWITCH
History Cube
Process
BIA499A • SQLCAT: Designing scalable and complex Analysis Service Cubes: Lessons learned from the Field
Two Real Time Cube methods - TradeoffsComponent MOLAP Switching ROLAP+MOLAP
Relation Tuning Low Must get right
AS Locking Must handle Not a big issues
Cache Usage Good Poor
Relational Concurrency N/A RCSI
Data Storage Best Compression ROLAP typically 2x MOLAP
Aggregate management Profiler + UBO Manual
Latency Minutes Seconds
Case Study: Sandvik FinanceDesigning scalable and complex Analysis Service Cubes
BIA499A • SQLCAT: Designing scalable and complex Analysis Service Cubes: Lessons learned from the Field
Challenge: Financial Consolidation
Need: Writeback with 5 second latency
Need: 100% Uptime during month closing
Need: 300 truly concurrent users
Need: Dynamic scale as company grows
BIA499A • SQLCAT: Designing scalable and complex Analysis Service Cubes: Lessons learned from the Field
Considerations
All about the hardware: Cubes a small, they will fit in memory if we get enough (dont use corporate SANMulti User: • Locking could be an issue, but all queries are
small (in control of the front end)• Processing Online may work.. But see later• Global company – what about latency?KISS: Review calculation script – CHECK!Security for auditing purpose
BIA499A • SQLCAT: Designing scalable and complex Analysis Service Cubes: Lessons learned from the Field
BIA499A • SQLCAT: Designing scalable and complex Analysis Service Cubes: Lessons learned from the Field
Latency of the link
Observations:• XMLA is flexible – but it is ”bloated” on bandwidth• Excel tends to be chatty clientResults: User will not NOT like the latency added by the speed of light• The ms required really adds up
Solution: Citrix farms with remote users. Gives great response times and good user experience
BIA499A • SQLCAT: Designing scalable and complex Analysis Service Cubes: Lessons learned from the Field
Partitioning for Real Time Setup
Financial Data
Historial Records
Current Month
Real Time
Historial Records(MOLAP / monthly)
Current Month(MOLAP / 10 min)
Current Month(ROLAP / ProActive)
Historial Records(MOLAP / monthly)
Current Month(MOLAP / 10 min)
Current Month(ROLAP / ProActive)
Read
Read
Writeback
BIA499A • SQLCAT: Designing scalable and complex Analysis Service Cubes: Lessons learned from the Field
Enryption techniques and overhead
Securing all lines and data:
• Crypt the XMLA traffic with SSL
• EFS on MOLAP data
• TDE on SQL Server data
• SQL Server TDS encryption
Overhead:
• Less than 10% on query performance
• Dimension processing times doubles!
• Fact Processing, only a few %
BIA499A • SQLCAT: Designing scalable and complex Analysis Service Cubes: Lessons learned from the Field
ROLAP – yes ROLAP!
No way to bring in data to MOLAP fast enough
• No – ProActive caching will not help.
• There is never a silent interval long enough
Solution: ROLAP partition with recent data
• Keep ROLAP partition small. Don’t want to touch the I/O system
• Avoids locking issue during processing
Keep the ROLAP small..
Technique:• Gradually shift data from ROLAP to MOLAP
”mid day” partition using ProActive cache
• Steps (every 10 min)
1. INSERT...SELECT from ROLAP partitions to MOLAP source table
2. Lock Cube
3. Process MOLAP (fast, only intra day data)
4. Modify View on ROLAP to exclude data
5. Unlock Cube
6. Delete Data (delay this, to avoid holding lock)
• Necessary assumption: All queries are short running
BIA499A • SQLCAT: Designing scalable and complex Analysis Service Cubes: Lessons learned from the Field
Summarizing (1/2)
You HAVE to get the design right if you want to scale
Partitioning is absolute crucial
• Partition for processing speed
• Partition for data latency (real time vs. Stale)
• Partition for archival of old data
• Secondary concern to above: partition for query speed
Hardware really matters for large cubes
• NAND devices are match made in heaven.
• Tests show two core Nehalem CPU can often support hundreds of users
• Carefully balance IOPS vs. Memory, consider hot portion of cube
BIA499A • SQLCAT: Designing scalable and complex Analysis Service Cubes: Lessons learned from the Field
Summarizing (2/2)
Locking is your enemy
• Consider processing server with sync vs. Individual servers processing
• Consider how to flip
• For super low latency, ROLAP is the way
Multi User Settings
• Just run them as per perf guide
• Consider adjusting CoordinatorBalancingFactor
Keep it Simple
• Remember the premise: AS is a ”send me any query” database.
• You really have to get this right at scale
BIA499A • SQLCAT: Designing scalable and complex Analysis Service Cubes: Lessons learned from the Field
Call to Action!Feedback really works – a story and “announcement”
Rules:
• Min 100GB cube and/or 100 concurrent users
• Max one pain / month
• Prefix mail subject: “Cube Feedback:”
Feedback, crucial questions:
• Who are you? (customer name / company profile)
• Measure/dimension size, users, cube project
• What are you trying to achieve?
• What does it cost you not to get there / lost opportunity?
My email: [email protected]
BIA499A • SQLCAT: Designing scalable and complex Analysis Service Cubes: Lessons learned from the Field
Complete the Evaluation Form to Win!
Win a Dell Mini Netbook – every day – just for handing in your completed form. Each session evaluation form represents a chance to win.
Pick up your evaluation form:• In each presentation room• At the PASS Booth near registration
Drop off your completed form:• Near the exit of each presentation room• At the PASS Booth near registration
Sponsored by Dell
for attending this session and the 2010 PASS Summit in Seattle