![Page 1: DBI 312. SELECT prod.. FROM Product,,,, WHERE …](https://reader035.vdocument.in/reader035/viewer/2022062515/56649c7c5503460f9492ffd4/html5/thumbnails/1.jpg)
Columnstore Indexes Unveiled
Eric N HansonPrincipal Program ManagerMicrosoft
DBI 312
![Page 2: DBI 312. SELECT prod.. FROM Product,,,, WHERE …](https://reader035.vdocument.in/reader035/viewer/2022062515/56649c7c5503460f9492ffd4/html5/thumbnails/2.jpg)
Alice, Today
![Page 3: DBI 312. SELECT prod.. FROM Product,,,, WHERE …](https://reader035.vdocument.in/reader035/viewer/2022062515/56649c7c5503460f9492ffd4/html5/thumbnails/3.jpg)
Alice, Today
![Page 4: DBI 312. SELECT prod.. FROM Product,,,, WHERE …](https://reader035.vdocument.in/reader035/viewer/2022062515/56649c7c5503460f9492ffd4/html5/thumbnails/4.jpg)
Alice, Today
SELECT prod ..FROM Product, ,,,WHERE ….
![Page 5: DBI 312. SELECT prod.. FROM Product,,,, WHERE …](https://reader035.vdocument.in/reader035/viewer/2022062515/56649c7c5503460f9492ffd4/html5/thumbnails/5.jpg)
Alice, Today
![Page 6: DBI 312. SELECT prod.. FROM Product,,,, WHERE …](https://reader035.vdocument.in/reader035/viewer/2022062515/56649c7c5503460f9492ffd4/html5/thumbnails/6.jpg)
Alice, Today
Mon Prod Cus Sales
Sept Cxtpo Aam 109.23
Oct Hifyw Cty 289.35
Nov Nrvan Rrst 453.89
Dec Cxtpo Cty 321.79
![Page 7: DBI 312. SELECT prod.. FROM Product,,,, WHERE …](https://reader035.vdocument.in/reader035/viewer/2022062515/56649c7c5503460f9492ffd4/html5/thumbnails/7.jpg)
Alice, Today
Mon Prod Cus Sales
Sept Cxtpo Aam 109.23
Oct Hifyw Cty 289.35
Nov Nrvan Rrst 453.89
Dec Cxtpo Cty 321.79
?
![Page 8: DBI 312. SELECT prod.. FROM Product,,,, WHERE …](https://reader035.vdocument.in/reader035/viewer/2022062515/56649c7c5503460f9492ffd4/html5/thumbnails/8.jpg)
Alice, Tomorrow
![Page 9: DBI 312. SELECT prod.. FROM Product,,,, WHERE …](https://reader035.vdocument.in/reader035/viewer/2022062515/56649c7c5503460f9492ffd4/html5/thumbnails/9.jpg)
Alice, Tomorrow
SELECT prod ..FROM Product, ,,,WHERE ….
![Page 10: DBI 312. SELECT prod.. FROM Product,,,, WHERE …](https://reader035.vdocument.in/reader035/viewer/2022062515/56649c7c5503460f9492ffd4/html5/thumbnails/10.jpg)
Alice, Tomorrow
Mon Prod Cus Sales
Sept Cxtpo Aam 109.23
Oct Hifyw Cty 289.35
Nov Nrvan Rrst 453.89
Dec Cxtpo Cty 321.79
![Page 11: DBI 312. SELECT prod.. FROM Product,,,, WHERE …](https://reader035.vdocument.in/reader035/viewer/2022062515/56649c7c5503460f9492ffd4/html5/thumbnails/11.jpg)
Alice, Tomorrow
SELECT custom ..FROM Cust, ,,,WHERE ….
![Page 12: DBI 312. SELECT prod.. FROM Product,,,, WHERE …](https://reader035.vdocument.in/reader035/viewer/2022062515/56649c7c5503460f9492ffd4/html5/thumbnails/12.jpg)
Alice, Tomorrow
Aha!
![Page 13: DBI 312. SELECT prod.. FROM Product,,,, WHERE …](https://reader035.vdocument.in/reader035/viewer/2022062515/56649c7c5503460f9492ffd4/html5/thumbnails/13.jpg)
Project Apollo
Accelerates data warehouse queriesNew columnstore indexImproved query execution
Makes SQL Server more interactiveEasy to useReduces TCO
![Page 14: DBI 312. SELECT prod.. FROM Product,,,, WHERE …](https://reader035.vdocument.in/reader035/viewer/2022062515/56649c7c5503460f9492ffd4/html5/thumbnails/14.jpg)
Data warehouse workload
Read-mostlyLoad large amounts of dataAppend new data incrementallyRarely update existing dataOften retain data for given time (e.g. 1, 3, 7 yrs)
Sliding window data management
![Page 15: DBI 312. SELECT prod.. FROM Product,,,, WHERE …](https://reader035.vdocument.in/reader035/viewer/2022062515/56649c7c5503460f9492ffd4/html5/thumbnails/15.jpg)
Sliding window
2010
2009
2008
2007
20112010
2009
2008
2010
2009
2008
2007
![Page 16: DBI 312. SELECT prod.. FROM Product,,,, WHERE …](https://reader035.vdocument.in/reader035/viewer/2022062515/56649c7c5503460f9492ffd4/html5/thumbnails/16.jpg)
Data warehouse schemas and queries
Star schemaFact tables put columnstore indexes hereDimension tables
Star joinsMost queries aggregate data
![Page 17: DBI 312. SELECT prod.. FROM Product,,,, WHERE …](https://reader035.vdocument.in/reader035/viewer/2022062515/56649c7c5503460f9492ffd4/html5/thumbnails/17.jpg)
Star schemaFactSales
DimCustomer
FactSales(CustomerKey int, ProductKey int,EmployeeKey int,StoreKey int,OrderDateKey int,SalesAmount money)
DimCustomer(CustomerKey int,FirstName nvarchar(50),LastName nvarchar(50),Birthdate date,EmailAddress nvarchar(50))
DimProduct …
DimDate
DimEmployee
DimStore
![Page 18: DBI 312. SELECT prod.. FROM Product,,,, WHERE …](https://reader035.vdocument.in/reader035/viewer/2022062515/56649c7c5503460f9492ffd4/html5/thumbnails/18.jpg)
Star join query
SELECT TOP 10 p.ModelName, p.EnglishDescription, SUM(f.SalesAmount) as SalesAmountFROM FactResellerSalesPart f, DimProduct p, DimEmployee eWHERE f.ProductKey=p.ProductKey AND e.EmployeeKey=f.EmployeeKey AND f.OrderDateKey >= 20030601 AND p.ProductLine = 'M' -- Mountain AND p.ModelName LIKE '%Frame%' AND e.SalesTerritoryKey = 1 GROUP BY p.ModelName, p.EnglishDescriptionORDER BY SUM(f.SalesAmount) desc;
![Page 19: DBI 312. SELECT prod.. FROM Product,,,, WHERE …](https://reader035.vdocument.in/reader035/viewer/2022062515/56649c7c5503460f9492ffd4/html5/thumbnails/19.jpg)
“Typical” data warehouse queries
Process large amounts of dataReporting queriesOften slow (minutes to hours)DBAs spend considerable effort
Designing indexes, tuning queriesBuilding summary tables, indexed views, OLAP cubes
![Page 20: DBI 312. SELECT prod.. FROM Product,,,, WHERE …](https://reader035.vdocument.in/reader035/viewer/2022062515/56649c7c5503460f9492ffd4/html5/thumbnails/20.jpg)
demo
Query Acceleration With Columnstore Indexes
![Page 21: DBI 312. SELECT prod.. FROM Product,,,, WHERE …](https://reader035.vdocument.in/reader035/viewer/2022062515/56649c7c5503460f9492ffd4/html5/thumbnails/21.jpg)
Columnar storage structure
Uses VertiPaq compression
C1 C2 C3 C5 C6C4
…
Pages
Row store:
Column store:
![Page 22: DBI 312. SELECT prod.. FROM Product,,,, WHERE …](https://reader035.vdocument.in/reader035/viewer/2022062515/56649c7c5503460f9492ffd4/html5/thumbnails/22.jpg)
Reduced I/O using columnstore indexes
Fetches only needed columns from diskColumns are compressedLess IOBetter buffer hit rates
C1
C2
C4 C5 C6
C3
SELECT region, sum (sales) …
![Page 23: DBI 312. SELECT prod.. FROM Product,,,, WHERE …](https://reader035.vdocument.in/reader035/viewer/2022062515/56649c7c5503460f9492ffd4/html5/thumbnails/23.jpg)
Advanced query execution technology
Batch mode execution of some operations
Processes rows in batchesGroups of batch operations in query plan
Efficient data representationHighly efficient algorithmsBetter parallelism
![Page 24: DBI 312. SELECT prod.. FROM Product,,,, WHERE …](https://reader035.vdocument.in/reader035/viewer/2022062515/56649c7c5503460f9492ffd4/html5/thumbnails/24.jpg)
Column segments
Column segment contains values from one column for a set of about 1M rowsColumn segments are compressedEach column segment stored in separate LOBColumn segment is unit of transfer from disk
C1 C2 C3 C5 C6C4
Set of about 1M rows
Column Segment
![Page 25: DBI 312. SELECT prod.. FROM Product,,,, WHERE …](https://reader035.vdocument.in/reader035/viewer/2022062515/56649c7c5503460f9492ffd4/html5/thumbnails/25.jpg)
Constraints on use with other indexes & partitioning
Base table must be clustered B-tree or heapColumnstore index:
nonclustered one per tablemust be partition-alignednot allowed on indexed viewcan’t be a filtered index
![Page 26: DBI 312. SELECT prod.. FROM Product,,,, WHERE …](https://reader035.vdocument.in/reader035/viewer/2022062515/56649c7c5503460f9492ffd4/html5/thumbnails/26.jpg)
Creating the columnstore index
Create the tableLoad data into the tableCreate a non-clustered columnstore index on all, or some, columns
CREATE NONCLUSTERED COLUMNSTORE INDEX ncci ON myTable(OrderDate, ProductID, SaleAmount)
Object Explorer
![Page 27: DBI 312. SELECT prod.. FROM Product,,,, WHERE …](https://reader035.vdocument.in/reader035/viewer/2022062515/56649c7c5503460f9492ffd4/html5/thumbnails/27.jpg)
Index dialog makes index creation easy
![Page 28: DBI 312. SELECT prod.. FROM Product,,,, WHERE …](https://reader035.vdocument.in/reader035/viewer/2022062515/56649c7c5503460f9492ffd4/html5/thumbnails/28.jpg)
Query optimization with columnstores
Cost-basedOptimizer chooses indexOptimizer chooses batch mode operators
![Page 29: DBI 312. SELECT prod.. FROM Product,,,, WHERE …](https://reader035.vdocument.in/reader035/viewer/2022062515/56649c7c5503460f9492ffd4/html5/thumbnails/29.jpg)
Using hints with columnstore indexes
• use the columnstore index
• use a different index
• ignore columnstore
select distinct (SalesTerritoryKey)from dbo.FactResellerSales with (index (ci))
select distinct (SalesTerritoryKey)from dbo.FactResellerSales with (index (ncci))
select distinct (SalesTerritoryKey)from dbo.FactResellerSalesoption(ignore_nonclustered_columnstore_index)
![Page 30: DBI 312. SELECT prod.. FROM Product,,,, WHERE …](https://reader035.vdocument.in/reader035/viewer/2022062515/56649c7c5503460f9492ffd4/html5/thumbnails/30.jpg)
Memory management
Memory management is automaticColumnstore persisted on diskNeeded columns fetched into memoryColumn segments flow between disk and memory
SELECT C2, SUM(C4)FROM TGROUP BY C2;
T.C2T.C4
T.C2T.C4
T.C2
T.C2
T.C2T.C1
T.C1
T.C1
T.C1
T.C1T.C3
T.C3
T.C3
T.C3
T.C3
T.C4
T.C4
T.C4
![Page 31: DBI 312. SELECT prod.. FROM Product,,,, WHERE …](https://reader035.vdocument.in/reader035/viewer/2022062515/56649c7c5503460f9492ffd4/html5/thumbnails/31.jpg)
New showplan elements for columnstores and batch processing
![Page 32: DBI 312. SELECT prod.. FROM Product,,,, WHERE …](https://reader035.vdocument.in/reader035/viewer/2022062515/56649c7c5503460f9492ffd4/html5/thumbnails/32.jpg)
![Page 33: DBI 312. SELECT prod.. FROM Product,,,, WHERE …](https://reader035.vdocument.in/reader035/viewer/2022062515/56649c7c5503460f9492ffd4/html5/thumbnails/33.jpg)
Columnstore index catalog tables
![Page 34: DBI 312. SELECT prod.. FROM Product,,,, WHERE …](https://reader035.vdocument.in/reader035/viewer/2022062515/56649c7c5503460f9492ffd4/html5/thumbnails/34.jpg)
Columnstore indexes: interoperability with the rest of SQL Server
Most things “just work” with columnstore indexesBackup and restoreMirroringLog shippingSSMSAdministration, tools
![Page 35: DBI 312. SELECT prod.. FROM Product,,,, WHERE …](https://reader035.vdocument.in/reader035/viewer/2022062515/56649c7c5503460f9492ffd4/html5/thumbnails/35.jpg)
Data type restrictions
Unsupported typesdecimal > 18 digitsBinaryBLOB(n)varchar(max)UniqueidentifierDate/time types > 8 bytesCLR
![Page 36: DBI 312. SELECT prod.. FROM Product,,,, WHERE …](https://reader035.vdocument.in/reader035/viewer/2022062515/56649c7c5503460f9492ffd4/html5/thumbnails/36.jpg)
Query performance restrictions
Outer joinsUnionsConsider modifying queries to hit “sweet spot”
Inner joinsStar joinsAggregation
![Page 37: DBI 312. SELECT prod.. FROM Product,,,, WHERE …](https://reader035.vdocument.in/reader035/viewer/2022062515/56649c7c5503460f9492ffd4/html5/thumbnails/37.jpg)
Loading new data into a columnstore index
Columnstore makes table read-onlyPartition switching allowedINSERT, UPDATE, DELETE, and MERGE not allowedTwo recommended methods for loading dataDisable, update, rebuildPartition switching
![Page 38: DBI 312. SELECT prod.. FROM Product,,,, WHERE …](https://reader035.vdocument.in/reader035/viewer/2022062515/56649c7c5503460f9492ffd4/html5/thumbnails/38.jpg)
Adding data to a table with a columnstore index
Method 1: Disable the columnstore indexDisable (or drop) the index
ALTER INDEX my_index ON MyTable DISABLE
Update the tableRebuild the columnstore index
ALTER INDEX my_index ON MyTable REBUILD
![Page 39: DBI 312. SELECT prod.. FROM Product,,,, WHERE …](https://reader035.vdocument.in/reader035/viewer/2022062515/56649c7c5503460f9492ffd4/html5/thumbnails/39.jpg)
Adding data to a table with a columnstore index
Method 2: Use PartitioningLoad new data into a staging tableBuild a columnstore index
CREATE NONCLUSTERED COLUMNSTORE INDEX my_index ON StagingT(OrderDate, ProductID, SaleAmount)
Split empty partitionSwitch the partition into the table
ALTER TABLE StagingT SWITCH TO T PARTITION 5
![Page 40: DBI 312. SELECT prod.. FROM Product,,,, WHERE …](https://reader035.vdocument.in/reader035/viewer/2022062515/56649c7c5503460f9492ffd4/html5/thumbnails/40.jpg)
Advanced near-real-time append
Partitioned table with columnstore indexKeep copy of data from latest partition in separate row-store table SWhile(True) {
Append new data to S for desired period (say 15 minutes)Columnstore index SSwap S with newest partitionDrop columnstore index on SInsert rows in S to “catch up” S to current time
}
![Page 41: DBI 312. SELECT prod.. FROM Product,,,, WHERE …](https://reader035.vdocument.in/reader035/viewer/2022062515/56649c7c5503460f9492ffd4/html5/thumbnails/41.jpg)
When to build a columnstore index
Read-mostly workloadMost updates are appending new dataWorkflow permits partitioning or index drop/rebuild
Typically a nightly load window
Queries often scan & aggregate lots of data
![Page 42: DBI 312. SELECT prod.. FROM Product,,,, WHERE …](https://reader035.vdocument.in/reader035/viewer/2022062515/56649c7c5503460f9492ffd4/html5/thumbnails/42.jpg)
What tables to columnstore index
Large fact tables(Maybe) very large dimension tables
![Page 43: DBI 312. SELECT prod.. FROM Product,,,, WHERE …](https://reader035.vdocument.in/reader035/viewer/2022062515/56649c7c5503460f9492ffd4/html5/thumbnails/43.jpg)
When not to build a columnstore index
Frequent updatesPartition switching or rebuilding index doesn’t fit workflowFrequent small look up queries
B-tree indexes may give better performance
Your workload does not benefit
![Page 44: DBI 312. SELECT prod.. FROM Product,,,, WHERE …](https://reader035.vdocument.in/reader035/viewer/2022062515/56649c7c5503460f9492ffd4/html5/thumbnails/44.jpg)
Summary: Apollo in a nutshell
Astonishing speedup for DW queries
Columnstore technology+
Advanced query processing
![Page 45: DBI 312. SELECT prod.. FROM Product,,,, WHERE …](https://reader035.vdocument.in/reader035/viewer/2022062515/56649c7c5503460f9492ffd4/html5/thumbnails/45.jpg)
Thank you!
![Page 46: DBI 312. SELECT prod.. FROM Product,,,, WHERE …](https://reader035.vdocument.in/reader035/viewer/2022062515/56649c7c5503460f9492ffd4/html5/thumbnails/46.jpg)
Database Platform (DAT) Resources
Try the new SQL Server Mission Critical BareMetal Hand’s on-Labs
Visit the updated website for SQL Server® Code Name “Denali” on www.microsoft.com/sqlserver and sign to be notified when the next CTP is availableFollow the @SQLServer Twitter account to watch for updates
Visit the SQL Server Product Demo Stations in the DBI Track section of the Expo/TLC Hall. Bring your questions, ideas and conversations!
• Microsoft® SQL Server® Security & Management • Microsoft® SQL Server® Optimization and Scalability• Microsoft® SQL Server® Programmability • Microsoft® SQL Server® Data Warehousing• Microsoft® SQL Server® Mission Critical • Microsoft® SQL Server® Data Integration
![Page 47: DBI 312. SELECT prod.. FROM Product,,,, WHERE …](https://reader035.vdocument.in/reader035/viewer/2022062515/56649c7c5503460f9492ffd4/html5/thumbnails/47.jpg)
Resources
www.microsoft.com/teched
Sessions On-Demand & Community Microsoft Certification & Training Resources
Resources for IT Professionals Resources for Developers
www.microsoft.com/learning
http://microsoft.com/technet http://microsoft.com/msdn
Learning
http://northamerica.msteched.com
Connect. Share. Discuss.
![Page 48: DBI 312. SELECT prod.. FROM Product,,,, WHERE …](https://reader035.vdocument.in/reader035/viewer/2022062515/56649c7c5503460f9492ffd4/html5/thumbnails/48.jpg)
Complete an evaluation on CommNet and enter to win!
![Page 49: DBI 312. SELECT prod.. FROM Product,,,, WHERE …](https://reader035.vdocument.in/reader035/viewer/2022062515/56649c7c5503460f9492ffd4/html5/thumbnails/49.jpg)
Scan the Tag to evaluate this session now on myTech•Ed Mobile
![Page 50: DBI 312. SELECT prod.. FROM Product,,,, WHERE …](https://reader035.vdocument.in/reader035/viewer/2022062515/56649c7c5503460f9492ffd4/html5/thumbnails/50.jpg)
© 2011 Microsoft Corporation. All rights reserved. Microsoft, Windows, Windows Vista and other product names are or may be registered trademarks and/or trademarks in the U.S. and/or other countries.The information herein is for informational purposes only and represents the current view of Microsoft Corporation as of the date of this presentation. Because Microsoft must respond to changing market conditions, it should not be interpreted to
be a commitment on the part of Microsoft, and Microsoft cannot guarantee the accuracy of any information provided after the date of this presentation. MICROSOFT MAKES NO WARRANTIES, EXPRESS, IMPLIED OR STATUTORY, AS TO THE INFORMATION IN THIS
PRESENTATION.
![Page 51: DBI 312. SELECT prod.. FROM Product,,,, WHERE …](https://reader035.vdocument.in/reader035/viewer/2022062515/56649c7c5503460f9492ffd4/html5/thumbnails/51.jpg)