White Paper
EMC Solutions Group
Abstract
This white paper describes a simulated multiple data-mart solution for the mid-size market, based on Microsoft SQL Server. The solution uses a cost-saving building-block approach to add servers, storage, or memory to scale, based on customers’ needs.
February 2013
SQL SERVER 2012 DATA WAREHOUSE EMC VNX5500 HB (R32), VMware vSphere 5, Microsoft Windows 2012
• High bandwidth for large size I/O in data warehouse
• Highly scalable building-block architecture for data warehouse • Virtualized with vSphere 5.1
SQL Server 2012 Data Warehouse EMC VNX5500 HB (R32), VMware vSphere 5, Microsoft Windows 2012
2
Copyright © 2013 EMC Corporation. All Rights Reserved.
EMC believes the information in this publication is accurate as of its publication date. The information is subject to change without notice.
The information in this publication is provided “as is.” EMC Corporation makes no representations or warranties of any kind with respect to the information in this publication, and specifically disclaims implied warranties of merchantability or fitness for a particular purpose.
Use, copying, and distribution of any EMC software described in this publication requires an applicable software license.
For the most up-to-date listing of EMC product names, see EMC Corporation Trademarks on EMC.com.
VMware, ESX, VMware vCenter, and VMware vSphere are registered trademarks or trademarks of VMware, Inc. in the United States and/or other jurisdictions.
All trademarks used herein are the property of their respective owners.
Part Number H11180
3 SQL Server 2012 Data Warehouse EMC VNX5500 HB (R32), VMware vSphere 5, Microsoft Windows 2012
Table of contents
Executive summary ............................................................................................................................... 6 Business case .................................................................................................................................. 6 Solution overview ............................................................................................................................ 6 Key results ....................................................................................................................................... 6
Introduction.......................................................................................................................................... 7 Purpose ........................................................................................................................................... 7 Scope .............................................................................................................................................. 7 Audience ......................................................................................................................................... 7 Terminology ..................................................................................................................................... 7
Key technology components ................................................................................................................ 8 Overview .......................................................................................................................................... 8 EMC VNX5500 series storage system with high bandwidth .............................................................. 8 EMC Storage Integrator (ESI) for Windows Suite ............................................................................... 8 VMware vSphere .............................................................................................................................. 9
VMware vSphere 5.1 ................................................................................................................... 9 VMware vCenter Server ................................................................................................................ 9 EMC PowerPath/VE ...................................................................................................................... 9
Microsoft SQL Server 2012 ............................................................................................................... 9
Solution architecture and design ........................................................................................................ 10 Overview ........................................................................................................................................ 10 Physical architecture ...................................................................................................................... 10
Building-block design ........................................................................................................................ 12 Overview ........................................................................................................................................ 12 Building-block design considerations ............................................................................................ 12
Proportional and predictive bandwidth ...................................................................................... 12 Virtual machine scalability ........................................................................................................ 12 Sufficient resources................................................................................................................... 13 Balanced disk utilization ........................................................................................................... 13
Building-block design details ......................................................................................................... 13 Deploying a building block ............................................................................................................. 15
Scale-up design ........................................................................................................................ 16 Scale-out design ....................................................................................................................... 16
SQL Server virtual machine and LUN allocation design ................................................................... 16 Hardware resources ....................................................................................................................... 18 Software resources ........................................................................................................................ 19 EMC VNX5500 HB storage system cache settings ........................................................................... 19
SQL Server 2012 Data Warehouse EMC VNX5500 HB (R32), VMware vSphere 5, Microsoft Windows 2012
4
Connectivity ................................................................................................................................... 20 Disk layout and SQL Server LUN configuration ................................................................................ 21
Storage System: ........................................................................................................................ 21 Hot Spares: ............................................................................................................................... 22 OS for VMs: ............................................................................................................................... 22 Tempdb: .................................................................................................................................... 22 Data LUNS: ................................................................................................................................ 22 Transaction Log LUNs: ............................................................................................................... 22
Host side I/O device and multipathing considerations ................................................................... 22 SQL Server 2012 and Windows 2012 considerations for DSS ......................................................... 23
Operating system and SQL Server instance settings .................................................................. 23 SQL Server 2012 startup options ............................................................................................... 23 SQL Server memory configuration .............................................................................................. 24 Database configurations ........................................................................................................... 24
VMware configuration .................................................................................................................... 24 ESXi HBA queue depth configuration ......................................................................................... 24 VMware vSphere configuration .................................................................................................. 24 VMware virtual machine configuration ....................................................................................... 24
EMC ESI v2.1 storage provisioning ..................................................................................................... 26 Objective ....................................................................................................................................... 26 ESI provisioning ............................................................................................................................. 26
High bandwidth read validation ......................................................................................................... 31 Overview ........................................................................................................................................ 31 Disk performance validation .......................................................................................................... 31 Application workloads ................................................................................................................... 32 Data warehouse performance validation ........................................................................................ 32
Data warehouse building-block validation ......................................................................................... 34 Overview ........................................................................................................................................ 34 SQL Server key performance indicators .......................................................................................... 34 Workload scale validation .............................................................................................................. 35
Typical DSS workload in profile W1 ........................................................................................... 35 Typical DSS workload in profile W2 ........................................................................................... 36
Scale out with building blocks ....................................................................................................... 37 Cumulative bandwidth and storage utilization ........................................................................... 39
Scale up with building blocks ........................................................................................................ 40 Bandwidth scalability ................................................................................................................ 40 Cumulative bandwidth and storage utilization ........................................................................... 42
Consolidated test results ............................................................................................................... 43 Tempdb consideration for DW/DSS-like workload .......................................................................... 45
5 SQL Server 2012 Data Warehouse EMC VNX5500 HB (R32), VMware vSphere 5, Microsoft Windows 2012
Capacity consideration .............................................................................................................. 45 Performance considerations ...................................................................................................... 45 Test results ................................................................................................................................ 46
Conclusion ......................................................................................................................................... 48 Summary ....................................................................................................................................... 48 Findings ......................................................................................................................................... 48
References.......................................................................................................................................... 49 White paper ................................................................................................................................... 49 Product documentation .................................................................................................................. 49 Other documentation ..................................................................................................................... 49
SQL Server 2012 Data Warehouse EMC VNX5500 HB (R32), VMware vSphere 5, Microsoft Windows 2012
6
Executive summary
Organizations in today’s increasingly competitive environment are quickly realizing the importance of a solid data warehouse platform that enables business intelligence (BI), in order to discover valuable insights from accumulated data.
Enterprise data warehouses that support BI tools have been growing rapidly with increased complexity. Many existing data warehouse configurations were not designed to deal with these issues. This solution addresses some of the common challenges for building data warehouse infrastructure. It helps customers achieve optimal performance with a building-block design deployed for Microsoft SQL Server.
The objective of this solution is to provide a building-block design for a small-to medium-sized virtualized data warehouse environment based on SQL Server 2012, VMware vSphere 5.1, and EMC® VNX5500® high bandwidth storage array.
Using the building-block design of compute, storage, and network resources, this solution can scale up or out, as needed, to accommodate the changing workloads and growing capacities of a typical compact, yet dynamic, data warehouse.
The solution uses an EMC VNX5500 with high bandwidth (VNX5500 HB) storage array to support high bandwidth reads for a large input/output (I/O) size (64 K-512 K) typically observed in data warehouse environment.
This solution demonstrates:
• A building-block design for data warehouse that provides an elastic infrastructure to scale up and/or scale out.
• That VNX5500 HB can provide high bandwidth especially for large element reads in a mission-critical data warehouse.
• Simplified storage management with EMC Storage Integrator (ESI).
Business case
Solution overview
Key results
7 SQL Server 2012 Data Warehouse EMC VNX5500 HB (R32), VMware vSphere 5, Microsoft Windows 2012
Introduction
This white paper describes the design and validation of a solution that uses building blocks for a data warehouse implementation of SQL Server 2012 on an EMC VNX5500 HB configuration in a virtualized environment.
This paper provides supporting technical collateral with customer and field guidance for an overview of one implementation of EMC’s designs for data warehouse on SQL server. SQL Query and Data Load Testing (DSS workloads) are generated through the database (not simulated I/O) to demonstrate more realistic throughput.
This white paper is intended for EMC employees, partners, and customers including IT planners, virtualization architects and SQL Server DBAs, administrators, and any other IT professionals involved in evaluating, acquiring, managing, operating, or designing infrastructure that uses EMC technologies.
Throughout this white paper we assume that you have some familiarity with the components and operations related to enterprise storage and virtualization technologies and their use in information infrastructures.
Table 1 defines several term used in this paper.
Table 1. Terminology
Term Definition
DSS Decision Support System (one type of data warehouse)
DW Data Warehouse
HBA Host Bus Adapters
VSI Virtual Storage Integrator
SAN Storage Area Network
SCSI Small Computer System Interface
VMFS Virtual Machine File System
Purpose
Scope
Audience
Terminology
SQL Server 2012 Data Warehouse EMC VNX5500 HB (R32), VMware vSphere 5, Microsoft Windows 2012
8
Key technology components
This solution uses the following key hardware and software components:
• EMC VNX5500 series storage system with high bandwidth
• EMC Storage Integrator (ESI) for Windows Suite
• VMware® vSphere™ ESXi
• Microsoft SQL Server 2012 Enterprise Edition
With industry-leading performance, EMC VNX5500 storage systems offer unified support for block and file usage protocols, a simple and intuitive management interface, “five-nines” reliability (equals roughly five minutes of unplanned downtime per year), and built-in features for supporting replication, disaster protection, and automatic storage tiering.
The solution uses a VNX5500 HB storage array to provide High Bandwidth Reads accommodating large element sizes up to 512k.
VNX supports the 512K RAID stripe element size that is specifically targeted at environments focusing on large random block reads. It is ideal for big data analytics processing and the data warehouse environment.
EMC Storage Integrator (ESI) for Windows Suite is a tool for Microsoft Windows and Microsoft applications administrators. The user interface for ESI is based on Microsoft Management Console (MMC). Therefore, you can run ESI as a standalone tool or as part of an MMC snap-in on a Windows platform.
ESI provides the ability to provision storage for Microsoft Windows or for Microsoft SharePoint sites. ESI supports the EMC VNX series, EMC VNXe® series, EMC CLARiiON® CX4 series, and EMC Symmetrix® VMAX® storage family.
In addition to physical environments, ESI also supports storage provisioning and discovery for Windows virtual machines running on Microsoft Hyper-V, Citrix XenServer, and VMware vSphere.
Overview
EMC VNX5500 series storage system with high bandwidth
EMC Storage Integrator (ESI) for Windows Suite
9 SQL Server 2012 Data Warehouse EMC VNX5500 HB (R32), VMware vSphere 5, Microsoft Windows 2012
For this solution, the Microsoft SQL Servers are fully virtualized using VMware vSphere 5.1. This section describes the virtualization infrastructure, which uses the following components and options:
• VMware vSphere 5.1
• VMware vCenter Server
• EMC PowerPath®/VE for VMware vSphere Version 5.7
VMware vSphere 5.1
VMware vSphere 5.1 is a complete, scalable, and powerful virtualization platform. It includes infrastructure services that transform IT hardware into a high-performance shared computing platform, and application services that help IT organizations deliver the highest levels of availability, security, and scalability.
VMware vCenter Server
VMware vCenter is the centralized management platform for vSphere environments, enabling control and visibility at every level of the virtual infrastructure.
EMC PowerPath/VE
EMC PowerPath/VE for VMware vSphere delivers PowerPath multipathing features that optimize VMware vSphere virtual environments. PowerPath/VE installs as a kernel module on the VMware ESXi host and works as a multipathing plug-in (MPP) that provides enhanced path management capabilities to ESXi hosts.
Microsoft SQL Server 2012 is Microsoft’s database management and analysis system for e-commerce, line-of-business, and data warehousing solutions. By enabling a modern data platform with SQL Server 2012, users can get built-in mission-critical capabilities and enable breakthrough insights across the organization with familiar analytics tools and enterprise-ready Big Data solutions.
VMware vSphere
Microsoft SQL Server 2012
SQL Server 2012 Data Warehouse EMC VNX5500 HB (R32), VMware vSphere 5, Microsoft Windows 2012
10
Solution architecture and design
This solution is a validated architecture designed to reflect realworld deployments. This section describes the overall architecture, key components, and resources that make up this solution and its environment.
Figure 1 depicts the physical architecture for this solution.
Figure 1. Architecture overview
Two 8 GB Fiber Chanel SAN switches connect two ESXi servers to the VNX5500 array.
Overview
Physical architecture
11 SQL Server 2012 Data Warehouse EMC VNX5500 HB (R32), VMware vSphere 5, Microsoft Windows 2012
The virtual machines created on the ESXi servers are designed using building blocks, which are detailed in the Building-block design section. A building block is defined to use a set of compute capacity (vCPU and random access memory (RAM)) and storage (disk number and appropriate RAID) as a block to support acceptable data warehouse performance (we1
In this solution, four virtual machines were built to host four SQL server 2012 instances. The building blocks were scaled up by adding to the same virtual machine, or scaled out to build in different virtual machines. A total of 11 building blocks with databases ranging from 500 GB, 1 TB to 2 TB of databases (total of 18 TB) were built in this solution.
used bandwidth as the key performance indicator in this solution to evaluate the data warehouse performance). A building block can be dynamically deployed to the virtualized environment to achieve the desired performance to scale up (adding to the same virtual machine) or scale out (adding with a new virtual machine).
1 In this white paper, "we" refers to the EMC Solutions engineering team that validated the solution.
SQL Server 2012 Data Warehouse EMC VNX5500 HB (R32), VMware vSphere 5, Microsoft Windows 2012
12
Building-block design
This section describes the use of a building-block approach to design storage for a data warehouse that provides flexibility and performance scalability.
It is critical that the infrastructure supporting the data warehouse—server, network, storage, and application—provides a robust, powerful, and flexible solution.
This section describes how to design a building block for data warehouses in a virtualized environment that provides predictable performance. The following criteria must be considered when designing a building block.
Proportional and predictive bandwidth
To achieve proportional and predictive bandwidth, consider the following design methodology:
• Design for the bandwidth, not for database capacity. In this solution, to achieve the goal of completing all the DSS queries in the test suit in a 12 ±2 hours window, the desired bandwidth is 100 MB/s for a 500 GB database, 200 MB/s for a 1 TB database, or 400 MB/s for a 2 TB database.
• Check disk performance. For a DSS workload with sequential 64K read only I/O, the 10K 600 GB SAS disks used in this solution provide average IOPS of 320 and bandwidth of 20 MB/disk. Bandwidth can be calculated for a given I/O size with the IOPS:
Bandwidth = Average I/O size X IOPS
• Calculate the building block storage requirement. With a DSS workload of read only I/O, on a RAID 5 ( 4+1) configuration, the 10K 600 GB SAS disks needed would be:
Disk number = required bandwidth / bandwidth per disk
For the 500 GB building block with bandwidth of 100 MB/s, five 10 K 600 GB SAS disks would be required.
For a 1 TB database building block with 200 MB/s bandwidth, 10 disks would be required.
For a 2 TB building block with 400 MB/s bandwidth, 20 disks would be required.
Virtual machine scalability
For virtual machine scalability, consider the following design:
• Virtual machine resources including vCPU and memory allocation should be part of the building block.
• The building block should be able to scale up (adding a block into the same virtual machine) and scale out (adding the building block to another virtual machine), all with minimum performance degradation.
Overview
Building-block design considerations
13 SQL Server 2012 Data Warehouse EMC VNX5500 HB (R32), VMware vSphere 5, Microsoft Windows 2012
Sufficient resources
For data warehouse building blocks, consider the following resource design:
• Sufficient disk utilization: The building block should be designed to sufficiently use the disk resource and leave room for any possible peak disk activities.
• Sufficient memory utilization: The building block should be designed with sufficient system memory to support the designed work load with anticipated peak load activities.
• Sufficient processor utilization: The building block is designed to have sufficient vCPU to support the designed work load and any anticipated peak load activities.
• Sufficient Tempdb design: The building block should be designed with sufficient capacity and performance to support the query workload of the database. The DSS workload has relative high demand for tempdb. For detailed information, see the section Tempdb consideration for DW/DSS-like workload.
Balanced disk utilization
Build database LUNs across as many buses as possible to avoid an unbalanced workload. This allocation can have potential high-availability benefits.
All building blocks listed here are designed for the following:
• The targeted bandwidth is 100 MB/s per LUN (RAID 5 (4+1) 10K 600 GB SAS disks). Use more disks for the building block if you want higher bandwidth.
• In this solution, the database size scaled for each LUN (RAID 5 (4+1)) is 500 GB. For a 1 TB database, two LUNs (10 disks) are created for the database files. For a 2 TB database, four LUNs (20 disks) are created.
• To support the targeted bandwidth of 100MB/s per LUN, a minimum of 2 vCPU and 8GB memory should be assigned in proportion to the building block. Or, 1 vCPU/4 GB memory per 50MB/s bandwidth.
Figure 2 shows the three building blocks we used in this solution based on this design principle. Table 2 lists the details of the three building blocks.
Building-block design details
SQL Server 2012 Data Warehouse EMC VNX5500 HB (R32), VMware vSphere 5, Microsoft Windows 2012
14
Figure 2. Three tested building blocks
Table 2. Details of the three building blocks
Building blocks 500 GB building block
1 TB building block 2 TB building block
Database size 500 GB 1 TB 2 TB
Targeted bandwidth (MB/s)
100 200 400
Database LUN design
1 x 2 TB data LUN
8 x 80 GB data file
2 x 2 TB data LUN
8 x 128 GB data file
4 x 2 TB data LUN
16 x 128 GB data file
Log design 1 log LUN
Log file size: 5 GB
1 log LUN
Log file size: 12 GB
1 log LUN
Log file size: 12 GB
Tempdb design 1 data LUN (1 x 100 GB data file)
1 Log LUN (2 GB log file )
1 data LUN (2 x 100 GB data file)
1 Log LUN (2 GB log file)
1 data LUN (4 x 100 GB data file)
1 Log LUN (2 GB log file)
Disk configuration 5 SAS disks 10 SAS disks 20 SAS disks
Memory ( GB) 8 16 32
vCPU (2.4 GHz) 2 4 8
15 SQL Server 2012 Data Warehouse EMC VNX5500 HB (R32), VMware vSphere 5, Microsoft Windows 2012
Note Log LUNs in data warehouse environment are not heavily utilized, so multiple building blocks on the same VM could potentially share the same log LUN.
Table 3 defines the minimum requirement for building blocks and virtual machine in this solution.
Table 3. Building-block design
Per virtual machine Memory (GB) CPU ( core) Number of disks RAID 5 (4+1)
Min. 16 4 5
/TB 16 4 10
/(100 MB/s) 8 2 5
500 GB ( 100 MB/s) 8 2 5
1TB( 200 MB/s) 16 4 10
2 TB(400 MB/s) 32 8 20
4 TB(800 MB/s) 64 16 40
6 TB(800 MB/s) 96 24 60
Figure 3 shows two ways to deploy building blocks:
1. Scale-up design: Put the building blocks on the same virtual machine.
2. Scale-out design: Put each building block into a different virtual machine.
Figure 3. Scale-up and scale-out building blocks
Deploying a building block
SQL Server 2012 Data Warehouse EMC VNX5500 HB (R32), VMware vSphere 5, Microsoft Windows 2012
16
Scale-up design
The scale-up design can potentially save the license cost for OS.
Because the vCPUs and memory resources are built into the building-block design, these resources will grow proportionally with the building-block deployment onto the same virtual machine to support the needs for that building block.
Scale-out design
In a scale-out deployment of building blocks, the vCPU and memory capacity of the ESXi server need to be considered to support the number of building blocks desired.
The system resources, such as vCPU and memory, are within the building block. Adding a building block in a scale-up (to the same virtual machine) or a scale-out (to a separate virtual machine) design requires the same resources unless built below the minimum requirement for a virtual machine. Thus, when having small building blocks (such as a database of less than 1 TB with 200 MB/s bandwidth), it is better to use the scale-up model to reduce waste of the virtual machine system resources, such as memory and CPU.
In a midsized environment such as this solution, organizations should carefully consider which approach best suits them.
Figure 4 and Table 4 show the virtual machine configuration and disk assignment for different databases used in this solution. We tried various ways to design and deploy a building block with reasonable performance.
SQL Server virtual machine and LUN allocation design
17 SQL Server 2012 Data Warehouse EMC VNX5500 HB (R32), VMware vSphere 5, Microsoft Windows 2012
Figure 4. Solution Building block deployment
Table 4. Solution building-block deployment on the virtual machine and ESXi server
Host Allocation CPU (core)
Memory (GB)
Database size (TB)
Number of disks
Bandwidth (MB/s)
ESXi 1 (40 cores, 256 GB RAM)
Virtual machine 1
16 64 2 20 400
Virtual machine 2
8 32 2 20 400
ESXi 2 (80 cores, 512 GB RAM)
Virtual machine 3
24 96 2 20 400
2 20 400
1 10 200
0.5 5 100
0.5 5 100
Virtual machine 4
24 96 2 20 400
SQL Server 2012 Data Warehouse EMC VNX5500 HB (R32), VMware vSphere 5, Microsoft Windows 2012
18
Host Allocation CPU (core)
Memory (GB)
Database size (TB)
Number of disks
Bandwidth (MB/s)
Total 4 virtual machines
72 288 18 180 3600
Note In one of the virtual machines, five databases with various size and
bandwidth requirements were all running heavy workload at the same time. Thus, they would compete for resources and might have contentions at peak time. Even in this case, the SQL Server in the virtual machine still maintained a reasonable performance.
Table 5 lists the hardware resources used in this solution.
Table 5. Hardware resources
Purpose Item Details Configuration
Server
ESXi server 1
CPU 2 x 10 core Intel Xeon Processors Model X2870 (2.40 GHz, 30 Mb cache, 130W )
Number cores 20
PCI-E slots 7 x PCIe 2.0 slots
Internal drives 2 x 600 GB 10k rpm SAS disk drives
Storage controller
1 LSI MegaRAID SAS 9261-8i
Host bus adapters
4 x Qlogic QLe8152 dual-port 10 Gb converged network adaptors
Network adapters
• 2 x Gigabit Ethernet LAN-on-motherboard (LOM) ports
• 2 x 10 Gigabit Ethernet ports
• 2 x dedicated out-of-band (OOB) management ports
RAM 256 GB PC2-5300FBD
Memory slot 32 GB DDR3 DIMMs with 64 DIMM slots
ESXi server 2
CPU 4 x 10 core Intel Xeon Processors Model X4870(2.40 GHz, 30Mb cache, 130W )
Number cores 40
PCI-E slots 10, 8 G2 and 2 G1
Internal drives 2 x 300 GB 10k-RPM SAS disk drives
Storage controller
1 Smart Array P410/1GB FBWC
Hardware resources
19 SQL Server 2012 Data Warehouse EMC VNX5500 HB (R32), VMware vSphere 5, Microsoft Windows 2012
Purpose Item Details Configuration
Host bus adapters
8 x Qlogic QLe8152 Dual Port 10 Gb Converged Network Adaptors
Network adapters
1 x 1GbE NC3751 Multifunction 4 ports
RAM 256 GB PC2-5300FBD
Memory slot 32 GB DDR3 DIMMs with 64 DIMM slots
Storage
EMC VNX5500 HB
Disk drives 216 SFF 600 GB 10K rpm SAS disk drives
Addition module
1 UltraFlex I/O SAS back-end module
Data protection
RAID 10 for data and tempdb, RAID 10 for log
Connectivity
Switch Cisco Nexus 5548P Switch
Protocol Fibre Channel (FC) from Server to Switch and from Switch to Storage
Table 6 lists the software resources used in this solution.
Table 6. Software resources
Software Version
EMC Unisphere® 1.2.0.1
EMC PowerPath/VE for VMware 5.7
Microsoft SQL Server 2012 Enterprise Edition
11.00.2100
EMC Solutions Enabler 7.4.0
EMC ESI 2.0
VMware vSphere 5 (Enterprise Plus) 5.1
EMC VSI 5.3
Microsoft Windows Server 2012 RTM
Benchmark Factory 5.8.1
For data warehouse storage implementation, the general recommendation for SP Cache is to dedicate 2,000 MB of memory from the storage processors to support host write caching. 500 MB of memory is allocated to support read-ahead caching. This cache configuration can provide performance improvements to write operations during data load window for data warehouse.
Figure 5 shows the memory allocation used in this solution.
Software resources
EMC VNX5500 HB storage system cache settings
SQL Server 2012 Data Warehouse EMC VNX5500 HB (R32), VMware vSphere 5, Microsoft Windows 2012
20
Figure 5. Memory allocation
Figure 6 shows the cache configuration used in this solution.
Figure 6. Cache configuration
Figure 7 shows a summary of the connections for each port on the two VNX5500 HB arrays to the Emulex controllers located in the ESXi servers. Eight dual-port CNA cards are on one of the servers. Four dual-port CNA cards are on the other server.
Connectivity
21 SQL Server 2012 Data Warehouse EMC VNX5500 HB (R32), VMware vSphere 5, Microsoft Windows 2012
Figure 7. Connectivity
Figure 8 illustrates the disk layout for this solution including:
• Disk configuration for 500 GB, 1 TB, and 2 TB building blocks for the Microsoft SQL Server Decision Support System (DSS) LUNs
• Disk configuration for the tempdb, hot spare OS LUNs, and database log file LUNs
The configuration design was based on the following principles:
• Distribute the data LUNs across as many Disk Array Enclosure (DAE) buses as possible
• Assign LUNs to SP A and SP B evenly to distribute the workload across the two SPs
Figure 8. Disk layout for this solution
We configured disk drives within the EMC VNX5500 HB storage system according to how they were used within the data warehouse system.
Storage System: We reserved four SAS drives for VNX storage system (FLAIR) use, shown as “Vault disks” in Figure 8.
Disk layout and SQL Server LUN configuration
SQL Server 2012 Data Warehouse EMC VNX5500 HB (R32), VMware vSphere 5, Microsoft Windows 2012
22
Hot Spares: We reserved six SAS drives as hot spares, shown as “HS” in Figure 8, which were ready to be automatically “switched in” to immediately rebuild any of the data drives in any of the RAID 5 groups. This minimizes the chance of data loss resulting from drive failures in one of the data LUNs.
OS for VMs: We reserved five SAS drives as RAID 5 (4+1) for the OS LUNs for SQL Server and management. These virtual machines are shown as “OS (VMs)” in Figure 8.
Tempdb: We used the tempdb for the temporary data and intermediate running results for the large volume queries. In this solution, we used 26 drives in the form of a 13+13 RAID 10 RAID group for the tempdb and log, shown as “TEMPDB” in Figure 8. The tempdb LUNs are designed for small, random I/O with high performance requirements.
Data LUNS: With the building-block design, we used 180 SAS drives in total for the database files.
To support 512 K large element size in VNX, we created all LUNs from the RAID 5 (4+1) group. The RAID 5 configuration consists of four data drives protected by one parity drive. Each 2,028 K chunk of database data stored on one of the data LUNs is striped in 512 K increments across four of the five drives in each RAID set. On the fifth drive, 512 K of parity information is computed and stored based on the data on the other four drives.
In this solution, we organized these 180 SAS drives into RAID 5 (4+1) sets, known as RAID groups in VNX terminology, colored blue, purple, light orange, and blue, and shown as “500 GB”, “1 TB”, and “2 TB” in Figure 8. All database building blocks follow the same layout approach within the array.
Transaction Log LUNs: The I/O pattern of the DSS workload is purely read. We considered the capacity only when designing the transaction log LUNs, but RAID 10 may benefit data loading in a DSS. In this solution, we used four SAS drives in the form of a 2+2 RAID 10 group for the transaction log, shown as “LOG” in Figure 8.
We used EMC PowerPath/VE version 5.7 for ESXi server as the host device multipathing management software. Under Windows Disk Manager, the EMC PowerPath service presents 24 uniquely identified data volumes. Figure 9 shows the multipathing for all data LUNs in EMC PowerPath.
Figure 9. Multipathing for all data LUNs
Host side I/O device and multipathing considerations
23 SQL Server 2012 Data Warehouse EMC VNX5500 HB (R32), VMware vSphere 5, Microsoft Windows 2012
Figure 10 shows the multipathing support across HBAs in EMC PowerPath in the PowerPath Viewer.
Figure 10. Multipathing support across the HBAs
For each of the 32 data drives configured, EMC PowerPath policies select one of the eight possible host-to-storage paths as the “preferred” path (shown as green under the Path column). In essence, each of the eight host-to-storage paths is assigned three of the 32 data LUNs and configured as the “primary access I/O channel” for those LUNs.
Operating system and SQL Server instance settings
In Windows 2012, the following settings were configured to support SQL Server 2012 DSS and OLAP workloads:
• Formatted all data and log LUNs using a 64 K allocation unit size.
• Used mount points for the data and log LUNs.
SQL Server 2012 startup options
The following options are recommended for data warehouse workload of SQL Server 2012:
• Added -E to the startup options. This increases the number of contiguous extents in each file allocated to a database table as it grows and improves sequential disk access.
SQL Server 2012 and Windows 2012 considerations for DSS
SQL Server 2012 Data Warehouse EMC VNX5500 HB (R32), VMware vSphere 5, Microsoft Windows 2012
24
For more information about the -E option, refer to Microsoft Knowledge Base Article 329526 File allocation extension in SQL Server 2000 (64-bit) and SQL Server 2005.
• Added -T1117 to the startup options. This trace flag ensures even growth of all files in a file group when autogrow option is enabled.
• Used the Lock pages in memory option for the SQL Server instances.
• Used -T834 startup parameter for SQL Server instance. It can improve throughput rates for many data warehouse workloads. This flag enables large-page allocations in memory for the SQL Server buffer pool.
Note The trace flags that this article describes are advanced tuning techniques. You should consider using these trace flags only after you do more basic and routine optimizations. For example, you should consider using these trace flags after you do the following optimizations.
SQL Server memory configuration
Do not reserve all the memory to the SQL Server instance; allocate the 2-6 GB memory to the OS.
Database configurations
For user databases, we:
• Used multiple data files for user database
• Disabled autogrow option and manually grew all data files
For tempdb, we:
• Pre-allocated space to ensure the same size for all files
• Put the temp log files on a dedicated LUN within the same pool
• Disabled the autogrow feature
We followed SQL Server best practices for database and tempdb sizing considerations. For more information, refer to the online SQL Server book entitled Capacity Planning for Tempdb.
ESXi HBA queue depth configuration
In this configuration, we set the queue depth to 256 in ESXi servers to achieve a maximum sequential bandwidth. For more information, refer to the VMware Knowledge Base article Changing the queue depth for QLogic and Emulex HBAs.
VMware vSphere configuration
VMware vCenter Server provides a scalable and extensible platform to centrally manage VMware vSphere environments, providing control and visibility at every level of the virtual infrastructure.
VMware virtual machine configuration
Two ESXi5 hosts were configured on the VNX5500 HB array, each with two virtual machines.
VMware configuration
25 SQL Server 2012 Data Warehouse EMC VNX5500 HB (R32), VMware vSphere 5, Microsoft Windows 2012
Table 7 shows the CPU and memory allocation for each SQL Server virtual machine. The CPU and memory allocation based on the approach that eight CPU+32 GB memory is designed to support 2 TB DSS performance. Therefore, 16 CPU+ 64 GB memory will support 4 TB DSS performance in total. Similarly, 24 CPU+96 GB memory will support 6 TB DSS performance in total.
Table 7. CPU and memory allocation
ESXi host Virtual machine VCPU number RAM
Server1 SQLTPCH01 8 32 GB
SQLTPCH02 16 64 GB
Server2 SQLTPCH03 24 96 GB
SQLTPCH04 24 96 GB
SQL Server 2012 Data Warehouse EMC VNX5500 HB (R32), VMware vSphere 5, Microsoft Windows 2012
26
EMC ESI v2.1 storage provisioning
ESI greatly simplifies the management, view, and provisioning of EMC storage in a Hyper-V/VMware environment. As part of the storage provisioning, ESI simplifies the various steps involved in creating a LUN, attaching the LUNs to the virtual machines.
ESI provides simple provisioning of LUNs to EMC storage in a virtualized environment. After the VMware Virtual Center Server is configured to ESI, ESI provides insight into the virtual machine disk resources on storage for all the ESXi hosts, data stores, SAN initiators, and virtual machines. Figure 11 shows the Virtual Center Server and VNX storage array across the entire solution environment.
Figure 11. ESI GUI
In this solution, ESI was used to add VNX storage and Virtual Center Server as server storage objects. New LUNs are created from VNX and attached to VMware virtual machines as RDM devices through ESI.
Objective
ESI provisioning
27 SQL Server 2012 Data Warehouse EMC VNX5500 HB (R32), VMware vSphere 5, Microsoft Windows 2012
We used the following steps to provision VNX LUNs for the vSphere virtual machines:
1. Add the storage array to ESI, and input the array connection and authentication information as shown in Figure 12.
Figure 12. Adding a storage in ESI
2. Add the VMware Virtual Center Server, and input the virtual center server connection and authentication information as shown in Figure 13.
SQL Server 2012 Data Warehouse EMC VNX5500 HB (R32), VMware vSphere 5, Microsoft Windows 2012
28
Figure 13. Adding a VMware virtual center server in ESI
3. Assign the LUN to the ESXi host. Figure 14 shows how a 10 GB LUN is connected to the ESXi host.
Figure 14. Assign LUNS to ESXi host in ESI
4. Create and provision LUNs to the VMware virtual machine with a wizard, which can automatically attach a RDM or VMDK to the virtual machine as shown in Figure 15.
29 SQL Server 2012 Data Warehouse EMC VNX5500 HB (R32), VMware vSphere 5, Microsoft Windows 2012
Figure 15. LUN creation and provisioning to virtual machine
5. Once a LUN is created, you can access the LUN in the virtual machine, as shown in Figure 16.
SQL Server 2012 Data Warehouse EMC VNX5500 HB (R32), VMware vSphere 5, Microsoft Windows 2012
30
Figure 16. LUN displayed in virtual machine after configuration in ESI
31 SQL Server 2012 Data Warehouse EMC VNX5500 HB (R32), VMware vSphere 5, Microsoft Windows 2012
High bandwidth read validation
This section describes how we tested the high bandwidth read in this solution environment. Each test scenario is described in more detail in later sections.
The following test validates the benefit of the large element size (up to 512k).
The SQLIO stress tool is used to emulate a typical I/O size in the DSS and data warehouse workloads. The test LUN is on five disks configured in a RAID 5 group.
As shown in Table 8, a 512 K element size with RDM disk type provided better performance for all the tested I/O sizes. The advantage is more visible with a large I/O size.
• When the I/O size is 512 K, the large stripe size configuration supported by the HB configuration can double the bandwidth to close to 400 MB/s for a five-disk RAID 5 (4+1) group.
• The VNX array with 180 SAS drives with 36 RAID 5 (4+1) groups: such RAID groups for the database LUNs could support up to 14,400 MB/s with sufficient system resource for SQL Server.
Table 8. SQLIO test results for RDM disks
Workload I/O size 64 K 128 K 512 K
VNX element size 64 K 512 K 64 K 512 K 64 K 512 K
Bandwidth (MB/s per LUN)
101 109 103 178 185 391
As shown in Figure 17, the wide-striped LUN with a 512 K stripe size can support almost double the bandwidth with an I/O size larger than 128 K.
Figure 17. Bandwidth doubled with a large I/O size in a wide-striped LUN
050
100150200250300350400
64K IO 128K IO 512K IO
Band
wid
th (
MB/
s)
Average I/O size
SQLIO test average I/O size vs. bandwidth
64k VNX element size
512k VNX element size
Overview
Disk performance validation
SQL Server 2012 Data Warehouse EMC VNX5500 HB (R32), VMware vSphere 5, Microsoft Windows 2012
32
We created all user database LUNs using a:
• RAID 5 group LUN
• Physical RDM disk type
We used the Quest Benchmark Factory workload to emulate actual DSS queries. Data warehouses need to support a broad range of queries from a wide-ranging audience, such as finance, marketing, operations, and research teams. Data warehousing can involve much larger data requests and benefit greatly from the increased total throughput potential of sequential disk scans.
The test environment is built for the SQL Server 2012 DSS application using Quest Benchmark Factory with a DW/DSS-like workload. It models the analysis of the business environment where trends are computed and refined data is produced to support sound business decisions.
The typical DW/DSS-like workload has an element size of 64 K. However, if the I/O size is larger, the bandwidth can be much higher. This section validates that the VNX5500 HB can support much a higher bandwidth when the I/O size is increased to 512 K.
The following test validates the benefit of the large element size with a DSS/DW workload created by Quest Benchmark Factory.
In our testing, when the columnstore index (refer to the Columnstore Indexes ) was placed on the DSS database, the same queries generated a large size I/O greater than the typical 64 K. By placing the columnstore index on a majority of the tables in a 500 GB database, we observed a sustainable 500 K element size in this test.
We conducted the test on a 500 GB database, created with a database LUN on a RAID 5 (4+1) storage group configured on 10K 600 GB SAS disks, using an RDM disk type with a 512 K element size on VNX5500 HB storage on a SQL Server virtual machine.
As shown in Figure 18, when the database is operating under a workload that generates sustainable I/O with 500 K, the bandwidth can reach approximately 350 MB/s for a single database LUN. The result is very close to that of using SQLIO tests on the same configuration with 512 K I/O size or about 70 MB/s per disk.
Note This validation used Columstore index, which resulted in larger I/O sizes. A traditional DSS application I/O size is 64 K.
Application workloads
Data warehouse performance validation
33 SQL Server 2012 Data Warehouse EMC VNX5500 HB (R32), VMware vSphere 5, Microsoft Windows 2012
Figure 18. Bandwidth doubled with a large I/O size in a wide-striped LUN
050100150200250300350400450500550
050
100150200250300350400450500550
16:03 16:36 17:09 17:42 18:16 18:49 19:22 19:56 20:29 21:02 21:36 22:09
I/O
Siz
e (K
)
Band
wid
th( M
B/s)
High bandwidth for large element size in data warehouse environment
Disk IO size (KB) Disk Read Bandwidth( MB/s)
SQL Server 2012 Data Warehouse EMC VNX5500 HB (R32), VMware vSphere 5, Microsoft Windows 2012
34
Data warehouse building-block validation
This section describes the application testing for the building blocks in this solution environment. Each test scenario is described in more detail in later sections.
Notes
• Benchmark results are highly dependent on workload, specific application requirements, and system design and implementation. Relative system performance will vary as a result of these and other factors. Therefore, this workload should not be used as a substitute for a specific customer application benchmark when critical capacity planning and/or product evaluation decisions are contemplated.
• The environment was rigorously controlled, and results obtained in other operating environments may vary significantly.
• EMC Corporation does not warrant that a user can or will achieve performances similar to these.
To determine the amount of bandwidth it takes SQL Server to handle DSS workloads, we recorded SQL Server throughput for client requests. To monitor the disk I/O activities, we tracked the following performance monitor counters on SQL Server:
• Disk bytes/sec
• Average CPU utilization (%)
From a SQL Server client perspective, we validated each test by comparing the results of select performance counters to the allowed Microsoft-specified criteria. We collected performance counter data at 10-second intervals for the duration of each test run.
The bandwidth and storage processor utilization were tracked during the test:
• Total bandwidth (MB/s)
• Storage processor utilization (%)
Table 9 lists the key performance counters and their expected values.
Table 9. Expected performance counters
Key performance indicators Explanation Expected value
Average Bandwidth or Disk Bytes/sec
Average SQL Server or storage supported bandwidth within the whole test run lifecycle, but exclude the post tasks of DW/DSS-like applications
As much as possible
Sustainable Bandwidth or Disk Bytes/sec
Average SQL Server or storage supported bandwidth of minimal tempdb activity
As much as possible
Overview
SQL Server key performance indicators
35 SQL Server 2012 Data Warehouse EMC VNX5500 HB (R32), VMware vSphere 5, Microsoft Windows 2012
Key performance indicators Explanation Expected value
Test duration Use (hour: minute: second) to measure the test duration, which is the total execution time of the 22 queries of DW/DSS-like application
As little as possible. In this solution, we targeted around 12 hours of execution duration for the test
Average virtual machine CPU utilization (%)
Average SQL virtual machine CPU utilization
Less than 75%, for a long period of time
Average VNX SP utilization (%) Storage Processor utilization Less than 70%, for a long period of time
The DSS application has limited long running queries (22 queries in total from Q1 to Q22), each query taking minutes, or even hours, to finish. The query performance depends on the database scale, server computing capacity, and memory and storage performance.
This section validates:
• The typical DSS workload pattern
• The disk and performance scalability under the same workload
A typical DSS application I/O pattern is 64 K, with a 100 percent read ratio on the data LUNs, as shown in Table 10.
Table 10. DSS query profile
DW/DSS query profile Value
Typical data LUN I/O size 64 K
Read:write ratio 100:00
Table 11 lists the two different 2 TB building block profiles used in this workload scale test.
Table 11. Building-block design profile for different building block profiles
Profile Database size (TB)
Disk number
Disk type
Virtual machine /SQL RAM(GB)
vCPU Virtual machine number
Concurrent user number
W1 2 20 SAS 32/28 8 1 1
W2 2 40 SAS 32/28 8 1 1
Typical DSS workload in profile W1
The typical DSS queries issued by application generally consume a lot of bandwidth. When using the profile W1 to test the workload performance, we observed the work load on the database and obtained a sustainable bandwidth of 400 MB/s, as shown in Figure 19.
Workload scale validation
SQL Server 2012 Data Warehouse EMC VNX5500 HB (R32), VMware vSphere 5, Microsoft Windows 2012
36
Figure 19. Bandwidth test for 2 TB database with 20 disks
Typical DSS workload in profile W2
When increasing the disk number to 40, for the same 2 TB database in a virtual machine with the same system resources (memory, CPU, and tempdb), the sustainable bandwidth is more than doubled, as shown in Figure 20. This proved that our initial design principle (design for bandwidth performance, not for database size) is critical to successful implementation of a DSS building-block design.
Figure 20. Bandwidth test for 2 TB database with 40 disks
37 SQL Server 2012 Data Warehouse EMC VNX5500 HB (R32), VMware vSphere 5, Microsoft Windows 2012
The purpose of this scale-out test was to validate that the performance of a building block can be linearly incremented when running on different virtual machines and SQL server instances, with no resource contention at the OS level, such as CPU and memory.
This test used the database block of the same size (2 TB block) to validate bandwidth increments when running that database on its own virtual machine and SQL server instance. We targeted each block for a sustainable bandwidth of 400 MB/s, configured with the following:
• Same database size: 2 TB
• Same number of disks: 20
• Same virtual machine configuration: 8 vCPUs, 32 GB of RAM
We designed the following scenarios to test the building-block approach:
• Scenario 1: Start the workload for a 2 TB database on each instance for all three different virtual machines, with a total 6 TB of workload running simultaneously to measure the aggregate bandwidth.
• Scenario 2: Start a 2 TB workload on one server, and add another 2 TB workload on the second server. Then add another 2 TB workload on the third virtual machine, until a total of 6 TB workload is reached, to measure the performance impact on the VNX Storage Processor (SP).
Note We ran all tests as separate workloads on separate SQL Server virtual machines, and all SQL Server tempdbs shared the same disks on the VNX array. Therefore, the performance impact of tempdb to the test was identical. The performance impact of tempdb is discussed in the section entitled Tempdb consideration for DW/DSS-like workload.
Table 12 shows the test profile details. In this test, we used the same building blocks of 2 TB with a 400 MB/s expected bandwidth.
Table 12. 2 TB building block profile details for scale-out testing
Building block
Database size (TB)
Disk number
Disk type
Virtual machine /SQL RAM(GB)
VCPU Database number
Virtual machine number
S1 2 20 SAS 32/28 8 1 1
S2 2 20 SAS 32/28 8 1 1
S3 2 20 SAS 32/28 8 1 1
Scale out with building blocks
SQL Server 2012 Data Warehouse EMC VNX5500 HB (R32), VMware vSphere 5, Microsoft Windows 2012
38
Table 13 shows the bandwidth aggregation of the three test profiles.
• S1: Run DW/DSS-like tests on one 2 TB block on the virtual machine. This is the baseline of the 2 TB building block, which gives a sustainable 400 MB/s bandwidth.
• S1+S2: Run DW/DSS-like tests on two 2 TB blocks on two separate virtual machines, which gives a sustainable 800 MB/s bandwidth.
• S1+S2+S3: Run three 2 TB blocks on separate virtual machines, which gives a sustainable 1,200 MB/s bandwidth.
Table 13. Linear growth of bandwidth and storage utilization in 2 TB building block scale-out testing
Profile Sustainable bandwidth (MB/s)
Maximum bandwidth
(MB/s) Average VNX SP utilization (%)
S1 403 645 11.8
S1+S2 810 1,301 19.2
S1+S2+S3 1,213 1,910 25.7
Figure 21 shows the bandwidth aggregation of the three building blocks. Scale-out with a single-user database on different virtual machines shows linear performance scalability. All resources (including disks, CPU and memory, and virtual machine) are separated in this solution. Therefore, the performance increments can be linear.
Figure 21. Scale out with 2 TB building blocks with linear bandwidth growth
Table 14 shows the test duration for the three building blocks and CPU utilization of each virtual machine. As the building block is designed with the CPU and memory, when it scales up, the CPU utilization is very close. There is almost no change after adding another building block into the environment. All instances finished the workload within the 12-hour window.
39 SQL Server 2012 Data Warehouse EMC VNX5500 HB (R32), VMware vSphere 5, Microsoft Windows 2012
Table 14. DW/DSS-like test duration and CPU utilization for the three building blocks
Profile Test duration (HH:MM:SS) CPU utilization (%)
S1 10:38:46 50.20%
S2 10:33:51 50.70%
S3 11:40:20 45.80%
Figure 22 shows that the three building blocks have similar test durations and CPU utilization.
Figure 22. Test duration and CPU utilization for building blocks in scale-out testing
Cumulative bandwidth and storage utilization
As shown in Figure 23, the average VNX storage processor utilization was linear after adding each building block into the data warehouse environment to scale out.
Figure 23. VNX storage utilization during scale-out testing
SQL Server 2012 Data Warehouse EMC VNX5500 HB (R32), VMware vSphere 5, Microsoft Windows 2012
40
The total bandwidth of the VNX array also had a very linear increment when adding building blocks to the environment to scale out, as shown in Figure 24.
Figure 24. VNX storage total bandwidth during scale out testing
The purpose of the scale-up test was to validate that the building block performance can be nearly linearly incremented with multiple databases running on the same virtual machine and SQL Server instance, with shared SQL Server resources such as tempdb.
This test demonstrates that we can deploy building blocks with a heavy load to the same SQL Server virtual machine to achieve a scalable aggregate throughput.
We designed the following scenarios to test the building block approach:
• Scenario 1: This scenario has three independent run tests:
Ran one 2 TB database on each instance
Ran two 2 TB databases on each instance
Ran three 2 TB databases on each instance
• Scenario 2: First we ran a 2 TB workload on one server, then added another 2 TB workload onto the same server, and then added a third 2 TB workload for a total of 6 TB workload to measure the performance impact on the VNX SP.
Bandwidth scalability
The validation test was based on the 2 TB building blocks. Table 15 lists the profile of the different virtual machines built with different number of building blocks. Each building block is of the same size (2 TB/32 GB RAM/8 vCPU).
Table 15. Virtual machine profile for the building blocks in scale-up testing
Profile Database size (TB)
Building block number
Disk number
Disk type
Virtual machine /SQL RAM (GB)
vCPU Virtual machine number
Scale up with building blocks
41 SQL Server 2012 Data Warehouse EMC VNX5500 HB (R32), VMware vSphere 5, Microsoft Windows 2012
Profile Database size (TB)
Building block number
Disk number
Disk type
Virtual machine /SQL RAM (GB)
vCPU Virtual machine number
M1 2 1 20 SAS 32/28 8 1
M2 2 2 40 SAS 64/58 16 1
M3 2 3 60 SAS 96/86 24 1
Table 16 shows the bandwidth scalability for scale-up deployment.
Table 16. Near-linear growth with 2 TB building blocks in scale-up testing
Profile Sustainable bandwidth (MB/s) Maximum bandwidth (MB/s)
M1 406 602
M2 785 1,164
M3 1,098 1,614
The bandwidth increment was nearly linear when we increased the building block from one 2 TB block (M1) to three 2 TB building blocks (M3) on a virtual machine, as shown in Figure 25.
Figure 25. Nearly linear bandwidth when scaling up with 2TB building blocks
In the scale-up testing, the DW/DSS-like test duration and the average CPU utilization were maintained while we proportionally increased the virtual machine resource (vCPU and RAM) and disk numbers, as shown in Table 17 and Figure 26.
Table 17. DW/DSS-like test duration and CPU utilization for the three building blocks
Profile Test duration (HH:MM:SS) CPU utilization (%)
M1 10:20:08 43.60
SQL Server 2012 Data Warehouse EMC VNX5500 HB (R32), VMware vSphere 5, Microsoft Windows 2012
42
Profile Test duration (HH:MM:SS) CPU utilization (%)
M2 10:42:45 42.20
M3 11:31:31 43.80
Figure 26. Test duration and CPU utilization for scale-up testing
Cumulative bandwidth and storage utilization
As shown in Figure 27, the average VNX storage processor utilization increased nearly linearly when we added each building block into the data warehouse environment to scale up.
Figure 27. VNX storage utilization when adding building blocks
The total bandwidth of the VNX array also had near-linear growth when we added the building blocks to the environment to scale up, as shown in Figure 28. In this case, the virtual machine with only one building block (M1) is running with a sustainable
43 SQL Server 2012 Data Warehouse EMC VNX5500 HB (R32), VMware vSphere 5, Microsoft Windows 2012
bandwidth of 400 MB/s, and M2 is running with two building blocks with 800 MB/s, while M3 with three building blocks is running with 1,200 MB/s.
Figure 28. VNX storage total bandwidth during scale-up testing
We performed the consolidated test with all 11 building blocks all under the maximum workload at the same time. The total 18 TB database capacity ran with 11 client connections on four virtual machines and SQL server instances as shown in Figure 29.
Figure 29. Consolidated test with building blocks
Table 18 shows the performance result for the consolidated test
Table 18. Consolidated test results
SQL VM
vCPU Memory/SQL allocated
Database size
Test duration (HH:MM:SS)
Average bandwidth (MB/s)
Sustainable bandwidth (MB/s)
Maximum bandwidth (MB/s)
Average VM CPU utilization (%)
SQL01 8 32/28 2 TB 11:30:20 339 402 599 61.20
SQL02 16 64/58 2 TB 11:40:37 652 801 1,180 42.50
Consolidated test results
SQL Server 2012 Data Warehouse EMC VNX5500 HB (R32), VMware vSphere 5, Microsoft Windows 2012
44
SQL VM
vCPU Memory/SQL allocated
Database size
Test duration (HH:MM:SS)
Average bandwidth (MB/s)
Sustainable bandwidth (MB/s)
Maximum bandwidth (MB/s)
Average VM CPU utilization (%)
2 TB 11:38:53
SQL03 24 96/86 500 GB 12:39:11 783 1,196 1,596 31.70
500 GB 12:53:40
1 TB 10:02:48
2 TB 14:06:11
2 TB 12:24:49
SQL04 24 96/86 2 TB 12:36:50 885 1,205 1,566 41.60
2 TB 12:56:47
2 TB 12:04:40
Table 19 shows the aggregated bandwidth and average utilization of the ports of the SPs.
Table 19. Consolidated test results in VNX
Average bandwidth (MB/s)
Sustainable bandwidth (MB/s)
Maximum bandwidth (MB/s)
Average VNX SP utilization (%)
2,296 3,305 3,535 64
Figure 30 shows the total bandwidth in MB/s during the consolidated test for the 11 building blocks under DW/DSS-like workloads.
Figure 30. Total bandwidth on VNX for the 18 TB 11 building block consolidated test
01000200030004000
14:2
6:41
14:5
7:41
15:2
7:41
15:5
8:41
16:2
8:41
17:0
1:41
17:3
1:41
18:0
2:41
18:3
2:41
19:0
3:41
19:3
6:41
20:0
6:41
20:3
6:41
21:0
6:41
21:3
6:41
22:0
8:41
22:3
9:41
23:0
9:41
23:4
0:41
0:10
:41
0:42
:41
1:12
:41
1:43
:41
2:13
:41
2:44
:41
3:14
:41
3:47
:41
4:17
:41
Total Bandwidth(MB/sec)
Total Bandwidth(MB/sec)
45 SQL Server 2012 Data Warehouse EMC VNX5500 HB (R32), VMware vSphere 5, Microsoft Windows 2012
Tempdb is heavily utilized when running the DSS workload to sort the intermediate execution results of large volume queries. Consider the capacity and performance at the same time when designing the tempdb on a large data warehouse.
Capacity consideration
The DW/DSS-like application required large tempdb space for the data files in the sorting of the intermediate result of the queries. The observed tempdb usage can be up to 150 GB per 1 TB user database with a bandwidth of 200 MB/s. The recommended LUN space is 20 percent of the user database.
Performance considerations
In this solution, the tempdb was located on the RAID 1/0 group, consisting of 26 SAS 10k 600 GB disks, and shared by all four SQL Server 2012 instances on four different virtual machines, with a total of 3.6 TB configured capacity for the tempdb LUN to support 11 DW/DSS-like user databases of 18 TB in total.
The observed performance impact of tempdb on the DW/DSS-like workload is:
• When the tempdb activity was high, the workload was waiting for tempdb to finish processing, so the total bandwidth on database LUNs became lower.
• When the tempdb activity was low, the workload was processed immediately, so the total bandwidth on the database LUN was high and sustainable.
Figure 31 shows the tempdb IOPS of one 2 TB building block on one virtual machine with a typical workload. When the two tempdb LUNs have high activity, the bandwidth on the database LUN becomes low. When tempdb has contention on one virtual machine or the shared disks for multiple virtual machines or SQL instances, it can impact the average bandwidth and the test duration for the database.
Figure 31. Tempdb impact on the bandwidth
To measure the performance impact of the tempdb data LUN on the DW/DSS-like application test duration and average bandwidth, we designed the following scenarios as detailed in Table 20.
Tempdb consideration for DW/DSS-like workload
SQL Server 2012 Data Warehouse EMC VNX5500 HB (R32), VMware vSphere 5, Microsoft Windows 2012
46
• T1: We ran the 2 TB workload independently. All 26 SAS RAID 10 disks supported the tempdb of this single building block workload exclusively.
• T2: We ran three 2 TB workloads on one virtual machine simultaneously. The 26 SAS RAID 10 disks supported the tempdb for the single SQL server instances. The contention was on the disk level and instance level.
• T3: We ran three 2 TB workloads on three virtual machines simultaneously. The 26 SAS RAID 10 disks supported the three tempdbs for the three different SQL server instances. The contention was on the disk level only.
Table 20. Configuration profiles for tempdb testing with different building blocks
Profile Total database size (TB)
Number of database x size
Virtual machine (SQL instance number
Tempdb disk number (shared pool)
Disk type/RAID type
vCPU
T1 2 1 x 2 TB 1 26 SAS/RAID 10 8
T2 6 3 x 2 TB 1 26 SAS/RAID 10 24
T3 6 3 x 2 TB 3 26 SAS/RAID 10 24
Test results
This section shows the tests result of the performance impact of tempdb. The following performance matrices are used.
Average test duration: The average DSS test duration of multiple workloads running simultaneously on different databases. If the test has three workloads, such as T2 and T3, the value is the average duration of the three tests.
Maximum test duration: The maximum test duration of all three workloads combined. If the test has three workloads, such as in the profiles T2 and T3, the value is the maximum duration of three independent workloads.
Normalized average bandwidth: The normal average bandwidth value calculated when normalizing the total bandwidth to the 2 TB building block.
Table 21 shows the test results for different profiles built with the building blocks.
Table 21. Tempdb impact to the building blocks with different profiles
Profile Average test duration (HH:MM:SS)
Maximum test duration (HH:MM:SS)
Normal average bandwidth (MB/s) per 2 TB
T1 10:20:08 10:20:08 379
T2 10:41:13 11:28:08 366
T3 11:07:34 11:22:20 339
• The baseline T1 showed no contention. The average LUN latency was below 10 ms. As shown in Figure 32, comparing with the baseline T1, the average bandwidth of the test profiles T2, T2, and T3 decreased and the test duration
47 SQL Server 2012 Data Warehouse EMC VNX5500 HB (R32), VMware vSphere 5, Microsoft Windows 2012
increased. When running multiple databases on the same instance, the average LUN latency of tempdb is expected to increase dramatically.
• Comparing the results of profiles T2 and T3 in this test, when the same building blocks shared the same instance (T2), the instance-level contention is very similar. This indicates that multiple building blocks running on the same instance, with a scale-up or scale-out profile, have similar scalability. The scale-up model shows slightly better overall performance by using a separate tempdb for each building block because each instance has its own tempdb database.
This condition can be relieved by adding more disk resources into tempdb.
Figure 32. Tempdb performance impact to average test duration and bandwidth
SQL Server 2012 Data Warehouse EMC VNX5500 HB (R32), VMware vSphere 5, Microsoft Windows 2012
48
Conclusion
This EMC solution illustrates the implementation of a building-block design and methodology on a data warehouse in a VMware virtualized environment, hosted by VNX5500 HB storage. The solution demonstrates that:
• A large element size, 512 K, can benefit the data warehouse workload, compared with a 64 K element size.
• Data warehouse workloads can be deployed with a predictable bandwidth using a building-block design.
• Using building-block scale-out deployment to separate virtual machine and SQL Server instances provides linear scalability of bandwidth; scale-up deployment on the same virtual machine or SQL Server instance can achieve near-linear performance scalability.
• ESI provides simplified configuration and management of the storage infrastructure.
The key findings of the tests show that:
• VNX5500 HB can achieve up to 400 MB/s for data warehouse I/Os (512 K) per LUN, in a RAID 5 (4+1) configuration. With 180 SAS drives, it can scale to 14,400 MB/s, therefore providing sufficient bandwidth for SQL Server.
• The building blocks for a DSS workload on VNX5500 HB can scale linearly with a bandwidth of 100 MB/s on each of the five SAS disks RAID 5 (4+1).
• The 180 disks supported by VNX5500 HB can sustain a DSS workload of up to 3,305 MB/s bandwidth and 3,535 MB/s maximum bandwidth, with the entire solution built using a building-block approach.
Summary
Findings
49 SQL Server 2012 Data Warehouse EMC VNX5500 HB (R32), VMware vSphere 5, Microsoft Windows 2012
References
For additional information, see the white paper EMC Mission Critical Infrastructure for Microsoft SQL Server 2012.
For additional information, see the product documents listed below.
• EMC Storage Integrator for Windows Product Guide
• EMC Storage Integrator for Windows Technical Notes
• EMC VSI for VMware vSphere: Storage Viewer Product Guide
For additional information, see the documents listed below.
• SQL Server Best Practices Article
• Tuning options for SQL Server when running in high performance workloads
• Optimizing tempdb Performance
• White Paper: Cisco Reference Configurations for Microsoft SQL Server 2012 Fast Track Data Warehouse 4.0 with EMC VNX5500 Series Storage Systems
White paper
Product documentation
Other documentation