emc integrated infrastructure for vmware integrated infrastructure for vmware ... solution can be...
TRANSCRIPT
EMC Integrated Infrastructure for VMware
Enabled by EMC Celerra NS-120
Proven Solutions Guide
EMC Integrated Infrastructure for VMware Enabled by EMC Celerra NS-120 Proven Solutions Guide 2
Copyright © 2009 EMC Corporation. All rights reserved. Published June 2009 EMC believes the information in this publication is accurate as of its publication date. The information is subject to change without notice. Benchmark results are highly dependent upon workload, specific application requirements, and system design and implementation. Relative system performance will vary as a result of these and other factors. Therefore, this workload should not be used as a substitute for a specific customer application benchmark when critical capacity planning and/or product evaluation decisions are contemplated. All performance data contained in this report was obtained in a rigorously controlled environment. Results obtained in other operating environments may vary significantly. EMC Corporation does not warrant or represent that a user can or will achieve similar performance expressed in transactions per minute. No warranty of system performance or price/performance is expressed or implied in this document. Use, copying, and distribution of any EMC software described in this publication requires an applicable software license. For the most up-to-date listing of EMC product names, see EMC Corporation Trademarks on EMC.com. All other trademarks used herein are the property of their respective owners. Part number: H6272
EMC Integrated Infrastructure for VMware Enabled by EMC Celerra NS-120 Proven Solutions Guide 3
Table of Contents
About this document.................................................................................................................................5
Overview...........................................................................................................................................5 Audience and purpose......................................................................................................................6 Scope ...............................................................................................................................................7 Prerequisites and supporting documentation ...................................................................................8
Chapter 1: Solution Design.......................................................................................................................9 Overview...........................................................................................................................................9 Business challenges and technology solution overview.................................................................10 Validated solution objectives ..........................................................................................................11 Key solution components and architecture.....................................................................................13 Expansion capabilities ....................................................................................................................15 Hardware and software resources..................................................................................................16
Chapter 2: Solution Components and Configuration ..............................................................................17 Overview.........................................................................................................................................17
Section A: Solution components.........................................................................................................18 Overview.........................................................................................................................................18 Key Components ............................................................................................................................19 Server components ........................................................................................................................20 Storage components ......................................................................................................................21
Section B: iSCSI Configuration...........................................................................................................22 Overview.........................................................................................................................................22 Prerequisites for iSCSI configuration..............................................................................................23 iSCSI LUN configuration.................................................................................................................24
Section C: Cisco Catalyst 3750-E switch configuration......................................................................27 Overview.........................................................................................................................................27 Cisco Catalyst 3750-E switch overview..........................................................................................28 Cisco Catalyst 3750-E switch configuration ...................................................................................29
Section D: VMware ESX server configuration ....................................................................................31 Overview.........................................................................................................................................31 VMware ESX server considerations ...............................................................................................32 VMware ESX server installation and configuration.........................................................................33 iSCSI configuration for the VMware ESX server ............................................................................35 Adding a datastore configuration....................................................................................................37 Configuring Disk/LUN properties ....................................................................................................39
Section E: EMC Avamar configuration ...............................................................................................41 Overview.........................................................................................................................................41 EMC Avamar deduplication overview.............................................................................................42 EMC Avamar installation and configuration....................................................................................43 Customization.................................................................................................................................44
Chapter 3: Environment Administration Tools .......................................................................................45 Overview.........................................................................................................................................45 Celerra ............................................................................................................................................46 ESX 3.5 Virtual Infrastructure Environment ....................................................................................54 Avamar ...........................................................................................................................................61
Chapter 4: Basic Administration Tasks...................................................................................................62 Overview.........................................................................................................................................62 Monitoring the environment ............................................................................................................63 Configuring additional storage........................................................................................................65 VMware ESX administration tasks .................................................................................................66 Backing up and restoring guest VMs..............................................................................................67
Chapter 5: Testing and Validation .........................................................................................................68 Overview.........................................................................................................................................68
EMC Integrated Infrastructure for VMware Enabled by EMC Celerra NS-120 Proven Solutions Guide 4
Parameters .....................................................................................................................................69 Methodology ...................................................................................................................................70 Results............................................................................................................................................71 Summary ........................................................................................................................................73
Conclusion..........................................................................................................................................74 Overview.........................................................................................................................................74 References .....................................................................................................................................75
EMC Integrated Infrastructure for VMware Enabled by EMC Celerra NS-120 Proven Solutions Guide 5
About this document
Overview
Introduction The EMC commitment to consistently maintain and improve quality is led by the
Total Customer Experience (TCE) program, which is driven by Six Sigma methodologies. As a result, Global Solutions Centers (GSC) has built customer integration labs to reflect real-world deployments in which TCE use cases are developed and executed. These use cases provide EMC with an insight into the challenges its customers currently face.
Contents This chapter contains the following topics:
Topic See Page
Audience and purpose 6
Scope 7
Prerequisites and supporting documentation 8
EMC Integrated Infrastructure for VMware Enabled by EMC Celerra NS-120 Proven Solutions Guide 6
Audience and purpose
Audience This document is intended to be used by
• EMC field personnel • EMC account teams, and • customers.
Document purpose
This guide provides guidance for designing a scalable, well-performing virtualization solution on EMC® Celerra® storage with local data protection using Avamar® for data deduplication and retention.
Intended usage This guide is an EMC internal and customer facing document that is intended to be
used in the • deployment, • customization, and • ongoing support of the EMC Integrated Infrastructure For VMware solution.
Additional usage
This document is also intended to be used as a guide for • selling • designing • building, and • supporting similar solutions to implement an integrated virtualization solution.
EMC Integrated Infrastructure for VMware Enabled by EMC Celerra NS-120 Proven Solutions Guide 7
Scope
Solution scope The scope of this solution encompasses the following:
• design and build details of an integrated VMware ESX solution utilizing Celerra
storage, Dell servers, Cisco IP switches and Avamar deduplication technology
Note: Both Dell 2950 servers and Dell R710 servers are approved for use within the solution. This document refers only to Dell 2950 servers.
• customization of the environment to meet the needs of the customer applications
unique to each deployment environment • validation testing of the integrated environment using I/Ometer, and • ongoing administration of the environment after deployment and customization.
Out of scope This solution constitutes a platform on which various applications can be run. How
the solution is configured depends on the storage and I/O requirements of the applications that are to be deployed. The solution can be built using server and switch hardware that is different from the hardware that was used to validate the solution; however, such alternatives must be properly qualified by EMC. The use of CLI commands to perform solution configuration is beyond the scope of this document. The information in this document is not intended to replace existing, detailed product implementation guides or pre-sale site evaluations.
EMC Integrated Infrastructure for VMware Enabled by EMC Celerra NS-120 Proven Solutions Guide 8
Prerequisites and supporting documentation
Technology It is assumed the reader is familiar with the following EMC, VMware, Dell and Cisco
products: • Celerra storage arrays • Avamar deduplication appliances • VMware ESX and Virtual Infrastructure 3.5 • Dell 2950 servers • Cisco 3750-E Ethernet switches
Supporting documents
It is also assumed the reader has an understanding of the following documents available on EMC Powerlink, VMware.com, and Cisco.com: • EMC Celerra NS-120 System Installation Guide • EMC Avamar 4.1 Server Software Installation Manual • VMware Server User’s Guide Version 2.0 • Catalyst 3750-E Switch Getting Started Guide
EMC Integrated Infrastructure for VMware Enabled by EMC Celerra NS-120 Integration Guide 9
Chapter 1: Solution Design
Overview
Introduction This chapter provides an overview of the validated solution design and design
assumptions.
Contents This chapter contains the following topics:
Topic See Page
Business challenges and technology solution overview 10
Solution validation objectives 11
Key solution components and architecture 13
Expansion capabilities 15
Hardware and software resources 16
EMC Integrated Infrastructure for VMware Enabled by EMC Celerra NS-120 Integration Guide 10
Business challenges and technology solution overview
Business need Customers need a validated hardware and software solution to simplify the
procurement, assembly, deployment, and management of their data centers.
Better performance and scalability
Often, customers need to deliver better data center performance and scalability while reducing costs. Virtualization is an essential technology to consider, since it enables customers to maximize the return on hardware and software investments.
Multiple data centers
Many customers have more than one data center to manage, including smaller, remote branch locations. Maximizing IT investments while delivering a consistent level of service across all locations is very difficult. Disparate locations increase maintenance and service costs, since infrastructures can vary across locations.
Solution design overview
This solution uses ESX Server 3.5 on Celerra storage, with Avamar deduplication. In this solution, Dell 2950 servers and Cisco 3750-E switches are used, although the solution can be modified for other server and network hardware vendors.
The technology solution
The EMC Integrated Infrastructure for VMware is a reference blueprint for an integrated “data center in a rack.” The table below lists the solution components.
Hardware and software Custom scripts to facilitate…
Operational best practices for…
• Dell 2950 servers1 • EMC Celerra NS-120 (NAS, iSCSI, FC) with
easy virtual provisioning • VMware for simplicity and efficiency • EMC Avamar deduplication to minimize the
amount of data to back up • Infrastructure services to ensure that the
solution is tied to and is manageable from internal systems
• Baseline build and configuration
• Customer customization and deployment
• Using the integrated infrastructure
• Managing system performance
• Performing backup and recovery
1 Dell 2950 servers and Dell R710 servers are approved for use within the solution. This document refers only to Dell 2950 servers.
EMC Integrated Infrastructure for VMware Enabled by EMC Celerra NS-120 Integration Guide 11
Solution validation objectives
Define hardware and software configuration
Base the configuration of the hardware and software on five ESX servers. The physical configuration can support from two to eight ESX servers, depending on application and storage requirements.
Define a baseline configuration
Define the initial pre-customization state to prepare the solution for deployment at a customer site. Note: To efficiently support the assembly and configuration of this solution in the field, the hardware and software need to be in a consistent state when the automation scripts are executed to further customize for customer use.
Develop automation scripts
Develop scripts to automate the customization of software and hardware components in the solution when being deployed into a customer environment. Note: Several aspects of the internal components (software and hardware) of the solution will require customer-driven configuration changes (for example: switch VLAN configuration).
Develop tear-down script
Develop a PowerShell script that can quickly tear down the customized environment and return it to the pre-customization state.
Perform initial test matrix
Use the results of these tests to provide information for the build plan and project documentation. Reference: For more information, refer to Chapter 4: Testing and Validation > Parameters > Workload Profile.
Perform workload testing
Using common load-generation tools such as I/Ometer, perform workload testing on the cluster and storage configuration. Workload testing defines the performance limits of the configuration to support deployment and provide consolidation recommendations in a customer’s operating environment.
EMC Integrated Infrastructure for VMware Enabled by EMC Celerra NS-120 Integration Guide 12
Provide VM guidelines
Using the integrated Avamar Datastore, provide recommendations for Virtual Machine (VM) backup. Reference: For more information, refer to EMC Avamar installation and configuration > Backups, in this chapter.
EMC Integrated Infrastructure for VMware Enabled by EMC Celerra NS-120 Integration Guide 13
Key solution components and architecture
Key solution components
This solution includes the following components: • Dell 2950 servers2 • EMC Celerra NS-120 (NAS, iSCSI, FC) with easy virtual provisioning • VMware for simplicity and efficiency • EMC Avamar deduplication appliance to minimize the amount of data to back up • Stackable Cisco IP switches to ensure that the solution is tied to and is
manageable from internal systems
Physical architecture
The figure below illustrates the overall physical architecture of the midsize solution.
2 Dell 2950 servers and Dell R710 servers are approved for use within the solution. This document refers only to Dell 2950 servers.
EMC Integrated Infrastructure for VMware Enabled by EMC Celerra NS-120 Integration Guide 14
Logical architecture
The figure below illustrates the logical architecture of the midsize solution.
EMC Integrated Infrastructure for VMware Enabled by EMC Celerra NS-120 Integration Guide 15
Expansion capabilities
Flexible deployment
The EMC Integrated Infrastructure For VMware solution is designed to be flexible, depending on the environment in which it is deployed. The combination of servers and storage that can be deployed is vast, with the servers ranging from two to eight Dell 2950s (or similar sized servers) with varying combinations of Disk Array Enclosures (DAEs). For this solution, NS storage capacity ranges from 1.5 TB to 71.5 TB maximum, depending on drive size and type.
Fixed rack deployment
In a fixed rack deployment, the options are limited to the space in the rack. In a standard 42U rack with five servers and three DAEs (as was used in the validated solution), there exists the ability to add • up to three additional servers, or • two additional DAEs, or • any combination of the two, up to the maximum capacity of the rack.
Space allocation dependencies
Space allocation for servers and storage in the rack depends on the applications deployed in the environment. Examples of space allocation requirements: • A database application will require more disk storage than a Web server. • A Web server will likely require more servers or Virtual Machines and less disk
space, depending on content.
Disk space requirements
Great care must be taken in allocating space to each VM to ensure that the maximum benefit is achieved in the least amount of physical space. Allocating space appropriately allows for future expansion.
EMC Integrated Infrastructure for VMware Enabled by EMC Celerra NS-120 Integration Guide 16
Hardware and software resources
Hardware resources
The hardware resources used to validate the solution are listed in the table below.
Equipment Quantity Configuration
Dell 2950 Servers 5 2 socket, quad core, 32 GB RAM, 2 x 73 GB internal disks - RAID (1), 10 network interfaces (1 GbE)
NS-120 Storage Array 1 Configured to support NFS and iSCSI protocols
Celerra DAE Storage Enclosures 3 (15 / 300 GB / 15k FC drives per enclosure)
Avamar Backup Datastore 1 Single-node deployment
Cisco 3750-E Switch 2 (48 x 1 GbE ports per)
Software resources
The software resources used to validate the solution are listed in the table below.
Software Version Configuration
Avamar Backup Software 4.1 Single-node configuration
VMware ESX 3.5 Update 3
Each VMware ESX server is configured with a predetermined IP address. This address is used to boot strap the customer configuration process.
VMware Virtual Center (DRS, VHA, VMotion) 2.5
If this is a standalone implementation of VMware Virtual Infrastructure, the Virtual Center Server will be installed in a virtual machine located on one of the Dell 2950 servers.
NS-120 Management Software 5.6 Required to configure iSCSI devices for VMware ESX servers.
EMC Integrated Infrastructure for VMware Enabled by EMC Celerra NS-120 Integration Guide 17
Chapter 2: Solution Components and Configuration
Overview
Introduction The EMC Integrated Infrastructure for VMware solution was designed and built to be
deployed as a standalone high availability (HA) VMware ESX cluster, using iSCSI storage on an EMC Celerra platform. Each component of the solution − storage, network, VMware ESX servers, and Avamar deduplication − was designed according to best practices to ensure a fault-tolerant HA solution that is efficient and scalable.
Contents This chapter contains the following sections:
Topic See Page
Section A: Solution components 18
Section B: iSCSI configuration 22
Section C: Cisco Catalyst 3750-E switch configuration 27
Section D: VMware ESX server configuration 31
Section E: EMC Avamar configuration 41
EMC Integrated Infrastructure for VMware Enabled by EMC Celerra NS-120 Integration Guide 18
Section A: Solution components
Overview
Introduction The key components in this solution are Dell servers and iSCSI storage.
Contents This section contains the following topics:
Topic See Page
Key Components 19
Server components 20
Storage components 21
EMC Integrated Infrastructure for VMware Enabled by EMC Celerra NS-120 Integration Guide 19
Key components
Introduction This section briefly describes the solution's EMC and VMware components.
EMC Celerra NS-120
The EMC Celerra NS-120 brings high availability to multi-protocol environments. With the EMC Celerra NS-120, you can connect to multiple storage networks via NAS, iSCSI, and Fibre Channel SAN with an integrated package that includes dedicated EMC CLARiiON® networked storage. In this solution environment, NAS and iSCSI storage are used by the ESX servers.
VMware ESX Server 3.5
VMware ESX 3.5 is the market-leading virtualization hypervisor in use across thousands of IT environments around the world. VMware ESX abstracts server processor, memory, storage and networking resources into multiple virtual machines, forming the foundation of the VMware Infrastructure 3 suite. VMware ESX is a “bare metal” hypervisor that partitions physical servers into multiple virtual machines. Each virtual machine represents a fully functional, complete system with processors, memory, networking, storage and BIOS.
EMC Avamar deduplication
EMC Avamar backup and recovery solutions utilize patented global data deduplication technology to identify redundant data at the source, minimizing backup data before it is sent over the network. Avamar deduplicated backups function like full backups and can be recovered in just one step, without restoring full backups and subsequent incrementals. In addition, Avamar verifies backup data recoverability and encrypts data for secure electronic backups.
EMC Integrated Infrastructure for VMware Enabled by EMC Celerra NS-120 Integration Guide 20
Server components
Server hardware
The EMC Integrated Infrastructure for VMware solution utilizes Dell 2950 servers.
BIOS settings To ensure optimal performance in a virtualized environment, the following BIOS
settings are modified as follows: • Enable virtualization to allow for 64-Bit Guest Operating Systems. • I/O Acceleration Technology (I/OAT) is enabled for data acceleration. • Local disk protection is configured with RAID.
Reference For additional information regarding the Dell hardware used in this solution, see the
References section at the end of this document.
EMC Integrated Infrastructure for VMware Enabled by EMC Celerra NS-120 Integration Guide 21
Storage components
Storage components
The EMC Integrated Infrastructure for VMware solution uses: • iSCSI on an EMC Celerra NS-120 array. The EMC Celerra provides access to
block and file data using native iSCSI and NFS. • iSCSI LUNs for the VMware ESX server datastores. With the introduction of native
iSCSI client support through the ESX Server VM Kernel software initiator, the ESX server is able to access up to 255 iSCSI LUNs from one or more Celerra iSCSI targets.
• Raw Device Mapping (RDM) disks for virtual machines.
Reference Installation and configuration of the Celerra NS-120 is performed to EMC
specifications as detailed in the VMware ESX Server Using EMC Celerra Storage Systems Solutions Guide, which can be found at the following URL: http://www.emc.com/collateral/hardware/technical-documentation/h5536-vmware-esx-srvr-using-celerra-stor-sys-wp.pdf
EMC Integrated Infrastructure for VMware Enabled by EMC Celerra NS-120 Integration Guide 22
Section B: iSCSI configuration
Overview
Introduction The solution uses iSCSI LUNs for storage purposes.
Contents This section contains the following topics:
Topic See Page
Prerequisites for iSCSI configuration 23
iSCSI LUN configuration 24
EMC Integrated Infrastructure for VMware Enabled by EMC Celerra NS-120 Integration Guide 23
Prerequisites for iSCSI configuration
Requirements The configuration of iSCSI requires that VMware ESX hosts have
• an IP network connection to access the IP storage, and • the iSCSI service enabled.
Configuration steps
The following steps must be completed before the VMware ESX hosts can see the Celerra devices.
Step Action
1 Using vCenter, enable the iSCSI client in the security profile, firewall properties interface of the client.
2 Configure the iSCSI software initiator on the VMware ESX server hosts.
3
Configure iSCSI LUNs and mask them to the IQN of the software initiator defined for this VMware ESX server host. The following illustration shows an example of masking an iSCSI LUN to the IQN of the software initiator.
References See the VMware iSCSI SAN Configuration Guide ESX Server 3.5, ESX Server 3i
version 3.5 VirtualCenter 2.5 located at http://vmware.com for additional details about configuring iSCSI for the VMware ESX server.
EMC Integrated Infrastructure for VMware Enabled by EMC Celerra NS-120 Integration Guide 24
iSCSI LUN configuration
LUN configurations
The Celerra iSCSI LUNs are configured as follows: • LUNs for creating data stores for virtual machines • LUNs used as Raw Device Mapping (RDM) disks for applications deployed in the
environment • LUNs used to store OS images and VM templates and clones
iSCSI wizard The following illustration shows an example of the wizard that allows LUN
configuration.
EMC Integrated Infrastructure for VMware Enabled by EMC Celerra NS-120 Integration Guide 25
Datastore requirement
The base configuration uses three 500 gigabyte (GB) RAID 5 LUNs on the Celerra to support up to 60 Windows 2003 virtual machines on five ESX servers.
RDM disk requirement
The RDM devices can be configured as either RAID 1 or RAID 5 LUNs, depending on the requirements of the application. • Use a RAID 1 configuration for database and logging applications. RAID 1 LUNs
provide optimal performance for write operations. They are suggested for I/O profiles that are write intensive.
• Use a RAID 5 configuration for other applications, such as file server and online
transaction processing (OLTP) applications.
VMFS requirement
Use RAID 5 LUNs for the virtual machine file system (VMFS). The VMFS stores: • OS images, and • VM templates and clones.
iSCSI access requirement
The EMC Integrated Infrastructure for VMware solution is designed to utilize VMware HA and DRS. To support these features, enable Multiple Access for all VMware ESX host iSCSI initiators when configuring the LUNs on the Celerra. Use the Celerra iSCSI Wizard or Celerra command line interface to configure the LUNs.
EMC Integrated Infrastructure for VMware Enabled by EMC Celerra NS-120 Integration Guide 26
Uncached write mechanism
The following recommendations apply to the use of the Celerra uncached write mechanism. • Mounting NFS with the uncached write mechanism enabled is recommended. • If planning to use more than eight NFS mounts (the default setting) in your
deployment, update the configuration setting with the Storage Add-on script during the customization phase of deployment or utilize the Celerra Manager wizards.
Benefit: The Celerra uncached write mechanism can enhance write performance to the Celerra file systems. This mechanism allows well-formed writes (for example, writes that are disk block aligned or whose size equals a multiple of the Celerra disk block size) to be sent directly to the disk without being cached on the Celerra Data Mover.
Send and receive buffers
NFS send and receive buffer sizes need to be configured in multiples of 32 KB. In the tested solution, the following default parameters were used: • Send buffer: 256KB • Receive buffer: 128KB
References • For information about using the customization utility, see the Customization section
of this document. • For further details on the Celerra Manager wizards available for Celerra
configuration, see Chapter 3: Environment Administration Tools. • For documents used in the design and planning of the environment configuration
see the References section of this document.
EMC Integrated Infrastructure for VMware Enabled by EMC Celerra NS-120 Integration Guide 27
Section C: Cisco Catalyst 3750-E switch configuration
Overview
Introduction Cisco Catalyst 3750-E switches are used for the IP network infrastructure Access
Layer.
Contents This section contains the following topic:
Topic See Page
Cisco Catalyst 3750-E switch overview 28
Cisco Catalyst 3750-E switch configuration 29
EMC Integrated Infrastructure for VMware Enabled by EMC Celerra NS-120 Integration Guide 28
Cisco Catalyst 3750-E switch overview
Overview The Cisco Catalyst 3750-E switch is a stackable GbE switch that
• allows the combination of up to nine physical switches into a single logical switch,
which can be managed as a single switch, and • enables port channeling of interfaces across the switch chassis.
Diagram: network configuration
The diagram below shows the network configuration used in the validated solution.
EMC Integrated Infrastructure for VMware Enabled by EMC Celerra NS-120 Integration Guide 29
Cisco Catalyst 3750-E switch configuration
Cabling In the solution as validated, the ESX server cabling is spread evenly across the two
physical switches, as described below: • On ESX-01, which has three NICs on the iSCSI network, two of the NICs are
cabled to the top 3750-E switch. • The remaining NIC is cabled to the bottom 3750-E switch.
Server links Aggregating the ESX Server links to the access layer switch and configuring them in
a Port Channel Group allows for increased utilization of server resources as well as redundancy for network connections. The NIC team is configured to load balance egress traffic on the source and destination IP address information. Benefit: This algorithm improves the overall link use of the ESX system by providing a more balanced distribution across the aggregated links.
Data Movers Half of the ports from each NS-120 Data Mover are connected to each physical
switch, as described below: • On Data Mover 1, interfaces CGE0 and CGE2 are connected to the top 3750-E
switch and CGE1 and CGE3 are cabled to the bottom with the same configuration used on the second Data Mover.
• The interfaces for each Data Mover are configured in a Port Channel on the switch,
one Port Channel per Data Mover since they are in an Active/Standby configuration.
Benefit: This configuration provides redundancy to insure continuation of traffic flow in the event of a NIC or switch failure.
IP-based port channeling
When configuring the 3750-E switch stack for both the ESX servers and the Celerra Data Movers, IP-based port channeling (802.3ad link aggregation) is used. Benefit: The ability of the 3750-E to port channel across switches in a stack allows the option of bundling the NICs together logically.
EMC Integrated Infrastructure for VMware Enabled by EMC Celerra NS-120 Integration Guide 30
Avamar Standalone node
The Avamar standalone node in this environment uses a single GbE connection to the storage network. NIC Teaming is not an option at this point so recovery from a NIC or switch failure requires the manual movement of the connection to an alternate NIC or alternate switch and/or port, depending on the failure.
Switching efficiency
To improve the switching efficiency and performance of the storage network in the environment, Jumbo Frame support is configured on the 3750-E stack. Benefit: Using an MTU of 9198 bytes, the packet size enables less processing overhead, which leads to a significant improvement of the amount of data that can be sent and received.
EMC Integrated Infrastructure for VMware Enabled by EMC Celerra NS-120 Integration Guide 31
Section D: VMware ESX server configuration
Overview
Introduction The validated solution uses VMware ESX servers for virtualization purposes.
Contents This section contains the following topics:
Topic See Page
VMware ESX server considerations 32
VMware ESX server installation and configuration 33
iSCSI configuration for the VMware ESX server 35
Adding a datastore configuration 37
Configuring Disk/LUN properties 39
EMC Integrated Infrastructure for VMware Enabled by EMC Celerra NS-120 Integration Guide 32
VMware ESX server considerations
Introduction When a VMware ESX server is used with EMC Celerra, it is critical to ensure proper
configuration of both the Celerra and the ESX server to ensure optimal performance and availability.
Required partitions
The VMware ESX 3.5 boot disk on the Dell PowerEdge 2950 is a RAID 1 LUN supported by the Dell PERC adapter. The ESX installation requires the following four partitions: • /boot (100 MB minimum) • Swap (544 MB recommended) • / (2560 MB recommended although ESX 3.5 update 3 supports up to 5 GB) • /var/log (200 MB recommended)
EMC Integrated Infrastructure for VMware Enabled by EMC Celerra NS-120 Integration Guide 33
VMware ESX server installation and configuration
Standard installation
Installation of the VMware ESX 3.5 software on the servers is standard with no variation from the instructions in the VMware documentation. Next step: After completing the software installation, configuration of network and storage settings must be completed.
ESX configuration
In this solution, three IP networks/vSwitches are configured: • One for vMotion • One for the Storage network • One for Public (client) access
Virtual Switch configuration
The following illustration shows a Virtual Switch configured for each network.
EMC Integrated Infrastructure for VMware Enabled by EMC Celerra NS-120 Integration Guide 34
VM kernel port Since iSCSI is used in this environment, a VM kernel port for software iSCSI is
created on the vSwitch configured for storage traffic. Recommendation: It is strongly recommended that a separate iSCSI network be used to segregate that traffic from client access and vMotion traffic for optimal performance and security.
ESX server best practices
Suggested best practices for modifications to the ESX environment are: • Align the VMFS partition on a 64K block for increased performance. It is aligned on
a 64K block by default if configured using the GUI. Setting the alignment is necessary only when using the CLI to create the VMFS datastore.
• Enable CDP, which provides more visibility into the switching environment. • Configure vCenter HA for high availability. In this environment, the vCenter server
is on a Virtual Machine in the cluster. This is a supported configuration but not required. The vCenter server can be on a server outside of the cluster environment, but HA is strongly recommended.
Network interface settings
Important: Ensure that speed and duplex settings are consistent between the ESX server NICs and the Ethernet switch to avoid mismatches that would result in either performance degradation or complete loss of network connectivity.
Use of DNS Once the environment build is complete and applications are deployed, use DNS to
enable proper address resolution for both the VMware Infrastructure environment and the general data center environment.
EMC Integrated Infrastructure for VMware Enabled by EMC Celerra NS-120 Integration Guide 35
iSCSI configuration for the VMware ESX server
Enabling the iSCSI initiator
Follow the steps in the table below to use the software iSCSI initiator in VMware ESX 3.5. Validated solution: In the test environment, build scripts configured the iSCSI initiator.
Step Action
1 On the Configuration tab of the ESX server, click Storage Adapters, select the iSCSI adapter you want to configure and click Properties.
2
In the iSCSI Initiator Properties dialog box, Click Configure. Result: The General Properties dialog box opens, displaying the initiator’s status, default name, and alias. The following illustration shows the General Properties dialog box.
3 Select Enabled to enable the initiator.
4
Enter the new name to change the default iSCSI name for your initiator. Note: You do not need to change the default name, however if you do, be sure to format the name properly, otherwise some storage devices might not recognize the software iSCSI initiator. Format: iqn.<year-month of domain registration>.<reverse domain name>:<device name>
5 Click OK to save your changes.
EMC Integrated Infrastructure for VMware Enabled by EMC Celerra NS-120 Integration Guide 36
iSCSI initiator view
The following illustration shows a view in vCenter of the configured SCSI initiators:
Viewing iSCSI initiator properties
Click Properties to view the properties of an iSCSI initiator. The following illustration shows an example of the iSCSI properties screen.
EMC Integrated Infrastructure for VMware Enabled by EMC Celerra NS-120 Integration Guide 37
Adding a datastore configuration
Introduction The Virtual Machines use a datastore created on the iSCSI device for storage
purposes.
Add a datastore for storage
Follow the steps in the table below to add a datastore for storage.
Step Action
1 Log in to the VI Client and select a server from the inventory panel.
2 Select the Configuration tab and click Storage.
3
Click Add Storage to open the Add Storage screen. The following illustration shows how to select a storage type.
EMC Integrated Infrastructure for VMware Enabled by EMC Celerra NS-120 Integration Guide 38
4
Select the Disk/LUN storage type and click Next to open the Select Disk/LUN page. Note: Opening this page can take a few seconds depending on the number of targets that you have. The following illustration shows the screen on which to select Disk/LUN.
5 Select the iSCSI device for the datastore and click Next to open the Current Disk Layout page.
6 Verify the current disk layout and click Next to open the Disk/LUN Properties page.
7
Enter a datastore name. Note: The datastore name appears in the VI Client and the label must be unique within the current Virtual Infrastructure instance.
Next step After adding the datastore, verify that the layout and properties are correct. See
Configuring Disk/LUN properties for step-by-step instructions.
EMC Integrated Infrastructure for VMware Enabled by EMC Celerra NS-120 Integration Guide 39
Configuringd/LUN properties
Configure disk/LUN properties
After adding the datastore, verify the layout and properties as described below.
Step Action
1 Click Next on the Select Disk/LUN page, to open the Disk/LUN - Formatting page. The following illustration shows the Disk/LUN – Formatting page.
2 If needed, adjust the file system values and capacity for the datastore and click Next to open the Ready to Complete page. Note: By default, all of the free space on the storage device is available.
3 Review the datastore configuration information and click Finish.
EMC Integrated Infrastructure for VMware Enabled by EMC Celerra NS-120 Integration Guide 40
Completed storage configuration
The following illustration shows the completed storage configuration for the ESX servers.
EMC Integrated Infrastructure for VMware Enabled by EMC Celerra NS-120 Integration Guide 41
Section E: EMC Avamar configuration
Overview
Introduction The solution as validated uses EMC Avamar software for deduplicated backups.
Contents This section contains the following topics:
Topic See Page
EMC Avamar deduplication overview 42
EMC Avamar installation and configuration 43
Customization 44
EMC Integrated Infrastructure for VMware Enabled by EMC Celerra NS-120 Integration Guide 42
EMC Avamar deduplication overview
Introduction EMC Avamar is backup and recovery software which, in the validated solution, is
installed as a single-node server, single-rack unit appliance.
Deduplication benefits
Avamar’s global data deduplication technology: • Eliminates the unnecessary transmission and storage of redundant data across
the network • Solves traditional backup challenges by reducing the size of the backup data at
the source which is done by storing only a single copy of sub-file data segments across all sites and servers
• Allows deduplicated backups to function like full backups, and can be recovered
in just one step while also verifying backup data recoverability
EMC Integrated Infrastructure for VMware Enabled by EMC Celerra NS-120 Integration Guide 43
EMC Avamar installation and configuration
Installation For the solution as validated, a standalone unit configuration is utilized. The single-
rack unit appliance is installed in the EMC rack and configured according to the EMC Avamar 4.1 Server Software Installation Manual.
Activation After installing and configuring the appliance per the single-node server instructions,
the Avamar client is installed on the VMs in the cluster and the VMs are activated on the Avamar server.
Backups In the validated solution, the Avamar node is configured to perform guest-based
backups based on Avamar recommendations for taking backups within virtualized environments. Guest-based backups provide the same file-level backup methods and procedures as for a physical machine. Note: Multiple backup jobs for the same client cannot be run, but multiple jobs across multiple clients will work.
Other types of backups
Other types of backups that Avamar provides, which were not used to validate this solution, include: • VMware Consolidated Backup (VCB) – Uses a centralized proxy server to
perform image-level backups for VMs running any operating system and file-level backups for VMs running Microsoft Windows.
• Service Console-Based – Image-level backup of VMs performed on the ESX
server.
EMC Integrated Infrastructure for VMware Enabled by EMC Celerra NS-120 Integration Guide 44
Customization
Introduction The flexibility and ease of customization of the EMC Integrated Infrastructure for
VMware solution allows it to be deployed in any environment and be up and running quickly, without having to touch each individual component of the solution.
Customized environment values
The following illustration shows the interactive Customization Utility, used to input customized environment values.
Customized ESX data
ESX configuration parameters can be configured by modifying the ESX-Addon.txt file in the customization script.
EMC Integrated Infrastructure for VMware Enabled by EMC Celerra NS-120 Integration Guide 45
Chapter 3: Environment Administration Tools
Overview
Introduction Once the EMC Integrated Infrastructure for VMware solution is deployed, the
environment is ready to begin hosting applications. Each component within the solution can be monitored and administered using the tools provided with the solution. This chapter addresses each component individually and provides an overview of the tools in place for regular administration of the environment.
Contents This chapter provides administration information for the following components:
Topic See Page
Celerra 46
ESX 3.5 Virtual Infrastructure Environment 54
Avamar 61
EMC Integrated Infrastructure for VMware Enabled by EMC Celerra NS-120 Integration Guide 46
Celerra
Introduction Once the environment is installed and configured, maintenance of the storage
environment should be performed with Celerra Manager or Navicli. This topic introduces the task wizards available within Celerra. Note: Navicli administration is beyond the scope of this document. If further information is required regarding environment administration using Navicli, please consult your EMC representative.
Wizards Various network, CIFS, file system and iSCSI tasks are executed quickly and easily
using the wizards included with Celerra Manager. The following illustration shows an example of the available wizards.
EMC Integrated Infrastructure for VMware Enabled by EMC Celerra NS-120 Integration Guide 47
New File System Wizard
The following series of figures illustrates the creation of a new file system using the New File System Wizard in Celerra Manager.
EMC Integrated Infrastructure for VMware Enabled by EMC Celerra NS-120 Integration Guide 48
New File System Wizard (continued)
EMC Integrated Infrastructure for VMware Enabled by EMC Celerra NS-120 Integration Guide 49
New File System Wizard (continued)
EMC Integrated Infrastructure for VMware Enabled by EMC Celerra NS-120 Integration Guide 50
New File System Wizard (continued)
EMC Integrated Infrastructure for VMware Enabled by EMC Celerra NS-120 Integration Guide 51
New File System Wizard (continued)
EMC Integrated Infrastructure for VMware Enabled by EMC Celerra NS-120 Integration Guide 52
New File System Wizard (continued)
EMC Integrated Infrastructure for VMware Enabled by EMC Celerra NS-120 Integration Guide 53
New File System Wizard (continued)
Monitor GUI The figure below illustrates the Monitor graphical user interface accessed via the
Tools menu in Celerra Manager.
EMC Integrated Infrastructure for VMware Enabled by EMC Celerra NS-120 Integration Guide 54
ESX 3.5 Virtual Infrastructure environment
Introduction Once the environment is deployed, various tasks required to monitor, maintain, and
add to the environment are performed by command line or by using the Virtual Infrastructure Client. This topic highlights the tools within Virtual Infrastructure Client. Note: Command line utilities are beyond the scope of this document.
The Performance screen
The figure below illustrates the Performance screen, which enables monitoring of the CPU, memory, disk, and network utilization of the ESX servers in the environment.
New Virtual Machine Wizard
To deploy applications in the environment, virtual machines need to be deployed. This is done with the Virtual Infrastructure Client.
EMC Integrated Infrastructure for VMware Enabled by EMC Celerra NS-120 Integration Guide 55
Specify typical or custom configuration
The figure below illustrates the New Virtual Machine wizard screen, where the VM machine configuration is selected.
Input VM name and location
The figure below illustrates the screen where the new VM name and location are input.
EMC Integrated Infrastructure for VMware Enabled by EMC Celerra NS-120 Integration Guide 56
Add Storage Wizard
The following two figures illustrate adding storage to an ESX server using the Add Storage wizard, accessed on the Configuration tab for the ESX server.
EMC Integrated Infrastructure for VMware Enabled by EMC Celerra NS-120 Integration Guide 57
Virtual Machine Properties
The figure below illustrates the Virtual Machine Properties screen, used to remove or edit resources associated with a VM, accessed with the Edit Settings option after right-clicking the VM icon.
Add Networking Wizard
The figure below illustrates how to add a vSwitch to an ESX server, using the Add Networking wizard, accessed on the Configuration tab under Networking.
EMC Integrated Infrastructure for VMware Enabled by EMC Celerra NS-120 Integration Guide 58
Connection type
The figure below illustrates the connection type specification screen.
Create a virtual switch
The figure below illustrates how to create a virtual switch.
EMC Integrated Infrastructure for VMware Enabled by EMC Celerra NS-120 Integration Guide 59
New Cluster Wizard
To add a new HA cluster to the existing environment, use the New Cluster wizard, accessed by right-clicking the data center name and selecting Add New Cluster. Note: For more detail on these features, see VMware Infrastructure - Automating High Availability (HA) Services with VMware HA and VMware Infrastructure Resource Management with VMware DRS.
Cluster name and features
The figure below illustrates the first screen in the New Cluster wizard, establishing the new cluster name and features.
EMC Integrated Infrastructure for VMware Enabled by EMC Celerra NS-120 Integration Guide 60
New Cluster Wizard summary screen
The figure below illustrates the New Cluster Wizard summary screen.
The figure below illustrates the HA and DRS cluster settings screen.
EMC Integrated Infrastructure for VMware Enabled by EMC Celerra NS-120 Integration Guide 61
Avamar
Introduction Once the virtual machine has been set up as a client, it can be managed through
the Avamar administration application. This topic highlights those applications.
Backup and restore
Backups can be taken at the file, folder or directory level by checking the appropriate box on the Backup & Restore screen. The figure below illustrates the Avamar Administrator screen.
EMC Integrated Infrastructure for VMware Enabled by EMC Celerra NS-120 Integration Guide 62
Chapter 4: Basic Administration Tasks
Overview
Introduction Chapter 3 provided an overview of the tools available to administer the EMC
Integrated Infrastructure for VMware solution. This chapter provides a brief overview of the basic administration tasks and correlates the tasks with the tools. Also provided are references to sources of detailed information on performing the tasks.
Contents This chapter contains the following topics:
Topic See Page
Monitoring the environment 63
Configuring additional storage 65
VMware ESX administration tasks 66
Backing up and restoring guest VMs 67
EMC Integrated Infrastructure for VMware Enabled by EMC Celerra NS-120 Integration Guide 63
Monitoring the environment
Components that can be monitored
It is necessary to monitoring the solution environment so each component and subsystem runs efficiently. The following components can be monitored with the appropriate tools: • ESX environment • Storage environment • IP switching, and • Avamar
ESX environment
Use VirtualCenter or ESX-Top commands to monitor the ESX environment. Third-party tools are available to perform additional monitoring: • CPU usage • Memory usage • Disk utilization • Network utilization Reference: Information on monitoring the ESX environment can be found at http://www.vmware.com.
Storage environment
Use Celerra Manager/Celerra Monitor to monitor the storage environment. Aspects of the storage environment that can be monitored include: • Disk utilization • I/OPS • Network utilization Note: If an existing EMC ControlCenter® (ECC) environment exists, the storage can be monitored using that tool also or instead. Reference: Information on using Celerra Manager/Celerra Monitor to monitor the storage environment can be found on EMC Powerlink: Managing EMC Celerra Statistics P/N 300-004-579, Rev A04 Version 5.6 December 2008
EMC Integrated Infrastructure for VMware Enabled by EMC Celerra NS-120 Integration Guide 64
IP switching Use any Cisco or third-party SNMP-based tools to monitor IP switching. MRTG was
used during solution validation. Aspects of IP switching that can be monitored include: • Interface utilization • Interface errors • Switch CPU utilization Reference: For information on MRTG, visit http://oss.oetiker.ch/mrtg/
Avamar Use the monitoring tool available in Avamar Administrator to monitor the Avamar
system. Aspects of Avamar that can be monitored include: • Backup schedules • Capacity utilization • Errors or failures Reference: Information on monitoring the Avamar system can be found in the EMC Avamar 4.1 Administrator’s Manual on EMC Powerlink.
EMC Integrated Infrastructure for VMware Enabled by EMC Celerra NS-120 Integration Guide 65
Configuring additional storage
Tasks The tasks involved in configuring additional iSCSI and NFS storage on the Celerra
and the CLARiiON include: • Adding more disk capacity (DAEs and disks) • Creating additional file systems and logical volumes • Modifying or growing existing logical volumes • Creating additional storage pools
Tools to use To perform these and other storage configuration tasks, use Celerra Manager and
Navisphere Manager.
Reference Refer to the VMware ESX Server Using EMC Celerra Storage Systems Solutions
Guide at the following URL: http://www.emc.com/collateral/hardware/technical-documentation/h5536-vmware-esx-srvr-using-celerra-stor-sys-wp.pdf
EMC Integrated Infrastructure for VMware Enabled by EMC Celerra NS-120 Integration Guide 66
VMware ESX administration tasks
Tasks Basic VMware ESX administration tasks include:
• Deploying VMs • Creating templates, clones, and snapshots • Creating users, groups, and permissions • Creating resource pools • Migrating VMs between ESX servers
Tools to use To perform these and other VMware administration tasks, use either:
• vCenter, available from the Virtual Infrastructure (VI) Client, or • ESXCFG command line tools
Reference Refer to Basic System Administration ESX Server 3.5, ESX Server 3i version 3.5
Virtual Center 2.5 at the following URL: http://www.vmware.com/pdf/vi3_35/esx_3/r35/vi3_35_25_admin_guide.pdf Additional VMware Infrastructure documentation can be found at the following URL: http://www.vmware.com/support/pubs/vi_pages/vi_pubs_35_3i_i.html
EMC Integrated Infrastructure for VMware Enabled by EMC Celerra NS-120 Integration Guide 67
Backing up and restoring guest VMs
Tasks Backing up and restoring guest VMs involves:
• Creating backup pools and schedules • Installing the Avamar agent on VMs • Verifying backups • Restoring files and folders
Tools Use Avamar Administrator to back up and restore guest VMs.
Reference Information on monitoring the Avamar system can be found in the EMC Avamar 4.1
Administrator’s Manual on EMC Powerlink.
EMC Integrated Infrastructure for VMware Enabled by EMC Celerra NS-120 Integration Guide 68
Chapter 5: Testing and Validation
Overview
Introduction This chapter describes the methodology and parameters used to test and validate
the EMC Integrated Infrastructure for VMware solution.
Contents This chapter contains the following topics:
Topic See Page
Parameters 69
Methodology 70
Results 71
Summary 73
EMC Integrated Infrastructure for VMware Enabled by EMC Celerra NS-120 Integration Guide 69
Parameters
Introduction This section reviews the parameters used to measure the EMC Integrated
Infrastructure for VMware solution’s performance and behavior under simulated load.
Testing environment
The solution was tested and validated in the EMC Global Solutions Customer Integration Labs using I/Ometer. I/Ometer is an I/O subsystem measurement and characterization tool configured to replicate the behavior of many different applications and measure performance and behavior under simulated loads.
Equipment A total of 60 VMs were used during the validation testing phase in the test
environment, consisting of • five ESX servers (Dell PE2950, four 2.50 GHz dual-core processors, 32 GB
memory), configured with • twelve Windows 2003 guest virtual machines, each with • four virtual CPUs and 3 GB memory.
Workload profile
The table below provides the I/Ometer workload profile defined for each VM:
Workload KB Read/Write Random/Sequential
20% Exchange 2003 4 KB 67% Read 100% Random
20% File Server 8 KB 90% Read 75% Random
20% OLTP 8 KB 67% Read 100% Random
20% Logging 64 KB 100% Write 100% Sequential
20% Video on Demand 512 KB 100% Read 100% Sequential
EMC Integrated Infrastructure for VMware Enabled by EMC Celerra NS-120 Integration Guide 70
Methodology
Introduction This section describes the testing methodology used to measure the EMC
Integrated Infrastructure for VMware solution’s performance and behavior.
Methodology Testing was executed incrementally, starting with six virtual machines per ESX
server and increasing until the maximum point was achieved.
Measurements The following measurements were taken using ESX-Top, and were used to generate
graphical representations of each metric. • Workload I/OPS • CPU utilization • Workload bandwidth utilization • Celerra array utilization Note: The Celerra results reflect I/Ometer running on 12 VMs on one ESX server using the workload profile stated above.
EMC Integrated Infrastructure for VMware Enabled by EMC Celerra NS-120 Integration Guide 71
Results
Workload I/OPS
The following illustration shows the workload I/OPS.
Workload MB/s The following illustration shows the workload bandwidth utilization
EMC Integrated Infrastructure for VMware Enabled by EMC Celerra NS-120 Integration Guide 72
Physical CPU utilization
The following illustration shows the virtual CPU utilization.
Celerra array utilization
The following illustration shows the Celerra array utilization.
Reference: For details, see “Measurements” in the “Methodology” topic.
EMC Integrated Infrastructure for VMware Enabled by EMC Celerra NS-120 Integration Guide 73
Summary
Summary Based on the results of the tested workload profile, the following determinations
were made: • As application workload (virtual machines) is added to the environment,
performance across all metrics eventually plateaus at 12 VMs per ESX server. • Based on the defined workload profile, this represents an aggressive environment
that performs very well under load.
• This aggressive environment lends itself to many varied configurations for combinations of mail server, file server, database server and VoD server deployments.
EMC Integrated Infrastructure for VMware Enabled by EMC Celerra NS-120 Integration Guide 74
Conclusion
Overview
Conclusion The EMC Integrated Infrastructure for VMware solution was designed and built as a
“data center in a rack,” which utilizes build and customization automation to deploy a standardized architecture to host numerous customer applications in a virtualized environment. This solution enables customers to deploy a best practices-based virtualization solution that maximizes the benefits of VMware ESX, EMC Celerra, and EMC Avamar.
Learn more To learn more about this and other solutions, contact an EMC representative or visit
www.EMC.com/solutions.
EMC Integrated Infrastructure for VMware Enabled by EMC Celerra NS-120 Integration Guide 75
References
EMC Celerra • Using EMC Celerra IP Storage with VMware Infrastructure 3.5 over iSCSI and
NFS Best Practices Planning
VMware ESX Server
• VMware ESX Server Optimization with EMC Celerra Performance Study • VMware ESX Server Using EMC Celerra Storage Systems Solutions Guide • VMware ESX Server 3.5 Resource Management Guide • VMware ESX Server 3.5 Administration Guide • VMware iSCSI SAN Configuration Guide ESX Server 3.5, ESX Server 3i version
3.5 VirtualCenter 2.5
EMC Avamar • EMC Avamar Backup Solutions for VMware ESX Server on Celerra NS Series
• EMC Avamar 4.1 Server Software Installation Manual
VMware infrastructure
• VMware Infrastructure Automating High Availability (HA) Services with VMware HA
• VMware Infrastructure Resource Management with VMware DRS
Dell servers • http://www.dell.com/downloads/global/power/dell2socket_vs_hp4socket_vmware.pdf
• http://www.dell.com/downloads/global/products/pedge/en/PE2950_SS_072007.pdf