giovanni marchetti sr. technical evangelist microsoft corp. wsv202
TRANSCRIPT
Considerations and Strategies for Deploying Virtual Clusters
Giovanni MarchettiSr. Technical EvangelistMicrosoft Corp.WSV202
Session Objectives
Brief intro to HPC ClustersWhy virtualize HPC clusters?Where do they make sense?How do we deploy and manage them?What configuration issues arise?How do we schedule jobs on them?Are they so exotic?
What Is High Performance Computing?
A branch of computer science that studies systems designs and programming techniques to extract the best computational performance out of microprocessors. It is usually associated with parallel computing either on specialized machines or on clusters of commodity machines.
Industries Using HPC
Education & ResearchGeological ServicesManufacturingFinancial ServicesMoviesMedicineArchaeology???!!!
Real-world Application Example
Courtesy ofUniversity of Bologna, Italy
CAT scan of a 3000 year-old catData and computation – intensiveLimited comms. among nodesTypical parametric sweep
MPINetwork
PrivateNetwork
PublicNetwork
Corporate IT Infrastructure
Compute NodeHead Node
AD
DNS
DHCP
Clients Monitoring SystemsManagement
Compute Cluster
Possible Cluster Topologies
Admin / User Cons
WDS
Job Scheduler
MPI
Management
NAT
Node Manager
MPI
Management
Compute Node
Node Manager
MPI
Management
The Opportunity
Low usage on physical servers (e.g. at night)Make computers compute for a living
Virtualized infrastructureTreat computing as just another functionIntegrate with “real” HPC clusters
Maintain isolation for security and ease of management reasons
Cycle scavenging does not
The Applications
Suitable for parametric sweeps (e.g., Montecarlo simulations) or web services
Many independent processes, possibly multi-threadedLarge set of inputsNot interactive
NOT suitable for MPI applicationsVirtualization comms. overhead (reduced with R2)Unpredictable latenciesLimited topologies
The Options
We have 3 alternatives:Install the HPC Pack on all the x64 serversDual bootUse Hyper-V and virtual machines with HPC 2008
Coming in R2 / v3 Boot from networkBoot from vhd
HPC Pack on all servers
ProsLittle overhead when not in useEasy to deploy and manageAlways ready
Cons:No isolation at all
Whoever can launch a job, can run anything on your computer / x64 workstation
Only works on the domain your computer is inHead node may not necessarily be in that domain
No control on resource consumption
Dual Boot
Pros:“Clean” solution, as long as you have a 2nd partition100% of resources available in designated timeNo perf. Overhead
Cons:Licenses: 2 per machine!!!100% of resources available ONLY in designated timeRequires careful management of OS transitions
Diskpart is powerful and dangerous.Limited isolation: must manage access to partitions carefully
Hyper-V
ProsComplete isolationControl resource allocation Rapid provisioning / de-provisioningDomains and VLANs for VM can differ from hostVirtualization rights included or free hyper-v server
Cons:Performance and management overheadMust have virtualization management solutionMust work on your own scheduler integration
Hyper-V: Quick Tips
Works only on x64 servers with DEP and VT (or AMD-V) enabled
Most x64 workstations can run it;Install “desktop experience”
Once installed, everything is a virtual machine, including the management partitionPatch early, patch often and be sure you know what to patch
Hyper-V Management
Requires Virtual Machine ManagerVMM will install agent, hyper-v role on targeted servers in trusted domainsCan manage servers in different domains
Deploy these patches on hyper-v hosts:KB956589 – perf. counters for synthetic storageKB956774 – BITS client handling volume IDs
Virtual Machine Manager Server
Connector
Self Service Web Portal
Administrator’s Console
Management Interfaces
SAN Storage
Operator’s Console
Web Console
Windows PowerShell
Operations Manager Server
Virtual Server Host
VM
VM
VMM Library Server
VM Template
ISO ScriptVHD
VMware VI3
Virtual Center Server
ESX Host
VM VM VM VM
VM VM VM VM
VM VM VM VM
Windows® PowerShell
VMM 2008 Architecture
VM Deployment
Build “prototype node” VMCreate hardware and o/s profile based on itExtract a template from it with SCVMM
Will run sysprep and make the node unavailableSelect target servers to run hyper-v
Note: RDMA adapters (e.g. Infiniband) will not be virtualized; they will be used as any other IPoIB interface
Deploy node template on target servers (GUI or powershell script)
VM Placement
CPU, Network,and Disk Load
Physical Disk and Memory Requirements
Configuration
VM
Result of Disk Capacity and Memory Check
Disk Capacity and Memory Check
Capacity PlanningTechnology
Normalized Host + VM Load
Existing Load Configuration
Host Performance and Configuration
RatingFunction
VM Placement
Memory is the main constraintVM can be moved and RAM allocation altered, but not dynamically
HPC VMs will most often either run at 100% CPU or 0% within the VM
Does not necessarily map to real cpu utilizationHPC job scheduler does not understand “50% of CPU”
VM Placement Considerations
Specify number of CPUs in VM and assume 100%Total Number of active logical CPUs ≤ number of physical CPUs on hostLogical will be mapped to physical by hyper-v scheduler
Hyper-V Architecture Provided by:
OS
MS / XenSource / Novell
ISV/IHV/OEM
Hyper-V
Windows hypervisor
“Designed for Windows” Server Hardware
Non hypervisor aware OS
Windows Server 2003, 2008
Applications
WindowsKernel VSC
WindowsKernel
Windows Server 2008
VSP
VMBus Emulation
Parent Partition
Kernel Mode
User Mode
Xen-enabledLinux Kernel Linux
VSCs
VMBus
Hypercall Adapter
Applications
Child Partitions
VMBus
Virtualization Stack
VMService
WMI Provider VM WorkerProcess
Applications
VM Placement Considerations
SCVMM hardware profiles ≠ hardware constraintsIndication of expected hw performance
Hyper-V always reports up to 1 quad-core cpu of underlying kind
Co-located tasks in VM may not actually beIdentical hardware profiles on same kind of host simplify life for the scheduler and the admin
Not always practicalPlace for performance, not density (in R2)
On the HPC side
Create host groupsRule of thumb: 1 per VM template deployed
Use job templates to enforce constraints for jobs on VM host group (e.g. max runtime)Schedule jobs at appropriate time
At <time> job submit <job parameters>Or specify host group, then activate the nodes in question
At <time> node resume <node list>
Rapid Provisioning / De-provisioning
Create several templatesDifferent kind of nodes for different kind of jobs> 1 template deployed to same physical host, 1 active
Test scalability by deploying N, 2N, etc… VMs. on same hostsNo need to delete VMs
Take nodes offline and shut them downWhen required, turn them back on
Provisioning on demand
1 more layer of complexityUse SCOM to track n. of tasks in the queueCause VM provisioning or activation when high threshold is reached (run psh script)De-provision or shut down when low threshold is reached
SCOM / SCVMM / Pro tipsMay cause VMs to be movedMay be irrelevant to particular HPC workload
VMs always run 100% or 0 %
Dynamically Grow / Shrink
Dynamically grow or shrink pool of resources for jobs in HPC Server 2008Ties in with dynamic provisioning / de-provisioning
Provision serversGrow pool available to running jobOffline servers -> shrink poolDe-provision or pause VMs
Job keeps running
SCOM rules for on-demand VMsdemo
Availability advantages
High availability of head nodes is quite complexSCVMM takes care of HA for VMs on clusters
Head node can be a VM tooIn this case, you’re protected against node failure only
What if time on a machine has run out, but you still need to compute?
Shrink pool, move VM to available node, bring backWith R2, live migration means you need not interrupt the computation either!
Moving Computing Nodesdemo
Call to Action
Just try it!www.microsoft.com/hpcwww.microsoft.com/hyper-vwww.microsoft.com/scvmm
questions & answers
www.microsoft.com/teched
Sessions On-Demand & Community
http://microsoft.com/technet
Resources for IT Professionals
http://microsoft.com/msdn
Resources for Developers
www.microsoft.com/learningMicrosoft Certification and Training Resources
www.microsoft.com/learning
Microsoft Certification & Training Resources
Resources
Related ContentVIR208 Virtual Machine Manager 2008: Technical Overview and R2 PreviewVIR209 How to Build an Efficient Application Infrastructure through VirtualizationVIR302 Hyper-V & Virtual Machine Manager 2008: Tips, Tricks, and Best PracticesVIR303 Hyper-V & System Center: Practical Orchestration & Process Automation
VIR03-INT Virtual Computing Clusters
VIR04-HOL Introduction to Hyper-V VIR05-HOL Introduction to Microsoft System Center Virtual Machine Manager 2008
Windows Server ResourcesMake sure you pick up your copy of Windows Server 2008 R2 RC from the Materials Distribution Counter
Learn More about Windows Server 2008 R2: www.microsoft.com/WindowsServer2008R2
Technical Learning Center (Orange Section): Highlighting Windows Server 2008 and R2 technologies•Over 15 booths and experts from Microsoft and our partners
Complete an evaluation on CommNet and enter to win!
© 2009 Microsoft Corporation. All rights reserved. Microsoft, Windows, Windows Vista and other product names are or may be registered trademarks and/or trademarks in the U.S. and/or other countries.The information herein is for informational purposes only and represents the current view of Microsoft Corporation as of the date of this presentation. Because Microsoft must respond to changing market conditions, it should not be interpreted to be a commitment on the part of Microsoft, and Microsoft cannot guarantee the accuracy of any information provided after the date of this presentation. MICROSOFT MAKES NO WARRANTIES, EXPRESS,
IMPLIED OR STATUTORY, AS TO THE INFORMATION IN THIS PRESENTATION.