© 2010 Quest Software, Inc. ALL RIGHTS RESERVED

Optimize Oracle RDBMS on VMware

Guy HarrisonDirector, R&D Melbournewww.guyharrison.net Guy.harrison@quest.com@guyharrison

2

Introductions

3

4

5

Agenda

• Motivations for Virtualization

• VMware ESX resource management:

– Memory

– CPU

– IO

• Paravirtualization (OVM) vs Hardware Assisted

Virtualization (ESX)

6

Motivations for Virtualization

•Better utilization of server resources

•Reduced power consumption

Server Consolidation

•Fewer physical machines

•Backup, cloning, rapid provisioning

Manageability

•Adjust resources on demand

•An complement to the physical “grid” vision

Elastic computing

7

Resistance to Database virtualization

•Virtual CPU & IO

•Sharing of virtual resources

Performance

•Large databases too big for a single VM

•RAC-style clustering problematic

Scale

•Oracle’s stance often misunderstood

•See MyOracleSupport 249212.1

Support

8

DB virtualization is happening

9

Dba-village.com

May 2009

Mar 2010

10

Oracle virtualization is lagging....

SQL Server

File and Print Servers

IIS

Active Directory

Apache

Exchange

Sharepoint

Java

Oracle

None of the Above

0% 10% 20% 30% 40% 50% 60% 70% 80% 90% 100%

Which of the following do you run in VMs?

Quest survey of vFoglight users , 2010

11

ESX Memory management

12

Managing ESX memory

• ESX can “overcommit” memory– Sum of all VM physical memory allocations > actual ESX physical

memory

• Memory is critical to Oracle server performance– SGA memory to reduce datafile IO– PGA memory to reduce sort & hash IO

• ESX uses four methods to share memory:– Memory Page Sharing– “Ballooning”– ESX swapping– Memory compression

• DBA needs to carefully configure to avoid disaster

13

Configuring VM memory

Relative Memory Priority for this VM

Minimum Memory for this VM

VMs Compete for memory in this range

Maximum memory for the VM (dynamic)

Monitoring VM memory

ESX Swap

ESX virtual memory

ESX physical memory

VM VM virtual memory

ESX swap

Effective VM physical memory

ESX and VM memory

ESX Swap

ESX virtual memory

ESX physical memory

VM

Apparent VM physical memory

ESX swap

Effective VM physical memory

Vmmemctl

“balloon”

VM SwapVM Swap

ESX Ballooning

17

ESX Ballooning

• As memory grows, ESX balloon driver (vmmemctl) forces VM to page out memory to VM swapfile

18

ESX Ballooning• Inside the VM, paging to the

swapfile is observed.• The guest OS will determine

which pages are paged out• If LOCK_SGA=TRUE, then

the SGA should not be paged.

ESX Swap

ESX virtual memory

ESX physical memory

ESX swap

ESX Swapping

VM virtual memory

VM

Effective VM physical memory

ESX Swap

ESX virtual memory

ESX physical memory

ESX swap

Apparent VM physical memory

VMEffective VM physical memory

ESX Swapping

• ESX swaps out VM memory to ESX swapfile

ESX Swapping

22

ESX Swapping• Within the VM, swapping

cannot be detected.• ESX will determine which

memory pages go to disk• Particularly occurs when

VMware tools are not installed

• Even if LOCK_SGA=TRUE, SGA memory might be on disk

23

Avoiding Ballooning and swapping

• memory reservations help avoid ballooning or ESX swapping

24

Other VMware memory management thoughts

• Memory page sharing – Multiple VMs can share an identical page of memory (Oracle code

pages, etc)

• Modern chipsets reduce memory management overhead– Multiple hardware page layers

• Memory compression (new in ESX 4.1)– Pages are compressed and written to cache rather than to disk

• Swapping is more expensive than ballooning– Slower to restore memory– OS and Oracle get no choice about what gets paged– “Double paging” can occur – guest and ESX both page a block of

memory

25

Ballooning vs. Swapping

Swingbench workload running on Oracle database – from VMWare whitepaper: http://www.vmware.com/files/pdf/perf-vsphere-memory_management.pdf

26

VMware memory recommendations

• Paging or swapping of PGA or SGA is almost always a Very Bad ThingTM.

• Use memory reservations to avoid swapping• Install VMware tools to minimize swapping

– Ballooning >> swapping (double paging, paging SGA, high swap-in latency, etc)

• Set Memory reservation = PGA+SGA+process Overhead• Be realistic about memory requirements:

– In physical machines, we are used to using all available memory– In VM, use only the memory you need, freeing up memory for other

VMs– Oracle advisories (or Spotlight) can show you how much memory is

needed– Reduce VM reservation and Oracle memory targets in tandem to

release memory

28

ESX CPU management

29

VMware CPU management

• If more virtual CPUs than ESX CPUs, then VCPUs must sometimes wait.

• Time “stops” inside the VM when this occurs

– For multi-CPU VMs, it’s (nearly) all or nothing).

• A VCPU can be in one of three states:– Associated with an ESX CPU but idle– Associated with an ESX CPU and executing instructions– Waiting for ESX CPU to become available

• Shares and reservations determine which VM wins access to the ESX CPUs

Configuring VM CPU

VMs compete for CPU in this range

Shares determine relative CPU allocated when competing

• “CPU Ready” is the amount of time VM spends waiting on ESX for CPU

• Inside the VM, CPU stats can be misleading

CPU utilization VM

32

SMP for vCPUs

• ESX usually has to schedule all vCPUs for a VM simultaneously

• The more CPUs the harder this is

• Some CPU is also needed for ESX

• More is therefore not always better

(Thanks to Carl Bradshaw for letting me reprint this diagram from his Oracle on VMWare whitepaper)

33

ESX CPU performance comparisons

Vs 2 core 1.8 GHz physical machine

VT enabled

34

Programmatic performance

PLSQL

PLSQL compiled

Java Stored Proc

Simple C program

Standalone Java

0 5 10 15 20 25 30 35 40 45 50

ESX 2 VCPU 2.26 GHz VT enabled

ESX 2 CPU 3.5 GHz no-VT

Physical 2 CPU 1.8 GHz

Elapsed Time (s)

35

Programmatic performance (2)

ESX 2 CPU 3.5 GHz no-VT

ESX 2 VCPU 2.26 GHz VT enabled

0 50 100 150 200 250 300 350

333

72

Elapsed time Pct relative to Physical CPU adjusted for GHz

36

ESX CPU recommendations

• Use up to date chipsets and ESX software

• Allocate as few VCPUs as possible to each VM

• Use reservations and shares to prioritise access to ESX CPU

• Performance of CPU critical workloads may be disappointing on older hardware

• Monitor ESX Ready time to determine the “penalty” of competing with other virtual machines

37

ESX IO managment

38

Typical VMWare disk configuration

39

Disk Resource Allocation

• Disk shares can be used to prioritize IO bandwidth.• However, ESX often does not know underlying

storage architecture

40

Performant VMware disk configuration

Optimal configuration

• See “Oracle Database Scalability in VMware® ESX” at www.vmware.com/oracle

• Each virtual disk directly mapped via RDM to dedicated RAID 0 (+1) group

41 Spindles!

42

ESX IO recomendations

• Follow normal best practice for physical disks• Avoid sharing disk workloads

– Dedicated datastores using VMFS • Align virtual disks to physical disks?

– Consider Raw Device Mapping (RDM)

• If you can’t optimize IO, avoid IO:– Tune, tune, tune SQL– Prefer indexed paths– Memory configuration– Don’t forget about temp IO (sorts, hash joins)

43

Shameless plugs

45

46

Paravirtualization vs Hardware Virtualization

47

Paravirtualization vs “Hardware Virtualization”

• Virtualization is not emulation....• Where-ever possible, Hypervisor runs native code from

OS against underlying hardware• Because a virtualized operating system is running outside

privileged x86 “ring 0”, direct calls to hardware need special handling.

• The three main approaches are:– Full Virtualization (VMWare on older hardware)– ParaVirtualization (Xen, Oracle VM)– Hardware Assisted Virtualization (Intel VT, AMD-V)

48

Full virtualization

Guest OS

Hypervisor

Hardware

Ring 0

• Hardware calls from the Guest are handled by the hypervisor by:– Catching the calls as they occur at

run time– Re-writing the VM image at load

time (binary translation)

• Requires no special hardware• Supports any guest OS• Relatively Poor performance• Used by ESX on older chip-sets

49

Hardware Assisted virtualization• Intel VT and AMD-V chips

add a “root mode”.• Guest can issue

instructions from non-root Ring 0.

• CPU can divert these to hypervisor

• No changes to OS required

• Good performance• Requires modern

chipsets

Hypervisor

Hardware

Guest

Ring 0

Root Mode Non-Root Mode

50

Paravirtualization

• Guest operating system is rewritten to translate device calls to “hypercalls”

• Hypercalls are handled by a special VM (dom0 in Xen/OVM)

• Good performance but requires modified guest OS

• Xen can use either paravirtualization or hardware assist

Hypervisor

Hardware

Guest(domU)

Ring 0Guest(dom0)

51

RAC and ESX

52

Paravirtualization, ESX and RAC

• Prior to 11.2.0.2, Oracle relied on paravirtualized kernels to maintain time synchronization for RAC clusters.

• From 11.2.0.2 Oracle uses Cluster Time Synchronization Service (CTSS) to maintain clock sync, and this works on ESX

• Therefore, Oracle supports RAC on Vmware ESX only from 11.2.0.2 onwards

• See Oracle MySupport Note 249212.1

53

References

• Latest version of this presentation:– http://www.slideshare.net/gharriso/optimize-oracle-on-vmware-5271

530

• My blog (www.guyharrison.net ):– http://guyharrison.squarespace.com/blog/2010/2/22/memory-

management-for-oracle-databases-on-vmware-esx.html– http://guyharrison.squarespace.com/blog/2010/4/9/esx-cpu-

optimization-for-oracle-databases.html– http://guyharrison.squarespace.com/blog/2010/7/12/stolen-cpu-on-

xen-based-virtual-machines.html• http://www.vmware.com/files/pdf/perf-vsphere-memory_management.p

df• http://www.vmware.com/files/pdf/Oracle_Databases_on_vSphere_Depl

oyment_Tips.pdf

54

Times Title Room Name Company

8:00am - 9:00amWhat DBAs Need to Know about Virtualization, Automation, and Cloud

307D Steve  Lemme Oracle

9:15am - 10:15am Optimizing Oracle databases on VMware 307D Guy Harrison Quest

10:30am - 11:30am Oracle VM, your unlimited playground 311B Gary  GordhamerGE Healthcare

11:45am - 12:15pm Why Virtualize Oracle Databases on VMware? 307D Michael West Vmware

1:00pm - 2:00pm Exadata for the Beginners 307D Rich Niemiec Rolta TUSC

2:15pm - 3:15pmReal Example for DBAs - Databases, Automation and the Cloud

307D Allen  Herndon Fedex

4:00pm - 5:00pmVirtually Complete: Real World Experiences in Virtualizing Tier-One Database Workloads.

307D Dan  YoungIndiana University

 

© 2010 Quest Software, Inc. ALL RIGHTS RESERVED

End of Presentation

너를 감사하십시요 Thank You Danke Schön

Gracias 有難う御座いました Merci

Grazie Obrigado 谢谢