Boost Compute Performance for VMs in a Cloud

Thang Man

1

Viet OpenStack Meetup 7th @Saigon 2015

$ cat /me.info

★ Open Source Cloud Computing Follower

mvthang

manvanthang

manthang.wordpress.com

2

Agenda

● Motivation

● VM placement in OpenStack

● Boost compute performance for VMs under KVM○ NUMA-aware scheduling

○ CPU Pinning

● Demo

3

Motivation

When / Why we consider running high performance VMs in a cloud?

And the challenges facing us?

4

Popular HPC Workloads

5

HPC on Cloud: The benefits of cost

6

On-Premises Cloud Services

HPC on Cloud: The benefits of elasticity

7

HPC on Cloud: The key challenges

8

The performance issues facing us…

Compute (processor, memory) (*)Network I/ODisk I/O“Noisy neighbours” effectetc.

(*) Our focus today :).

VM Placement in OpenStack

VM or cloud instance placement is one of the most crucial components of most clouds. It has not only impact on the performance of VMs, but also the efficient resource usage of the cloud.

9

Instance Creation Flow

10

Nova Scheduler

11

Using Filter Scheduler by default

● NUMATopologyFilter● AggregateInstanceExtraSpecsFilter

Why NUMA?

12

NUMA helps to surpass the scalability, bandwidth and latency limits of the SMP (symmetric multiprocessing) architecture, e.g. UMA.

NUMA Placement

13

Non-Optimal NUMA PlacementOptimal NUMA Placement

NUMA Downsides

14

Good: Low latency(local memory access)

Bad: High latency(remote memory access)

Some KVM-NUMA Suggestions

15

● Pinning guest vCPUs to pCPU cores○ No share those pinned cores with other VMs or Host.○ Help avoid CPU cache misses.

● Don't assign over compute resources to guests○ Guest vCPUs <= available pCPU cores per node.○ Guest memory < available memory per node.

⇒ Say “NO” to overcommit!

● In OpenStack, using Host Aggregates to separate VMs with dedicated resourcing requirements from those without dedicated resources.

Demo in OpenStack Kilo (1)

16

● Reserving dedicated cores for host processes.

# cat /etc/default/grub

GRUB_CMDLINE_LINUX_DEFAULT="quiet splash isolcpus=4-23"

● Reserving dedicated cores for VMs.

● Enabling the required scheduler filters.

# cat /etc/nova/nova.conf

[DEFAULT]

vcpu_pin_set=4-23

scheduler_default_filters=NUMATopologyFilter,AggregateInstanceExtraSpecsFilter,...

Demo in OpenStack Kilo (2)

17

● Creating a Host Aggregate to add all hosts configured for CPU pinning to.

● Creating a performance focused Flavor to target this Host Aggregate.

● Launching an instance with above performance focused Flavor!

Benchmark Results

18

CERN: Tuning hypervisors for High Throughput Computinghttp://openstack-in-production.blogspot.com/2015/08/kvm-and-hyper-v-comparison-for-high.html

Reference

19

● http://docs.openstack.org/● http://redhatstackblog.redhat.com/2015/05/05/cpu-pinning-and-numa-

topology-awareness-in-openstack-compute/● https://moinakg.wordpress.com/2013/06/05/findings-by-google-on-numa-

performance/● http://frankdenneman.nl/2015/02/27/memory-deep-dive-numa-data-locality/● http://blogs.technet.com/b/windowsserver/archive/2012/04/05/windows-

server-8-beta-hyper-v-amp-scale-up-virtual-machines-part-1.aspx● http://blog.platform9.com/virtual-machine-placement-openstack/● http://www.slideshare.net/AmazonWebServices/intro-to-high-performance-

computing-in-the-aws-cloud

Thank You!Q&A

20