virt1430be performance tuning & monitoring for or distribution€¦ · performance tuning &...

David Klee, Heraflux TechnologiesThomas LaRock, SolarWinds

VIRT1430BE

#VMworld #VIRT1430BE

Performance Tuning & Monitoring forVirtualized Database Servers

VMworld 2017 Content: Not fo

r publication or distri

bution

• This presentation may contain product features that are currently under development.

• This overview of new technology represents no commitment from VMware to deliver these features in any generally available product.

• Features are subject to change, and must not be included in contracts, purchase orders, or sales agreements of any kind.

• Technical feasibility and market demand will affect final delivery.

• Pricing and packaging for any new technologies or features discussed or presented have not been determined.

Disclaimer

2#VIRT1430BE CONFIDENTIAL



bution

Virtualizing Applications Track Sessions and Offerings

• 30 Breakout Sessions with 2 Panels & 3 Quick Talks

• 10 BCA Meet-The-Experts sessions (15min 1-on-1 appts)

• 2 Birds-of-a-Feather special invitation receptions (Oracle & SAP)

• 5 Group Discussions

• 3 Saturday - Full Day Applications Bootcamps

• Sign up for the Independent Oracle User Group (IOUG) VMware Special Interest Group (SIG)

www.ioug.org/vmware



bution

http://www.ioug.org/vmware

The Percentage of Applications in Virtualized Infrastructure Has Increased Dramatically Over the Last Few Years

(VMware Core Metrics Survey 2016)

4

NA EU dAP BRIC SMB COMM ENT

80% 81% 75% 84% 75% 81% 86%

57% 70% 66% 71% 59% 70% 68%

52% 55% 49% 58% 48% 51% 60%

61% 44% 43% 51% 41% 56% 60%

36% 51% 48% 55% 32% 45% 59%

32% 29% 40% 38% 32% 35% 34%

38% 22% 24% 31% 24% 33% 34%

26% 28% 30% 36% 24% 37% 30%

18% 29% 41% 40% 21% 31% 35%

19% 20% 26% 29% 18% 24% 26%

388 289 139 208 401 217 406

Region Company Size

81%

65%

53%

52%

46%

33%

30%

29%

29%

22%

Microsoft SQL

Custom/Industry-Specific Business Critical Application

Microsoft Exchange

Microsoft SharePoint

SAP

Oracle Databases

IBM Middleware

Oracle Applications

High Performance Computing

Oracle Middleware

% Respondents Running the Application in Virtualized Infrastructure

> Total

< Total

N = 1024

#VIRT1430BE CONFIDENTIAL



bution

Where Can I Learn More?

▪ Business Critical Applications VMware.com Homepage Page

• https://www.vmware.com/solutions/business-critical-apps.html

▪ VMware – DellEMC Collaborative Collateral and DBTA Surveys

• http://www.dbta.com/emc

▪ Blogs

• davidklee.net

• thomaslarock.com

• vSphere Blog

• https://blogs.vmware.com/vsphere/

• One Stop Shop - All Oracle on VMware SDDC

• https://blogs.vmware.com/apps/2017/01/oracle-vmware-collateral-one-stop-shop.html

• VMware IOUG Special Interest Group

• http://vmsig.org/




bution

https://www.vmware.com/solutions/business-critical-apps.html

http://www.dbta.com/emc

http://davidklee.net/

https://thomaslarock.com/

https://blogs.vmware.com/apps/2017/01/oracle-vmware-collateral-one-stop-shop.html

Why Are You Here?

• You have database servers that are problematic when virtualized

• You have database servers you want to virtualize, but can’t

• You have database servers you are afraid to virtualize

• DBAs are scary when they’re mad!




bution




bution

A Little about David Klee

8

@kleegeek

davidklee.net

heraflux.com

linkedin.com/in/davidaklee

• Speaker, Technologist, DBA, Sysadmin, Geek

• Loves the convergence of infrastructure and data




bution

A Little about Heraflux Technologies

Biz Critical DB Virtualization

Performance & Capacity

Health & Efficiency

HA & DR

Cloud

www.heraflux.com




bution

http://www.heraflux.com/

A Little about Me

• Thomas has over 20 years experience in roles including programmer, developer, analyst, and DBA.

• He enjoys working with data, probably too much to be healthy, really.

#VIRT1430BE CONFIDENTIAL 10



bution

solarwinds.com/products

A Little about SolarWinds




bution

Agenda

13

1 Database Performance Basics

2 Solution Techniques

3 Questions




bution

Agenda

14



3 Questions




bution

Agenda

15



3 Questions




bution

Only Four True Resource Bottlenecks Exist

16

A basic understanding of four resource bottlenecks will help you decide the costs, benefits,

and risks of your overall infrastructure architecture and design




bution

#hardtruth

17

“Bad code and design will bring even the best

hardware to its knees.”

–Thomas LaRock




bution

• Result is wasted time (and money) in a never-ending cycle

– Often due to poor configuration choices!

• As a VM admin, your goal is to make certain the infrastructure is NOT the bottleneck

– Shift bottleneck onto the code instead!

Why Database Performance Matters

18

“Bad code and design will bring even the best hardware to its knees.”

–Thomas LaRock




bution

SQL Server on Linux




bution

20

Don’t believe the FUD!




bution

Why Your Database is *THE* Resource Bottleneck

• Historical design for consolidation (not for performance)

– This meant every server was treated the same

21

THIS BECAME

THAT




bution

Storage latency Networking

Biggest SQL Server Infrastructure Pain Points

22

vCPUconfiguration

vCPU overallocation




bution

Solution Techniques

• Storage latency validation

• VM construction

– Right-sizing VM

– vCPU sizing

– vRAM sizing

– Disk configurations

– Network

• Performance metric collection




bution

Storage Latency

• Every database transaction must be written to disk to complete

• Round trip latency to storage critical

• Max IOPs, throughput less critical

• Round Robin threshold IOPs = 1




bution

VM Construction

• Click through defaults are bad!

• vNUMA

• PVSCSI

• VMXNET3




bution

vCPU Sizing

• CPU allocations

– Often first step in performance tuning

– VM Ready Time

– Co-Stop

– Entitlement vs. Demand Ratio




bution

vCPU Sizing

• CPU allocations


– VM Ready Time

– Co-Stop





bution

vCPU Sizing

• CPU allocations


– VM Ready Time

– Co-Stop


• Out of balance VMs

or

• Too much activity on the host




bution

vCPU Ready Time




bution

Co-Stop Time




bution

Demand vs. Entitlement




bution

vCPU Right-Sizing

• “Right” amount of vCPU and vRAM resources

• Physical world = Size for requirements at end of life

• Virtual world = Size for your requirements right now




bution

vCPU Right-Sizing

• “Right” amount of vCPU and vRAM resources

• Physical world = Size for requirements at end of life

• Virtual world = Size for your requirements right now

• Idle vCPUs can slow application performance of all VMs on the host

• Profile workload resource consumption with DBAs

• Repeat “right-sizing” analysis quarterly




bution

34

Do they need all eight cores?




bution

vCPU vNUMA Sizing

• Fit VM inside one physical NUMA node

– vCPU count & RAM

• Split VM if too large

• NUMA layers

• Physical server

• Hypervisor

• Windows OS

• SQL Server 1x12 CPU Socket

1x12 CPU Socket

VM 1x10

vCPUs

VM 2x8

vCPUs vCPUs




bution

4x4 8x2 2x8 1x16

CPU Sizing – vNUMA Results

• Example: 16 vCPU VM

• What’s better?

– 2 vSocket x 8 vCore?




• Varies by workload, hardware

• Test it for yourself!

36

0

200000

400000

600000

800000

1000000

1200000

1 2 3 4 5

TR

AN

SA

CT

ION

S /

MIN

# USERS

HammerDB & vNUMA - 16 vCPUs




bution

vCPU Hot Add




bution

vSphere 6.5 Auto vNUMA Management




bution

VMware Recommends…

• Large vNUMA-aware VMs – X vSockets, 1 vCore (no longer applies)

• vSphere will now extend vNUMA automatically

• Change in behavior on reboot on mixed host

• Changes parallelism expectations within DB




bution

Memory Allocations

• Memory allocations

– No host memory overcommit at all!

– Reserve all guest memory at VM level (shrinks VM swap file)




bution

Active Memory Counter

41

VMware.com

SQL Server on VMware Best Practices Guide PDF

bit.ly/2mXT32R




bution

Disk I/O

• SQL Server extremely latency sensitive!

• Often legacy storage networks in place; not optimized for virtualized workloads

• Performance factors such as:

– Multipathing

– Interconnects

– Controller cache amounts, ratios

– SSD caching vs. tier

– Raw disk pool speed




bution

Storage – Maximums

• Test raw performance

– DiskSpd Batchhfxte.ch/diskspd

• Collect metrics:

– I/Os per second (IOPs)

– Latency (ms)

– Throughput (MB/s)

• Ceiling vs. Steady State

• NOT a replacement forSQL Server workloadtesting

– Distributed Replay

43

0

1000

2000

3000

4000

5000

6000

1 2 3 4 5 6 7 8

MB

/S (

TH

RO

UG

HP

UT

)WORKER THREADS PER CPU

DiskSpd – MB/s

RAND R SEQ R RAND W SEQ W




bution

http://heraflux.com/go/diskspd

Storage Benchmarking Layers

44

Controller

Controller

LUN

LUN

LUN

LUN

Disk Pool

VM

VM

VM

VM




bution

Blade Server Interconnects


Blade Enclosure

% ?VM

VM

VM

VM

VM

VM

VM

VM

VM

VM

VM

VM

VM

VM

VM

VM

VM

VM

VM

VM

VM

VM

VM

VM

VM

VM

VM

VM

VM

VM

VM

VM

VM



bution

0

200

400

600

800

1000

1200

1400

1600

1800

2000

12:00

AM

12:00

AM

12:00

AM

12:00

AM

12:00

AM

12:00

AM

12:00

AM

12:00

AM

12:00

AM

12:00

AM

12:00

AM

12:00

AM

12:00

AM

12:00

AM

12:00

AM

12:00

AM

12:00

AM

12:00

AM

12:00

AM

12:00

AM

12:00

AM

12:00

AM

12:00

AM

12:00

AM

Thro

ug

hp

ut

(MB

/s)

Time Slice

Avg Aggregate Blade Chassis HBA Throughput

Series1




bution

0

200

400

600

800

1000

1200

1400

1600

1800

2000

12:00

AM

12:00

AM

12:00

AM

12:00

AM

12:00

AM

12:00

AM

12:00

AM

12:00

AM

12:00

AM

12:00

AM

12:00

AM

12:00

AM

12:00

AM

12:00

AM

12:00

AM

12:00

AM

12:00

AM

12:00

AM

12:00

AM

12:00

AM

12:00

AM

12:00

AM

12:00

AM

12:00

AM

Thro

ug

hp

ut

(MB

/s)

Time Slice


Series1 Series2




bution

0

200

400

600

800

1000

1200

1400

1600

1800

2000

12:00

AM

12:00

AM

12:00

AM

12:00

AM

12:00

AM

12:00

AM

12:00

AM

12:00

AM

12:00

AM

12:00

AM

12:00

AM

12:00

AM

12:00

AM

12:00

AM

12:00

AM

12:00

AM

12:00

AM

12:00

AM

12:00

AM

12:00

AM

12:00

AM

12:00

AM

12:00

AM

12:00

AM

Thro

ug

hp

ut

(MB

/s)

Time Slice


Series1 Series2 Series3




bution

0

200

400

600

800

1000

1200

1400

1600

1800

2000

12:00

AM

12:00

AM

12:00

AM

12:00

AM

12:00

AM

12:00

AM

12:00

AM

12:00

AM

12:00

AM

12:00

AM

12:00

AM

12:00

AM

12:00

AM

12:00

AM

12:00

AM

12:00

AM

12:00

AM

12:00

AM

12:00

AM

12:00

AM

12:00

AM

12:00

AM

12:00

AM

12:00

AM

Thro

ug

hp

ut

(MB

/s)

Time Slice


Series1 Series2 Series3 Series4




bution

0

200

400

600

800

1000

1200

1400

1600

1800

2000

12:00

AM

12:00

AM

12:00

AM

12:00

AM

12:00

AM

12:00

AM

12:00

AM

12:00

AM

12:00

AM

12:00

AM

12:00

AM

12:00

AM

12:00

AM

12:00

AM

12:00

AM

12:00

AM

12:00

AM

12:00

AM

12:00

AM

12:00

AM

12:00

AM

12:00

AM

12:00

AM

12:00

AM

Thro

ug

hp

ut

(MB

/s)

Time Slice



2x 8Gb HBA

Maximum

Aggregate

Throughput




bution

0

200

400

600

800

1000

1200

1400

1600

1800

2000

12:00

AM

12:00

AM

12:00

AM

12:00

AM

12:00

AM

12:00

AM

12:00

AM

12:00

AM

12:00

AM

12:00

AM

12:00

AM

12:00

AM

12:00

AM

12:00

AM

12:00

AM

12:00

AM

12:00

AM

12:00

AM

12:00

AM

12:00

AM

12:00

AM

12:00

AM

12:00

AM

12:00

AM

Thro

ug

hp

ut

(MB

/s)

Time Slice



1x 8Gb Max

2x 8Gb HBA

Maximum

Aggregate

Throughput




bution

0

50

100

150

200

250

300

0

200

400

600

800

1000

1200

1400

1600

1800

2000

12:00

AM

12:00

AM

12:00

AM

12:00

AM

12:00

AM

12:00

AM

12:00

AM

12:00

AM

12:00

AM

12:00

AM

12:00

AM

12:00

AM

12:00

AM

12:00

AM

12:00

AM

12:00

AM

12:00

AM

12:00

AM

12:00

AM

12:00

AM

12:00

AM

12:00

AM

12:00

AM

12:00

AM

Av

g L

UN

late

nc

y (

ms)

Thro

ug

hp

ut

(MB

/s)

Time Slice


Series1 Series2 Series3 Series4 Series5




bution

Performance Metric Collection

• Communication

– Talking between storage, server, DBA teams

– “Why do we see spikes at 7PM each night”; “can we stagger backups?”

• Access to vCenter

– DBAs need this access, or similar tools!

– Transparency helps to break down silos

• All layers matter

– Metric overlay




bution

Tips & TricksFor Monster workloads



bution

Don’t Run Everything at Once




bution

Baseline/Benchmark for Performance




bution

Be Mindful of Thin Provisioning




bution

Disk Performance and Datastore Options




bution

Avoid Over-Allocation of CPU and Memory




bution

Capacity Planning




bution

Read It!

61

Master It!




bution

Agenda

62



3 Questions




bution

For More Information

63

@kleegeek

heraflux.com

davidklee.net

linked.com/in/davidaklee

David Klee Thomas LaRock

davidklee.net/resources




bution



bution

virt1430be performance tuning & monitoring for or distribution€¦ · performance tuning &...

Documents