introduction to achieving performance and scale on...

Introduction to Achieving Performance and Scale on Large OpenStack Clouds

Alec Hothan, Principal Engineer

DEVNET-1693

• OpenStack Performance and Scale: challenges and existing tools

• A new generation of fully integrated tools to measure tenant network and storage at scale

• Design of these tools

• What they measure and how

• How to read results

• Free and open-source: install and use the tools yourself

Agenda

OpenStack Performance and ScaleTools and challenges

© 2016 Cisco and/or its affiliates. All rights reserved. Cisco Public 4DEVNET-1693

OpenStack Performance Metrics

Control Plane Metrics

Resource operations

Storage/Data PlaneTraffic metrics

Host system metrics

Networking

Compute Storage

tenants, userscpu, memory… networks

routers…

volumesimages…

Bare Metal

© 2016 Cisco and/or its affiliates. All rights reserved. Cisco Public 5DEVNET-1693

Measuring these metrics is challenging

Abundance of tools

Complex measuring conditions

Automation is key

ps, top, strace, ptrace, perf…

iperf, netperf, nuttcp…

fio, jmeter, siege, wrk, cb

OpenStack rally, tempest

Cloudbench, PerfKitBenchmarker…

OpenStack variants/integration

Cloud scale

Failure conditions

Repeatable

Consistent

© 2016 Cisco and/or its affiliates. All rights reserved. Cisco Public 6DEVNET-1693

Example: web services and user experience

Density

Throughput

Latency

HTTP servers per compute node

HTTP requests per second

Latency percentile requirements:

User perception…

90th %ile 99th %ile Max

Fast

< 0.1 sec

Noticeable

< 1 sec

Slow

< 10 sec

© 2016 Cisco and/or its affiliates. All rights reserved. Cisco Public 7DEVNET-1693

Latency histogram (each traffic generator)

Latency

histogram

min

90th %ile

99th %ile

max

© 2016 Cisco and/or its affiliates. All rights reserved. Cisco Public 8DEVNET-1693

Latency histogram aggregation

+N x latency histograms

(distributed)

Overall system latency

histograms

© 2016 Cisco and/or its affiliates. All rights reserved. Cisco Public 9DEVNET-1693

Latency histograms at cloud scale

Number of HTTP servers per cloud 1000

Number of users per HTTP server x 1000

Number of requests per user/sec x 1

Test duration in seconds x 100

Total number of requests 100M

Number of latency histograms to consolidate

(1000 users/HTTP traffic generator)1000

A new generation of performance/scale tools

© 2016 Cisco and/or its affiliates. All rights reserved. Cisco Public 11DEVNET-1693

Tools characteristics

Fully integrated

End to end

Accurate and precise

Tight integration with OpenStack

Automated

VM and user level metrics

All batteries included

< msec precision

Correct consolidation of metrics

Simple and AccessibleInstallation

Usage

Result interpretation

© 2016 Cisco and/or its affiliates. All rights reserved. Cisco Public 12DEVNET-1693

OpenStack tools for performance/scale

Control Plane API-level

Performance

Single Flow Performance

KloudBuster

Networking

Compute Storage

Rally

High Scale Performance

VMTP

✅

✅

✅ ✅

✅ ✅

(coming soon)NFV Data Plane✅

© 2016 Cisco and/or its affiliates. All rights reserved. Cisco Public 13DEVNET-1693

Neutron Tenant Network Flows

VMs can run on same or

different compute nodes

North-South flows (NAT)

East-West flows

© 2016 Cisco and/or its affiliates. All rights reserved. Cisco Public

Networking

Compute Storage

14DEVNET-1693

VMTP

• Single flow VM-level performance

• TCP

• UDP unicast and multicast

• ICMP

• Tenant network flows

• East-West and North-South

• Intra-node and inter-node

Validation and Performance Measurement of Network Flows

• Encapsulation and Neutron implementation agnostic

• Also supports

• Provider network

• SR-IOV NICs

© 2016 Cisco and/or its affiliates. All rights reserved. Cisco Public 15DEVNET-1693

VMTP Throughput Chart

© 2016 Cisco and/or its affiliates. All rights reserved. Cisco Public 16DEVNET-1693

KloudBuster

• Configure:

• Number of HTTP servers (1 per VM)

• Number of HTTP clients per server

• HTTP requests/sec

• VM placement policy

• Stages and orchestrates everything

• Simple runs

• Progression runs (HTTP servers)

• Real time latency monitoring

OpenStack Data Plane Performance at Scale

• Aggregates and consolidates all results

• Total throughput

• Micro-sec precision latency histograms• HDR (High Dynamic Range) histogram

library

Networking

Compute Storage

© 2016 Cisco and/or its affiliates. All rights reserved. Cisco Public

VM VM VM

VM

proxy

VMVM

KB

17DEVNET-1693

Data Plane Scale Overview (East-West 1 cloud)

N x http servers

N x (http traffic generators (m users))

East-West L3 traffic

N x m flows

staging

orchestration

results aggregation

Internal network

tenant

© 2016 Cisco and/or its affiliates. All rights reserved. Cisco Public

proxy VM VM VM VM

VM VMVM

VM

KB

18DEVNET-1693

Data Plane Scale Overview (North-South 2 clouds)

N x http servers

N x (http traffic generators (m users))

North-South traffic

N x m flows

staging

orchestration

results aggregation

© 2016 Cisco and/or its affiliates. All rights reserved. Cisco Public 19DEVNET-1693

Latency Chart: 1K to 20K HTTP Connections

© 2016 Cisco and/or its affiliates. All rights reserved. Cisco Public 20DEVNET-1693

Real-Time Latency Monitoring (vPC failure)One protected

interface going down

Latency stacked

iso-percentile

scrolling layers

© 2016 Cisco and/or its affiliates. All rights reserved. Cisco Public 21DEVNET-1693

KloudBuster

• Configure:

• VM count

• Cinder volumes (block storage) or ephemeral disks

• Storage workload definition

• Stages and orchestrates everything

• Progression runs (VM count)

OpenStack Storage Performance at Scale

• Aggregates and consolidates all results

• Total throughput

• micro-second precision latency histograms (HDR)

Networking

Compute Storage

© 2016 Cisco and/or its affiliates. All rights reserved. Cisco Public 22DEVNET-1693

KloudBuster Storage Scale Test Overview

ephemeral

or block storagestorage network

tenant networkstaging

orchestration

results aggregation

N x Fio testers

© 2016 Cisco and/or its affiliates. All rights reserved. Cisco Public 23DEVNET-1693

Generated Charts (NFVi profile/Ceph backend)

progressions

concurrent VMs

Latency iso-percentile

line charts

workload results tabs

© 2016 Cisco and/or its affiliates. All rights reserved. Cisco Public 24DEVNET-1693

High IOPs profile (61K IOPs random read Cinder)

© 2016 Cisco and/or its affiliates. All rights reserved. Cisco Public 25DEVNET-1693

High Throughput profile (3,743 MB/s seq read Cinder)

Installation and User Interfaces

© 2016 Cisco and/or its affiliates. All rights reserved. Cisco Public

Installation and User InterfacesVMTP KloudBuster

Docker Container docker pull berrypatch/vmtp docker pull berrypatch/kloudbuster

PyPI installation (pip) pip install vmtp pip install kloudbuster

VM Image (REST/Web server) http://storage.apps.openstack.org/images/kloudbuster_v6.qcow2

git clone git clone https://github.com/openstack/vmtp.git git clone https://github.com/openstack/kloudbuster.git

CLI ☑️ ☑️

REST ☑️

Web UI ☑️

27DEVNET-1693

Q & A

© 2016 Cisco and/or its affiliates. All rights reserved. Cisco Public

What’s next

• Meet us at the Cisco Live! DevNet OpenStack Demo Zone

• Video demos of VMTP and KloudBuster

• Contact: ahothan@cisco.com

• VMTP: https://github.com/openstack/vmtp

• KloudBuster: https://github.com/openstack/kloudbuster

29DEVNET-1693

© 2016 Cisco and/or its affiliates. All rights reserved. Cisco Public

Complete Your Online Session Evaluation

Don’t forget: Cisco Live sessions will be available for viewing on-demand after the event at CiscoLive.com/Online

• Give us your feedback to be entered into a Daily Survey Drawing. A daily winner will receive a $750 Amazon gift card.

• Complete your session surveys through the Cisco Live mobile app or from the Session Catalog on CiscoLive.com/us.

30DEVNET-1693

© 2016 Cisco and/or its affiliates. All rights reserved. Cisco Public

Continue Your Education

• Demos in the Cisco campus

• Walk-in Self-Paced Labs

• Lunch & Learn

• Meet the Engineer 1:1 meetings

• Related sessions

31DEVNET-1693

Please join us for the Service Provider Innovation Talk featuring:

Yvette Kanouff | Senior Vice President and General Manager, SP Business

Joe Cozzolino | Senior Vice President, Cisco Services

Thursday, July 14th, 2016

11:30 am - 12:30pm, In the Oceanside A room

What to expect from this innovation talk

• Insights on market trends and forecasts

• Preview of key technologies and capabilities

• Innovative demonstrations of the latest and greatest products

• Better understanding of how Cisco can help you succeed

Register to attend the session live now or

watch the broadcast on cisco.com

Thank you