monitor open stack environments from the bottom up and front to back

Monitor Open stack environments from the bottom up and front to back

Roger Ruttimann VP Engineering, GroundWork OpenSource November 17, 2015

What’s ahead of us

Overview: The Impact of Virtualization on IT operations

How Open Stack fits into the Virtualization landscape

Monitoring the changed landscape of IT infrastructure

New Monitoring concepts

Selection of the right tools to fit the right process

Conclusion

2 OSCM 2015 Monitor OpenStack environments from bottom up and front to back by Roger Ruttimann

Start with the facts: Open Stack is real!

It’s all OpenSource Has 5000 active members, growing daily Conferences draw close to 10,000 participants Many corporate sponsors

3

OpenStack deployments are not just happening in a far away land ... Not secret potion or evil conspiracy Not just a geek movement Serious deployments for IT Operations not just in the Silicon Valley

It’s disruptive and requires serious re-tooling for IT operations

Let’s review the challanges and what tools best fit the new realities

Box Spy

OSCM 2015 Monitor Open Stack environments from bottom up and front to back by Roger Ruttimann

Once upon a time

SysAdmin task: Add To IT infrastructure {

Deploy servers into datacenter Provisioning applications Define monitoring for each element Monitoring was done with SSH checks and port gets

} repeat Checking for up down and send email


Switch

Router

Switch

Apps

Apps

Apps

Apps

Apps

Firewall

Server Server Server

Monitor

Port get

ssh

snmp

snmp

snmp

virtualization

Then came along compute virtualization

5

Datacenter compute optimization •  Server resources were virtualized to

improve efficiency which was < 30% •  Products like ESX allowed resource

optimization not disturbing the provisioning process

Rollout •  VM machines were provisioned •  Applications installed •  Monitoring defined as before New was an API to the virtualization manager (example VSphere API) to poll data

OSCM 2015 Monitor OpenStack environments from bottom up and front to back by Roger Ruttimann

Switch

Router

Switch

Apps

Apps

Apps

Apps

Apps

Firewall


Monitor

Port get

ssh

snmp

snmp

snmp

•  REST API

But it didn’t stop there

6

Software Defined Everything •  Compute (Hypervisors) •  Storage (SDS) •  Network (SDN)

Hybrid Cloud Public/Private •  Amazon WebServices •  Rackspace •  Azure

Change created blind spots in coverage Suddenly ssh/snmp methods don’t cover all

KVM

RHEV-M VCenter

virtualization

Switch

Router

Switch

Apps

Apps

Apps

Apps

Apps

Firewall


Monitor

Port get

ssh

snmp

snmp

snmp

•  REST API

Virtualized Network

Virtualized Storage

Switches Switches Switches

•  REST API

•  REST API


Infrastructure landscape completely changed

7

Silo

vie

w

Hyp

ervi

sor m

anag

er v

iew

Apps VMs

Devices

Files Shared I/O Fog

Network & Storage OSCM 2015 Monitor OpenStack environments from bottom up and front to back by Roger Ruttimann

Silo

vie

w

Silo

vie

w

Silo

vie

w

Silo

vie

w

Silo

vie

w

And the moment you think you’ve seen it all …

8

… DevOps is pushing the envelope even more

Linux Containers

Are the new kids in town

Application Isolation Rapid deployment Elastic Scalability

API for deployment:

Monitoring is somebody else’s job

virtualization

Switch Switch

Apps

Apps

Apps

Apps

Apps


Monitor

Port get

ssh

•  REST API

Virtualized Network

Virtualized Storage

Switches Switches Switches

•  REST API

•  REST API

Apps Apps Apps

Server

Docker Host

Apps

Server

Docker Host

Apps Apps Apps Apps

Server

Docker Host

Apps Apps Apps Apps


Don’t panic

9

What happened over the last 5 years: •  Virtualize everything •  Private and Public clouds

•  API centric world not just for Applications

•  REST API is a standard •  CORBA, SOAP & proprietary API’s are

now classified as dinosaurs

•  Browser based UI / JavaScript is king •  Learn to speak REST & JSON

Diverse Virtualization stacks

vSphere API

ESX

NSX

vSan

OpenStack API

Nova

Neutron

Cinder Glance

AWS API

EC2

VPC

S3

oVirt API

Linux KVM

Network

NFS


How does Open Stack fit in?

10

Service oriented architecture Pluggable hypervisors, network, storage to support wide range of technologies Elastic Compute Units for better virtualization efficiency Standardized API’s to all services

Sounds really good doesn’t it ?

Object Store--Swift

Image--Glance Compute/

Hypervisor--Nova

Network— Quantum-Neutron Storage--Cinder

Dashboard--Horizon

Identity--Keystone

API API API API API

API

But what’s the reality ?

Façade Service Façade Service


Open Stack Profile Editor


ManageIQ/ Cloud Forms

Ansible

API

Manager Stack

API

HP Helion API

VMWare VRealize

Mirantis Fuel

API

API

Provisioning/Configuration

Servers Servers

API is good but comprehensive coverage is better

13

Object Store

Image

Computer

Network

Identity

Storage

Dashboard

Open Stack API


Container metrics

Performance data

Virtual infrastructure

Legacy network

Servers

Storage racks

Applications, Infrastructure checks

ssh/SNMP/port get

Unified View Standardized Data Collection

Unified View

14

Silo tools •  Impossible to correlate •  Naming mismatch •  Over-monitoring •  Encourages: “It’s not my problem

check your system” •  No Big Picture Dashboards

Need for •  Stack monitoring bottom to top •  Aliasing of names •  Combine best of breed collectors •  Correlated metrics across all

infrastructure •  Dashboard for each customer/client


Stack monitoring for Open Stack

15

Function Source Host name Alias

Network, Storage Netflow SNMP

Hardware IPMI SNMP

Drac-server1 os-eng-h1

Identity server check-mysql check-port

server1:port os-eng-h1

Operating System

check-proc check-mem check-load

server1 os-eng-h1

OpenStack-API Hypervisor CPU/Mem metrics, Network storage

10.10.0.1 os-eng-h1

OpenStack API VM CPU/mem, Network, storage 10.10.10.123

eng-slicer-1

Operating system

check-proc check-mem check-load

eng-cent6-actg

eng-slicer-1

DockerHost Memory, CPU eng-cent6-actg eng-slicer-1

Container Memory, CPU, Procs FAC3443DA77 Load-bal-01

Application Check_https

172.28.102.51 Load-bal-01

Hyp

ervi

sor

VM

A

PP

Automation / Continuous configuration discovery

16

Adding Virtual machines, containers is automatic


V C

Hypervisor

So is the monitoring

Management Monitor System

Configuration collector

Data collector

Plugins

Dashboards Data

C C V V V V V C

Monitoring is like a flight recorder -- collecting all active data

Continuous operations by .. synchronization of configuration

re-configuration of plugins Collection of data (availability & performance)

Open Stack API

Icinga2 setup

OpenStack Monitoring: Tools selection


OpenStack Linux OS Platform

Hypervisor Hardware IPMI Port

Agent

OpenStack Controller Keystone

OpenStack Storage Cinder

Unified Monitoring View

Data Management Collector Collector Collector

Fuel

Murano Application Software Catalog

Icinga2

1)  Fuel, Open Stack deployment 2)  Murano, Application deployment 3)  Fuel Plugin, Configuration generate 4)  Fuel Plugin, Configure monitoring

Docker

Node 2

Node 3

Docker

VM 1

OPEN TSDB

Icinga2

VM 1 VM 2 VM 3 VM 4

Box Spy

Box Spy

Docker OPEN TSDB

Performance Data

18

Monitoring systems

Grafana Dashboards

VCenter


Centralized Data Collector •  Monitoring send Perf Data to single

API Expandable storage cluster •  Expand on demand

Dashboard to visualize and drill down •  Historic, raw performance data •  Group alike metrics to find outliers

Reference Architecture monitoring

Aliasing and data normalization in backend

VCenter

API

Data Collector

Hub

Collector Collector Collector

Collect

Data Integration

Normalization Aliasing

API

Integrate

API

Visualize


Growing number of API’s require integration Hub

Benefits

20

Automation •  Reduced cost of maintenance •  Dynamic configuration •  Streamlined operations

Personnel •  System Administrator & Dev Ops

functions merge

Capacity & Resource planning •  Complete Bottom (network) to Top

(application) data collection •  Allocate resources at the right level

Hardware •  No vendor lock-in •  Transparency through API


Lessons learned

21

Virtualization and containerization require new monitoring techniques Legacy hardware/software will be around for a while so don’t throw tools away Everything API -- needs integration Aliasing is hard but maybe easier than cross silo cooperation Use the best tool for the task Don’t over-monitor to cover a gap


Conclusion

22

Open Source tools are the driving force for innovation

DevOps just selects the best tool to do the job

A single monitoring tool to “rule them all” doesn’t exist

Integration is a complex task – Don’t expect teams will agree on naming, process, and workflow

Automation and pragmatism will prevail as DevOps has demonstrated

Virtualize everything will continue

Automation and continuous discovery is necessary for rapid scale out

Open Stack is the Open Source Virtualization platform but Monitoring coverage needs major improvements


23

Thank you

Roger Ruttimann [email protected]


monitor open stack environments from the bottom up and front to back

Documents