symantec sdn deployment

Symantec SDN Deployment

Jasmeet Sidhu, Rudrajit TapadarCloud Platform Engineering

Class of Service

Copyright © 2015 Symantec Corporation 2

Class of Service

• Dev– For developers to get familiar with OpenStack cloud– Each developer has a project

• Production– For teams to onboard their members– Each team has a project– Manage user roles– Manage production workloads

Copyright © 2015 Symantec Corporation3

Self-Service User Onboarding


Self-Service User Onboarding

• Zero tickets for user onboarding– Provide sign up capabilities on Horizon

• Provide easy networking on Dev CoS– Hide all complexities– Automatically create network– Allocate routable subnets by using Contrail VNC APIs– Create security group with proper rules– Create unique domain names for instances by using Designate for routable

IPs


Load Balancer as a Service


Load Balancer as a Service

•Out of the box

– Icehouse, v1 APIs

– Launch HA Proxy service instances on a single AZ

– SSL Support: Wildcard cert

• Symantec fixes

–Multiple AZ, SSL Passthrough, Stats and Metrics

• Performance:

–~6.5 Gbps throughput with 10K parallel connections, VIP with 2 members

–20K HTTPS requests/sec for 10K parallel connections with 1 million requests, 1K response size

• Tuning - haproxy.cfg: maxconn 50K, nbproc 4, ulimit-n 200K, Cipher

• Pain points

–No control over ha proxy cfg

–No control over resource allocations (cpu, etc)Copyright © 2015 Symantec Corporation

7

Baremetal on Overlay


Baremetal on Overlay

•Applications that run on baremetal but needs to be on the overlay– Example: swift proxy and data nodes– Launch them inside network namespaces– Plug them to the vRouter– East-West Traffic

• Manual Setup via scripts– Nova is not aware but Contrail is.– Multiple nics sitting on multiple networks– Static IPs


Availability::Control Plane


Control Plane Availability

• Goal - 99.95% Availability

• 5 SDN controller VMs distributed over 3 racks

• 5 Cassandra database baremetal nodes distributed over 3 racks– RF of 3 for analytics– RF of 5 for config– Compaction throughput 256 Mbps

• Deployment Automation: Puppet

• Issues seen: DB Timeouts, Version mismatch, admin token


Failed Customer Interactions


Failed Customer Interactions

• Measure the control plane availability

• Use Symantec’s Logging-Monitoring-Metering as a Service to parse Neutron logs

• Compare response codes: 5XX counted as failures

• Dashboards!


Availability::Data Plane


Data Plane Availability

• Work in progress..

–FIP Availability

–vDNS

–Link Local

–Private Network


Seamless Upgrades


Upgrade 1.20 to 2.0.1

• Goal - Zero Downtime

• Controller upgrades– No in-place upgrades– Build a parallel control plane with new release– Add them to the VIP pool and gradually decommission old controllers

• Database upgrades– Add new DB nodes one by one to the existing cluster– Repair the DB– Decommission old DB node one by one

• Compute upgrades– Automate unloading and loading of kernel module in all computes


Health Monitoring


Health Monitoring

• Volta –Logging •Logstash•Elasticsearch

–Metrics•InfluxDB•Statsd•Collectd)

–RESTful APIs make it easy:•Response Codes, Bytes Transfered, Time, Verb, etc.

• OpsView / Zabbix


Troubleshooting


Troubleshooting

• Most incidents are trivial– Known issues– Trivial fixes/workarounds

• Some incidents are complex– RCA is very involved– Might have to wait for next code release for a fix– Quick and dirty solution – use auto healing scripts for workarounds

•Periodically check system health (Synthetic Transactions)•Remediate known bugs•Fix problems as they are detected, Save pagers, run 24x7!(MX Encapsulation, Dead processes, etc.)


Thank you!

Copyright © 2014 Symantec Corporation. All rights reserved. Symantec and the Symantec Logo are trademarks or registered trademarks of Symantec Corporation or its affiliates in the U.S. and other countries. Other names may be trademarks of their respective owners.

This document is provided for informational purposes only and is not intended as advertising. All warranties relating to the information in this document, either express or implied, are disclaimed to the maximum extent allowed by law. The information in this document is subject to change without notice.

symantec sdn deployment

Engineering

symantec corporation

control plane copyright

overlay copyright

data plane copyright

class of service copyright

seamless upgrades copyright

routable ips copyright

production workloads