kubernetes and openstack at scale · upstream test suites: ... - placement: "test" #...
TRANSCRIPT
KUBERNETES AND OPENSTACK AT SCALE
Will it blend?
Stephen Gordon (@xsgordon)Principal Product Manager, Red Hat
May 8th, 2017
KUBERNETES AND OPENSTACK AT SCALE #OPENSTACKSUMMIT #REDHAT
ONCE UPON A TIME...Part 1
● 1000 OpenShift Container Platform 3.3 / Kubernetes 1.3 nodes on OpenStack infrastructure
● Presented methodology and results in Barcelona:○ https://www.cncf.io/blog/2016/08/23/deploying-1000-
nodes-of-openshift-on-the-cncf-cluster-part-1/● Goals were:
○ Push limits○ Identify best practices○ Document best practices○ Fix issues
KUBERNETES AND OPENSTACK AT SCALE #OPENSTACKSUMMIT #REDHAT
FOR OUR NEXT TRICK!Part 2
● Goals:○ 2048 OpenShift Container Platform 3.5 / Kubernetes 1.5
nodes on OpenStack infrastructure○ Network ingress tier saturation test○ Overlay2 graph driver w/ SELinux test○ Persistent volume scalability and performance test of
Container Native Storage (glusterfs)
KUBERNETES AND OPENSTACK AT SCALE #OPENSTACKSUMMIT #REDHAT
●
○
●
○
KUBERNETES AND OPENSTACK AT SCALE #OPENSTACKSUMMIT #REDHAT
OPENSTACK KUBERNETES
KUBERNETES AND OPENSTACK AT SCALE #OPENSTACKSUMMIT #REDHAT
OPENSTACK OPENSHIFT
KUBERNETES AND OPENSTACK AT SCALE #OPENSTACKSUMMIT #REDHAT
CONCEPTUAL ARCHITECTURE
●
●●●
●
PREPARATION
KUBERNETES AND OPENSTACK AT SCALE #OPENSTACKSUMMIT #REDHAT
WHERE TO TEST?
KUBERNETES AND OPENSTACK AT SCALE #OPENSTACKSUMMIT #REDHAT
HOW TO TEST?System Verification Test suite (SVT)
● Red Hat OpenShift Performance and Scalability team’s upstream test suites:○ Application Performance○ Application Scalability○ OpenShift Performance○ OpenShift Scalability (incl. cluster-loader)○ Networking Performance○ Reliability/Longevity
● Also includes some additional tools e.g. image provisioner● https://github.com/openshift/svt
KUBERNETES AND OPENSTACK AT SCALE #OPENSTACKSUMMIT #REDHAT
ARCHITECTUREBaremetal Cluster (100 nodes)
OpenShift-on-OpenStack Cluster (2048 nodes)
KUBERNETES AND OPENSTACK AT SCALE #OPENSTACKSUMMIT #REDHAT
ARCHITECTURE (cont.)
● Software:○ Red Hat OpenStack Platform 10, based on “Newton”○ OpenShift Container Platform 3.5 (built around K8S 1.5)○ Red Hat Enterprise Linux 7.3 (mostly…)
● Deployment:○ Deployed OpenStack + Ceph using TripleO○ Deployed OpenShift Container Platform using openshift-ansible.
● Applying previous learnings○ Storage architecture○ Image formatting○ Pre-baked images (see image_provisioner tool)
NETWORK INGRESS/ROUTING
KUBERNETES AND OPENSTACK AT SCALE #OPENSTACKSUMMIT #REDHAT
NETWORK INGRESS/ROUTING TIERTesting HAProxy Performance
● Load generator itself runs in a pod.
● Added SNI and TLS variants to the test suite.
● Configuration by passing in configmaps.
● Focused in on HTTP with keepalive and TLS terminated at the edge.
projects: - num: 1 basename: centos-stress ifexists: delete tuning: default templates: - num: 1 file: ./content/quickstarts/stress/stress-pod.json parameters: - RUN: "wrk" # which app to execute inside WLG pod - RUN_TIME: "120" # benchmark run-time in seconds - PLACEMENT: "test" # Placement of the WLG pods based on node label - WRK_DELAY: "100" # maximum delay between client requests in ms - WRK_TARGETS: "^cakephp-" # extended RE (egrep) to filter target routes - WRK_CONNS_PER_THREAD: "1" # how many connections per worker thread/route - WRK_KEEPALIVE: "y" # use HTTP keepalive [yn] - WRK_TLS_SESSION_REUSE: "y" # use TLS session reuse [yn] - URL_PATH: "/" # target path for HTTP(S) requests
KUBERNETES AND OPENSTACK AT SCALE #OPENSTACKSUMMIT #REDHAT
NETWORK INGRESS/ROUTING TIERTesting HAProxy Performance (cont.)
● 1p-mix-cpu*: nbproc=1, run on any CPU● 1p-mix-cpu0: nbproc=1, run on core 0● 1p-mix-cpu1: nbproc=1, run on core 1● 1p-mix-cpu2: nbproc=1, run on core 2● 1p-mix-cpu3: nbproc=1, run on core 3● 1p-mix-mc10x: nbproc=1, run on any core,
sched_migration_cost=5000000● 2p-mix-cpu*: nbproc=2, run on any core● 4p-mix-cpu02: nbproc=4, run on core 2
NETWORK
KUBERNETES AND OPENSTACK AT SCALE #OPENSTACKSUMMIT #REDHAT
NETWORK PERFORMANCETesting OpenShift-sdn (OVS+VXLAN) Performance
● OpenShift includes and uses OpenShift-sdn (OpenvSwitch + VXLAN) by default:○ Provides full multi-tenancy○ Is fully pluggable (as is ingress/routing tier)○ Supports all four footprints (physical/virtual/private/public)
● Web-based workloads are mostly transactional● Focused microbenchmark on a ping-pong test of varying payload sizes
KUBERNETES AND OPENSTACK AT SCALE #OPENSTACKSUMMIT #REDHAT
NETWORK PERFORMANCETesting OpenShift-sdn (OVS+VXLAN) Performance (cont.)
● Tested mix of payload sizes and stream counts.
● tcp_rr-XXB-Yi○ XX = # of bytes○ Y = # of instances
(streams)● Slimmed down version of
RFC2544
STORAGE
KUBERNETES AND OPENSTACK AT SCALE #OPENSTACKSUMMIT #REDHAT
OVERLAY2 w/ SELINUXNext on storage wars...
● Until recently RHEL used Device Mapper for docker’s storage graph driver○ Overlay support added in RHEL 7.2○ Overlay2 supported added in RHEL 7.3○ Overlay2 support w/ SELinux added upstream and expected in RHEL 7.4
■ https://lkml.org/lkml/2016/7/5/409○ Device Mapper remains default in RHEL for now, Overlay2 default in Fedora
26■ https://fedoraproject.org/wiki/Changes/DockerOverlay2
● Let’s try it out!
KUBERNETES AND OPENSTACK AT SCALE #OPENSTACKSUMMIT #REDHAT
OVERLAY2 w/ SELINUXResults
● Single base image for all pods
● 240 pods on the node (rate limited creation)
● Reasonable memory savings
KUBERNETES AND OPENSTACK AT SCALE #OPENSTACKSUMMIT #REDHAT
OVERLAY2 w/ SELINUXResults
KUBERNETES AND OPENSTACK AT SCALE #OPENSTACKSUMMIT #REDHAT
CONTAINER NATIVE STORAGEApproach
● OpenShift Container Platform supports a wide variety of volume providers via the standard Kubernetes volume interface
● Red Hat Container Native Storage is a Gluster-based persistent volume provider deployed on OpenShift
● Used the NVMe disks as “bricks” for Gluster, exposed 1G persistent volumes ● Container Native Storage nodes marked unschedulable for other OpenShift
pods● Ran throughput numbers for create/delete operations, as well as API
parallelism
KUBERNETES AND OPENSTACK AT SCALE #OPENSTACKSUMMIT #REDHAT
CONTAINER NATIVE STORAGEResults
● CNS allocated volumes in constant time
● Consistent with results for other persistent volume providers
NEXT STEPS
KUBERNETES AND OPENSTACK AT SCALE #OPENSTACKSUMMIT #REDHAT
NEXT STEPSTo infinity, and beyond!
● Filed 40+ bugs across a variety of projects and components● Scaling and Performance Guide, new with OpenShift Container Platform 3.5● Getting Involved:
○ “Kubernetes Ops on OpenStack” forum session■ Wednesday, May 10, 1:50pm-2:30pm■ Hynes Convention Center MR102
○ K8S SIG Scalability○ K8S SIG OpenStack
● Interested in seeing what OpenShift Container Platform on Red Hat OpenStack Platform incl. some real example applications?
KUBERNETES AND OPENSTACK AT SCALE #OPENSTACKSUMMIT #REDHAT
REFERENCES
● Part 1: https://www.cncf.io/blog/2016/08/23/deploying-1000-nodes-of-openshift-on-the-cncf-cluster-part-1/
● Part 2: https://www.cncf.io/blog/2017/03/28/deploying-2048-openshift-nodes-cncf-cluster-part-2/
● Overlay2 and Device Mapper https://developers.redhat.com/blog/2016/10/25/docker-project-can-you-have-overlay2-speed-and-density-with-devicemapper-yep/
● Red Hat Performance and Scale Trello: https://trello.com/b/M1bpo55E/scalability
THANK YOUplus.google.com/+RedHat
linkedin.com/company/red-hat
youtube.com/user/RedHatVideos
facebook.com/redhatinc
twitter.com/RedHatNews