rancher kubernetes cluster using csi driver for dell emc vxflex … · 2021. 1. 31. · using...

000066

White Paper

Rancher Kubernetes Cluster using CSI Driver for Dell EMC VxFlex family

Abstract This white paper provides guidance on the deployment of a Rancher Kubernetes

Cluster on to the Dell EMC VxFlex family and the integration of the VxFlex CSI

driver for persistent volumes, for customers requiring an on-premises container

platform solution.

May 2020

Revisions

2 Rancher Kubernetes Cluster using CSI Driver for Dell EMC VxFlex family | 000066

Revisions

Date Description

April 2020 Initial release

May 2020 Updated Section 3: Solution Architecture

Acknowledgements

This paper was produced by the following:

Author: Sanjay Puttaswamy

Support: David J Adams and Shashikiran Chidambara

Others: Sridhar Subramanian, Shalini G, Kevin Jones and Simon Stevens

The information in this publication is provided “as is.” Dell Inc. makes no representations or warranties of any kind with respect to the information in this

publication, and specifically disclaims implied warranties of merchantability or fitness for a particular purpose.

Use, copying, and distribution of any software described in this publication requires an applicable software license.

Copyright © 2020 Dell Inc. or its subsidiaries. All Rights Reserved. Dell, EMC, Dell EMC and other trademarks are trademarks of Dell Inc. or its

subsidiaries. Other trademarks may be trademarks of their respective owners. [5/14/2020] [White Paper] [000066]

Table of contents


Table of contents

Revisions............................................................................................................................................................................. 2

Acknowledgements ............................................................................................................................................................. 2

Table of contents ................................................................................................................................................................ 3

Executive summary ............................................................................................................................................................. 4

1 Introduction ................................................................................................................................................................... 5

1.1 Objective ............................................................................................................................................................. 5

1.2 Audience ............................................................................................................................................................. 5

1.3 Terminology ........................................................................................................................................................ 5

2 Product overview .......................................................................................................................................................... 6

2.1 VxFlex family ...................................................................................................................................................... 6

2.1.1 VxFlex OS .................................................................................................................................................. 6

2.2 Rancher for Kubernetes ..................................................................................................................................... 7

2.2.1 How Rancher delivers production-grade Kubernetes at scale ................................................................... 7

3 Solution infrastructure .................................................................................................................................................. 9

3.1 Logical architecture ............................................................................................................................................ 9

3.2 Two-layer network topology .............................................................................................................................. 10

4 Installation of Rancher Kubernetes cluster ................................................................................................................ 12

4.1 Requirements ................................................................................................................................................... 12

4.2 Install Kubernetes cluster using RKE ............................................................................................................... 14

4.3 Install Rancher server ....................................................................................................................................... 16

5 VxFlex OS Container Storage Interface driver ........................................................................................................... 19

5.1 Overview ........................................................................................................................................................... 19

5.2 Prerequisites ..................................................................................................................................................... 19

5.3 Install CSI drives for VxFlex OS ....................................................................................................................... 20

6 Conclusion .................................................................................................................................................................. 23

A Appendix ..................................................................................................................................................................... 24

A.1 Configuration details ......................................................................................................................................... 24

A.2 Rancher Kubernetes cluster details.................................................................................................................. 25

A.2.1 cluster.yaml file details ............................................................................................................................. 25

B Technical support and resources ............................................................................................................................... 27

B.1 Related resources............................................................................................................................................. 27

Executive summary


Executive summary

Container technology enables development teams to quickly provision isolated applications without the

traditional complexities. For many companies, to boost productivity and time to value, the use of containers

starts with the departments that are focused on software development.

Kubernetes orchestration provides capabilities such as auto scaling, security, and managing containerized

applications. In order to run containerized applications within a Kubernetes cluster, a persistent and stable

data store is required, one that can outlast the pod or even the node it is running on.

Rancher is a Kubernetes management platform that simplifies cluster installation and operations, whether

they are on-premises, in the cloud or at the edge, giving DevOps teams the freedom to build and run

containerized applications anywhere.

The VxFlex family offers key value propositions for traditional and cloud-native production workloads,

deployment flexibility, linear scalability, predictable high performance, and enterprise-grade resilience.

This paper discusses how to deploy a Kubernetes cluster using Rancher Kubernetes Engine (RKE) on a

VxFlex Family cluster and the use of the VxFlex OS CSI driver to provision persistent storage for Kubernetes

managing containerized applications.

Introduction


1 Introduction

1.1 Objective The goal of this paper is to provide a simple approach to creating a Kubernetes cluster using Rancher

Kubernetes Engine and managed using Rancher as the container orchestration layer on Dell EMC VxFlex

family platform to meet the performance, scalability, resiliency, and availability requirements. The document

also outlines information about leveraging Dell EMC VxFlex OS CSI driver to dynamically provision persistent

volumes on a Rancher managed Kubernetes cluster.

1.2 Audience The audience for this paper includes sales engineers, field consultants, IT administrators, customers, and

anyone else interested in configuring and deploying a Kubernetes cluster using RKE and using the VxFlex OS

CSI driver to dynamically provision persistent volumes in a Rancher managed Kubernetes cluster.

Readers are expected to have an understanding and working knowledge of Containers, Kubernetes, and

VxFlex OS.

1.3 Terminology The following table defines acronyms and terms that are used throughout this document:

Terms and definitions

Term Definition

CA Certificate Authority

CSI Container Storage Interface

DNS Domain Name System

MDM Meta Data Manager

OS Operating System

PV Persistent Volume

PVC Persistent Volume Claim

RKE Rancher Kubernetes Engine

SDC Storage Data Client for VxFlex

SDS Storage Data Server for VxFlex

SSD Solid-State Disk for VxFlex

TLS Transport Layer Security

VLAN Virtual Local Area Network

VM Virtual Machine

Product overview


2 Product overview

2.1 VxFlex family The VxFlex software-defined storage platform is designed to deliver flexibility, elasticity and simplicity with

predictable performance and resiliency at scale by combining compute as well as high-performance storage

resources in a managed unified fabric. VxFlex is an ideal platform for modernizing infrastructure to improve IT

agility. The VxFlex Family currently includes VxFlex Ready Nodes, VxFlex appliance, and VxFlex integrated

rack.

VxFlex family

2.1.1 VxFlex OS VxFlex OS is the software foundation of VxFlex software-defined storage. It is a scale-out block storage

service that is designed to deliver flexibility, elasticity and simplicity with predictable high performance and

resiliency at scale.

VxFlex OS consists of the following components:

Storage Data Client (SDC)

• Provides front-end volume access to applications and file system

• Installed on servers consuming storage

• Maintains peer-to-peer connections to every SDS managing a pool of storage

Storage Data Server (SDS)

• Abstracts local storage, maintains storage pools, and presents volumes to the SDCs

• Installed on servers contributing local storage to the VxFlex OS cluster

Product overview


Meta Data Manager (MDM)

• Oversees storage cluster configurations, monitoring, rebalances, and rebuilds

• Highly available, independent cluster installed on three or five different nodes

• May reside alongside SDCs or SDSs, or on separate nodes

• Sits outside the data path

Gateway

• Performs installation and configuration checks

• Acts as an endpoint for API calls and passes them to MDM

2.2 Rancher for Kubernetes Rancher is the enterprise computing platform to run Kubernetes on-premises, in the cloud and at the edge. It

addresses the operational and security challenges of managing multiple Kubernetes clusters everywhere.

Rancher also provides IT operators and development teams with integrated tools for building, deploying, and

running cloud-native workloads.

Rancher not only deploys production-grade Kubernetes clusters from data center to cloud to the edge, it also

unites them with centralized authentication, access control and observability. Rancher lets you streamline

cluster deployment on bare metal, edge devices, private clouds, public clouds, or vSphere and secure them

using global security policies. Use Helm or Rancher App Catalog to deploy and manage applications across

any or all these environments, ensuring multi-cluster consistency with a single deployment.

2.2.1 How Rancher delivers production-grade Kubernetes at scale

Rancher's recipe for production quality Kubernetes at scale

Product overview


Certified Kubernetes Distributions

Rancher supports any certified Kubernetes distribution. For on-premises workloads, Rancher offers the RKE -

a CNCF-certified Kubernetes distribution that runs entirely within Docker containers. It works on bare-metal

and virtualized servers. RKE solves the problem of installation complexity, a common issue in the Kubernetes

community. With RKE, the installation and operation of Kubernetes is both simplified and easily automated,

and it is entirely independent of the operating system.

For the public cloud, all the major distributions are supported, including EKS, AKS, and GKE. For edge,

branch and desktop workloads Rancher offers K3s, a certified lightweight distribution of Kubernetes.

Simplified Cluster Operations

Rancher provides simple, consistent cluster operations including provisioning, version management, visibility

and diagnostics, monitoring and alerting, and centralized audit.

Security, Policy, and User Management

Rancher lets you automate processes and applies a consistent set of user access and security policies for all

your clusters, no matter where they are running.

Shared Tools and Services

Rancher provides a rich catalog of services for building, deploying, and scaling containerized applications,

including app packaging, CI/CD, logging, monitoring, and service mesh.

Solution infrastructure


3 Solution infrastructure This solution could be deployed on any of the VxFlex family. In this solution, it is deployed in a two-layer

configuration using compute only nodes deployed with VMware ESXi hypervisor and dedicated storage only

nodes to provide the required storage capacity.

3.1 Logical architecture The diagram below shows the logical architecture diagram of the RKE cluster deployed on a two-layer VxFlex

cluster setup with four storage only nodes and three compute only nodes.

Logical architecture

Each Storage Only node includes two Intel Xeon Skylake 8-core processors, 192 GB RAM, and six 3.84 TB

SSDs. From the VxFlex OS standpoint, the Red Hat Enterprise Linux 7.6 Storage only nodes run the SDS

component of VxFlex OS to provide storage capacity. A single protection domain is created from these four

SDS systems. A single storage pool is created within this Protection Domain from which persistent storage

volumes can be provisioned for Kubernetes pods. The three ESXi 6.7 Compute only nodes run the SDC

components of the VxFlex OS in order to access the volumes created in the storage pool.

The RKE cluster is deployed as virtual machines on a VMware vSphere 6.7 cluster. For Rancher to work, the

minimum hardware requirement to run Kubernetes node components is one CPU and 1 GB of memory.

Considering CPU and Memory, it is recommended to host the different roles of the Kubernetes cluster (etcd,

controlplane, and workers) on different nodes so that they can scale independently from one another. In this

solution, each virtual machine is configured with 4 virtual CPUs, 8 GB RAM, and thick provisioned disks on

the VxFlex backend storage.



The management node hosts the vCenter appliance, VxFlex OS Gateway, DNS, and workstation for RKE.

For detailed configuration of two-layer VxFlex cluster nodes, see A.1 Configuration details.

3.2 Two-layer network topology The following diagram depicts the two-layer network architecture that is based upon VxFlex OS best

practices:

Network architecture



• Two Dell S5048F switches are configured with VLT to provide fault tolerance and enable connectivity with

other switches. Storage nodes, Compute nodes, Management, and other Teamed Networks use Link

Aggregation Groups (LAG).

• Two dual port 25 Gb Mellanox NICs on each server provide 4 x 25 Gb ports.

• VLAN 51 and 52 are dedicated to SDS-SDS communication. Both VLANs are isolated at switch level to

provide high availability and avoid a single point of failure.

• VLAN 53 and 54 are dedicated to SDS-SDC communication. Both VLANs are isolated at switch level to

provide high availability and avoid a single point of failure.

• On storage nodes, 2 x 10 Gb ports are NIC teamed on VLAN 55 is dedicated to VxFlex OS management.

• On compute nodes, 2 x 25 Gb ports are NIC teamed to provide high availability. A NIC teaming network is

used to create Virtual Networks in vSphere. VLAN 56 is configured to provide connectivity with the

customer network, VLAN 57 is dedicated vMotion, and VLAN 50 is dedicated to Hypervisor management.

Installation of Rancher Kubernetes cluster


4 Installation of Rancher Kubernetes cluster This section includes the Kubernetes installation using RKE and the configuration of VxFlex OS CSI driver on

a two-layer VxFlex cluster.

Rancher supports Kubernetes clusters running Ubuntu, CentOS, Oracle Linux, RancherOS, and RedHat

Enterprise Linux. For information about the supported Operating System, Docker, and Rancher versions, see

Rancher - All Supported Versions.

4.1 Requirements The requirements for building Rancher Kubernetes cluster using RKE are as follows:

Requirement for RKE cluster on VxFlex Family

Name Version Description Reference

Rancher server 2.3.3 Rancher server will be used from the Workstation VM.

https://releases.rancher.com/server-charts/latest

Rancher Kubernetes Engine (RKE)

1.0.0 RKE is used from Workstation VM.

https://github.com/rancher/rke/releases/latest

Kubectl 1.13.12 Kubectl to interact with Kubernetes cluster.

https://kubernetes.io/docs/tasks/tools/install-kubectl/

Docker 18.06.3 Docker is installed on each RHEL node.

https://download.docker.com/linux/centos/docker-ce.repo

3 RHEL 7.6 nodes with SELinux and swap disabled

7.6 Ensure the nodes must be access using SSH and the required ports must be opened before cluster installation.

https://rancher.com/docs/rke/latest/en/os/#ports

VxFlex OS CSI 1.1.3 VxFlex OS CSI is used from the Workstation VM.

https://github.com/dell/csi-vxflexos

The RKE binary is available for Windows and Linux operating system. In this solution, RKE is run from a

Linux workstation virtual machine. RKE connects to the nodes using SSH key pairs.

https://rancher.com/support-maintenance-terms/all-supported-versions/rancher-v2.3.4/https://rancher.com/support-maintenance-terms/all-supported-versions/rancher-v2.3.4/https://rancher.com/support-maintenance-terms/all-supported-versions/rancher-v2.3.4/https://releases.rancher.com/server-charts/latesthttps://github.com/rancher/rke/releases/latesthttps://kubernetes.io/docs/tasks/tools/install-kubectl/https://kubernetes.io/docs/tasks/tools/install-kubectl/https://download.docker.com/linux/centos/docker-ce.repohttps://download.docker.com/linux/centos/docker-ce.repohttps://rancher.com/docs/rke/latest/en/os/#portshttps://github.com/dell/csi-vxflexos



Nodes connect using SSH key

The SSH login used for node access must have access to the Docker socket /var/run/docker.sock on

the machine, in order to provide this access, the account must be added as a member of the docker group on

the node

Note: The root account is not disabled by default. It is recommended not to use root as the SSH user, refer to

Bugzilla 1527565 for details.

Procedure:

1. Create Linux user account on every node.

# useradd -m -G docker rke

# su - rke

# mkdir $HOME/.ssh

# chmod 700 $HOME/.ssh

# touch $HOME/.ssh/authorized_keys

2. Test Docker socket access.

$ docker version

https://bugzilla.redhat.com/show_bug.cgi?id=1527565



3. Create an SSH key pair using the following command from the Workstation VM where RKE binary

exists:

$ ssh-keygen

The following files are created:

$HOME/.ssh/id_rsa (SSH private key, keep this secure)

$HOME/.ssh/id_rsa.pub (SSH public key)

4. Copy the SSH public key to the Kubernetes nodes to provide access to the nodes, using the created

SSH key pair:

$ cat .ssh/id_rsa.pub | ssh node1 "cat >> .ssh/authorized_keys"

5. Test SSH connectivity by running the following command from the Workstation VM, replacing

“hostname” with each of the Kubernetes nodes IP or hostname.

$ ssh -i $HOME/.ssh/id_rsa rke@hostname docker version

4.2 Install Kubernetes cluster using RKE RKE is a fast, simple Kubernetes installer used to install Kubernetes on Linux nodes.

1. Download the latest RKE installer applicable to the operating system being used and keep the RKE

binary in a rke user home directory (/home/rke).

2. Move rke_linux-amd64 binary as rke and make the RKE binary executable and confirm.

$ mv rke_linux-amd64 rke

$ chmod +x rke

$ rke --version

3. Create the cluster.yml file by running "./rke config” and answering the questions., this file

contains all information required to build the Kubernetes cluster, such as node connection information

and roles (controlplane, etcd, worker) to apply to each node.

[rke@rancher ~]$ ./rke config

[+] Cluster Level

Private Key Path [~/.ssh/id_rsa]:

[+] Number of Hosts [1]: 3

[+] SSH Address of host (1) [none]: rk8s-w1

..

[+] Network Plugin Type (flannel, calico, weave, canal) [canal]:

[+] Authentication Strategy [x509]:

[+] Authorization Mode (rbac, none) [rbac]:



[+] Kubernetes Docker image [rancher/hyperkube:v1.16.3-rancher1]:

rancher/hyperkube:v1.13.12-rancher1

[+] Cluster domain [cluster.local]:

[+] Service Cluster IP Range [10.43.0.0/16]:

[+] Cluster Network CIDR [10.42.0.0/16]:

[+] Cluster DNS Service IP [10.43.0.10]:

[+] Add addon manifest URLs or YAML files [no]: no

[rke@rancher ~]$

4. Check that the cluster.yml file is created in the same directory as RKE was run.

[rke@rancher ~]$ ls -ltr

-rwxrwxr-x 1 rke rke 46135151 Dec 25 04:42 rke

-rw-r----- 1 rke rke 4717 Dec 26 05:21 cluster.yml

[rke@rancher ~]$

5. Edit the cluster.yml file to enable the additional feature gates before Kubernetes cluster creation.

The details of the additional feature gates that are updated in cluster.yml as extra_args are

shown in A.2.1 cluster.yaml file details.

6. Build the RKE cluster using the following command:

[rke@rancher ~]$ ./rke up

INFO[0000] Running RKE version: v1.0.0

INFO[0000] Initiating Kubernetes cluster

INFO[0000] [certificates] Generating admin certificates and kubeconfig

INFO[0000] Successfully Deployed state file at [./cluster.rkestate]

INFO[0000] Building Kubernetes cluster

INFO[0000] [dialer] Setup tunnel for host [rk8s-w3]



..

..

INFO[0048] [addons] Successfully saved ConfigMap for addon rke-

ingress-controller to Kubernetes

INFO[0048] [addons] Executing deploy job rke-ingress-controller

INFO[0053] [ingress] ingress controller nginx deployed successfully

INFO[0053] [addons] Setting up user addons

INFO[0053] [addons] no user addons defined

INFO[0053] Finished building Kubernetes cluster successfully

[rke@rancher ~]$

Note: It is very important to add Kubernetes add-ons like feature-gates, service arguments, volume binds,

and environment variables before building Rancher Kubernetes cluster using rke up command. The

Kubernetes feature-gates must be enabled before installing the CSI Driver for VxFlex OS.



7. Check that the nodes are in a ready state after a successful Kubernetes cluster creation. The output

below shows that all the nodes have both master and worker roles installed.

[rke@rancher ~]$ kubectl get nodes

NAME STATUS ROLES AGE VERSION

rk8s-w1 Ready controlplane,etcd,worker 9h v1.13.12



[rke@rancher ~]$

4.3 Install Rancher server Rancher server is installed using the Helm package manager for Kubernetes.

1. Install Helm and Tiller with a service account. For detailed instruction, see the Helm Quickstart Guide.

2. Add the Helm chart repository that contains charts to install Rancher.

helm repo add rancher-latest https://releases.rancher.com/server-

charts/latest

3. Create a namespace for Rancher as cattle-system.

kubectl create namespace cattle-system

The cert-manager is a Kubernetes add-on to automate the management and issuance of TLS certificates

from various issuing sources. Rancher relies on cert-manager to issue certificates from Rancher’s own

generated CA or to request Let us Encrypt certificates.

4. Install cert-manager by following these steps:

$ kubectl apply --validate=false -f

https://raw.githubusercontent.com/jetstack/cert-manager/release-

0.12/deploy/manifests/00-crds.yaml

$ kubectl create namespace cert-manager

$ kubectl label namespace cert-manager certmanager.k8s.io/disable-

validation=true

$ helm repo add jetstack https://charts.jetstack.io

$ helm repo update

$ helm install \

> --name cert-manager \

> --namespace cert-manager \

> --version v0.12.0 \

> jetstack/cert-manager

https://v2.helm.sh/docs/using_helm/#installing-helmhttps://releases.rancher.com/server-charts/latesthttps://releases.rancher.com/server-charts/latesthttps://raw.githubusercontent.com/jetstack/cert-manager/release-0.12/deploy/manifests/00-crds.yamlhttps://raw.githubusercontent.com/jetstack/cert-manager/release-0.12/deploy/manifests/00-crds.yaml



5. Check the cert-manager namespace for running pods to verify it is deployed correctly.

$ kubectl get pods -n cert-manager

NAME READY STATUS RESTARTS AGE

cert-manager-75cf57777c-ztw9f 1/1 Running 0 2m2s

cert-manager-cainjector-f54c57bf8-wkc2z 1/1 Running 0 2m2s

cert-manager-webhook-76794c6967-84gb6 1/1 Running 0 2m2s

6. Install Rancher with Helm.

[root@rancher ~]# helm install rancher-latest/rancher \

> --name rancher \

> --namespace cattle-system \

> --set hostname=rancher.rack9lab.com

NAME: rancher

LAST DEPLOYED: Tue Jan 21 03:21:58 2020

NAMESPACE: cattle-system

STATUS: DEPLOYED

..

..

NOTES:

Rancher Server has been installed.

NOTE: Rancher may take several minutes to fully initialize. Please standby

while Certificates are being issued and Ingress comes up.

Check out Rancher docs at https://rancher.com/docs/rancher/v2.x/en/

Browse to https://rancher.rack9lab.com

Happy Containering!

[root@rancher ~]#

7. Check the cattle-system namespace for running pods to verify if the namespace is deployed correctly.

[root@rancher helm]# kubectl get pods –cattle-system

NAMESPACE NAME READY STATUS

RESTARTS AGE

cattle-system cattle-cluster-agent-85db6685cd-vvft9 1/1 Running

0 7h19m

cattle-system cattle-node-agent-6c8nw 1/1 Running

0 7h19m

cattle-system cattle-node-agent-jhw2n 1/1 Running

0 7h19m

cattle-system cattle-node-agent-wh78g 1/1 Running

0 7h19m

cattle-system rancher-5d549cb9c8-2xmj5 1/1 Running

0 7h23m

cattle-system rancher-5d549cb9c8-8bbzs 1/1 Running

0 7h23m

cattle-system rancher-5d549cb9c8-bh22p 1/1 Running

0 7h23m

Result: Rancher is up and running.



8. Browse to https://rancher.rack9lab.com to access the functional Rancher server.

Result: The three nodes are listed in the local cluster that represent the Rancher Management Cluster.

VxFlex OS Container Storage Interface driver


5 VxFlex OS Container Storage Interface driver

5.1 Overview The CSI driver for VxFlex OS is a plug-in that is installed into Kubernetes to provide persistent storage using

VxFlex OS storage system. The CSI driver for VxFlex OS and Kubernetes communicate using the Container

Storage Interface (CSI) protocol. The CSI driver for VxFlex OS supports Persistent volume (PV) capabilities,

Dynamic and Static PV provisioning, and Snapshot capabilities.

The Helm chart installs the CSI driver for Dell EMC VxFlex OS using a shell script. This script installs the CSI

driver container image along with the required Kubernetes sidecar containers.

The controller section of the Helm chart installs the following components in a Stateful Set in the namespace

vxflexos:

• CSI driver for Dell EMC VxFlex OS

• Kubernetes Provisioner, which provisions the volumes

• Kubernetes Attacher, which attaches the volumes to the containers

• Kubernetes Snapshotter, which provides snapshot support

The node section of the Helm chart installs the following component in a Daemon Set in the namespace

vxflexos:

• CSI driver for Dell EMC VxFlex OS

• Kubernetes Registrar, which handles the driver registration

5.2 Prerequisites Before you install the CSI Driver for VxFlex OS, verify that the following requirements are installed and

configured:

• Kubernetes v1.13.x

• Docker configured for bi-directional mount (MountFlags=shared)

• Kubernetes Feature Gates enabled for Snapshots and CSI support

• Helm and Tiller that is installed on a Master node

• VxFlex OS SDC installed on all Kubernetes nodes

• Kubernetes secret created for VxFlex OS credentials

• Helm values (yaml template) for things like VxFlex OS system name or ID, default gateway, and

MDM IP addresses, default storage pool.

For detailed information, see GitHub.

https://github.com/dell/csi-vxflexos



5.3 Install CSI drives for VxFlex OS 1. Download the installation source files from GitHub.

# git clone https://github.com/dell/csi-vxflexos

2. Create the namespace called vxflexos.

# kubectl create namespace vxflexos

3. Collect information from the VxFlex OS SDC (Storage Data Client) by running the

get_vxflexos_info.sh script.

4. Copy the csi-vxflexos/values.yaml into a file in the same directory as the install.vxflexos

named myvalues.yaml, to customize settings for installation.

5. Edit myvalues.yaml to set the parameters like VxFlex OS system name or ID, default gateway and

MDM IP addresses, default storage pool.

6. Run the sh install.vxflexos command to proceed with the installation.

[root@rancher helm]# sh install.vxflexos

Kubernetes version v1.13.12

Kubernetes master nodes:

Kubernetes minion nodes: 10.10.56.81 10.10.56.82 10.10.56.83

Verifying the SDC installation.

..

..

NAME: vxflexos

LAST DEPLOYED: Tue Jan 21 10:42:49 2020

NAMESPACE: vxflexos

STATUS: DEPLOYED

..

..

running 4 / 4


vxflexos-controller-0 4/4 Running 0 31s

vxflexos-node-k8s5t 2/2 Running 0 31s

vxflexos-node-ks5jd 2/2 Running 0 31s

vxflexos-node-xvbb5 2/2 Running 0 31s

CSIDrivers:

NAME AGE

vxflexos 41s

CSINodeInfos:

NAME AGE

rk8s-w1 32s

rk8s-w2 13s

rk8s-w3 17s

StorageClasses:

NAME PROVISIONER AGE

vxflexos (default) csi-vxflexos 41s

vxflexos-xfs csi-vxflexos 41s

https://github.com/dell/csi-vxflexoshttps://github.com/dell/csi-vxflexos



No resources found.

installing volumesnapshotclass

volumesnapshotclass.snapshot.storage.k8s.io/vxflexos-snapclass created

VolumeSnapshotClasses:

NAME AGE

vxflexos-snapclass 0s

[root@rancher helm]#

7. Check the vxflexos namespace for running pods to verify it is deployed correctly.

[root@rancher ~]# kubectl get pods -n vxflexos


vxflexos-controller-0 4/4 Running 0 31d

vxflexos-node-k8s5t 2/2 Running 0 31d

vxflexos-node-ks5jd 2/2 Running 0 31d

vxflexos-node-xvbb5 2/2 Running 0 31d

[root@rancher ~]#

For details information about CSI driver installation, see GitHub.

8. Check that the VxFlex OS CSI driver is running on the Rancher Kubernetes cluster and the VxFlex OS

Storage classes are listed in the Rancher menu to be used by any application launched in the Rancher

Kubernetes cluster.

8. Test the deployment workflow of a simple pod on VxFlex OS storage.

The helm chart contains two PersistentVolumeClaim definitions. The Storage Class vxflexos yields a

mounted ext4 file system and another Storage Class vxflex-xfs gives a xfs file system. For the

complete reference to test CSI drives on VxFlex OS, see GitHub.

root@rancher ~]# kubectl get pods -n test -o wide

NAME READY STATUS RESTARTS AGE IP NODE

NOMINATED NODE READINESS GATES

vxflextest-0 1/1 Running 661 31d 10.10.56.82 rk8s-w2

[root@rancher ~]#

https://github.com/dell/csi-vxflexoshttps://github.com/dell/csi-vxflexos



[root@rancher helm]# kubectl get pvc -n test

NAME STATUS VOLUME CAPACITY ACCESS MODES STORAGECLASS

AGE

pvol0 Bound k8s-59c253473c 8Gi RWO vxflexos

71s

pvol1 Bound k8s-59c264333c 16Gi RWO vxflexos-xfs

71s

[root@rancher helm]#

9. Verify that the persistent volumes are created in VxFlex cluster using the VxFlex OS GUI.

10. Verify the following output from Rancher UI and confirm that the VxFlex OS Persistent Volumes are being

used by Kubernetes pods running under test namespace.

Conclusion


6 Conclusion The Kubernetes ecosphere continues to grow in huge strides, providing more stability, security, and automatic

service discovery. Streamlining some of the basic operations, such as the Kubernetes cluster setup and

outline along with dynamically provisioning persistent storage using Rancher and the VxFlex OS CSI on

VxFlex family empowers admins to deploy their Kubernetes environment quickly for developers and end-

users, enabling uninterrupted utilization of infrastructure.

Appendix


A Appendix

A.1 Configuration details

Note: Dell EMC PowerEdge R740xd server. Available storage on storage nodes is the only difference

between storage and compute nodes. SSDs on storage nodes are been used to form a storage pool and

volumes that are used by compute nodes.

Storage only nodes

Hardware Configuration

CPU Cores 2 x 8 Intel® Xeon® Gold 6134 CPU @ 3.20 GHz

Memory 12 x 16 GB DDR4

NIC 2 x MLNX 25 GbE 2P ConnectX4LX Adpt

1 x Intel®2P X710/2P I350 rNDC

Storage BOSS S1 Controller 2 x 120 GB SATA SSD

Dell HBA330 controller 6 x 3.84 TB SAS SSD

operating system RHEL 7.6

Firmware Version 3.21.26.22

VxFlex OS 3.0.1

Compute only nodes

Hardware Configuration

CPU Cores 2 x 8 Intel®Xeon® Gold 6134 CPU @ 3.20 GHz

Memory 12 x 16 GB DDR4

NIC 2 x MLNX 25 GbE 2P ConnectX4LX Adpt

1 x Intel®2P X710/2P I350 rNDC

Storage BOSS S1 Controller 2 x 120 GB SATA SSD

Dell HBA330 controller 2 x 1.92 TB SAS SSD

Hypervisor ESXi 6.7

Firmware Version 3.21.26.22

VxFlex OS 3.0.1

Appendix


A.2 Rancher Kubernetes cluster details

Rancher Management cluster details

Components Items Details

Hardware Virtualized Hardware vSphere 6.7

CPU 4 vCPU

RAM 8 GB

Hard Disk 40 GB

NIC VMware Virtual NIC

Software Operating System RHEL 7.6

Container Runtime Docker-18.06.3-ce

Rancher v2.3.3

Rancher Kubernetes v1.13.12

RKE v1.0.0

VxFlex OS CSI driver v1.1.3

A.2.1 cluster.yaml file details

[rke@rancher ~]$ cat cluster.yml

..

..

kube-api:

image: ""

extra_args:

anonymous-auth: "false"

# enable-admission-plugins:

"ServiceAccount,NamespaceLifecycle,LimitRanger,PersistentVolumeLabel,DefaultSto

rageClass,ResourceQuota,DefaultTolerationSeconds,AlwaysPullImages,DenyEscalatin

gExec,NodeRestriction,EventRateLimit,PodSecurityPolicy"

profiling: "false"

service-account-lookup: "true"

tls-cipher-suites:

"TLS_ECDHE_ECDSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256,

TLS_ECDHE_ECDSA_WITH_CHACHA20_POLY1305,TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384,TL

S_ECDHE_RSA_WITH_CHACHA20_POLY1305,TLS_ECDHE_ECDSA_WITH_AES_256_GCM_SHA384,TLS_

RSA_WITH_AES_256_GCM_SHA384,TLS_RSA_WITH_AES_128_GCM_SHA256"

feature-gates:

"VolumeSnapshotDataSource=true,KubeletPluginsWatcher=true,CSINodeInfo=true,CSID

riverRegistry=true,BlockVolume=true,CSIBlockVolume=true"

extra_binds: []

extra_env: []

service_cluster_ip_range: 10.43.0.0/16

service_node_port_range: ""

pod_security_policy: false

always_pull_images: false

secrets_encryption_config: null

Appendix


audit_log: null

admission_configuration: null

event_rate_limit: null

kube-controller:

image: ""

extra_args:

profiling: "false"

address: "127.0.0.1"

terminated-pod-gc-threshold: "1000"

feature-gates:



extra_binds: []

extra_env: []

cluster_cidr: 10.42.0.0/16

service_cluster_ip_range: 10.43.0.0/16

scheduler:

image: ""

extra_args:

profiling: "false"

address: "127.0.0.1"

feature-gates:



extra_binds: []

extra_env: []

kubelet:

image: ""

extra_args:

streaming-connection-idle-timeout: "1800s"

authorization-mode: "Webhook"

protect-kernel-defaults: "true"

make-iptables-util-chains: "true"

event-qps: "0"

anonymous-auth: "false"

feature-gates:



tls-cipher-suites:

"TLS_ECDHE_ECDSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256,

TLS_ECDHE_ECDSA_WITH_CHACHA20_POLY1305,TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384,TL

S_ECDHE_RSA_WITH_CHACHA20_POLY1305,TLS_ECDHE_ECDSA_WITH_AES_256_GCM_SHA384,TLS_

RSA_WITH_AES_256_GCM_SHA384,TLS_RSA_WITH_AES_128_GCM_SHA256"

extra_binds: []

extra_env: []

cluster_domain: cluster.local

infra_container_image: ""

cluster_dns_server: 10.43.0.10

fail_swap_on: false

generate_serving_certificate: false

kubeproxy:

image: ""

extra_args: {}

extra_binds: []

extra_env: []

..

[rke@rancher ~]$

Technical support and resources


B Technical support and resources

• Dell.com/support is focused on meeting customer needs with proven services and support.

• Dell EMC VxFlex product overview and White papers on DellEMC.com provides expertise that helps

to ensure customer success on Dell EMC VxFlex family.

• Storage technical documents and videos provide expertise that helps to ensure customer success on

Dell EMC storage platforms.

B.1 Related resources

Note: Few links might require registration to access.

• Dell EMC VxFlex Product Overview

• VxFlex with Kubernetes

• CSI Driver for VxFlex OS Product Guide

• Rancher Product Overview

• Rancher Installation

• Rancher Support Matrix
http://www.dell.com/supporthttps://www.dellemc.com/en-us/hyperconverged-infrastructure/vxflex.htmhttp://www.dell.com/storageresourceshttps://www.dellemc.com/en-us/collaterals/unauth/data-sheets/products/converged-infrastructure/vxflex_data_sheet.pdfhttps://www.dellemc.com/en-us/collaterals/unauth/data-sheets/products/ready-solutions/vxflex_kubernetes_solution_brief.pdfhttps://github.com/dell/csi-vxflexos/blob/master/CSI%20Driver%20for%20VxFlex%20OS%20Product%20Guide.pdfhttps://rancher.com/products/rancher/https://rancher.com/docs/rancher/v2.x/en/installation/https://rancher.com/support-maintenance-terms/all-supported-versions/rancher-v2.3.5/

rancher kubernetes cluster using csi driver for dell emc vxflex … · 2021. 1. 31. · using...

Documents