rancher kubernetes cluster using csi driver for dell emc ... › ... ›...
TRANSCRIPT
000066
White Paper
Rancher Kubernetes Cluster using CSI Driver for Dell EMC VxFlex family
Abstract This white paper provides guidance on the deployment of a Rancher Kubernetes
Cluster on to the Dell EMC VxFlex family and the integration of the VxFlex CSI
driver for persistent volumes, for customers requiring an on-premises container
platform solution.
May 2020
Revisions
2 Rancher Kubernetes Cluster using CSI Driver for Dell EMC VxFlex family | 000066
Revisions
Date Description
April 2020 Initial release
May 2020 Updated Section 3: Solution Architecture
Acknowledgements
This paper was produced by the following:
Author: Sanjay Puttaswamy
Support: David J Adams and Shashikiran Chidambara
Others: Sridhar Subramanian, Shalini G, Kevin Jones and Simon Stevens
The information in this publication is provided “as is.” Dell Inc. makes no representations or warranties of any kind with respect to the information in this
publication, and specifically disclaims implied warranties of merchantability or fitness for a particular purpose.
Use, copying, and distribution of any software described in this publication requires an applicable software license.
Copyright © 2020 Dell Inc. or its subsidiaries. All Rights Reserved. Dell, EMC, Dell EMC and other trademarks are trademarks of Dell Inc. or its
subsidiaries. Other trademarks may be trademarks of their respective owners. [5/14/2020] [White Paper] [000066]
Table of contents
3 Rancher Kubernetes Cluster using CSI Driver for Dell EMC VxFlex family | 000066
Table of contents
Revisions............................................................................................................................................................................. 2
Acknowledgements ............................................................................................................................................................. 2
Table of contents ................................................................................................................................................................ 3
Executive summary ............................................................................................................................................................. 4
1 Introduction ................................................................................................................................................................... 5
1.1 Objective ............................................................................................................................................................. 5
1.2 Audience ............................................................................................................................................................. 5
1.3 Terminology ........................................................................................................................................................ 5
2 Product overview .......................................................................................................................................................... 6
2.1 VxFlex family ...................................................................................................................................................... 6
2.1.1 VxFlex OS .................................................................................................................................................. 6
2.2 Rancher for Kubernetes ..................................................................................................................................... 7
2.2.1 How Rancher delivers production-grade Kubernetes at scale ................................................................... 7
3 Solution infrastructure .................................................................................................................................................. 9
3.1 Logical architecture ............................................................................................................................................ 9
3.2 Two-layer network topology .............................................................................................................................. 10
4 Installation of Rancher Kubernetes cluster ................................................................................................................ 12
4.1 Requirements ................................................................................................................................................... 12
4.2 Install Kubernetes cluster using RKE ............................................................................................................... 14
4.3 Install Rancher server ....................................................................................................................................... 16
5 VxFlex OS Container Storage Interface driver ........................................................................................................... 19
5.1 Overview ........................................................................................................................................................... 19
5.2 Prerequisites ..................................................................................................................................................... 19
5.3 Install CSI drives for VxFlex OS ....................................................................................................................... 20
6 Conclusion .................................................................................................................................................................. 23
A Appendix ..................................................................................................................................................................... 24
A.1 Configuration details ......................................................................................................................................... 24
A.2 Rancher Kubernetes cluster details.................................................................................................................. 25
A.2.1 cluster.yaml file details ............................................................................................................................. 25
B Technical support and resources ............................................................................................................................... 27
B.1 Related resources............................................................................................................................................. 27
Executive summary
4 Rancher Kubernetes Cluster using CSI Driver for Dell EMC VxFlex family | 000066
Executive summary
Container technology enables development teams to quickly provision isolated applications without the
traditional complexities. For many companies, to boost productivity and time to value, the use of containers
starts with the departments that are focused on software development.
Kubernetes orchestration provides capabilities such as auto scaling, security, and managing containerized
applications. In order to run containerized applications within a Kubernetes cluster, a persistent and stable
data store is required, one that can outlast the pod or even the node it is running on.
Rancher is a Kubernetes management platform that simplifies cluster installation and operations, whether
they are on-premises, in the cloud or at the edge, giving DevOps teams the freedom to build and run
containerized applications anywhere.
The VxFlex family offers key value propositions for traditional and cloud-native production workloads,
deployment flexibility, linear scalability, predictable high performance, and enterprise-grade resilience.
This paper discusses how to deploy a Kubernetes cluster using Rancher Kubernetes Engine (RKE) on a
VxFlex Family cluster and the use of the VxFlex OS CSI driver to provision persistent storage for Kubernetes
managing containerized applications.
Introduction
5 Rancher Kubernetes Cluster using CSI Driver for Dell EMC VxFlex family | 000066
1 Introduction
1.1 Objective The goal of this paper is to provide a simple approach to creating a Kubernetes cluster using Rancher
Kubernetes Engine and managed using Rancher as the container orchestration layer on Dell EMC VxFlex
family platform to meet the performance, scalability, resiliency, and availability requirements. The document
also outlines information about leveraging Dell EMC VxFlex OS CSI driver to dynamically provision persistent
volumes on a Rancher managed Kubernetes cluster.
1.2 Audience The audience for this paper includes sales engineers, field consultants, IT administrators, customers, and
anyone else interested in configuring and deploying a Kubernetes cluster using RKE and using the VxFlex OS
CSI driver to dynamically provision persistent volumes in a Rancher managed Kubernetes cluster.
Readers are expected to have an understanding and working knowledge of Containers, Kubernetes, and
VxFlex OS.
1.3 Terminology The following table defines acronyms and terms that are used throughout this document:
Terms and definitions
Term Definition
CA Certificate Authority
CSI Container Storage Interface
DNS Domain Name System
MDM Meta Data Manager
OS Operating System
PV Persistent Volume
PVC Persistent Volume Claim
RKE Rancher Kubernetes Engine
SDC Storage Data Client for VxFlex
SDS Storage Data Server for VxFlex
SSD Solid-State Disk for VxFlex
TLS Transport Layer Security
VLAN Virtual Local Area Network
VM Virtual Machine
Product overview
6 Rancher Kubernetes Cluster using CSI Driver for Dell EMC VxFlex family | 000066
2 Product overview
2.1 VxFlex family The VxFlex software-defined storage platform is designed to deliver flexibility, elasticity and simplicity with
predictable performance and resiliency at scale by combining compute as well as high-performance storage
resources in a managed unified fabric. VxFlex is an ideal platform for modernizing infrastructure to improve IT
agility. The VxFlex Family currently includes VxFlex Ready Nodes, VxFlex appliance, and VxFlex integrated
rack.
VxFlex family
2.1.1 VxFlex OS VxFlex OS is the software foundation of VxFlex software-defined storage. It is a scale-out block storage
service that is designed to deliver flexibility, elasticity and simplicity with predictable high performance and
resiliency at scale.
VxFlex OS consists of the following components:
Storage Data Client (SDC)
• Provides front-end volume access to applications and file system
• Installed on servers consuming storage
• Maintains peer-to-peer connections to every SDS managing a pool of storage
Storage Data Server (SDS)
• Abstracts local storage, maintains storage pools, and presents volumes to the SDCs
• Installed on servers contributing local storage to the VxFlex OS cluster
Product overview
7 Rancher Kubernetes Cluster using CSI Driver for Dell EMC VxFlex family | 000066
Meta Data Manager (MDM)
• Oversees storage cluster configurations, monitoring, rebalances, and rebuilds
• Highly available, independent cluster installed on three or five different nodes
• May reside alongside SDCs or SDSs, or on separate nodes
• Sits outside the data path
Gateway
• Performs installation and configuration checks
• Acts as an endpoint for API calls and passes them to MDM
2.2 Rancher for Kubernetes Rancher is the enterprise computing platform to run Kubernetes on-premises, in the cloud and at the edge. It
addresses the operational and security challenges of managing multiple Kubernetes clusters everywhere.
Rancher also provides IT operators and development teams with integrated tools for building, deploying, and
running cloud-native workloads.
Rancher not only deploys production-grade Kubernetes clusters from data center to cloud to the edge, it also
unites them with centralized authentication, access control and observability. Rancher lets you streamline
cluster deployment on bare metal, edge devices, private clouds, public clouds, or vSphere and secure them
using global security policies. Use Helm or Rancher App Catalog to deploy and manage applications across
any or all these environments, ensuring multi-cluster consistency with a single deployment.
2.2.1 How Rancher delivers production-grade Kubernetes at scale
Rancher's recipe for production quality Kubernetes at scale
Product overview
8 Rancher Kubernetes Cluster using CSI Driver for Dell EMC VxFlex family | 000066
Certified Kubernetes Distributions
Rancher supports any certified Kubernetes distribution. For on-premises workloads, Rancher offers the RKE -
a CNCF-certified Kubernetes distribution that runs entirely within Docker containers. It works on bare-metal
and virtualized servers. RKE solves the problem of installation complexity, a common issue in the Kubernetes
community. With RKE, the installation and operation of Kubernetes is both simplified and easily automated,
and it is entirely independent of the operating system.
For the public cloud, all the major distributions are supported, including EKS, AKS, and GKE. For edge,
branch and desktop workloads Rancher offers K3s, a certified lightweight distribution of Kubernetes.
Simplified Cluster Operations
Rancher provides simple, consistent cluster operations including provisioning, version management, visibility
and diagnostics, monitoring and alerting, and centralized audit.
Security, Policy, and User Management
Rancher lets you automate processes and applies a consistent set of user access and security policies for all
your clusters, no matter where they are running.
Shared Tools and Services
Rancher provides a rich catalog of services for building, deploying, and scaling containerized applications,
including app packaging, CI/CD, logging, monitoring, and service mesh.
Solution infrastructure
9 Rancher Kubernetes Cluster using CSI Driver for Dell EMC VxFlex family | 000066
3 Solution infrastructure This solution could be deployed on any of the VxFlex family. In this solution, it is deployed in a two-layer
configuration using compute only nodes deployed with VMware ESXi hypervisor and dedicated storage only
nodes to provide the required storage capacity.
3.1 Logical architecture The diagram below shows the logical architecture diagram of the RKE cluster deployed on a two-layer VxFlex
cluster setup with four storage only nodes and three compute only nodes.
Logical architecture
Each Storage Only node includes two Intel Xeon Skylake 8-core processors, 192 GB RAM, and six 3.84 TB
SSDs. From the VxFlex OS standpoint, the Red Hat Enterprise Linux 7.6 Storage only nodes run the SDS
component of VxFlex OS to provide storage capacity. A single protection domain is created from these four
SDS systems. A single storage pool is created within this Protection Domain from which persistent storage
volumes can be provisioned for Kubernetes pods. The three ESXi 6.7 Compute only nodes run the SDC
components of the VxFlex OS in order to access the volumes created in the storage pool.
The RKE cluster is deployed as virtual machines on a VMware vSphere 6.7 cluster. For Rancher to work, the
minimum hardware requirement to run Kubernetes node components is one CPU and 1 GB of memory.
Considering CPU and Memory, it is recommended to host the different roles of the Kubernetes cluster (etcd,
controlplane, and workers) on different nodes so that they can scale independently from one another. In this
solution, each virtual machine is configured with 4 virtual CPUs, 8 GB RAM, and thick provisioned disks on
the VxFlex backend storage.
Solution infrastructure
10 Rancher Kubernetes Cluster using CSI Driver for Dell EMC VxFlex family | 000066
The management node hosts the vCenter appliance, VxFlex OS Gateway, DNS, and workstation for RKE.
For detailed configuration of two-layer VxFlex cluster nodes, see A.1 Configuration details.
3.2 Two-layer network topology The following diagram depicts the two-layer network architecture that is based upon VxFlex OS best
practices:
Network architecture
Solution infrastructure
11 Rancher Kubernetes Cluster using CSI Driver for Dell EMC VxFlex family | 000066
• Two Dell S5048F switches are configured with VLT to provide fault tolerance and enable connectivity with
other switches. Storage nodes, Compute nodes, Management, and other Teamed Networks use Link
Aggregation Groups (LAG).
• Two dual port 25 Gb Mellanox NICs on each server provide 4 x 25 Gb ports.
• VLAN 51 and 52 are dedicated to SDS-SDS communication. Both VLANs are isolated at switch level to
provide high availability and avoid a single point of failure.
• VLAN 53 and 54 are dedicated to SDS-SDC communication. Both VLANs are isolated at switch level to
provide high availability and avoid a single point of failure.
• On storage nodes, 2 x 10 Gb ports are NIC teamed on VLAN 55 is dedicated to VxFlex OS management.
• On compute nodes, 2 x 25 Gb ports are NIC teamed to provide high availability. A NIC teaming network is
used to create Virtual Networks in vSphere. VLAN 56 is configured to provide connectivity with the
customer network, VLAN 57 is dedicated vMotion, and VLAN 50 is dedicated to Hypervisor management.
Installation of Rancher Kubernetes cluster
12 Rancher Kubernetes Cluster using CSI Driver for Dell EMC VxFlex family | 000066
4 Installation of Rancher Kubernetes cluster This section includes the Kubernetes installation using RKE and the configuration of VxFlex OS CSI driver on
a two-layer VxFlex cluster.
Rancher supports Kubernetes clusters running Ubuntu, CentOS, Oracle Linux, RancherOS, and RedHat
Enterprise Linux. For information about the supported Operating System, Docker, and Rancher versions, see
Rancher - All Supported Versions.
4.1 Requirements The requirements for building Rancher Kubernetes cluster using RKE are as follows:
Requirement for RKE cluster on VxFlex Family
Name Version Description Reference
Rancher server 2.3.3 Rancher server will be used from the Workstation VM.
https://releases.rancher.com/server-charts/latest
Rancher Kubernetes Engine (RKE)
1.0.0 RKE is used from Workstation VM.
https://github.com/rancher/rke/releases/latest
Kubectl 1.13.12 Kubectl to interact with Kubernetes cluster.
https://kubernetes.io/docs/tasks/tools/install-kubectl/
Docker 18.06.3 Docker is installed on each RHEL node.
https://download.docker.com/linux/centos/docker-ce.repo
3 RHEL 7.6 nodes with SELinux and swap disabled
7.6 Ensure the nodes must be access using SSH and the required ports must be opened before cluster installation.
https://rancher.com/docs/rke/latest/en/os/#ports
VxFlex OS CSI 1.1.3 VxFlex OS CSI is used from the Workstation VM.
https://github.com/dell/csi-vxflexos
The RKE binary is available for Windows and Linux operating system. In this solution, RKE is run from a
Linux workstation virtual machine. RKE connects to the nodes using SSH key pairs.
Installation of Rancher Kubernetes cluster
13 Rancher Kubernetes Cluster using CSI Driver for Dell EMC VxFlex family | 000066
Nodes connect using SSH key
The SSH login used for node access must have access to the Docker socket /var/run/docker.sock on
the machine, in order to provide this access, the account must be added as a member of the docker group on
the node
Note: The root account is not disabled by default. It is recommended not to use root as the SSH user, refer to
Bugzilla 1527565 for details.
Procedure:
1. Create Linux user account on every node.
# useradd -m -G docker rke
# su - rke
# mkdir $HOME/.ssh
# chmod 700 $HOME/.ssh
# touch $HOME/.ssh/authorized_keys
2. Test Docker socket access.
$ docker version
Installation of Rancher Kubernetes cluster
14 Rancher Kubernetes Cluster using CSI Driver for Dell EMC VxFlex family | 000066
3. Create an SSH key pair using the following command from the Workstation VM where RKE binary
exists:
$ ssh-keygen
The following files are created:
$HOME/.ssh/id_rsa (SSH private key, keep this secure)
$HOME/.ssh/id_rsa.pub (SSH public key)
4. Copy the SSH public key to the Kubernetes nodes to provide access to the nodes, using the created
SSH key pair:
$ cat .ssh/id_rsa.pub | ssh node1 "cat >> .ssh/authorized_keys"
5. Test SSH connectivity by running the following command from the Workstation VM, replacing
“hostname” with each of the Kubernetes nodes IP or hostname.
$ ssh -i $HOME/.ssh/id_rsa rke@hostname docker version
4.2 Install Kubernetes cluster using RKE RKE is a fast, simple Kubernetes installer used to install Kubernetes on Linux nodes.
1. Download the latest RKE installer applicable to the operating system being used and keep the RKE
binary in a rke user home directory (/home/rke).
2. Move rke_linux-amd64 binary as rke and make the RKE binary executable and confirm.
$ mv rke_linux-amd64 rke
$ chmod +x rke
$ rke --version
3. Create the cluster.yml file by running "./rke config” and answering the questions., this file
contains all information required to build the Kubernetes cluster, such as node connection information
and roles (controlplane, etcd, worker) to apply to each node.
[rke@rancher ~]$ ./rke config
[+] Cluster Level
Private Key Path [~/.ssh/id_rsa]:
[+] Number of Hosts [1]: 3
[+] SSH Address of host (1) [none]: rk8s-w1
..
[+] Network Plugin Type (flannel, calico, weave, canal) [canal]:
[+] Authentication Strategy [x509]:
[+] Authorization Mode (rbac, none) [rbac]:
Installation of Rancher Kubernetes cluster
15 Rancher Kubernetes Cluster using CSI Driver for Dell EMC VxFlex family | 000066
[+] Kubernetes Docker image [rancher/hyperkube:v1.16.3-rancher1]:
rancher/hyperkube:v1.13.12-rancher1
[+] Cluster domain [cluster.local]:
[+] Service Cluster IP Range [10.43.0.0/16]:
[+] Cluster Network CIDR [10.42.0.0/16]:
[+] Cluster DNS Service IP [10.43.0.10]:
[+] Add addon manifest URLs or YAML files [no]: no
[rke@rancher ~]$
4. Check that the cluster.yml file is created in the same directory as RKE was run.
[rke@rancher ~]$ ls -ltr
-rwxrwxr-x 1 rke rke 46135151 Dec 25 04:42 rke
-rw-r----- 1 rke rke 4717 Dec 26 05:21 cluster.yml
[rke@rancher ~]$
5. Edit the cluster.yml file to enable the additional feature gates before Kubernetes cluster creation.
The details of the additional feature gates that are updated in cluster.yml as extra_args are
shown in A.2.1 cluster.yaml file details.
6. Build the RKE cluster using the following command:
[rke@rancher ~]$ ./rke up
INFO[0000] Running RKE version: v1.0.0
INFO[0000] Initiating Kubernetes cluster
INFO[0000] [certificates] Generating admin certificates and kubeconfig
INFO[0000] Successfully Deployed state file at [./cluster.rkestate]
INFO[0000] Building Kubernetes cluster
INFO[0000] [dialer] Setup tunnel for host [rk8s-w3]
INFO[0000] [dialer] Setup tunnel for host [rk8s-w2]
INFO[0000] [dialer] Setup tunnel for host [rk8s-w1]
..
..
INFO[0048] [addons] Successfully saved ConfigMap for addon rke-
ingress-controller to Kubernetes
INFO[0048] [addons] Executing deploy job rke-ingress-controller
INFO[0053] [ingress] ingress controller nginx deployed successfully
INFO[0053] [addons] Setting up user addons
INFO[0053] [addons] no user addons defined
INFO[0053] Finished building Kubernetes cluster successfully
[rke@rancher ~]$
Note: It is very important to add Kubernetes add-ons like feature-gates, service arguments, volume binds,
and environment variables before building Rancher Kubernetes cluster using rke up command. The
Kubernetes feature-gates must be enabled before installing the CSI Driver for VxFlex OS.
Installation of Rancher Kubernetes cluster
16 Rancher Kubernetes Cluster using CSI Driver for Dell EMC VxFlex family | 000066
7. Check that the nodes are in a ready state after a successful Kubernetes cluster creation. The output
below shows that all the nodes have both master and worker roles installed.
[rke@rancher ~]$ kubectl get nodes
NAME STATUS ROLES AGE VERSION
rk8s-w1 Ready controlplane,etcd,worker 9h v1.13.12
rk8s-w2 Ready controlplane,etcd,worker 9h v1.13.12
rk8s-w3 Ready controlplane,etcd,worker 9h v1.13.12
[rke@rancher ~]$
4.3 Install Rancher server Rancher server is installed using the Helm package manager for Kubernetes.
1. Install Helm and Tiller with a service account. For detailed instruction, see the Helm Quickstart Guide.
2. Add the Helm chart repository that contains charts to install Rancher.
helm repo add rancher-latest https://releases.rancher.com/server-
charts/latest
3. Create a namespace for Rancher as cattle-system.
kubectl create namespace cattle-system
The cert-manager is a Kubernetes add-on to automate the management and issuance of TLS certificates
from various issuing sources. Rancher relies on cert-manager to issue certificates from Rancher’s own
generated CA or to request Let us Encrypt certificates.
4. Install cert-manager by following these steps:
$ kubectl apply --validate=false -f
https://raw.githubusercontent.com/jetstack/cert-manager/release-
0.12/deploy/manifests/00-crds.yaml
$ kubectl create namespace cert-manager
$ kubectl label namespace cert-manager certmanager.k8s.io/disable-
validation=true
$ helm repo add jetstack https://charts.jetstack.io
$ helm repo update
$ helm install \
> --name cert-manager \
> --namespace cert-manager \
> --version v0.12.0 \
> jetstack/cert-manager
Installation of Rancher Kubernetes cluster
17 Rancher Kubernetes Cluster using CSI Driver for Dell EMC VxFlex family | 000066
5. Check the cert-manager namespace for running pods to verify it is deployed correctly.
$ kubectl get pods -n cert-manager
NAME READY STATUS RESTARTS AGE
cert-manager-75cf57777c-ztw9f 1/1 Running 0 2m2s
cert-manager-cainjector-f54c57bf8-wkc2z 1/1 Running 0 2m2s
cert-manager-webhook-76794c6967-84gb6 1/1 Running 0 2m2s
6. Install Rancher with Helm.
[root@rancher ~]# helm install rancher-latest/rancher \
> --name rancher \
> --namespace cattle-system \
> --set hostname=rancher.rack9lab.com
NAME: rancher
LAST DEPLOYED: Tue Jan 21 03:21:58 2020
NAMESPACE: cattle-system
STATUS: DEPLOYED
..
..
NOTES:
Rancher Server has been installed.
NOTE: Rancher may take several minutes to fully initialize. Please standby
while Certificates are being issued and Ingress comes up.
Check out Rancher docs at https://rancher.com/docs/rancher/v2.x/en/
Browse to https://rancher.rack9lab.com
Happy Containering!
[root@rancher ~]#
7. Check the cattle-system namespace for running pods to verify if the namespace is deployed correctly.
[root@rancher helm]# kubectl get pods –cattle-system
NAMESPACE NAME READY STATUS
RESTARTS AGE
cattle-system cattle-cluster-agent-85db6685cd-vvft9 1/1 Running
0 7h19m
cattle-system cattle-node-agent-6c8nw 1/1 Running
0 7h19m
cattle-system cattle-node-agent-jhw2n 1/1 Running
0 7h19m
cattle-system cattle-node-agent-wh78g 1/1 Running
0 7h19m
cattle-system rancher-5d549cb9c8-2xmj5 1/1 Running
0 7h23m
cattle-system rancher-5d549cb9c8-8bbzs 1/1 Running
0 7h23m
cattle-system rancher-5d549cb9c8-bh22p 1/1 Running
0 7h23m
Result: Rancher is up and running.
Installation of Rancher Kubernetes cluster
18 Rancher Kubernetes Cluster using CSI Driver for Dell EMC VxFlex family | 000066
8. Browse to https://rancher.rack9lab.com to access the functional Rancher server.
Result: The three nodes are listed in the local cluster that represent the Rancher Management Cluster.
VxFlex OS Container Storage Interface driver
19 Rancher Kubernetes Cluster using CSI Driver for Dell EMC VxFlex family | 000066
5 VxFlex OS Container Storage Interface driver
5.1 Overview The CSI driver for VxFlex OS is a plug-in that is installed into Kubernetes to provide persistent storage using
VxFlex OS storage system. The CSI driver for VxFlex OS and Kubernetes communicate using the Container
Storage Interface (CSI) protocol. The CSI driver for VxFlex OS supports Persistent volume (PV) capabilities,
Dynamic and Static PV provisioning, and Snapshot capabilities.
The Helm chart installs the CSI driver for Dell EMC VxFlex OS using a shell script. This script installs the CSI
driver container image along with the required Kubernetes sidecar containers.
The controller section of the Helm chart installs the following components in a Stateful Set in the namespace
vxflexos:
• CSI driver for Dell EMC VxFlex OS
• Kubernetes Provisioner, which provisions the volumes
• Kubernetes Attacher, which attaches the volumes to the containers
• Kubernetes Snapshotter, which provides snapshot support
The node section of the Helm chart installs the following component in a Daemon Set in the namespace
vxflexos:
• CSI driver for Dell EMC VxFlex OS
• Kubernetes Registrar, which handles the driver registration
5.2 Prerequisites Before you install the CSI Driver for VxFlex OS, verify that the following requirements are installed and
configured:
• Kubernetes v1.13.x
• Docker configured for bi-directional mount (MountFlags=shared)
• Kubernetes Feature Gates enabled for Snapshots and CSI support
• Helm and Tiller that is installed on a Master node
• VxFlex OS SDC installed on all Kubernetes nodes
• Kubernetes secret created for VxFlex OS credentials
• Helm values (yaml template) for things like VxFlex OS system name or ID, default gateway, and
MDM IP addresses, default storage pool.
For detailed information, see GitHub.
VxFlex OS Container Storage Interface driver
20 Rancher Kubernetes Cluster using CSI Driver for Dell EMC VxFlex family | 000066
5.3 Install CSI drives for VxFlex OS 1. Download the installation source files from GitHub.
# git clone https://github.com/dell/csi-vxflexos
2. Create the namespace called vxflexos.
# kubectl create namespace vxflexos
3. Collect information from the VxFlex OS SDC (Storage Data Client) by running the
get_vxflexos_info.sh script.
4. Copy the csi-vxflexos/values.yaml into a file in the same directory as the install.vxflexos
named myvalues.yaml, to customize settings for installation.
5. Edit myvalues.yaml to set the parameters like VxFlex OS system name or ID, default gateway and
MDM IP addresses, default storage pool.
6. Run the sh install.vxflexos command to proceed with the installation.
[root@rancher helm]# sh install.vxflexos
Kubernetes version v1.13.12
Kubernetes master nodes:
Kubernetes minion nodes: 10.10.56.81 10.10.56.82 10.10.56.83
Verifying the SDC installation.
..
..
NAME: vxflexos
LAST DEPLOYED: Tue Jan 21 10:42:49 2020
NAMESPACE: vxflexos
STATUS: DEPLOYED
..
..
running 4 / 4
NAME READY STATUS RESTARTS AGE
vxflexos-controller-0 4/4 Running 0 31s
vxflexos-node-k8s5t 2/2 Running 0 31s
vxflexos-node-ks5jd 2/2 Running 0 31s
vxflexos-node-xvbb5 2/2 Running 0 31s
CSIDrivers:
NAME AGE
vxflexos 41s
CSINodeInfos:
NAME AGE
rk8s-w1 32s
rk8s-w2 13s
rk8s-w3 17s
StorageClasses:
NAME PROVISIONER AGE
vxflexos (default) csi-vxflexos 41s
vxflexos-xfs csi-vxflexos 41s
VxFlex OS Container Storage Interface driver
21 Rancher Kubernetes Cluster using CSI Driver for Dell EMC VxFlex family | 000066
No resources found.
installing volumesnapshotclass
volumesnapshotclass.snapshot.storage.k8s.io/vxflexos-snapclass created
VolumeSnapshotClasses:
NAME AGE
vxflexos-snapclass 0s
[root@rancher helm]#
7. Check the vxflexos namespace for running pods to verify it is deployed correctly.
[root@rancher ~]# kubectl get pods -n vxflexos
NAME READY STATUS RESTARTS AGE
vxflexos-controller-0 4/4 Running 0 31d
vxflexos-node-k8s5t 2/2 Running 0 31d
vxflexos-node-ks5jd 2/2 Running 0 31d
vxflexos-node-xvbb5 2/2 Running 0 31d
[root@rancher ~]#
For details information about CSI driver installation, see GitHub.
8. Check that the VxFlex OS CSI driver is running on the Rancher Kubernetes cluster and the VxFlex OS
Storage classes are listed in the Rancher menu to be used by any application launched in the Rancher
Kubernetes cluster.
8. Test the deployment workflow of a simple pod on VxFlex OS storage.
The helm chart contains two PersistentVolumeClaim definitions. The Storage Class vxflexos yields a
mounted ext4 file system and another Storage Class vxflex-xfs gives a xfs file system. For the
complete reference to test CSI drives on VxFlex OS, see GitHub.
root@rancher ~]# kubectl get pods -n test -o wide
NAME READY STATUS RESTARTS AGE IP NODE
NOMINATED NODE READINESS GATES
vxflextest-0 1/1 Running 661 31d 10.10.56.82 rk8s-w2
<none> <none>
[root@rancher ~]#
VxFlex OS Container Storage Interface driver
22 Rancher Kubernetes Cluster using CSI Driver for Dell EMC VxFlex family | 000066
[root@rancher helm]# kubectl get pvc -n test
NAME STATUS VOLUME CAPACITY ACCESS MODES STORAGECLASS
AGE
pvol0 Bound k8s-59c253473c 8Gi RWO vxflexos
71s
pvol1 Bound k8s-59c264333c 16Gi RWO vxflexos-xfs
71s
[root@rancher helm]#
9. Verify that the persistent volumes are created in VxFlex cluster using the VxFlex OS GUI.
10. Verify the following output from Rancher UI and confirm that the VxFlex OS Persistent Volumes are being
used by Kubernetes pods running under test namespace.
Conclusion
23 Rancher Kubernetes Cluster using CSI Driver for Dell EMC VxFlex family | 000066
6 Conclusion The Kubernetes ecosphere continues to grow in huge strides, providing more stability, security, and automatic
service discovery. Streamlining some of the basic operations, such as the Kubernetes cluster setup and
outline along with dynamically provisioning persistent storage using Rancher and the VxFlex OS CSI on
VxFlex family empowers admins to deploy their Kubernetes environment quickly for developers and end-
users, enabling uninterrupted utilization of infrastructure.
Appendix
24 Rancher Kubernetes Cluster using CSI Driver for Dell EMC VxFlex family | 000066
A Appendix
A.1 Configuration details
Note: Dell EMC PowerEdge R740xd server. Available storage on storage nodes is the only difference
between storage and compute nodes. SSDs on storage nodes are been used to form a storage pool and
volumes that are used by compute nodes.
Storage only nodes
Hardware Configuration
CPU Cores 2 x 8 Intel® Xeon® Gold 6134 CPU @ 3.20 GHz
Memory 12 x 16 GB DDR4
NIC 2 x MLNX 25 GbE 2P ConnectX4LX Adpt
1 x Intel®2P X710/2P I350 rNDC
Storage BOSS S1 Controller 2 x 120 GB SATA SSD
Dell HBA330 controller 6 x 3.84 TB SAS SSD
operating system RHEL 7.6
Firmware Version 3.21.26.22
VxFlex OS 3.0.1
Compute only nodes
Hardware Configuration
CPU Cores 2 x 8 Intel®Xeon® Gold 6134 CPU @ 3.20 GHz
Memory 12 x 16 GB DDR4
NIC 2 x MLNX 25 GbE 2P ConnectX4LX Adpt
1 x Intel®2P X710/2P I350 rNDC
Storage BOSS S1 Controller 2 x 120 GB SATA SSD
Dell HBA330 controller 2 x 1.92 TB SAS SSD
Hypervisor ESXi 6.7
Firmware Version 3.21.26.22
VxFlex OS 3.0.1
Appendix
25 Rancher Kubernetes Cluster using CSI Driver for Dell EMC VxFlex family | 000066
A.2 Rancher Kubernetes cluster details
Rancher Management cluster details
Components Items Details
Hardware Virtualized Hardware vSphere 6.7
CPU 4 vCPU
RAM 8 GB
Hard Disk 40 GB
NIC VMware Virtual NIC
Software Operating System RHEL 7.6
Container Runtime Docker-18.06.3-ce
Rancher v2.3.3
Rancher Kubernetes v1.13.12
RKE v1.0.0
VxFlex OS CSI driver v1.1.3
A.2.1 cluster.yaml file details
[rke@rancher ~]$ cat cluster.yml
..
..
kube-api:
image: ""
extra_args:
anonymous-auth: "false"
# enable-admission-plugins:
"ServiceAccount,NamespaceLifecycle,LimitRanger,PersistentVolumeLabel,DefaultSto
rageClass,ResourceQuota,DefaultTolerationSeconds,AlwaysPullImages,DenyEscalatin
gExec,NodeRestriction,EventRateLimit,PodSecurityPolicy"
profiling: "false"
service-account-lookup: "true"
tls-cipher-suites:
"TLS_ECDHE_ECDSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256,
TLS_ECDHE_ECDSA_WITH_CHACHA20_POLY1305,TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384,TL
S_ECDHE_RSA_WITH_CHACHA20_POLY1305,TLS_ECDHE_ECDSA_WITH_AES_256_GCM_SHA384,TLS_
RSA_WITH_AES_256_GCM_SHA384,TLS_RSA_WITH_AES_128_GCM_SHA256"
feature-gates:
"VolumeSnapshotDataSource=true,KubeletPluginsWatcher=true,CSINodeInfo=true,CSID
riverRegistry=true,BlockVolume=true,CSIBlockVolume=true"
extra_binds: []
extra_env: []
service_cluster_ip_range: 10.43.0.0/16
service_node_port_range: ""
pod_security_policy: false
always_pull_images: false
secrets_encryption_config: null
Appendix
26 Rancher Kubernetes Cluster using CSI Driver for Dell EMC VxFlex family | 000066
audit_log: null
admission_configuration: null
event_rate_limit: null
kube-controller:
image: ""
extra_args:
profiling: "false"
address: "127.0.0.1"
terminated-pod-gc-threshold: "1000"
feature-gates:
"VolumeSnapshotDataSource=true,KubeletPluginsWatcher=true,CSINodeInfo=true,CSID
riverRegistry=true,BlockVolume=true,CSIBlockVolume=true"
extra_binds: []
extra_env: []
cluster_cidr: 10.42.0.0/16
service_cluster_ip_range: 10.43.0.0/16
scheduler:
image: ""
extra_args:
profiling: "false"
address: "127.0.0.1"
feature-gates:
"VolumeSnapshotDataSource=true,KubeletPluginsWatcher=true,CSINodeInfo=true,CSID
riverRegistry=true,BlockVolume=true,CSIBlockVolume=true"
extra_binds: []
extra_env: []
kubelet:
image: ""
extra_args:
streaming-connection-idle-timeout: "1800s"
authorization-mode: "Webhook"
protect-kernel-defaults: "true"
make-iptables-util-chains: "true"
event-qps: "0"
anonymous-auth: "false"
feature-gates:
"VolumeSnapshotDataSource=true,KubeletPluginsWatcher=true,CSINodeInfo=true,CSID
riverRegistry=true,BlockVolume=true,CSIBlockVolume=true"
tls-cipher-suites:
"TLS_ECDHE_ECDSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256,
TLS_ECDHE_ECDSA_WITH_CHACHA20_POLY1305,TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384,TL
S_ECDHE_RSA_WITH_CHACHA20_POLY1305,TLS_ECDHE_ECDSA_WITH_AES_256_GCM_SHA384,TLS_
RSA_WITH_AES_256_GCM_SHA384,TLS_RSA_WITH_AES_128_GCM_SHA256"
extra_binds: []
extra_env: []
cluster_domain: cluster.local
infra_container_image: ""
cluster_dns_server: 10.43.0.10
fail_swap_on: false
generate_serving_certificate: false
kubeproxy:
image: ""
extra_args: {}
extra_binds: []
extra_env: []
..
[rke@rancher ~]$
Technical support and resources
27 Rancher Kubernetes Cluster using CSI Driver for Dell EMC VxFlex family | 000066
B Technical support and resources
• Dell.com/support is focused on meeting customer needs with proven services and support.
• Dell EMC VxFlex product overview and White papers on DellEMC.com provides expertise that helps
to ensure customer success on Dell EMC VxFlex family.
• Storage technical documents and videos provide expertise that helps to ensure customer success on
Dell EMC storage platforms.
B.1 Related resources
Note: Few links might require registration to access.
• Dell EMC VxFlex Product Overview
• VxFlex with Kubernetes
• CSI Driver for VxFlex OS Product Guide
• Rancher Product Overview
• Rancher Installation
• Rancher Support Matrix