![Page 1: Running Legacy VM’s along with containers in Kubernetes · Kubernetes Custom Resource Definitions API • It enables to run VMs along with containers on existing Kubernetes nodes](https://reader031.vdocument.in/reader031/viewer/2022041014/5ec4662c7cd94d146313fb85/html5/thumbnails/1.jpg)
Copyright©2019 NTT Corp. All Rights Reserved.
Running Legacy VM’s along with containers in Kubernetes
Kunal Kushwaha
NTT Open Source Software Center
Delusion or Reality?
![Page 2: Running Legacy VM’s along with containers in Kubernetes · Kubernetes Custom Resource Definitions API • It enables to run VMs along with containers on existing Kubernetes nodes](https://reader031.vdocument.in/reader031/viewer/2022041014/5ec4662c7cd94d146313fb85/html5/thumbnails/2.jpg)
2Copyright©2019 NTT Corp. All Rights Reserved.
About me
• Work @ NTT Open Source Software Center
• Collaborator (Core developer) for libpod (podman)
• Contributor KubeVirt, buildkit and other related projects
• Docker Community Leader @ Tokyo Chapter
![Page 3: Running Legacy VM’s along with containers in Kubernetes · Kubernetes Custom Resource Definitions API • It enables to run VMs along with containers on existing Kubernetes nodes](https://reader031.vdocument.in/reader031/viewer/2022041014/5ec4662c7cd94d146313fb85/html5/thumbnails/3.jpg)
3Copyright©2019 NTT Corp. All Rights Reserved.
Growth of Containers in Companies
Credits: CNCF website
Adoption of containers in production has significantly increased
![Page 4: Running Legacy VM’s along with containers in Kubernetes · Kubernetes Custom Resource Definitions API • It enables to run VMs along with containers on existing Kubernetes nodes](https://reader031.vdocument.in/reader031/viewer/2022041014/5ec4662c7cd94d146313fb85/html5/thumbnails/4.jpg)
4Copyright©2019 NTT Corp. All Rights Reserved.
Growth of Container Orchestration usage
Credits: CNCF website
Adoption of container orchestrator like Kubernetes have also increased significantly on public as well private clouds.
![Page 5: Running Legacy VM’s along with containers in Kubernetes · Kubernetes Custom Resource Definitions API • It enables to run VMs along with containers on existing Kubernetes nodes](https://reader031.vdocument.in/reader031/viewer/2022041014/5ec4662c7cd94d146313fb85/html5/thumbnails/5.jpg)
5Copyright©2019 NTT Corp. All Rights Reserved.
Infrastructure landscape
• The application infrastructure is fragmented as most of old application still running on traditional infrastructure.
• Fragmentation means more work & increase in cost
VM Platform
app-1 app-2 app-N
VM Platform
app-1
app-2
kernel
app-Napp-1
app-2 app-M
Existing Products New Products
VM VM VM
![Page 6: Running Legacy VM’s along with containers in Kubernetes · Kubernetes Custom Resource Definitions API • It enables to run VMs along with containers on existing Kubernetes nodes](https://reader031.vdocument.in/reader031/viewer/2022041014/5ec4662c7cd94d146313fb85/html5/thumbnails/6.jpg)
6Copyright©2019 NTT Corp. All Rights Reserved.
What keeps applications away from Containers
• Lack of knowledge / Too complex to migrate in containers.
• Dependency on custom kernel parameters.
• Application designed for a custom kernel.
• Application towards the end of life.
Companies prefer to re-write application, rather than directly migrating them to
containers.
https://dzone.com/guides/containers-orchestration-and-beyond
![Page 7: Running Legacy VM’s along with containers in Kubernetes · Kubernetes Custom Resource Definitions API • It enables to run VMs along with containers on existing Kubernetes nodes](https://reader031.vdocument.in/reader031/viewer/2022041014/5ec4662c7cd94d146313fb85/html5/thumbnails/7.jpg)
7Copyright©2019 NTT Corp. All Rights Reserved.
Ideal World
• Applications in VM and containers can be managed with same control plane • Management/ Governance Policies like RBAC, Network etc. can same for all application• Intercommunication between application over containers and VM possible.
VM Platform
app-1 app-2 app-N
app-1
app-2
kernel
app-Napp-1
app-2 app-M
VM VM VM
![Page 8: Running Legacy VM’s along with containers in Kubernetes · Kubernetes Custom Resource Definitions API • It enables to run VMs along with containers on existing Kubernetes nodes](https://reader031.vdocument.in/reader031/viewer/2022041014/5ec4662c7cd94d146313fb85/html5/thumbnails/8.jpg)
8Copyright©2019 NTT Corp. All Rights Reserved.
“Lift & Shift” Strategy of Migration
• Original terminology coined for migrating in-
house application to Cloud.
• Also known as re-hosting application.
• The lift and shift migration approach is about migrating your application and
associated data to the target platform with minimal or no changes.
• Making VMs part of Kubernetes infrastructure along with containers, will help Lift &
Shift strategy for migrating applications running in VMs to Kubernetes.
*Pic credit: Google Cloud
![Page 9: Running Legacy VM’s along with containers in Kubernetes · Kubernetes Custom Resource Definitions API • It enables to run VMs along with containers on existing Kubernetes nodes](https://reader031.vdocument.in/reader031/viewer/2022041014/5ec4662c7cd94d146313fb85/html5/thumbnails/9.jpg)
9Copyright©2019 NTT Corp. All Rights Reserved.
KubeVirt Overview
• KubeVirt extends Kubernetes by adding resource types for VMs through
Kubernetes Custom Resource Definitions API
• It enables to run VMs along with containers on existing Kubernetes nodes
• VMs run inside regular Kubernetes pods, where they have access to
standard pod networking and storage, and managed using standard
Kubernetes tools such as kubectl
• Build on mature technology like KVM, qemu, libvirtd, Kubernetes
![Page 10: Running Legacy VM’s along with containers in Kubernetes · Kubernetes Custom Resource Definitions API • It enables to run VMs along with containers on existing Kubernetes nodes](https://reader031.vdocument.in/reader031/viewer/2022041014/5ec4662c7cd94d146313fb85/html5/thumbnails/10.jpg)
10Copyright©2019 NTT Corp. All Rights Reserved.
KubeVirt Goals
Leverage KubeVirt and
Kubernetes to manage
virtual machines for
impractical-to-containerize apps.
Combine existing
virtualized workloads with
new container workloads on the one platform.
Support development of
new micro-service
applications in containers
that interact with existing virtualized applications.
![Page 11: Running Legacy VM’s along with containers in Kubernetes · Kubernetes Custom Resource Definitions API • It enables to run VMs along with containers on existing Kubernetes nodes](https://reader031.vdocument.in/reader031/viewer/2022041014/5ec4662c7cd94d146313fb85/html5/thumbnails/11.jpg)
11Copyright©2019 NTT Corp. All Rights Reserved.
KubeVirt Control Flow & Architecture
• Virt-API and Virt-Controller are added to Master Node.
• Virt-Handler is on each worker node, responsible to launch VM in a pod.
• Containerized-data-importer prepare persistent Volumes
![Page 12: Running Legacy VM’s along with containers in Kubernetes · Kubernetes Custom Resource Definitions API • It enables to run VMs along with containers on existing Kubernetes nodes](https://reader031.vdocument.in/reader031/viewer/2022041014/5ec4662c7cd94d146313fb85/html5/thumbnails/12.jpg)
12Copyright©2019 NTT Corp. All Rights Reserved.
Important Features of KubeVirt
• KubeVirt features
• Can be installed and removed in existing k8s cluster.
• Supports multiple network and storage options, suitable for migration
• VMs run as part of pod, so utilize all other k8s components like DNS, RBAC, Network
Policies etc.
• VM capabilities
• Run VM with images in qemu qcow2 format, same as in OpenStack
• latest device support
• Q35 machine support.
![Page 13: Running Legacy VM’s along with containers in Kubernetes · Kubernetes Custom Resource Definitions API • It enables to run VMs along with containers on existing Kubernetes nodes](https://reader031.vdocument.in/reader031/viewer/2022041014/5ec4662c7cd94d146313fb85/html5/thumbnails/13.jpg)
13Copyright©2019 NTT Corp. All Rights Reserved.
KubeVirt Evaluation Process
![Page 14: Running Legacy VM’s along with containers in Kubernetes · Kubernetes Custom Resource Definitions API • It enables to run VMs along with containers on existing Kubernetes nodes](https://reader031.vdocument.in/reader031/viewer/2022041014/5ec4662c7cd94d146313fb85/html5/thumbnails/14.jpg)
14Copyright©2019 NTT Corp. All Rights Reserved.
Evaluation Viewpoint
VM to K8s Image migration
• Import into k8s PV or Container Image
• Understand problems/limitations of system
Configuration & Deployment
• Design VM to match original requirements / environment
• Understand problems/limitations /workarounds
Operational & Functional Validation
• Service creation
• App functionality/ accessibility / restriction
Reliability
• Time to recover from failure
• Maintenance downtime/disruption
![Page 15: Running Legacy VM’s along with containers in Kubernetes · Kubernetes Custom Resource Definitions API • It enables to run VMs along with containers on existing Kubernetes nodes](https://reader031.vdocument.in/reader031/viewer/2022041014/5ec4662c7cd94d146313fb85/html5/thumbnails/15.jpg)
15Copyright©2019 NTT Corp. All Rights Reserved.
Important KubeVirt Objects
VirtualMachine (VM) :
represents a virtual machine in the runtime environment of Kubernetes.
VirtualMachineInstanceReplicaSet (VMRS) :
Tries to ensures that a specified number of virtual machine replicas are running at any time.
DataVolume :
Data Volumes(DV) are an abstraction on top of Persistent Volume Claims(PVC) and the
Containerized Data Importer(CDI)
ContainerRegistryDisk :
local ephemeral disk for booting VMI. Any changes to disk are not persisted across reboot.
![Page 16: Running Legacy VM’s along with containers in Kubernetes · Kubernetes Custom Resource Definitions API • It enables to run VMs along with containers on existing Kubernetes nodes](https://reader031.vdocument.in/reader031/viewer/2022041014/5ec4662c7cd94d146313fb85/html5/thumbnails/16.jpg)
16Copyright©2019 NTT Corp. All Rights Reserved.
Migration of VM to KubeVirtApp in V
M •Prepare VM
for Migration
•Consistent
data state
Export
& B
uild I
mage
•Export the VM
Disk &
convert in
qcow2 format
• Import in
Persistent
Volume (PV)
Or
• Build Docker
imagePre
pare
k8s
Manifest
•Prepare yaml
file for VM
Definition in
KubeVirt
Deplo
y
•Deploy
application
with kubectl
apply
Expose
Serv
ice
•Create
Service
•Expose the
service to
outer network
![Page 17: Running Legacy VM’s along with containers in Kubernetes · Kubernetes Custom Resource Definitions API • It enables to run VMs along with containers on existing Kubernetes nodes](https://reader031.vdocument.in/reader031/viewer/2022041014/5ec4662c7cd94d146313fb85/html5/thumbnails/17.jpg)
17Copyright©2019 NTT Corp. All Rights Reserved.
Measuring Parameters
• Image Migration
• Configuration & Deployment
• Maintenance
• Reliability of service
EasyMedium
Hard
Migration effort
Configuration
&
Deployment
Maintenance
Reliability
![Page 18: Running Legacy VM’s along with containers in Kubernetes · Kubernetes Custom Resource Definitions API • It enables to run VMs along with containers on existing Kubernetes nodes](https://reader031.vdocument.in/reader031/viewer/2022041014/5ec4662c7cd94d146313fb85/html5/thumbnails/18.jpg)
18Copyright©2019 NTT Corp. All Rights Reserved.
replica: 1Use Cases
![Page 19: Running Legacy VM’s along with containers in Kubernetes · Kubernetes Custom Resource Definitions API • It enables to run VMs along with containers on existing Kubernetes nodes](https://reader031.vdocument.in/reader031/viewer/2022041014/5ec4662c7cd94d146313fb85/html5/thumbnails/19.jpg)
19Copyright©2019 NTT Corp. All Rights Reserved.
Evaluation Use Case
• Monolithic Application (Single VM)
• 3 Tier Web Application (Multiple VM)
• HA with multi network Architecture
![Page 20: Running Legacy VM’s along with containers in Kubernetes · Kubernetes Custom Resource Definitions API • It enables to run VMs along with containers on existing Kubernetes nodes](https://reader031.vdocument.in/reader031/viewer/2022041014/5ec4662c7cd94d146313fb85/html5/thumbnails/20.jpg)
20Copyright©2019 NTT Corp. All Rights Reserved.
replica: 1
Monolithic Application
![Page 21: Running Legacy VM’s along with containers in Kubernetes · Kubernetes Custom Resource Definitions API • It enables to run VMs along with containers on existing Kubernetes nodes](https://reader031.vdocument.in/reader031/viewer/2022041014/5ec4662c7cd94d146313fb85/html5/thumbnails/21.jpg)
21Copyright©2019 NTT Corp. All Rights Reserved.
Monolithic application
Monolithic Appin VM
http://my-company-intranet.com
Users
VM Platform (oVirt / ESXi ..)
DNS
Company Network
DNS
NIC
![Page 22: Running Legacy VM’s along with containers in Kubernetes · Kubernetes Custom Resource Definitions API • It enables to run VMs along with containers on existing Kubernetes nodes](https://reader031.vdocument.in/reader031/viewer/2022041014/5ec4662c7cd94d146313fb85/html5/thumbnails/22.jpg)
22Copyright©2019 NTT Corp. All Rights Reserved.
Monolithic application
Monolithic Appin VM
http://my-company-intranet.com
Users
VM Platform (oVirt / ESXi ..)
DNS
Company Network
DNS
NIC
Persistent data
• Application stores the data in file
based DB locally of disk
![Page 23: Running Legacy VM’s along with containers in Kubernetes · Kubernetes Custom Resource Definitions API • It enables to run VMs along with containers on existing Kubernetes nodes](https://reader031.vdocument.in/reader031/viewer/2022041014/5ec4662c7cd94d146313fb85/html5/thumbnails/23.jpg)
23Copyright©2019 NTT Corp. All Rights Reserved.
Monolithic application
Monolithic AppRunning in VM
http://my-company-intranet.com
Users
VM Platform (oVirt / ESXi ..)
DNS
Company Network
DNS
Application Type Standalone application with file based DB.
Requirements• Persistent Storage• Networking• Volume Backup
Policies• No auto re-creation of VM• Health Check
![Page 24: Running Legacy VM’s along with containers in Kubernetes · Kubernetes Custom Resource Definitions API • It enables to run VMs along with containers on existing Kubernetes nodes](https://reader031.vdocument.in/reader031/viewer/2022041014/5ec4662c7cd94d146313fb85/html5/thumbnails/24.jpg)
24Copyright©2019 NTT Corp. All Rights Reserved.
Migration process: Image Migration
Image Migration is simple process
- Depending on disk size, it may be time consuming.
• Converting vm-disk to kubevirt compatible format• img, qcow2, iso etc are supported formats *
• Conversion can be done with any v2v or p2v tools
• Importing disk to KubeVirt (Kubernetes)
*github.com/kubevirt/containerized-data-importer/blob/master/doc/supported_operations.md
$ qemu-img convert –f vdi monolithic.vdi –O qcow2 mono.qcow2
$ virtctl image-upload –pvc-name-monolithic-vm-disk \--pvc-size=64Gi\--image-path=/home/kunal/images/mono.qcow2 \--uploadproxy-url=https://172.20.20.51:5002
![Page 25: Running Legacy VM’s along with containers in Kubernetes · Kubernetes Custom Resource Definitions API • It enables to run VMs along with containers on existing Kubernetes nodes](https://reader031.vdocument.in/reader031/viewer/2022041014/5ec4662c7cd94d146313fb85/html5/thumbnails/25.jpg)
25Copyright©2019 NTT Corp. All Rights Reserved.
Migration process: VM definition
• Depending on original VM configuration,
writing VM yaml file could be tough.[1]
• Translation of old VM configuration to
new VM yaml is done manually.
template:spec:
terminationGracePeriodSeconds: 30 domain:
devices:disks:- disk:bus: virtioname: pvcdisk
volumes:- name: pvcdiskpersistentVolumeClaim:
claimName: monolithic-vm-disknetworks:
- name: defaultpod: {}
• Key definitions• run strategy : defines vm state after object
creation (running, manual etc)
• Volume
• Network
apiVersion: kubevirt.io/v1alpha3kind: VirtualMachinemetadata:
labels:kubevirt.io/vm: monolithic-app
name: monolithic-appspec:
runStrategy: manual
[1] : OpenShift supports KubeVirt templates, which is helpful
![Page 26: Running Legacy VM’s along with containers in Kubernetes · Kubernetes Custom Resource Definitions API • It enables to run VMs along with containers on existing Kubernetes nodes](https://reader031.vdocument.in/reader031/viewer/2022041014/5ec4662c7cd94d146313fb85/html5/thumbnails/26.jpg)
26Copyright©2019 NTT Corp. All Rights Reserved.
Migration process: Service Definition
apiVersion: v1kind: Servicemetadata:name: intranet-services
spec:ports:- name: nodeport
nodePort: 30000port: 27017targetPort: 80
selector:kubevirt.io/vm: monolithic-app
type: NodePort
Common to Kubernetes
• All solutions of Service Discovery of Kubernetes shall work with KubeVirt
VMs too.
Sample service definition
![Page 27: Running Legacy VM’s along with containers in Kubernetes · Kubernetes Custom Resource Definitions API • It enables to run VMs along with containers on existing Kubernetes nodes](https://reader031.vdocument.in/reader031/viewer/2022041014/5ec4662c7cd94d146313fb85/html5/thumbnails/27.jpg)
27Copyright©2019 NTT Corp. All Rights Reserved.
After Migration: Monolithic application
pod
Worker Node
Kubelet
Monolithic App
libvirtd
Virt-launcher
Service
Users
http://my-company-intranet.com
IngressOr
NodePort
DNS
Company Network
Virthandler
![Page 28: Running Legacy VM’s along with containers in Kubernetes · Kubernetes Custom Resource Definitions API • It enables to run VMs along with containers on existing Kubernetes nodes](https://reader031.vdocument.in/reader031/viewer/2022041014/5ec4662c7cd94d146313fb85/html5/thumbnails/28.jpg)
28Copyright©2019 NTT Corp. All Rights Reserved.
Migration process: Maintenance
• Backup/snapshot management.
• PersistentVolume (PV) is provided by K8s storage providers.
• Managed in similar way as PersistentVolume of K8s.
• Patch management/VM upgrade
• Traditional way (ssh / config manager)
• On failure
• Depending on Run strategy, action can be defined.
Kubernetes/KubeVirt do not add much value for maintenance phase for this
kind of application
![Page 29: Running Legacy VM’s along with containers in Kubernetes · Kubernetes Custom Resource Definitions API • It enables to run VMs along with containers on existing Kubernetes nodes](https://reader031.vdocument.in/reader031/viewer/2022041014/5ec4662c7cd94d146313fb85/html5/thumbnails/29.jpg)
29Copyright©2019 NTT Corp. All Rights Reserved.
Conclusion: Monolithic application migration
- Migration process : Easy.
- online migration : No.
- Security : Good
- As good as Kubernetes
- RBAC policies
- Network policies
- Maintenance: Medium
- Reliability with Kubernetes : Good
Lesson learnt
- VM maintenance changes w.r.t. Kubernetes.
- Be expert in Kubernetes.
• Easy to migrate and maintain application in Kubernetes
EasyMediumHard
Migration effort
Configuration
&
Deployment
Maintenance
Reliability
![Page 30: Running Legacy VM’s along with containers in Kubernetes · Kubernetes Custom Resource Definitions API • It enables to run VMs along with containers on existing Kubernetes nodes](https://reader031.vdocument.in/reader031/viewer/2022041014/5ec4662c7cd94d146313fb85/html5/thumbnails/30.jpg)
30Copyright©2019 NTT Corp. All Rights Reserved.
3-Tire Web Application
![Page 31: Running Legacy VM’s along with containers in Kubernetes · Kubernetes Custom Resource Definitions API • It enables to run VMs along with containers on existing Kubernetes nodes](https://reader031.vdocument.in/reader031/viewer/2022041014/5ec4662c7cd94d146313fb85/html5/thumbnails/31.jpg)
31Copyright©2019 NTT Corp. All Rights Reserved.
3 Tier Web Application
Fro
nte
nd
http://webservices-intranet.com
Users
VM Platform (oVirt / ESXi ..)
DNS
Company Network
DNS
NIC
Application
Logic
Backend
![Page 32: Running Legacy VM’s along with containers in Kubernetes · Kubernetes Custom Resource Definitions API • It enables to run VMs along with containers on existing Kubernetes nodes](https://reader031.vdocument.in/reader031/viewer/2022041014/5ec4662c7cd94d146313fb85/html5/thumbnails/32.jpg)
32Copyright©2019 NTT Corp. All Rights Reserved.
3 Tier Web Application
Fro
nte
nd
http://webservices-intranet.com
Users
VM Platform (oVirt / ESXi ..)
DNS
Company Network
DNS
NIC
Application
Logic
Backend
• Frontend & Application logic do not
store data locally.
• Backend store all data of application
• External network connect only frontend
No data stored
Persistent data
Scalable
![Page 33: Running Legacy VM’s along with containers in Kubernetes · Kubernetes Custom Resource Definitions API • It enables to run VMs along with containers on existing Kubernetes nodes](https://reader031.vdocument.in/reader031/viewer/2022041014/5ec4662c7cd94d146313fb85/html5/thumbnails/33.jpg)
33Copyright©2019 NTT Corp. All Rights Reserved.
Fro
nte
nd
Application
Logic
Backend
3 Tier Web Application
http://my-company-intranet.com
Users
VM Platform (oVirt / ESXi ..)
DNS
Company Network
DNS
eth0
Backend
DB
Application Type 3 tier web architecture.
Requirements
• Application and Frontend should be scalable.• Persistent Storage for Backend• Networking
• Inter-VM & external communication• Volume Backup
Policies• Auto re-create of Application & Frontend VM• No auto re-creation of VM for Backend• Health Check
![Page 34: Running Legacy VM’s along with containers in Kubernetes · Kubernetes Custom Resource Definitions API • It enables to run VMs along with containers on existing Kubernetes nodes](https://reader031.vdocument.in/reader031/viewer/2022041014/5ec4662c7cd94d146313fb85/html5/thumbnails/34.jpg)
34Copyright©2019 NTT Corp. All Rights Reserved.
Migration process: Image Migration
$ cat DockerfileFROM kubevirt/container-disk-v1alphaADD frontend-disk.qcow2 /disk
$ docker build –t kunalkushwaha/frontend-disk:v1
• ContainerDisk type suites better for immutable application types.
• Extra temporary storage can be provided using EmptyDisk type.
• PersistentVolume(PV) for storing persistent data in application.
• Frontend and Application VM imported as
ContainerDisk
• ContainerDisk is created using Dockerfile
with special Base Image provided by
KubeVirt.
![Page 35: Running Legacy VM’s along with containers in Kubernetes · Kubernetes Custom Resource Definitions API • It enables to run VMs along with containers on existing Kubernetes nodes](https://reader031.vdocument.in/reader031/viewer/2022041014/5ec4662c7cd94d146313fb85/html5/thumbnails/35.jpg)
35Copyright©2019 NTT Corp. All Rights Reserved.
Migration process: VM Definition
spec:replica: 1devices:
disks:- disk:
bus: virtioname: containerdisk
- disk:bus: virtioname: configdisk
volumes:- name: containerdiskcontainerDisk:
image: kunalkushwaha/frontend-vm-disk:v1- name: configdiskcloudInitNoCloud:userDataBase64: $(cat app-scripts.sh | base64 -w0)
.
Frontend and Application logic are created as VMReplicaSet
• To make Frontend and Application scalable, defined as VMReplicaSet(VMRS).
• Though VMs created using ContainerDisk are not compatible with live-migration.
• Data/Configuration can be passed to application
in VM using cloudInit or ConfigMap during VM
creation.Sample VMReplicaSet definition
![Page 36: Running Legacy VM’s along with containers in Kubernetes · Kubernetes Custom Resource Definitions API • It enables to run VMs along with containers on existing Kubernetes nodes](https://reader031.vdocument.in/reader031/viewer/2022041014/5ec4662c7cd94d146313fb85/html5/thumbnails/36.jpg)
36Copyright©2019 NTT Corp. All Rights Reserved.
Migration process: Service Definition
apiVersion: v1kind: Servicemetadata:
name: web-serverspec:ports:- name: nodeportnodePort: 30002port: 27019targetPort: 80
selector:kubevirt.io/vm: frontend-app
type: NodePort
apiVersion: v1kind: Servicemetadata:
name: application-serverspec:
ports:- name: clusterip
port: 27021targetPort: 80
selector:kubevirt.io/vm: application-app
type: ClusterIP
Frontend Application & Backend
• Hostname of old topology system becomes service name
• Frontend exposed with NodePort
• Application and Backend as ClusterIP (accessed within Cluster)
![Page 37: Running Legacy VM’s along with containers in Kubernetes · Kubernetes Custom Resource Definitions API • It enables to run VMs along with containers on existing Kubernetes nodes](https://reader031.vdocument.in/reader031/viewer/2022041014/5ec4662c7cd94d146313fb85/html5/thumbnails/37.jpg)
37Copyright©2019 NTT Corp. All Rights Reserved.
After Migration: 3 Tier Web Application
Worker Node
Kubelet
Service
Users
http://webservices-intranet.com
NodePort
DNS
Company Network
Frontend
libvirtd
Virt-launcher
Application
libvirtd
Virt-launcherVirt
handler
Service
Backend
libvirtd
Virt-launcher
Service
Frontend
libvirtd
Virt-launcher
Application
libvirtd
Virt-launcher
![Page 38: Running Legacy VM’s along with containers in Kubernetes · Kubernetes Custom Resource Definitions API • It enables to run VMs along with containers on existing Kubernetes nodes](https://reader031.vdocument.in/reader031/viewer/2022041014/5ec4662c7cd94d146313fb85/html5/thumbnails/38.jpg)
38Copyright©2019 NTT Corp. All Rights Reserved.
Migration process: Maintenance
VMReplicaSet are easy to scale, same as Pod replicaset, But no
rolling updates supported.
• Blue-Green deployment for updating immutable VMs outside of KubeVirt.
• Scale with updated image.
• Delete old image instances
• Scale down
• Use traditional approach for updating Stateful VM instances.
• ssh, config management
![Page 39: Running Legacy VM’s along with containers in Kubernetes · Kubernetes Custom Resource Definitions API • It enables to run VMs along with containers on existing Kubernetes nodes](https://reader031.vdocument.in/reader031/viewer/2022041014/5ec4662c7cd94d146313fb85/html5/thumbnails/39.jpg)
39Copyright©2019 NTT Corp. All Rights Reserved.
Conclusion: 3 Tier Web Application
- Migration process : Medium
- Online migration : No
- Maintenance : Good
- Reliability with Kubernetes : Good
Lesson learnt
- Name resolution/ Fixed IP reference in application
config, do not work.
- Hostname of VMs will be services of VM instance.
- Be expert in Kubernetes.
• Maintaining & scaling stateless VMs becomes very easy.
EasyMediumHard
Migration effort
Configuration
&
Deployment
Maintenance
Reliability
![Page 40: Running Legacy VM’s along with containers in Kubernetes · Kubernetes Custom Resource Definitions API • It enables to run VMs along with containers on existing Kubernetes nodes](https://reader031.vdocument.in/reader031/viewer/2022041014/5ec4662c7cd94d146313fb85/html5/thumbnails/40.jpg)
40Copyright©2019 NTT Corp. All Rights Reserved.
HA Architecture
![Page 41: Running Legacy VM’s along with containers in Kubernetes · Kubernetes Custom Resource Definitions API • It enables to run VMs along with containers on existing Kubernetes nodes](https://reader031.vdocument.in/reader031/viewer/2022041014/5ec4662c7cd94d146313fb85/html5/thumbnails/41.jpg)
41Copyright©2019 NTT Corp. All Rights Reserved.
HA Architecture Patterns
• Active-Standby with Shared Disk
• Active-Standby with Shared nothing
• Active-Active with Shared nothing*
Master Standby
Master Standby
Master Master*Please see appendices
![Page 42: Running Legacy VM’s along with containers in Kubernetes · Kubernetes Custom Resource Definitions API • It enables to run VMs along with containers on existing Kubernetes nodes](https://reader031.vdocument.in/reader031/viewer/2022041014/5ec4662c7cd94d146313fb85/html5/thumbnails/42.jpg)
42Copyright©2019 NTT Corp. All Rights Reserved.
HA Architecture (Active-Standby)
Standbyhttp://ha-services.com
Users
VM Platform (oVirt / ESXi ..)
DNS
Company Network
DNS
NIC
Master
VIP
![Page 43: Running Legacy VM’s along with containers in Kubernetes · Kubernetes Custom Resource Definitions API • It enables to run VMs along with containers on existing Kubernetes nodes](https://reader031.vdocument.in/reader031/viewer/2022041014/5ec4662c7cd94d146313fb85/html5/thumbnails/43.jpg)
43Copyright©2019 NTT Corp. All Rights Reserved.
Active-Standby with Shared Disk
When one node become unresponsive. How it can be ensured if it is not
updating disk / Corrupting data?
• Shoot The Other Node In The Head (STONITH)
• Data consistency is hard to achieve with this architecture in KubeVirt /Kubernetes
• Fencing mechanism like STONITH, not available in Kubernetes/KubeVirt yet.
Master Standby
![Page 44: Running Legacy VM’s along with containers in Kubernetes · Kubernetes Custom Resource Definitions API • It enables to run VMs along with containers on existing Kubernetes nodes](https://reader031.vdocument.in/reader031/viewer/2022041014/5ec4662c7cd94d146313fb85/html5/thumbnails/44.jpg)
44Copyright©2019 NTT Corp. All Rights Reserved.
Active-Standby with Shared Disk
When one node become unresponsive. How it can be ensured if it is not
updating disk / Corrupting data?
• Shoot The Other Node In The Head (STONITH)
• Data consistency is hard to achieve with this architecture in KubeVirt /Kubernetes
• Fencing mechanism like STONITH, not available in Kubernetes/KubeVirt yet.
Master Standby
Hello there!
![Page 45: Running Legacy VM’s along with containers in Kubernetes · Kubernetes Custom Resource Definitions API • It enables to run VMs along with containers on existing Kubernetes nodes](https://reader031.vdocument.in/reader031/viewer/2022041014/5ec4662c7cd94d146313fb85/html5/thumbnails/45.jpg)
45Copyright©2019 NTT Corp. All Rights Reserved.
Active-Standby with Shared Disk
When one node become unresponsive. How it can be ensured if it is not
updating disk / Corrupting data?
• Shoot The Other Node In The Head (STONITH)
• Data consistency is hard to achieve with this architecture in KubeVirt /Kubernetes
• Fencing mechanism like STONITH, not available in Kubernetes/KubeVirt yet.
Master Standby
Hello there!
No Response
Let me take over
![Page 46: Running Legacy VM’s along with containers in Kubernetes · Kubernetes Custom Resource Definitions API • It enables to run VMs along with containers on existing Kubernetes nodes](https://reader031.vdocument.in/reader031/viewer/2022041014/5ec4662c7cd94d146313fb85/html5/thumbnails/46.jpg)
46Copyright©2019 NTT Corp. All Rights Reserved.
Active-Standby with Shared Disk
When one node become unresponsive. How it can be ensured if it is not
updating disk / Corrupting data?
• Shoot The Other Node In The Head (STONITH)
• Data consistency is hard to achieve with this architecture in KubeVirt /Kubernetes
• Fencing mechanism like STONITH, not available in Kubernetes/KubeVirt yet.
Master Standby
Hello there!
No Response
Let me take over
What if “He is still
writing on disk?”
![Page 47: Running Legacy VM’s along with containers in Kubernetes · Kubernetes Custom Resource Definitions API • It enables to run VMs along with containers on existing Kubernetes nodes](https://reader031.vdocument.in/reader031/viewer/2022041014/5ec4662c7cd94d146313fb85/html5/thumbnails/47.jpg)
47Copyright©2019 NTT Corp. All Rights Reserved.
Active-Standby with Shared Disk
When one node become unresponsive. How it can be ensured if it is not
updating disk / Corrupting data?
• Shoot The Other Node In The Head (STONITH)
• Data consistency is hard to achieve with this architecture in KubeVirt /Kubernetes
• Fencing mechanism like STONITH, not available in Kubernetes/KubeVirt yet.
Master Standby
Hello there!
No Response
Let me take over
What if “He is still
writing on disk?”
PowerOff
![Page 48: Running Legacy VM’s along with containers in Kubernetes · Kubernetes Custom Resource Definitions API • It enables to run VMs along with containers on existing Kubernetes nodes](https://reader031.vdocument.in/reader031/viewer/2022041014/5ec4662c7cd94d146313fb85/html5/thumbnails/48.jpg)
48Copyright©2019 NTT Corp. All Rights Reserved.
Active-Standby with Shared Disk
When one node become unresponsive. How it can be ensured if it is not
updating disk / Corrupting data?
• Shoot The Other Node In The Head (STONITH)
• Data consistency is hard to achieve with this architecture in KubeVirt /Kubernetes
• Fencing mechanism like STONITH, not available in Kubernetes/KubeVirt yet.
Master Standby
Hello there!
No Response
Let me take over
What if “He is still
writing on disk?”
PowerOff
Lack of fencing mechanism, restrict migration of applications
implemented with STONITH like solution
![Page 49: Running Legacy VM’s along with containers in Kubernetes · Kubernetes Custom Resource Definitions API • It enables to run VMs along with containers on existing Kubernetes nodes](https://reader031.vdocument.in/reader031/viewer/2022041014/5ec4662c7cd94d146313fb85/html5/thumbnails/49.jpg)
49Copyright©2019 NTT Corp. All Rights Reserved.
Active-Standby without Shared Disk
Master Standby
postgresql postgresql
Pacemaker Pacemaker
corosync corosync
VIP
Read-Write Read OnlyService - LAN
DATA - LAN
InterConnect - LAN
![Page 50: Running Legacy VM’s along with containers in Kubernetes · Kubernetes Custom Resource Definitions API • It enables to run VMs along with containers on existing Kubernetes nodes](https://reader031.vdocument.in/reader031/viewer/2022041014/5ec4662c7cd94d146313fb85/html5/thumbnails/50.jpg)
50Copyright©2019 NTT Corp. All Rights Reserved.
Service - LAN
DATA - LAN
InterConnect - LAN
VIP
Active-Standby without Shared Disk
Master Standby
postgresql postgresql
Pacemaker Pacemaker
corosync corosync
• PG-REX is a solution based on PostgreSQL & Pacemaker.
• Based on streaming replication feature.
• Open Source tool for easier setup*
*https://osdn.net/projects/pg-rex/
![Page 51: Running Legacy VM’s along with containers in Kubernetes · Kubernetes Custom Resource Definitions API • It enables to run VMs along with containers on existing Kubernetes nodes](https://reader031.vdocument.in/reader031/viewer/2022041014/5ec4662c7cd94d146313fb85/html5/thumbnails/51.jpg)
51Copyright©2019 NTT Corp. All Rights Reserved.
Migration process: VM Definition of HA models
• Multus ( a meta CNI plugin) used for providing multiple network
interfaces to VMs of KubeVirt.
• Uses NetworkAttachment (CNI CRD) for implementing multiple
networks.
• Apart from Persistent Volume, this use case requires multiple Network segments.
• Preparation of network is required before using them in VM Definition i.e. defining
NetworkAttchmentDefinition.
![Page 52: Running Legacy VM’s along with containers in Kubernetes · Kubernetes Custom Resource Definitions API • It enables to run VMs along with containers on existing Kubernetes nodes](https://reader031.vdocument.in/reader031/viewer/2022041014/5ec4662c7cd94d146313fb85/html5/thumbnails/52.jpg)
52Copyright©2019 NTT Corp. All Rights Reserved.
Migration process: VM Definition of HA models
• Multus ( a meta CNI plugin) used for providing multiple network
interfaces to VMs of KubeVirt.
• Uses NetworkAttachment (CNI CRD) for implementing multiple
networks.
• Apart from Persistent Volume, this use case requires multiple Network segments.
• Preparation of network is required before using them in VM Definition i.e. defining
NetworkAttchmentDefinition.
apiVersion: v1kind: NetworkAttachmentDefinitionmetadata:
name: pgrex-s-lanspec:
config:’{“cniVersion”: “0.3.0”,“type”: “macvlan”,“master”:”enp0s8”,... < CNI plugin >..
}’
pgrex-s-lan.yaml
![Page 53: Running Legacy VM’s along with containers in Kubernetes · Kubernetes Custom Resource Definitions API • It enables to run VMs along with containers on existing Kubernetes nodes](https://reader031.vdocument.in/reader031/viewer/2022041014/5ec4662c7cd94d146313fb85/html5/thumbnails/53.jpg)
53Copyright©2019 NTT Corp. All Rights Reserved.
Migration process: VM Definition for PG-REX
• Works on fixed IP address, but troubleshooting is hard.
• Using macvlan network, network with narrow range of IP is be created for all
segments.
• HA components communicate with VM IP’s instead of services.
• Extra logic required to ensure user request goes to Master VM only.*
• Need reconfiguration, if VM’s moved from current node.
• With private hosted Kubernetes, its hard to get Fixed IP over cluster.
• Service cannot have custom ClusterIP in different segment.
• Migration in KubeVirt is possible with hackish solution.
*Leader election to mark Active VM. (https://kubernetes.io/blog/2016/01/simple-leader-election-with-kubernetes/)
![Page 54: Running Legacy VM’s along with containers in Kubernetes · Kubernetes Custom Resource Definitions API • It enables to run VMs along with containers on existing Kubernetes nodes](https://reader031.vdocument.in/reader031/viewer/2022041014/5ec4662c7cd94d146313fb85/html5/thumbnails/54.jpg)
54Copyright©2019 NTT Corp. All Rights Reserved.
After Migration: Active-Standby without Shared Disk
Worker Node
Kubelet
Service
Users
http://ha-services.com
NodePort
DNS
Virthandler
master
libvirtd
virt-launcher
standby
libvirtd
virt-launcher
Fixed IP
![Page 55: Running Legacy VM’s along with containers in Kubernetes · Kubernetes Custom Resource Definitions API • It enables to run VMs along with containers on existing Kubernetes nodes](https://reader031.vdocument.in/reader031/viewer/2022041014/5ec4662c7cd94d146313fb85/html5/thumbnails/55.jpg)
55Copyright©2019 NTT Corp. All Rights Reserved.
Migration process: Maintenance
• Backup/snapshot management.
• PersistentVolume (PV) is provided by K8s storage providers.
• Managed in similar way as PersistentVolume of K8s.
• Patch management/VM upgrade
• Traditional way (ssh / config manager)
• On failure
• Application logic of smooth failover works.
Maintenance approach of Application VMs do not change much, though
little added complexity in connecting the VMs
![Page 56: Running Legacy VM’s along with containers in Kubernetes · Kubernetes Custom Resource Definitions API • It enables to run VMs along with containers on existing Kubernetes nodes](https://reader031.vdocument.in/reader031/viewer/2022041014/5ec4662c7cd94d146313fb85/html5/thumbnails/56.jpg)
56Copyright©2019 NTT Corp. All Rights Reserved.
Conclusion: HA Architecture
- Migration process : Hard
- Online migration : No
- Maintenance : Medium
- Reliability with Kubernetes : Good
Lesson learnt
- Configuration changes are not apparent.
- Look beyond standard Kubernetes pod communication
channels
- Be expert in Kubernetes.
Simply Lift & Shift do not work for application with complex topology
EasyMediumHard
Migration effort
Configuration
&
Deployment
Maintenance
Reliability
![Page 57: Running Legacy VM’s along with containers in Kubernetes · Kubernetes Custom Resource Definitions API • It enables to run VMs along with containers on existing Kubernetes nodes](https://reader031.vdocument.in/reader031/viewer/2022041014/5ec4662c7cd94d146313fb85/html5/thumbnails/57.jpg)
57Copyright©2019 NTT Corp. All Rights Reserved.
Conclusion: HA Architecture (cont’d)
• Migrating shared disk DB Cluster might not be wise at
this moment.
• Data consistency need to be maintained by
application only.
• Particularly for DB, shared nothing kind of
configuration there are few solution which works on
KubeVirt like environment.
• PG-REX
• Works with hack
• Crunchy
• A Kubernetes Operator based PostgreSQL solution.
• Not for migrating existing DB nodes.
No perfect solution for migrating DB VMs to Kubernetes.
EasyMediumHard
Migration effort
Configuration
&
Deployment
Maintenance
Reliability
![Page 58: Running Legacy VM’s along with containers in Kubernetes · Kubernetes Custom Resource Definitions API • It enables to run VMs along with containers on existing Kubernetes nodes](https://reader031.vdocument.in/reader031/viewer/2022041014/5ec4662c7cd94d146313fb85/html5/thumbnails/58.jpg)
58Copyright©2019 NTT Corp. All Rights Reserved.
Overall Conclusion
• KubeVirt works including multiple networks.
• Migration steps can be automated for VM Definition;
But IP addresses aren’t portable.
• HA is currently tough; it requires non-standard(hackish) configuration.
![Page 59: Running Legacy VM’s along with containers in Kubernetes · Kubernetes Custom Resource Definitions API • It enables to run VMs along with containers on existing Kubernetes nodes](https://reader031.vdocument.in/reader031/viewer/2022041014/5ec4662c7cd94d146313fb85/html5/thumbnails/59.jpg)
59Copyright©2019 NTT Corp. All Rights Reserved.
Challenges & Future Work for Kubernetes/KubeVirt
• Challenges
• Reliable fencing mechanism
• Support for service IP other than default network segment
• Future work
• VM Definition generator from old VM configuration e.g. OVA file.
![Page 60: Running Legacy VM’s along with containers in Kubernetes · Kubernetes Custom Resource Definitions API • It enables to run VMs along with containers on existing Kubernetes nodes](https://reader031.vdocument.in/reader031/viewer/2022041014/5ec4662c7cd94d146313fb85/html5/thumbnails/60.jpg)
60Copyright©2019 NTT Corp. All Rights Reserved.
Alternatives
- Virtlet
- Project with similar goal, but implemented as Container
Runtime Interface(CRI) instead of CRD.
- KubeVirt is more active project compared to Virtlet.
- Kata Container runtime?
- Not an alternative.
- Though it uses VM level isolation, but designed to run
docker/container type workload (Single application)
![Page 61: Running Legacy VM’s along with containers in Kubernetes · Kubernetes Custom Resource Definitions API • It enables to run VMs along with containers on existing Kubernetes nodes](https://reader031.vdocument.in/reader031/viewer/2022041014/5ec4662c7cd94d146313fb85/html5/thumbnails/61.jpg)
61Copyright©2019 NTT Corp. All Rights Reserved.
Summary
Running Legacy VM’s along with containers in Kubernetes
Delusion or Reality?
• Yes, it is possible in near future.
• It will not be simple Lift & Shift, but shall be less expensive than rewriting or
restructuring in containers.
• Automating migration will be daunting task.
• Application specific details are unique
• Kubernetes/KubeVirt specific changes could be automated with some
declarative objects.
![Page 62: Running Legacy VM’s along with containers in Kubernetes · Kubernetes Custom Resource Definitions API • It enables to run VMs along with containers on existing Kubernetes nodes](https://reader031.vdocument.in/reader031/viewer/2022041014/5ec4662c7cd94d146313fb85/html5/thumbnails/62.jpg)
Copyright©2019 NTT Corp. All Rights Reserved.
Thank you
![Page 63: Running Legacy VM’s along with containers in Kubernetes · Kubernetes Custom Resource Definitions API • It enables to run VMs along with containers on existing Kubernetes nodes](https://reader031.vdocument.in/reader031/viewer/2022041014/5ec4662c7cd94d146313fb85/html5/thumbnails/63.jpg)
63Copyright©2019 NTT Corp. All Rights Reserved.
Appendices
![Page 64: Running Legacy VM’s along with containers in Kubernetes · Kubernetes Custom Resource Definitions API • It enables to run VMs along with containers on existing Kubernetes nodes](https://reader031.vdocument.in/reader031/viewer/2022041014/5ec4662c7cd94d146313fb85/html5/thumbnails/64.jpg)
64Copyright©2019 NTT Corp. All Rights Reserved.
Evaluation Environment
Architecture: x86_64
Model name: Intel(R) Xeon(R) W-2123 CPU @ 3.60GHz
Hypervisor : KVM
Virtualization: full
Kernel: 4.18
OS: Fedora Server 29
Memory : 4GB
Kubernetes version : v1.12.2KubeVirt Version : v0.17.0CDI version : v1.9.0
Kubernetes Master
Architecture: x86_64
Model name: Intel(R) Xeon(R) W-2123 CPU @ 3.60GHz
Hypervisor : KVM
Virtualization: full
Kernel: 4.18
OS: Fedora Server 29
Memory : 12GB
Kubernetes Worker Node x 2
Software version
![Page 65: Running Legacy VM’s along with containers in Kubernetes · Kubernetes Custom Resource Definitions API • It enables to run VMs along with containers on existing Kubernetes nodes](https://reader031.vdocument.in/reader031/viewer/2022041014/5ec4662c7cd94d146313fb85/html5/thumbnails/65.jpg)
65Copyright©2019 NTT Corp. All Rights Reserved.
HA Architecture (Active-Active without Shared Disk)
Master-1http://ha-services.com
Users
VM Platform (oVirt / ESXi ..)
DNS
Company Network
DNS
NIC
Master-2
VIP
![Page 66: Running Legacy VM’s along with containers in Kubernetes · Kubernetes Custom Resource Definitions API • It enables to run VMs along with containers on existing Kubernetes nodes](https://reader031.vdocument.in/reader031/viewer/2022041014/5ec4662c7cd94d146313fb85/html5/thumbnails/66.jpg)
66Copyright©2019 NTT Corp. All Rights Reserved.
Data Segment
HA Segment
VIP Segment
Active-Active without Shared Disk
VIP
Master - 1 Master - 2
heartbeat heartbeat
HA proxy HA proxy
mysqld mysqld
![Page 67: Running Legacy VM’s along with containers in Kubernetes · Kubernetes Custom Resource Definitions API • It enables to run VMs along with containers on existing Kubernetes nodes](https://reader031.vdocument.in/reader031/viewer/2022041014/5ec4662c7cd94d146313fb85/html5/thumbnails/67.jpg)
67Copyright©2019 NTT Corp. All Rights Reserved.
Migration Process: VM Definition for MySQL Active-Active
• Define network for each segment.
• Define ports for each segment too.
• Defining multiple network VMs is same as pods using meta CNI plugins like
multus.
• Using cloudInit, its easy to make and try changes in application configuration
![Page 68: Running Legacy VM’s along with containers in Kubernetes · Kubernetes Custom Resource Definitions API • It enables to run VMs along with containers on existing Kubernetes nodes](https://reader031.vdocument.in/reader031/viewer/2022041014/5ec4662c7cd94d146313fb85/html5/thumbnails/68.jpg)
68Copyright©2019 NTT Corp. All Rights Reserved.
• Define network for each segment.
• Define ports for each segment too.
Migration Process: VM Definition for MySQL Active-Active
• Defining multiple network VMs is same as pods using meta CNI plugins like
multus.
• Using cloudInit, its easy to make and try changes in application configuration
interfaces:- bridge:{}
name: default- bridge {}
name: green-netports:- name: heartbeat
port: 694 - bridge: {}
name: orange-netports:
.
.
.
networks:- name: default
pod:{}- multus:
networkName: green-networkname: green-net
VM.yaml
![Page 69: Running Legacy VM’s along with containers in Kubernetes · Kubernetes Custom Resource Definitions API • It enables to run VMs along with containers on existing Kubernetes nodes](https://reader031.vdocument.in/reader031/viewer/2022041014/5ec4662c7cd94d146313fb85/html5/thumbnails/69.jpg)
69Copyright©2019 NTT Corp. All Rights Reserved.
Migration Process: Service Definition
• Traditionally application services are bind to particular NIC.
• These setting required to bind on hostname (or all NICs e.g. 0.0.0.0)
• Firewall rules need to ease out the restriction as static network is missing.
• These security settings move out of VM i.e. Network Policy for k8s.
• Configuration changes required in original VM
• e.g. Bind of host instead of specific interface (IP)
• Firewall rules requires to be updated
• Changes makes VM less secure.
![Page 70: Running Legacy VM’s along with containers in Kubernetes · Kubernetes Custom Resource Definitions API • It enables to run VMs along with containers on existing Kubernetes nodes](https://reader031.vdocument.in/reader031/viewer/2022041014/5ec4662c7cd94d146313fb85/html5/thumbnails/70.jpg)
70Copyright©2019 NTT Corp. All Rights Reserved.
After Migration: Active-Active without Shared Disk
Worker Node
Kubelet
Service
Users
http://ha-services.com
NodePort
DNS
Virthandler
MySQL master 1
libvirtd
virt-launcher
MySQL master 2
libvirtd
virt-launcher
Communication via IP