agile hpc with vmware cloud foundation & ubercloud€¦ · grid engine, kubernetes) ... app os...

1 Confidential │ ©2020 VMware, Inc.

Agile HPC with VMware Cloud Foundation & UberCloud

Thomas Francis, UberCloud

Mohan Potheri, VMware

Confidential │ ©2020 VMware, Inc.

Agenda

2

•  Traditional HPC architecture

•  Path to agile HPC

•  vSphere announcements for Kubernetes

•  Containers and HPC with UberCloud

•  Example deployment


Traditional HPC Architecture

3

•  Login / Head Node

•  Scheduler

•  Storage Gateway

•  NFS Storage

•  Compute Nodes

•  Accelerator Nodes

•  Parallel Filesystem (Lustre)

•  High Speed Interconnects

Confidential │ ©2020 VMware, Inc. 4

  Variety of Devices (CPU, RAM, Storage, Networking/RDMA, GPU, FPGA)

  Workload Management Options (Moab, PBS, Grid Engine, Kubernetes)

  Dormant infrastructure that is only refreshed every 3 years

Slower innovation

Complex resource and application management

Higher Costs

Siloed operations

80% of budget spend on keeping the lights on

  Complex Infrastructure   Business Impact

Enterprise HPC Today

4


Virtual Machine Benefits

5

hardware

hypervisor

VM OS App

Virtual Machines offer: •  Multi-cluster aggregation

(heterogeneity)

•  Multi-tenant data security

•  Fault isolation

•  Reproducibility

•  Fault resiliency

•  Dynamic load balancing

•  Performance

hardware


The Road to a Modern Data Center Architecture for HPC

6

HW

Legacy Apps

Cloud Native Apps

From traditional Data Center to unified, scalable Data Center operations…

Combining existing tools with new data center technologies •  VMware VCF virtualizes Data Center resources •  Kubernetes is the orchestrator •  Docker Containers are the new kernel •  UberCloud Containers bridge to HPC •  Enabling HPC and AI to converge •  Preserving existing CAPEX investment •  Addressing rapidly evolving business requirements •  Accelerating the HPC application lifecycle


VMware vSphere 7 & VCF 4: Unifying Kubernetes and VMs vSAN

IT Operator Developer

Kubernetes is embedded into the control plane of vSphere

vCen

ter S

erve

r

AI/ML Business Critical

DB & Analytics Time-critical

Namespaces

VMware Cloud Foundation Services

Tanzu Runtime Services

Hybrid Infrastructure Services VM Service Storage Service Registry Service

vSphere Pod Service Network Service Hybrid Mobility Service

Tanzu Kubernetes Grid Service

vSphere vSAN NSX vRealize


VMware + UberCloud Provide:

8

•  Hybrid and multi-clouds

•  HPC, Big Data, AI, Enterprise apps and microservices

•  Commercial, OpenSource, in-house codes

•  CAE, Life Sciences, EDA, Finance, Oil & Gas, …

Choice Agility Security

•  Unified operations •  Rapid delivery and

response •  Cost efficiency

•  HW flexibility

•  Package once, move to any cloud

•  On-demand Scaling of resources

•  High Availability & Auto-Recovery

•  Safer applications •  Governance •  Chain of custody

•  Threat mitigations

•  Multitenancy

•  HW+Network+SW isolation

•  Additional Login/Password for app container protection

•  iLO with Trusted Platform Module offer enhanced security


Containers: The New Application Deployment Standard

Application

Operating System

Run Time

Middleware

Infrastructure Integration

•  Portable across hybrid infrastructure

•  Continuous updates

•  Can bridge HPC & Enterprise IT

Cloud Infrastructure


Automated Container Factory: CI/CD to build containers

•  Templatized HPC containers

•  Automated Build

•  Integrated Testing

•  Controlled Versioning

10


Deployment

11

Capabilities On Premise Off Premise* Public Cloud Interactive Access (allowing full engineering workflow)

✔ ✔ ✔

Batch Access ✔ ✔ ✔

Data Isolation (by user/group)

✔ (per cluster)

✔ (per cluster)

✔ (per cluster)

Access Security ✔ (inside network)

✔ (subnet & VPN)

✔ (VPN)

Integrated Security ✔ ✔ ✔ (Automated Threat Scanning, Controlled Code Deployment, Encrypted Communications, Role-based access

control, Policy Management and Enforcement, Image Signing)

RDMA Support ✔ ✔ ✔

GPU Support ✔ ✔ ✔

Cluster Resizing ✔ ✔ ✔

Multi-User Access ✔ ✔ ✔

On-Demand Access ✔ ✔ ✔

Single Image on platforms ✔ ✔ ✔

Certification ✔ ✔ ✔ (Certified plug-ins and Infrastructure, Certified ISV apps as containers, Certified professionals)

Automation ✔ ✔ ✔ (Built-in app reliability/High Availability, Policy-based automation, Auto health check, Auto Load Balance/Deploy/

Recovery, Containerized Application Lifecycle Management, Automated Deployment/DevOps)

Orchestration ✔ (UGE, K8s, others)

✔ (UGE, K8s, others)

✔ (UGE, K8s, others)

Add, remove compute nodes (containers), or terminate service.

Execute Docker Run to start Login Container and Compute Containers

Configure multi-node container environment

Load UberCloud Docker Container Image on all VMs

Pull UberCloud Docker Container Image from Registry to Cluster Storage

Install Docker EE (Runtime)

Configure virtual network, devices, and storage

Create new cluster in vSphere and add hosts login and compute resources

Deploy VCSA and Configure vCenter (management server(s))

Deploy ESXi on servers (management and compute)


Video Demo

12


HPC Case Studies

13

50 HPC Case Studies with UberCloud’s HPC Containers since 2015

•  Selection of 21 Containerized Engineering Case Studies: https://info.theubercloud.com/ubercloud-container-compendium

Using HPC Software Containers based on Docker


Thank You

14

agile hpc with vmware cloud foundation & ubercloud€¦ · grid engine, kubernetes) ... app os...

Documents