performance compute solving hybrid cloud for high€¦ · os • linux clouds • azure • gcp •...

22
Solving Hybrid Cloud for High Performance Compute TUT1195 Brian Farrell Director Eoghan Cotter Systems Specialist

Upload: others

Post on 26-Sep-2020

4 views

Category:

Documents


0 download

TRANSCRIPT

Page 1: Performance Compute Solving Hybrid Cloud for High€¦ · OS • Linux Clouds • Azure • GCP • AWS • Openstack • Bare metal • KVM • Native Batch • SLURM • PBS / Moab

Solving Hybrid Cloud for High Performance Compute

TUT1195

Brian Farrell Director Eoghan Cotter Systems Specialist

Page 2: Performance Compute Solving Hybrid Cloud for High€¦ · OS • Linux Clouds • Azure • GCP • AWS • Openstack • Bare metal • KVM • Native Batch • SLURM • PBS / Moab

Securelinx has been successfully delivering Open Source innovation and value for customers throughout Ireland & UK for over 15 years.

• Based in Dublin & founded 2002

• Highly technically skilled in Linux

• 10 full-time engineers

• Very strong industry reputation

• Key Industry Partnerships

• Enterprise customers all sectors

Securelinx

Page 3: Performance Compute Solving Hybrid Cloud for High€¦ · OS • Linux Clouds • Azure • GCP • AWS • Openstack • Bare metal • KVM • Native Batch • SLURM • PBS / Moab

Securelinx & HPC

• Design and delivery of HPC systems to multiple customers

• Provide HPC Managed Services to Multiple Organisations

• 3rd Level Escalation Support

• Support Provided Includes:

○ Infrastructure Support

○ Application Support

○ Addressing Researcher’s Concerns

Page 4: Performance Compute Solving Hybrid Cloud for High€¦ · OS • Linux Clouds • Azure • GCP • AWS • Openstack • Bare metal • KVM • Native Batch • SLURM • PBS / Moab

Solving Hybrid Cloud for HPC• Initial Problem

• Tooling

• Scheduling

• Demo

• Roadmap

Page 5: Performance Compute Solving Hybrid Cloud for High€¦ · OS • Linux Clouds • Azure • GCP • AWS • Openstack • Bare metal • KVM • Native Batch • SLURM • PBS / Moab

Proof of Concept

Page 6: Performance Compute Solving Hybrid Cloud for High€¦ · OS • Linux Clouds • Azure • GCP • AWS • Openstack • Bare metal • KVM • Native Batch • SLURM • PBS / Moab

Hybrid HPC

Page 7: Performance Compute Solving Hybrid Cloud for High€¦ · OS • Linux Clouds • Azure • GCP • AWS • Openstack • Bare metal • KVM • Native Batch • SLURM • PBS / Moab

SHC4HPC

Page 8: Performance Compute Solving Hybrid Cloud for High€¦ · OS • Linux Clouds • Azure • GCP • AWS • Openstack • Bare metal • KVM • Native Batch • SLURM • PBS / Moab

Securelinx Hybrid Cloud for HPC

Page 9: Performance Compute Solving Hybrid Cloud for High€¦ · OS • Linux Clouds • Azure • GCP • AWS • Openstack • Bare metal • KVM • Native Batch • SLURM • PBS / Moab

Securelinx Hybrid Cloud for HPC

Page 10: Performance Compute Solving Hybrid Cloud for High€¦ · OS • Linux Clouds • Azure • GCP • AWS • Openstack • Bare metal • KVM • Native Batch • SLURM • PBS / Moab

Infrastructure

Page 11: Performance Compute Solving Hybrid Cloud for High€¦ · OS • Linux Clouds • Azure • GCP • AWS • Openstack • Bare metal • KVM • Native Batch • SLURM • PBS / Moab

Infrastructure• Implementation will be specific to source and destination infrastructure.

– Follow Best Practises• Understand Connectivity -

– Where are the services all resources need access to eg;• Authentication• Scheduler• Orchestration• Storage

Page 12: Performance Compute Solving Hybrid Cloud for High€¦ · OS • Linux Clouds • Azure • GCP • AWS • Openstack • Bare metal • KVM • Native Batch • SLURM • PBS / Moab

Images

Page 13: Performance Compute Solving Hybrid Cloud for High€¦ · OS • Linux Clouds • Azure • GCP • AWS • Openstack • Bare metal • KVM • Native Batch • SLURM • PBS / Moab

SHC4HPC - Image Pipeline

Page 14: Performance Compute Solving Hybrid Cloud for High€¦ · OS • Linux Clouds • Azure • GCP • AWS • Openstack • Bare metal • KVM • Native Batch • SLURM • PBS / Moab

Scheduler

Page 15: Performance Compute Solving Hybrid Cloud for High€¦ · OS • Linux Clouds • Azure • GCP • AWS • Openstack • Bare metal • KVM • Native Batch • SLURM • PBS / Moab

SHC4HPC - Batch integrationSHC4HPC

**

* roadmap

Page 16: Performance Compute Solving Hybrid Cloud for High€¦ · OS • Linux Clouds • Azure • GCP • AWS • Openstack • Bare metal • KVM • Native Batch • SLURM • PBS / Moab

Demo

Page 17: Performance Compute Solving Hybrid Cloud for High€¦ · OS • Linux Clouds • Azure • GCP • AWS • Openstack • Bare metal • KVM • Native Batch • SLURM • PBS / Moab

Storage

Page 18: Performance Compute Solving Hybrid Cloud for High€¦ · OS • Linux Clouds • Azure • GCP • AWS • Openstack • Bare metal • KVM • Native Batch • SLURM • PBS / Moab

Storage• Work with users so they understand that using the cloud resources means

for there jobs• Ideas:

– Multiple Blob Storage Accounts Extend Performance– Directly connect users to on prem storage– Build ephemeral HPC storage in the remote cloud

Page 19: Performance Compute Solving Hybrid Cloud for High€¦ · OS • Linux Clouds • Azure • GCP • AWS • Openstack • Bare metal • KVM • Native Batch • SLURM • PBS / Moab

SummaryOS

• LinuxClouds

• Azure• Openstack• Bare metal• KVM• Native

Batch• SLURM

Orchestration• Build + Scripts

Page 20: Performance Compute Solving Hybrid Cloud for High€¦ · OS • Linux Clouds • Azure • GCP • AWS • Openstack • Bare metal • KVM • Native Batch • SLURM • PBS / Moab

SHC4HPC Gen 2

Roadmap

Page 21: Performance Compute Solving Hybrid Cloud for High€¦ · OS • Linux Clouds • Azure • GCP • AWS • Openstack • Bare metal • KVM • Native Batch • SLURM • PBS / Moab

RoadmapOS

• LinuxClouds

• Azure• GCP• AWS• Openstack• Bare metal• KVM• Native

Batch• SLURM• PBS / Moab

Orchestration• Build + Scripts• SaltStack

Architecture• Open Source• Separate Service

Page 22: Performance Compute Solving Hybrid Cloud for High€¦ · OS • Linux Clouds • Azure • GCP • AWS • Openstack • Bare metal • KVM • Native Batch • SLURM • PBS / Moab

Questions?