lxcloud

CERN IT DepartmentCH-1211 Genève 23

Switzerlandwww.cern.ch/it

PES CERN's Cloud Computing infrastructure

CERN's Cloud Computing Infrastructure

Tony Cass, Sebastien Goasguen, Belmiro Moreira, Ewan Roche, Ulrich Schwickerath, Romain Wartel

Cloudview conference, Porto, 2010

See also related presentations: HEPIX spring and autumn meeting 2009, 2010 Virtualization vision, Grid Deployment Board (GDB) 9/9/2009 Batch virtualization at CERN, EGEE09 conference, Barcelona

CERNs Cloud computing infrastructure - a status report - 2



PES Outline and disclaimer

An introduction to CERN

Why virtualization and cloud computing ?

Virtualization of batch resources at CERN

Building blocks and current status

Image management systems: ISF and ONE

Status of the project and first numbers

Disclaimer: We are still in the testing and evaluation phase. No final decision has been taken yet on what we are going to use in the future.

All given numbers and figures are preliminary




PES

European Organization for Nuclear Research

The world’s largest particle physics laboratory

Located on Swiss/French border

Funded/staffed by 20 member states in 1954

With many contributors in the USA

Birth place of World Wide Web

Made popular by the movie “Angels and Demons”

Flag ship accelerator LHC

http://www.cern.ch

Introduction to CERN

http://www.cern.ch/




PES Introduction to CERN: LHC and the experiments

LHCb Alice

ATLAS CMS

Circumference of LHC: 26 659 mMagnets : 9300Temperature: -271.3°C (1.9 K)Cooling: ~60T liquid HeMax. Beam energy: 7TeVCurrent beam energy: 3.5TeV

TOTEMLHCf




PES Introduction to CERN

Data Signal/Noise ratio 10-9

Data volume:High rate * large number of channels * 4 experiments

15 PetaBytes of new data each year

Compute powerEvent complexity * Nb. events * thousands users

100 k CPUs (cores)Worldwide analysis & fundingComputing funding locally in major regions & countriesEfficient analysis everywhere

GRID technology




PES LCG computing GRID

Number of sites:

T0: 1 (CERN), 20% T1: 11 round the worldT2: ~160

http://lcg.web.cern.ch/lcg/public/

Requried computing capacity:~100 000 processors




PES The CERN Computer Center

Disk and tape: 1500 disk servers5PB disk space16PB tape storage

Computing facilities:>20.000 CPU cores (batch only)Up to ~10000 concurrent jobsJob throughput ~200 000/day

http://it-dep.web.cern.ch/it-dep/communications/it_facts__figures.htm




PES Why virtualization and cloud computing ?

Service consolidation:

Improve resource usage by squeezing mostly unused machines onto single big hypervisors Decouple hardware life cycle from applications running on the box Ease management by supporting life migration

Virtualization of batch resources:

Decouple jobs and physical resources Ease management of the batch farm resources Enable the computer center for new computing models

This presentation is about virtualization of batch resources only




PES Batch virtualization

Type 1: Run my jobs in your VM

Type 2: Run my jobs in my VM

CERN batch farm lxbatch:~3000 physical hosts~20000 CPU cores>70 queues




PES

Type 3:Give me my infrastructurei.e a VM or a batch of VMs

Towards cloud computing




PES Philosophy

Internal cloud

development other/cloud applicationsT0

I

(far) future ?

hypervisor cluster

Batch

Physical SLC4 WN

Physical SLC5 WN

SLC4 WN SLC5 WN

Batch

Near future: (

(SLC = Scientific Linux CERN)




PES Visions beyond the current plan

Reusing/sharing images between different sites (phase 2) HEPIX working group founded in autumn 2009 to define rules and boundary conditions (https://www.hepix.org/)

Experiment specific images (phase 3)

Use of images which are customized for specific experiments

Use of resources in a cloud like operation mode (phase 4)

Images directly join experiment controlled scheduling systems (phase 5)

Controlled by experiment Spread across sites




PES Virtual batch: basic ideas

Platform Infrastructure Sharing Facility (ISF)

For high level VM management

Virtual batch worker nodes:

Clones of real worker nodes, same setup

Mix with physical resources

Dynamically join the batch farm as normal worker nodes

Limited lifetime: stop accepting jobs after 24h

Destroy when empty

Only one user job per VM at a time

Note: The limited lifetime allows for a fully automated system which dynamically adapts to the current needs, and automatically deploys intrusive updates.




PES Virtual batch: basic ideas, technical



Images:

staged on hypervisors

master images, instances use LVM snapshots

Start with few different flavors only

Image creation:

Derived from a centrally managed “golden node”

Regularly updated and distributed to get updates “in”







Images distribution:

Only shared file system available is AFS

Prefer peer to peer methods (more on that later)

SCP wave

Rtorrent







VM placement and management system

Use existing solutions

Testing both a free and a commercial solution

OpenNebula (ONE)

Platform's Infrastructure Sharing Facility (ISF)




PES Batch virtualization: architecture

Batch systemmanagement

VM worker nodesWith limited lifetime VM management system

Hypervisors / HW resources

VM kiosk

Centrally managedGolden nodes

Job submissionCE/interactive

Grey: physical resourceColored: different VMs




PES Status of building blocks (test system)

Submission and batch managemen

Hypervisor cluster

VM kiosk and image distribution

VM management system

Initial deployment OK OK OK OK

Central management OK OK Mostly

implemented

ISF OK, ONE

missing

Monitoring and alarming OK Switched off

for testsmissing missing




PES Image distribution: SCP wave versus rtorrent

Preliminary !

Slow nodes,(under investigation)

(BT = bit torrent)




PES VM placement and management

OpenNebula (ONE): Basic model:

Single ONE master

Communication with hypervisors via ssh only

(Currently) no special tools on the hypervisors

Some scalability issues at the beginning (50 VM at the beginning)

Addressing issues as they turn up

Close collaboration with developers, ideas for improvements

Managed to start more than 7,500 VMs




PES Scalability tests: some first numbers

“One shot” test with OpenNebula: Inject virtual machine requests And let them die Record the number of alive machines seen by LSF every 30s

Units: 1h5min




PES VM placement and management

Platforms Infrastructure Sharing Facility (ISF)

One active ISF master node, plus fail-over candidates

Hypervisors run an agent which talks to XEN

Needed to be packaged for CERN

Resource management layer similar to LSF

Scalability expected to be good but needs verification

Tested with 2 racks (96 machines) so far, ramping up

Filled with ~2.000 VMs so far (which is the maximum)

See: http://www.platform.com




PES Screen shots: ISF

Status: 1 of 2 racks available

VM status:

Note: one out of 2 racks enabled for demonstration purpose




PES Screen shots: ISF




PES Summary

Virtualization efforts at CERN are proceeding. Still some work to be done.

Main challenges

Scalability considerations

provisioning systems

of the batch system

No decision on provisioning system to be used yet

Reliability and speed of image distribution

General readiness for production (hardening)

Seamless integration into the existing infrastructure




PES Outlook

What's next ?

Continue testing of ONE and ISF in parallel

Solve remaining (known) issues

Release first VMs for testing by our users soon




PES Questions ?

?




PES Philosophy

Hypervisor cluster(physical resources)

OpenNebula



pVMO

VM provisioning system(s)




PES Details: Some additional explanations ...

“Golden node”:A centrally managed (i.e. Quattor controlled) standard worker node

which Is a virtual machine Does not accept jobs Receives regular updates

Purpose: creation of VM images

“Virtual machine worker node”: A virtual machine derived from a golden node Not updated during their life time Dynamically adds itself to the batch farm Accepts jobs for only 24h Runs only one user job at a time Destroys itself when empty




PES VM kiosk and Image distribution

Boundary conditions at CERN

Only available shared file system is AFS Network infrastructure with a single 1GE connection

No dedicated fast network for transfers that could be used (eg 10GE, IB or similar)

Tested options:

Scp wave: Developed at Clemson university Based on simple scp

rtorrent: Infrastructure developed at CERN Each node starts serving blocks it already hosts




PES Details: VM kiosk and Image distribution

Local image distribution model at CERN

Approved images are moved to a central image repository (the “kiosk”) Hypervisors check regularly for new images on the kiosk node The new image is transferred to a temporary area on the hypervisors When the transfer is finished, a new LV is created The new image is unpacked into the new LV The current production image is renamed (via lvrename) The new image is renamed to become the production image

Note: may need a sophisticated locking strategy in this process

Virtual machines are instantiated as LVM snapshots of a base image.This is process is very fast

Replacing a production image:




PES Image distribution with torrrent: transfer speed

7GB file compressed452 target nodes

90% finished after 25minPrelim

inary

Slow nodes, under investigation




PES Image distribution: total distribution speed

7GB file compressed452 target nodes

All done after 1.5h

→ Unpacking still needs some tuning !

Preliminary

lxcloud

Technology

sites cern

vm cern

barcelona cern

department ch

status report

new computing

grid technology cern

cern computer centerdisk