cloud – grid integration · on-going projects in cloud • open source cloud middleware...
TRANSCRIPT
S MADHUSUDHANA RAOCENTRE FOR DEVELOPMENT OF ADVANCED COMPUTING, CHENNAI
CLOUD – GRID Integration
• Introduction –CDAC
• Cloud Infrastructure @ CDAC
• GRID-GARUDA
• Integration of CLOUD & GRID
AGENDA
• C-DAC : seeded at Pune Centre in 1987
– High Performance Computing (HPC) – the focal area then
• Later started
– Multilingual Computing (1988)
– Advanced Computing Training School (ACTS) (1993)
• Subsequently added activities
– Healthcare
– e-Governance
• Multimedia
– Geo-Matics
• Merger of 3 premier societies of DIT with C-DAC in December, 2002
– ( ERDCI- 1982 – 88, NCST- 1985, CEDTI-1990 )
– ( Electronics, Software, Training & skills development)
C-DAC an overview
• 10 Locations
• 10 Labs
• 2500 members
C-DAC an overview
• High Performance Computing & Grid Computing
– Hardware, Software, Systems, Applications, Research, Technology,
Infrastructure
• Multilingual Computing and Heritage Computing
– Tools, Fonts, Products, Solutions, Research, Technology
Development
• Health Informatics
– Hospital Information System, Telemedicine, Decision Support
System, Tools, Traditional Knowledge-base and DSS for Medicine
• Software Technologies, including FOSS
– OSS, Multimedia, ICT for masses, E-Governance, Geo-Matics, ICT4D
Thematic focus areas of C-DAC
• Professional Electronics including VLSI & Embedded Systems
– Digital Broadband and Wireless Systems, Network Technologies,
Power Electronics, Real-Time Systems, Control Electronics,
Embedded Systems, VLSI/ASIC Design, Agri Electronics, Strategic
Electronics
• Cyber Security & Cyber Forensics
– Cyber Security tools, technologies & solution development, Research
& Training
Thematic focus areas of C-DAC
Kaleidoscope of C-DAC products
RUSSIA
TAJIKISTAN
UZBEKISTAN
GHANA
MAURITIUS
MYANMAR
SEYCHELLES
TANZANIA
International Presence
GRID - GARUDA
GARUDA India Map
GARUDA
• India’s National Grid Computing Initiative
• Distributed across 45 institutions in 17 locations.
• Total:1560 CPUs and 15.2 TF
• Network : National Knowledge Network (linking all research & educational
institutes on high speed backbone )
• Certificate : IGCA (Indian Grid Certificate Authority)
• Applications include - Disaster Management; Computer Aided Engineering;
Climate Modeling; Medical and Health Care; Collaborative Class Room
Garuda – Some Applications
• Development and parallelization of 2D model based Seismic Waveform inversion algorithm and software for estimating the elastic properties of the earth.
• Portal which provides a web-interface for Bioinformatics applications in Sequence Analysis and Molecular Modeling running on a High Performance Computer. iMolDock
• Molecular Modeling docking application, which helps the user to find out appropriate ligands for particular proteins. EQ-Check
• To tackle problems related to checking and design of earthquake resistant structures, potentially saving money, structures and lives. SD2000
• SD2000 is a supercomputing based Decision Support System using Mathematical, Statistical and Artificial Intelligence more..
CLOUD INFRASTRUCTURE
Private Cloud Infrastructure
Make Model Processor RAM HDD Quantity
HP DL 380 G5
Dual Socket Quad Core Intel Xeon
X5460 @ 3.16GHz 32 GB 4 x 146GB 3
HP DL 380G5
Dual Socket Quad Core Intel Xeon
X5460 @ 3.16GHz 16 GB 2 x 250GB 1
HP DL 160 G5
Dual Socket Quad Core Intel Xeon
X5460 @ 3.16GHz 16 GB 2 x 250GB 40
HP DL 160G5
Quad Core Intel Xeon E5405 @
2.00GHz 2 GB 2 x 250GB 1
HP DL 360G4
Dual Socket Dual Core Intel Xeon @
3.60GHz 3 GB 160GB 8
HCL Global infinity Dual Core Intel Xeon @ 3.20GHz 2 GB 2 x 146GB 4
TOTAL 57
Computational resources @ three locations Hyderabad, Bengaluru, Chennai
Make Description Capacity
20 x 300 GB 15K RPM SAS disks and 27 x 750 GB SATA disks 20 + 6 TB
EMC2 2 TB
TOTAL28 TB
Computational resources
Network : NATIONAL KNOWLEDGE NETWORK
Underlying Technical components
CDAC Chennai Cloud powered by
•BOSS Advanced Server 1.0
•Xen Hypervisor 3.2
•Eucalyptus 2.0
•Hadoop
•Appscale
•Hyperic HQ
•CDAC Scripts for Metering & Billing integrated
with Eucalyptus
•CDAC Scripts for Elasticity integrated with
Eucalyptus
Garuda Grid powered by
•Globus Toolkit 4.*
•Torque
•Gridway
ON-GOING PROJECTS In Cloud
• Open Source Cloud Middleware development
• End to End Security for Cloud
• Integration of Garuda (India’s National Grid Computing Initiative)
with Cloud infrastructure @ Chennai.
• Cloud stack for e-Governance projects (State Data centre
pilot@Kerala)
Integration Requirements
• The cloud resources hosting virtualized grid environment through the images
bundled with the grid middleware should be customized for execution of high
performance, parallel processing distributed grid applications without any
issues or hindrances.
• The grid environment should be capable of hosting cloud instances and
applications.
• The virtual grid hosted over cloud should be incorporated into the virtual
organization formation across cloud and grid or within the cloud.
Integration - Requirements
• Resource management across networking barriers
• Execution management
• Monitoring and control of jobs & instances
• Backup and disaster recovery, live migration
• Enhanced security
• Common user interface
SCOPE OF THE INTEGRATION
• Private cloud lacks with resources resulting in “Cloud burst”
• Integrates cloud middleware with grid middleware along with its schedulers
• Optimal utilization of the grid resources by deploying cloud appliances and
applications into the grid
• Return on investment to the grid service providers
• High performance grid jobs are executed over virtual environment on cloud
instance
• Dedicated resource for grid not needed.
Integrator with Gridway
• Interfacing Gridway with Integrator.
• If the job state is pending gridway must invoke integrator
• Additional no of nodes to be created in cloud resources
• Image & Security
• Eucalyptus VM Creation
• Network Management between grid and cloud
• Execution Management
• Data Management
• Gridway has DRMAA (Distributed Resource Management Application API) for
user interface
Integrator
Submits Job
Meta Scheduler (Gridway)
Computing Nodes
Selects Resource
& Submit
If Job state is
pending
Eucalyptus Cloud
Request to
create VM
Cloud Resource
Creating VM in
Cloud
Integrator with Eucalyptus
• Interfacing Eucalyptus with Integrator
• If Eucalyptus cloud can't create VM, it must invoke integrator
• Additional number of nodes to be created in grid resource
• Image & Network Management
• Security Management between grid and cloud
• VM Creation in grid resources
Integrator
Request
Meta Scheduler (Gridway)
Request to
Create VM
Eucalyptus Cloud
Creating VM in
Grid Nodes
Computing Nodes Cloud Resource
Creating VM in
Cloud Nodes
CLOUD GRID INTEGRATION ARCHITECTURE
INTEGRATION MODULE
• Hierarchy of both grid and cloud is almost the same
• Components of Integrator
• Execution Manager
• Resource Manager
• Network Manager
• Imaging component
• Security component
• Portal
GRID JOB EXECUTION IN CLOUD
• Grid jobs under execution when need additional resources from cloud, request
is sent by meta scheduler to integrator
• Resource management component in integrator approaches cloud controller
• Bundled grid image from imaging component deployed into cloud with
concurrence from the cloud controller
• Cloud controller through cluster and node controller deploys the image on n
number of virtual instances
• Local scheduler runs on newly formed grid cluster over cloud resources and is
directly controlled by the meta scheduler at the grid site.
• Virtual organization can be formed across the new grid cluster in cloud and
clusters functioning at the grid site
CLOUD APPLICATION IN GRID
• Cloud image bundled and residing in the cloud controller fetched by integrator
imaging component
• Deployed into grid through meta scheduler after concurrence from meta
scheduler
• Cloud controller along with integrator – resource manager raises a request
• Meta scheduler deploys image bundle and application is hosted on the
instances
• Grid instances contains node controller and all nodes controllers operating in
grid site is controlled by a Cluster controller deployed internally
• Cluster controller directly controlled by Cloud controller operating in the cloud
site.
• Integrator is responsible for Metering of Cloud instances, Elasticity, Backup and
disaster recovery, Elastic Ip addressing of instances deployed onto Grid.
Image Management
• Image Management is mainly used to manage the virtual images.
• Image Management module maintains the metadata as an xml DB which
contains all the image related information.
• Virtual Image is the image which has created with operating system and
applications.
Security Management
• It is mainly used to authenticate and authorize the grid and cloud users.
• The security management should have the intelligent for mapping both the
grid and cloud users to access the resource.
Network Management
• It is responsible for assigning the IP address, host name and managing the
network translation between grid and cloud resources
REFERENCES
• http://www.stratuslab.org/doku.php, http://garudaindia.in
• http://arxiv.org/ftp/arxiv/papers/0903/0903.2525.pdf
• http://alp.access-company.com/images/overview/architecture_lg.gif
• http://www.it20.info/misc/pictures/hypervisorcomparison-xen.jpg
• http://www.gridway.org/doku.php?id=documentation:release_5.6:ia
• http://gdp.globus.org/gt4-tutorial/multiplehtml/ch01s04.html
• http://open.eucalyptus.com/, http://www.hpcinthecloud.com/
• “Application Level Interoperability between Clouds and Grids”, Andre Merzky, Katerina
Stamou, Shantenu Jha, uting Conference 978-0-7695-3677-4/09 $25.00 © 2009 IEEE
• “Using Clouds to address Grid Limitations”, Giacomo V. Mc Evoy Bruno Schulze,
Laboratório Nacional de Computação Científica Av. Getúlio Vargas, 333 – Quitandinha
25651-075 Petrópolis, RJ, Brazil MGC’08, December 1-5, 2008 Leuven, Belgium.
Copyright 2008 ACM 978-1-60558-365-5/08/12...
• “Integration of Cloud and Grid Middleware at DGRZR”, Stefan Freitag, Robotics
Research Institute Dortmund University of Technology International Symposium on Grid
Computing 2010.
