kite
DESCRIPTION
KITE. A down to earth approach to cloud computing. Marco Konijnenburg SIG SE @ Nikhef 14 november 2013. Contents. Context Predecessor Goals Design. Processing large datasets. Datasets are over 10 GB in size Collection of thousands of spectra - PowerPoint PPT PresentationTRANSCRIPT
Softw
are
Engi
neer
ing
@ F
OM In
stitu
te A
MOL
F
Marco Konijnenburg
SIG SE @ Nikhef
14 november 2013
KITE
A down to earth approach to cloud computing
Softw
are
Engi
neer
ing
@ F
OM In
stitu
te A
MOL
F
2
Contents
• Context
• Predecessor
• Goals
• Design
Softw
are
Engi
neer
ing
@ F
OM In
stitu
te A
MOL
F
3
Processing large datasets• Datasets are over 10 GB in size
• Collection of thousands of spectra
• Each spectrum can be processed individually
Softw
are
Engi
neer
ing
@ F
OM In
stitu
te A
MOL
F
4
Compute
• Use general purpose compute clusters
• Use dedicated compute cluster
• Splittable compute jobs
Softw
are
Engi
neer
ing
@ F
OM In
stitu
te A
MOL
F
5
Cloud• HPC –Cloud Calligo @ SARA
– Open Nebula implementation
• Configurable systems– Mix various systems (like different OS-es)
• Start and stop systems at will– Using VM’s
Softw
are
Engi
neer
ing
@ F
OM In
stitu
te A
MOL
F
6
COMMIT
Job workersJob office (headnode)
Calligo
Firewall
Proxy
Internet
ToPoS
J ob administratorJob administrator
Softw
are
Engi
neer
ing
@ F
OM In
stitu
te A
MOL
F
7
Job submission application• Connect to headnode
• Specify script , data and storage for MS processing
• Specify number of tokens (job parts)
• Submit job– Receive email of job
status– View progress
Softw
are
Engi
neer
ing
@ F
OM In
stitu
te A
MOL
F
8
Cloud UI
Softw
are
Engi
neer
ing
@ F
OM In
stitu
te A
MOL
F
9
Improve• Easier interface for non computer
scientists– Start and stop preconfigured group of VM’s
• Interface from desktop app to web app– Multi platform– Always up to date– Needs a login to remember user settings
Softw
are
Engi
neer
ing
@ F
OM In
stitu
te A
MOL
F
10
Overview
Softw
are
Engi
neer
ing
@ F
OM In
stitu
te A
MOL
F
11
Kite log in• Use existing techniques
– Browser login, SSO
– Using Shibboleth to connect to SURFConext
Softw
are
Engi
neer
ing
@ F
OM In
stitu
te A
MOL
F
12
Softw
are
Engi
neer
ing
@ F
OM In
stitu
te A
MOL
F
13
KRON• Kite relay for Open Nebula
• Only command and data passing
• Reports to Kite when (re)started
Softw
are
Engi
neer
ing
@ F
OM In
stitu
te A
MOL
F
14
KRON
Softw
are
Engi
neer
ing
@ F
OM In
stitu
te A
MOL
F
15
KRON (expanded)
Softw
are
Engi
neer
ing
@ F
OM In
stitu
te A
MOL
F
16
Communication
Softw
are
Engi
neer
ing
@ F
OM In
stitu
te A
MOL
F
17
Issues & Wishes• XML RPC requires extra log in, not
SURFConext’ed
• Within one group VM’s ‘see’ each other
• Data in and out
• Scale it up to multiple clouds, possibly multiple types
• Scale it up to users from other institutes
Softw
are
Engi
neer
ing
@ F
OM In
stitu
te A
MOL
F
18
The end
Thank you for your attention
Contact:
MARCO.KONIJNENBURG
HTTP://WWW.LINKEDIN.COM/IN/MARCOKONIJNENBURG