the building of thai grid
DESCRIPTION
This slide present some technical challenges and experiences in building national scale gridTRANSCRIPT
Building a National Grid Infrastructure: A ThaiGrid Experience
Putchong Uthayopas,Department of Computer Engineering,Faculty of Engineering,Kasetsart University, [email protected]
The Need for National E-Infrastructure
• Efficient Resources Management for Government
• Information dissemination for people
• Emergency Response– Natural Disaster: Flooding, earth
quake, Tsunami• Better Quality of Live– Environmental monitoring,
protection
National E-Infrastructure• Fast computer
– massive computing power • Large and fast storage for data
intensive applications• Fast network
– Information sharing– Collaboration
• Middleware– Security, connectivity, access
• Portal– Hiding the complexity of access,
data movement, and execution from users
August 29,2008 TGCC 2008, Khon Khan University , Thailand
Introduction to ThaiGrid
• A National Project under Software Industry Promotion Agency (Public Organization) , Ministry of Information and Communication Technology
• Started in 2005 from 14 member organizations
• Expanded to 22 organizations in 2008
TGCC 2008, Khon Khan University , Thailand
ThaiGrid : an Enabling Platform for Innovation
August 29,2008
Thai Grid Infrastructure
2.5 G
bps
1 Gb
ps
1 G
bps 1
Gbp
s
1 Gbps
2.5 Gbps155 Mbps
310
Mbp
s15
5 M
bps
310
Mbp
s155
Mbps
155 Mbps
155
Mbp
s
155 Mbps
155
Mbp
s
19 sitesAbout 1000 CPU core.
August 29,2008 TGCC 2008, Khon Khan University , Thailand
ThaiGrid Usage• ThaiGrid provides about 290
years of computing time for members– 9 years on the grid– 280 years on tera
• 41 projects from 8 areas are being support on Teraflop machine
• More small projects on each machines
TGCC 2008, Khon Khan University , ThailandAugust 29,2008
Medicinal Herb Research• Partner
– Cheminormetics Center, Kasetsart Univesity (Chak Sangma and team)
• Objective– Using 3D-molecular databse and virtual
screening to verify the traditional medicinal herb
• Benefit– Scientific proof of the ancient
traditional drug – Benefit poor people that still rely on the
drug from medicinal herb – Potential benefit for local
pharmaceutical industry
TGCC 2008, Khon Khan University , Thailand
Virtual Screening
Infrastructure
Lab Test
August 29,2008
NanoGrid
• Objective– Platform that support computational Nano science
research• Technology used
– AccelRys Materials Studio– Cluster Scheduler: Sun Grid Engine and Torque
TGCC 2008, Khon Khan University , Thailand
AccelRys Materials Studio
Materials Studio Gateway
Cluster Scheduler Cluster Scheduler
Grid Infrastructure
ThaiGrid MS-Gateway
MS-Gateway
Computing ResourcesComputing Resources
1
2
3
August 29,2008
Building ThaiGrid
Challenges
• Size and Scale• Manageability– Deployment– Configuration– Operation
• Software and Hardware Compatibility
Principle Design Goal
• Openness• Usability• Functionality• Performance
Some statistics about Thai Grid Infrastructure
• 21 clusters from 16 sites around the country• More than than ~1,000 cores• A real super computer 4.6 Teraflops machines– 200 nodes of 800 cores of Intel Xeon 3.0GHz
• ThaiGrid provides more than 111 years of computing time for member– 7 years on the grid and more than 104 years on tera
server• More than 30-40 projects from 8 areas are being
support
Grid System Architecture
• Clusters– Satellite Sets
• 16 clusters delivered from ThaiGrid for initial members
• Composed of 5 nodes of IBM eServer xSeries 336 – Intel Xeon 2.8Ghz (Dual Processor)– x86_64 architecture– Memory: 4 GB (DDR2 SDRAM)
– Other sets• Various type of servers and
number of nodes • Provided by member institutes of
ThaiGrid
Grid Software Architecture
Cluster OS:Linux / Rocks 4.0 – 4.3 on x86/x86_64 machines
Grid Middleware:Globus Toolkit 4.0
Management/Monitoring:
SCMS
Grid Scheduler:GridWay
Programming:Ninf-G
MPICH-G2Globus WSRF
Grid Applications
Management/Monitoring:
SCMS,Ganglia
Grid Scheduler:SGE, PBS,
Condor, SCMS
Programming:MPICH
OpenMPI
Thai
Grid
Rol
l (P
rovi
de a
uto-
confi
gura
tion)
UNINET (10Gbps)
Production CA Infrastructure
CA RA OnlineRepository
Secure WebInterface
CA Operator RA Operator User
secure
secure
face-to-faceVTC or Skype
Experiences
• Building a quick deployment technology is a key!– NPACI Rocks is a key technology being used– Collaboration SDSC/ TNGC under PRAGMA– ThaiGrid Roll is built to facilitate the fast deployment
time• Building a good infrastructure monitoring and
management system– Know what your grid are doing.– Develop system monitoring, accounting, testing tools
Grid Monitoring and Scheduling
• Using SCMSWeb to monitor and probe for Grid status– Grid services probe is mainly used to categorize services
level in ThaiGrid– Used in resource certification system
• Grid Meta Scheduler– Meta scheduler is installed in central site– GridWay 5.2 is installed and experiment in some projects
• Drug Design Grid• Information is pulled from SCMSWeb
C C C C
H
C C C C
H C C C C
H
C C C C
HREN
GCC
Grid Scheduler
Grid as a Super Cluster
TGCC 2008, Khon Khan University , ThailandAugust 29,2008
Experiences
• Strong standard compliance is critical to make the systems interoperable
• Site certification program– Level 1: Network firewall
compliance– Level 2: Middleware
compliance (MPI/G2, Ninf-g)– Level 3: Job scheduling
compliance– Level 4: Data Grid compliance
TERA Cluster
August 29,2008 TGCC 2008, Khon Khan University , Thailand
Edge Switch 1Gbps Ethernet
FESunyata
FEAraya
WinHPC(FE)
TERA(FE)
SPARE1(FE)
SPARE2(FE)
FS1
FS2
FS3
FS4
4 nodes 4 nodes 64nodes
96 nodes +
16 sparenodes
200 Ports Gigabit Ethernet switch
Storage Tier 5TB Lustre FS
Anatta(FE)
15nodes
KU Fiber Backbone (1Gbps Fiber)
2.5Gbps to UninetStorage 48 TB
1 Gbps Ethernet/Fiber
• 1 Frontend (HP ProLiant DL360 G5 Server) and 192 computer nodes– Intel Xeon 3.2
GHz (Dual core, Dual processor)
– Memory 4 GB (8GB for Frontend & infiniband nodes)
– 70x4 GB SCSI HDD (RAID1)
• 4 Storage Servers– Lustre file
system for TERA cluster's storage
– Attached with Smart Array P400i Controller for 5TB space
Experience
• For a large user group, users will cluster them self based on their application interest. Create a divers set of requirement in term of Operating system, Scheduler and End user application– Single large cluster may not be used effectively
• Partition large cluster into several set of partitionthat can be modified seems to work well.– We have two set of infiniband for parallel job– Unix and windows for single batch/interactive
Future
• The world are moving toward Cloud– More cost effective– Better manage through centralized control– More elastic and on demand
• Thailand cover with cloud– E-government cloud– National Crisis Management Cloud– Research Cloud– Education Cloud
• Support mobile learning
• Nationwide broadband infrastructure is needed– 4G/4G , Wifi, ADSL , Fiber to Home
Acknowledgement• I would like to Thank you
– Dr. Vara Varavithya for being the co-founder of the project and spent numerous hours helping on laying out many technical detail.
– Dr. Arnon Rungsawang and ThaiGrid staffs for the hard work during the project to prove that we can do it!
– All the university contact persons for helping make our dream come true for a while
– Kasetsart University that allow me to do this for our country.
– MICT, SIPA, NECTEC, NANOTECH, NSTDA for their support
• Any honor and award should go to them for their contributions . I can not do this alone!
Thank you