indiana university's lustre wan: empowering production workflows on the teragrid and beyond...
TRANSCRIPT
![Page 1: Indiana University's Lustre WAN: Empowering Production Workflows on the TeraGrid and beyond Craig Stewart and Stephen C. Simms Indiana University stewart@iu.edu](https://reader036.vdocument.in/reader036/viewer/2022062516/56649e2a5503460f94b17f04/html5/thumbnails/1.jpg)
Indiana University's Lustre WAN: Empowering Production Workflows
on the TeraGrid and beyond
Craig Stewart and Stephen C. SimmsIndiana University
![Page 2: Indiana University's Lustre WAN: Empowering Production Workflows on the TeraGrid and beyond Craig Stewart and Stephen C. Simms Indiana University stewart@iu.edu](https://reader036.vdocument.in/reader036/viewer/2022062516/56649e2a5503460f94b17f04/html5/thumbnails/2.jpg)
License terms• Please cite as: Stewart, C.A. and S.C. Simms. 2010. Indiana University's
Lustre WAN: Empowering Production Workflows on the TeraGrid and beyond. (Presentation) TeraGrid Forum (Distributed virtual meeting, 20 May 2010). Available from: http://hdl.handle.net/2022/13912
• Except where otherwise noted, by inclusion of a source url or some other note, the contents of this presentation are © by the Trustees of Indiana University. This content is released under the Creative Commons Attribution 3.0 Unported license (http://creativecommons.org/licenses/by/3.0/). This license includes the following terms: You are free to share – to copy, distribute and transmit the work and to remix – to adapt the work under the following conditions: attribution – you must attribute the work in the manner specified by the author or licensor (but not in any way that suggests that they endorse you or your use of the work). For any reuse or distribution, you must make clear to others the license terms of this work.
2
![Page 3: Indiana University's Lustre WAN: Empowering Production Workflows on the TeraGrid and beyond Craig Stewart and Stephen C. Simms Indiana University stewart@iu.edu](https://reader036.vdocument.in/reader036/viewer/2022062516/56649e2a5503460f94b17f04/html5/thumbnails/3.jpg)
NSF initial funding in 2005, expanded with IU funds
Aggregate 936 formatted Terabytes Lustre storage
14.5 GB/s aggregate writeShort term storage
The Data Capacitor Project
![Page 4: Indiana University's Lustre WAN: Empowering Production Workflows on the TeraGrid and beyond Craig Stewart and Stephen C. Simms Indiana University stewart@iu.edu](https://reader036.vdocument.in/reader036/viewer/2022062516/56649e2a5503460f94b17f04/html5/thumbnails/4.jpg)
![Page 5: Indiana University's Lustre WAN: Empowering Production Workflows on the TeraGrid and beyond Craig Stewart and Stephen C. Simms Indiana University stewart@iu.edu](https://reader036.vdocument.in/reader036/viewer/2022062516/56649e2a5503460f94b17f04/html5/thumbnails/5.jpg)
IU’s Data Capacitor WAN• 1 pair Dell PowerEdge 2950 for MDS• 2 pair Dell PowerEdge 2950 for OSS
– 2 x 3.0 GHz Dual Core Xeon– Myrinet 10G Ethernet– Dual port Qlogic 2432 HBA (4 x FC)– 2.6 Kernel (RHEL 5)
• DDN S2A9550 Controller– Over 2.4 GB/sec measured
throughput– 360 Terabytes of spinning SATA disk
• Currently running Lustre 1.6.7.2• Upgrading to 1.8.1.1 in May
• Announced production at LUG 2008• Allocated on Project by Project basis
![Page 6: Indiana University's Lustre WAN: Empowering Production Workflows on the TeraGrid and beyond Craig Stewart and Stephen C. Simms Indiana University stewart@iu.edu](https://reader036.vdocument.in/reader036/viewer/2022062516/56649e2a5503460f94b17f04/html5/thumbnails/6.jpg)
IU UID Mapping
Lightweight
Not everyone needs / wants kerberos
Not everyone needs / wants encryption
Only change MDS code
Want to maximize clients we can serve
Simple enough to port the code forward
![Page 7: Indiana University's Lustre WAN: Empowering Production Workflows on the TeraGrid and beyond Craig Stewart and Stephen C. Simms Indiana University stewart@iu.edu](https://reader036.vdocument.in/reader036/viewer/2022062516/56649e2a5503460f94b17f04/html5/thumbnails/7.jpg)
IU UID Mapping cont’d
• UID lookups on the MDS call a pluggable kernel module– Binary tree stored in memory– Based on NID or NID range– Remote UID mapped to Effective UID
![Page 8: Indiana University's Lustre WAN: Empowering Production Workflows on the TeraGrid and beyond Craig Stewart and Stephen C. Simms Indiana University stewart@iu.edu](https://reader036.vdocument.in/reader036/viewer/2022062516/56649e2a5503460f94b17f04/html5/thumbnails/8.jpg)
Username
IP T
able
s
ClientNID/UID
1.4.x1.6.x1.8.1
Kern
el
Mod
Kern
el
Mem
ory
Pat
ched
MD
SNID - Remote UID - Local UID
Client UIDs/etc/passwd
TGCDBUsername
NID Ranges
SQLite
![Page 9: Indiana University's Lustre WAN: Empowering Production Workflows on the TeraGrid and beyond Craig Stewart and Stephen C. Simms Indiana University stewart@iu.edu](https://reader036.vdocument.in/reader036/viewer/2022062516/56649e2a5503460f94b17f04/html5/thumbnails/9.jpg)
UID Mapping
• Userspace – Kernel Space Barrier– Only crossed when we update the table
• Create a Forest of Binary Trees– Forward and Inverse Lookups for each UID– Time consumed for lookup is predictable
• Speed over Space• Consume memory rather than on the fly lookups• Every UID node consumes 6 Ints• 300 Users approximately 300KB
![Page 10: Indiana University's Lustre WAN: Empowering Production Workflows on the TeraGrid and beyond Craig Stewart and Stephen C. Simms Indiana University stewart@iu.edu](https://reader036.vdocument.in/reader036/viewer/2022062516/56649e2a5503460f94b17f04/html5/thumbnails/10.jpg)
IU’s Lustre WAN on the TeraGrid
• 8 Sites currently mounting IU DC-WAN– IU, LONI, NCSA, NICS, PSC, Purdue, SDSC, TACC
• 5 Sites mounting on compute resources– IU, LONI, NCSA, PSC, TACC
• Average of 93% capacity for the last quarter• 2009 uptime of 96%
– Filesystem availability to users
• PBs of aggregate writes and reads in NSF FY 2010
![Page 11: Indiana University's Lustre WAN: Empowering Production Workflows on the TeraGrid and beyond Craig Stewart and Stephen C. Simms Indiana University stewart@iu.edu](https://reader036.vdocument.in/reader036/viewer/2022062516/56649e2a5503460f94b17f04/html5/thumbnails/11.jpg)
One Degree Imager (ODI)
HPSS
WIYN Telescope
Tuc
son,
Ariz
ona
1726 miles
![Page 12: Indiana University's Lustre WAN: Empowering Production Workflows on the TeraGrid and beyond Craig Stewart and Stephen C. Simms Indiana University stewart@iu.edu](https://reader036.vdocument.in/reader036/viewer/2022062516/56649e2a5503460f94b17f04/html5/thumbnails/12.jpg)
Ethnographic Video for Instruction and AnalysisEVIA
Samba
Video Acquisiton
Server
HPSS
Compression/AnnotationServer
1 mile
346 miles
Ann
Arb
or,
Mic
higa
n
![Page 13: Indiana University's Lustre WAN: Empowering Production Workflows on the TeraGrid and beyond Craig Stewart and Stephen C. Simms Indiana University stewart@iu.edu](https://reader036.vdocument.in/reader036/viewer/2022062516/56649e2a5503460f94b17f04/html5/thumbnails/13.jpg)
Linked Environments for Atmospheric Discovery LEAD
Big RedCompute Resource
Data TransferServer
2 miles
![Page 14: Indiana University's Lustre WAN: Empowering Production Workflows on the TeraGrid and beyond Craig Stewart and Stephen C. Simms Indiana University stewart@iu.edu](https://reader036.vdocument.in/reader036/viewer/2022062516/56649e2a5503460f94b17f04/html5/thumbnails/14.jpg)
Center for the Remote Sensing of Ice Sheets (CReSIS) Workflow
• gg
U of Kansas
Greenland
IU Quarry Cluster
HPSS
517 miles
Law
renc
e, K
ansa
s
![Page 15: Indiana University's Lustre WAN: Empowering Production Workflows on the TeraGrid and beyond Craig Stewart and Stephen C. Simms Indiana University stewart@iu.edu](https://reader036.vdocument.in/reader036/viewer/2022062516/56649e2a5503460f94b17f04/html5/thumbnails/15.jpg)
Samba
CRYO Electron Microscopy
3 miles
HPSS
Big Red
Electron microscope
![Page 16: Indiana University's Lustre WAN: Empowering Production Workflows on the TeraGrid and beyond Craig Stewart and Stephen C. Simms Indiana University stewart@iu.edu](https://reader036.vdocument.in/reader036/viewer/2022062516/56649e2a5503460f94b17f04/html5/thumbnails/16.jpg)
EOS and Plasma Pasta
879 miles
3 miles
Simulation Machine Analysis Machine
Aus
tin, T
exas
HPSS
![Page 17: Indiana University's Lustre WAN: Empowering Production Workflows on the TeraGrid and beyond Craig Stewart and Stephen C. Simms Indiana University stewart@iu.edu](https://reader036.vdocument.in/reader036/viewer/2022062516/56649e2a5503460f94b17f04/html5/thumbnails/17.jpg)
Computational Fluid Dynamics
Pitt
sbur
gh,
PA
410 miles
Big Red
PopleOpenMPParaview
![Page 18: Indiana University's Lustre WAN: Empowering Production Workflows on the TeraGrid and beyond Craig Stewart and Stephen C. Simms Indiana University stewart@iu.edu](https://reader036.vdocument.in/reader036/viewer/2022062516/56649e2a5503460f94b17f04/html5/thumbnails/18.jpg)
Gas Giant Planet Research
Urbana, IL
Pittsburgh, PA
410 miles
147 miles
Starkville, MS
607 miles
HPSS
Visualization
![Page 19: Indiana University's Lustre WAN: Empowering Production Workflows on the TeraGrid and beyond Craig Stewart and Stephen C. Simms Indiana University stewart@iu.edu](https://reader036.vdocument.in/reader036/viewer/2022062516/56649e2a5503460f94b17f04/html5/thumbnails/19.jpg)
Beyond the TeraGrid
• Dresden– ZIH (Technische Universitaet Dresden)
• Denmark– Risø – National Laboratory for Sustainable Energy
• Finland– Metsähovi Radio Observatory
![Page 20: Indiana University's Lustre WAN: Empowering Production Workflows on the TeraGrid and beyond Craig Stewart and Stephen C. Simms Indiana University stewart@iu.edu](https://reader036.vdocument.in/reader036/viewer/2022062516/56649e2a5503460f94b17f04/html5/thumbnails/20.jpg)
Many Thanks• Josh Walgenbach, Justin Miller, Nathan Heald, James McGookey,
Resat Payli***, Suresh Marru, Robert Henschel, Scott Michael, Tom Johnson, Chuck Horowitz, Don Berry, Scott, Teige, David Morgan, Matt Link (IU)
• Kit Westneat (DDN)• Oracle support and engineering• Michael Kluge, Guido Juckeland, Matthias Mueller (ZIH,Dresden)• Thorbjorn Axellson (CReSIS)• Greg Pike and ORNL• Doug Balog, Josephine Palencia, and PSC• Trey Breckenridge, Roger Smith, Joey Jones
(Mississippi State University)
Support for this work provided by the National Science Foundation is gratefully acknowledged and appreciated (CNS-0521433). Any opinions expressed are those of the authors and do not necessarily reflect the views of the NSF
![Page 21: Indiana University's Lustre WAN: Empowering Production Workflows on the TeraGrid and beyond Craig Stewart and Stephen C. Simms Indiana University stewart@iu.edu](https://reader036.vdocument.in/reader036/viewer/2022062516/56649e2a5503460f94b17f04/html5/thumbnails/21.jpg)
Thank you!
Questions?
http://datacapacitor.iu.edu