la-ur-12-23586 glusterfs · terrell perrotti (south carolina state university) mentors: david...
TRANSCRIPT
![Page 1: LA-UR-12-23586 GlusterFS · Terrell Perrotti (South Carolina State University) Mentors: David Kennel (DCS-1) Greg Lee (DCS-1) Instructors: Dane Gardner (NMC) Andree Jacobson (NMC)](https://reader033.vdocument.in/reader033/viewer/2022042310/5ed7a91a48b98015c202158b/html5/thumbnails/1.jpg)
LA-UR-12-23586
Team Members: Matthew Broomfield (New Mexico Tech)
Eric Boyer (Michigan Tech) Terrell Perrotti (South Carolina State University)
Mentors: David Kennel (DCS-1)
Greg Lee (DCS-1)
Instructors: Dane Gardner (NMC)
Andree Jacobson (NMC)
GlusterFS One Storage Server to Rule Them All
LA-UR-12-23586
![Page 2: LA-UR-12-23586 GlusterFS · Terrell Perrotti (South Carolina State University) Mentors: David Kennel (DCS-1) Greg Lee (DCS-1) Instructors: Dane Gardner (NMC) Andree Jacobson (NMC)](https://reader033.vdocument.in/reader033/viewer/2022042310/5ed7a91a48b98015c202158b/html5/thumbnails/2.jpg)
LA-UR-12-23586
l Introduction to GlusterFS l Services l Administration l Performance l Conclusions l Future Work
Outline
![Page 3: LA-UR-12-23586 GlusterFS · Terrell Perrotti (South Carolina State University) Mentors: David Kennel (DCS-1) Greg Lee (DCS-1) Instructors: Dane Gardner (NMC) Andree Jacobson (NMC)](https://reader033.vdocument.in/reader033/viewer/2022042310/5ed7a91a48b98015c202158b/html5/thumbnails/3.jpg)
LA-UR-12-23586
• GlusterFS is a Linux based distributed file system, designed to be highly scalable and serve many clients
What is GlusterFS?
![Page 4: LA-UR-12-23586 GlusterFS · Terrell Perrotti (South Carolina State University) Mentors: David Kennel (DCS-1) Greg Lee (DCS-1) Instructors: Dane Gardner (NMC) Andree Jacobson (NMC)](https://reader033.vdocument.in/reader033/viewer/2022042310/5ed7a91a48b98015c202158b/html5/thumbnails/4.jpg)
LA-UR-12-23586
• No centralized metadata server • Scalability • Open Source • Dynamic and live service modifications • Can be used over Infiniband or Ethernet • Can be tuned for speed and/or resilience • Flexible administration
Why Use GlusterFS?
![Page 5: LA-UR-12-23586 GlusterFS · Terrell Perrotti (South Carolina State University) Mentors: David Kennel (DCS-1) Greg Lee (DCS-1) Instructors: Dane Gardner (NMC) Andree Jacobson (NMC)](https://reader033.vdocument.in/reader033/viewer/2022042310/5ed7a91a48b98015c202158b/html5/thumbnails/5.jpg)
LA-UR-12-23586
l Enterprise environments • Virtualization
l High Performance Computing (HPC) l Works with Mac, Linux, and Windows clients
Where Is It Useful?
![Page 6: LA-UR-12-23586 GlusterFS · Terrell Perrotti (South Carolina State University) Mentors: David Kennel (DCS-1) Greg Lee (DCS-1) Instructors: Dane Gardner (NMC) Andree Jacobson (NMC)](https://reader033.vdocument.in/reader033/viewer/2022042310/5ed7a91a48b98015c202158b/html5/thumbnails/6.jpg)
LA-UR-12-23586
• Individual nodes export bricks ( directories ) to GlusterFS
• GlusterFS combines bricks into virtual volumes
How Does It Work?
![Page 7: LA-UR-12-23586 GlusterFS · Terrell Perrotti (South Carolina State University) Mentors: David Kennel (DCS-1) Greg Lee (DCS-1) Instructors: Dane Gardner (NMC) Andree Jacobson (NMC)](https://reader033.vdocument.in/reader033/viewer/2022042310/5ed7a91a48b98015c202158b/html5/thumbnails/7.jpg)
LA-UR-12-23586
• Different GlusterFs volume types:
GlusterFS Volume Types
![Page 8: LA-UR-12-23586 GlusterFS · Terrell Perrotti (South Carolina State University) Mentors: David Kennel (DCS-1) Greg Lee (DCS-1) Instructors: Dane Gardner (NMC) Andree Jacobson (NMC)](https://reader033.vdocument.in/reader033/viewer/2022042310/5ed7a91a48b98015c202158b/html5/thumbnails/8.jpg)
LA-UR-12-23586
• GlusterFS has a built in quota service and uses POSIX ACLs for user control
• POSIX ACLs • Can set individual users or group permissions
• Quotas • Given via directory
• Both apply to Mac and windows clients (NFS/SAMBA)
Data Control
![Page 9: LA-UR-12-23586 GlusterFS · Terrell Perrotti (South Carolina State University) Mentors: David Kennel (DCS-1) Greg Lee (DCS-1) Instructors: Dane Gardner (NMC) Andree Jacobson (NMC)](https://reader033.vdocument.in/reader033/viewer/2022042310/5ed7a91a48b98015c202158b/html5/thumbnails/9.jpg)
LA-UR-12-23586
l GlusterFS volumes can be exported via NFSv3 • POSIX ACLs are lost when exporting directly via NFS • Enable POSIX ACLs by mounting via GlusterFS and exporting
via NFS
• SAMBA allows Windows users to modify NTFS permissions on files
Exporting
![Page 10: LA-UR-12-23586 GlusterFS · Terrell Perrotti (South Carolina State University) Mentors: David Kennel (DCS-1) Greg Lee (DCS-1) Instructors: Dane Gardner (NMC) Andree Jacobson (NMC)](https://reader033.vdocument.in/reader033/viewer/2022042310/5ed7a91a48b98015c202158b/html5/thumbnails/10.jpg)
LA-UR-12-23586
l Snapshots • We used the rsnapshot utility to enable snapshots • Use cron jobs to specify snapshot intervals and locations
l Auditing • The auditd utility can be used in conjunction with GlusterFS • Shows detailed file interactions
• GlusterFS built-in logging support • Performance • Diagnostics • Events (Warnings, Errors, General Information)
Data Support
![Page 11: LA-UR-12-23586 GlusterFS · Terrell Perrotti (South Carolina State University) Mentors: David Kennel (DCS-1) Greg Lee (DCS-1) Instructors: Dane Gardner (NMC) Andree Jacobson (NMC)](https://reader033.vdocument.in/reader033/viewer/2022042310/5ed7a91a48b98015c202158b/html5/thumbnails/11.jpg)
LA-UR-12-23586
l GlusterFS has an intuitive CLI • Allows for quick volume tuning, shrinking, and expanding
while the system is online and available • Easily integrated into current infrastructure
l Pitfalls • Latency induced when mounting and exporting • GlusterFS mounting/unmounting occasionally hung • Metadata is distributed, thus harder to remove
Administration
![Page 12: LA-UR-12-23586 GlusterFS · Terrell Perrotti (South Carolina State University) Mentors: David Kennel (DCS-1) Greg Lee (DCS-1) Instructors: Dane Gardner (NMC) Andree Jacobson (NMC)](https://reader033.vdocument.in/reader033/viewer/2022042310/5ed7a91a48b98015c202158b/html5/thumbnails/12.jpg)
LA-UR-12-23586
• Open Source • Can use commodity hardware • Can use 1 or 10Gbps Ethernet
Total Cost of Operation
![Page 13: LA-UR-12-23586 GlusterFS · Terrell Perrotti (South Carolina State University) Mentors: David Kennel (DCS-1) Greg Lee (DCS-1) Instructors: Dane Gardner (NMC) Andree Jacobson (NMC)](https://reader033.vdocument.in/reader033/viewer/2022042310/5ed7a91a48b98015c202158b/html5/thumbnails/13.jpg)
LA-UR-12-23586
l Base-N Distributed across N nodes
l Striped-X-N Striped across X nodes on an N node volume
l Replicated-X-N Replicated across X nodes on
an N node volume
l Hybrid-X-Y-N Striped across X nodes, Distributed across Y nodes on an N node volume
Performance Testing Key
![Page 14: LA-UR-12-23586 GlusterFS · Terrell Perrotti (South Carolina State University) Mentors: David Kennel (DCS-1) Greg Lee (DCS-1) Instructors: Dane Gardner (NMC) Andree Jacobson (NMC)](https://reader033.vdocument.in/reader033/viewer/2022042310/5ed7a91a48b98015c202158b/html5/thumbnails/14.jpg)
LA-UR-12-23586
Write Performance
![Page 15: LA-UR-12-23586 GlusterFS · Terrell Perrotti (South Carolina State University) Mentors: David Kennel (DCS-1) Greg Lee (DCS-1) Instructors: Dane Gardner (NMC) Andree Jacobson (NMC)](https://reader033.vdocument.in/reader033/viewer/2022042310/5ed7a91a48b98015c202158b/html5/thumbnails/15.jpg)
LA-UR-12-23586
Write Performance
![Page 16: LA-UR-12-23586 GlusterFS · Terrell Perrotti (South Carolina State University) Mentors: David Kennel (DCS-1) Greg Lee (DCS-1) Instructors: Dane Gardner (NMC) Andree Jacobson (NMC)](https://reader033.vdocument.in/reader033/viewer/2022042310/5ed7a91a48b98015c202158b/html5/thumbnails/16.jpg)
LA-UR-12-23586
Write Performance
![Page 17: LA-UR-12-23586 GlusterFS · Terrell Perrotti (South Carolina State University) Mentors: David Kennel (DCS-1) Greg Lee (DCS-1) Instructors: Dane Gardner (NMC) Andree Jacobson (NMC)](https://reader033.vdocument.in/reader033/viewer/2022042310/5ed7a91a48b98015c202158b/html5/thumbnails/17.jpg)
LA-UR-12-23586
Read Performance
![Page 18: LA-UR-12-23586 GlusterFS · Terrell Perrotti (South Carolina State University) Mentors: David Kennel (DCS-1) Greg Lee (DCS-1) Instructors: Dane Gardner (NMC) Andree Jacobson (NMC)](https://reader033.vdocument.in/reader033/viewer/2022042310/5ed7a91a48b98015c202158b/html5/thumbnails/18.jpg)
LA-UR-12-23586
Read Performance
![Page 19: LA-UR-12-23586 GlusterFS · Terrell Perrotti (South Carolina State University) Mentors: David Kennel (DCS-1) Greg Lee (DCS-1) Instructors: Dane Gardner (NMC) Andree Jacobson (NMC)](https://reader033.vdocument.in/reader033/viewer/2022042310/5ed7a91a48b98015c202158b/html5/thumbnails/19.jpg)
LA-UR-12-23586
Read Performance
![Page 20: LA-UR-12-23586 GlusterFS · Terrell Perrotti (South Carolina State University) Mentors: David Kennel (DCS-1) Greg Lee (DCS-1) Instructors: Dane Gardner (NMC) Andree Jacobson (NMC)](https://reader033.vdocument.in/reader033/viewer/2022042310/5ed7a91a48b98015c202158b/html5/thumbnails/20.jpg)
LA-UR-12-23586
Ls Performance
![Page 21: LA-UR-12-23586 GlusterFS · Terrell Perrotti (South Carolina State University) Mentors: David Kennel (DCS-1) Greg Lee (DCS-1) Instructors: Dane Gardner (NMC) Andree Jacobson (NMC)](https://reader033.vdocument.in/reader033/viewer/2022042310/5ed7a91a48b98015c202158b/html5/thumbnails/21.jpg)
LA-UR-12-23586
• Replicated volumes can self-heal
Fault Tolerance
FILE A
FILE B
FILE A FILE B
FILE A FILE B
BRICK 1
BRICK 2
BRICK 1
BRICK 2
![Page 22: LA-UR-12-23586 GlusterFS · Terrell Perrotti (South Carolina State University) Mentors: David Kennel (DCS-1) Greg Lee (DCS-1) Instructors: Dane Gardner (NMC) Andree Jacobson (NMC)](https://reader033.vdocument.in/reader033/viewer/2022042310/5ed7a91a48b98015c202158b/html5/thumbnails/22.jpg)
LA-UR-12-23586
• GlusterFS proved to have widespread capabilities as a virtual file system
• Scalability is very dependent upon the underlying hardware
• Lack of built-in encryption and security paradigm • Best suited in a general purpose computing
environment
Conclusions
![Page 23: LA-UR-12-23586 GlusterFS · Terrell Perrotti (South Carolina State University) Mentors: David Kennel (DCS-1) Greg Lee (DCS-1) Instructors: Dane Gardner (NMC) Andree Jacobson (NMC)](https://reader033.vdocument.in/reader033/viewer/2022042310/5ed7a91a48b98015c202158b/html5/thumbnails/23.jpg)
LA-UR-12-23586
• GlusterFS over Infiniband • Geo-replication • Unified File and Object Storage • Apache Hadoop • Scalability for 1000’s of nodes • Using other filesystems on top of GlusterFS • Testing different RAID types
Future Research
![Page 24: LA-UR-12-23586 GlusterFS · Terrell Perrotti (South Carolina State University) Mentors: David Kennel (DCS-1) Greg Lee (DCS-1) Instructors: Dane Gardner (NMC) Andree Jacobson (NMC)](https://reader033.vdocument.in/reader033/viewer/2022042310/5ed7a91a48b98015c202158b/html5/thumbnails/24.jpg)
LA-UR-12-23586
• We would like to thank: • Los Alamos National Laboratory • New Mexico Consortium/PRObE • National Science Foundation • National Nuclear Security Administration • Gary Grider and the HPC division • Carol Hogsett and Josephine Olivas • Our mentors, David Kennel and Greg Lee • Our instructors, Dane Gardner and Andree Jacobson
Acknowledgments
![Page 25: LA-UR-12-23586 GlusterFS · Terrell Perrotti (South Carolina State University) Mentors: David Kennel (DCS-1) Greg Lee (DCS-1) Instructors: Dane Gardner (NMC) Andree Jacobson (NMC)](https://reader033.vdocument.in/reader033/viewer/2022042310/5ed7a91a48b98015c202158b/html5/thumbnails/25.jpg)
LA-UR-12-23586
Questions??