![Page 1: GlusterFS Challenges and Futures - SNIA · 2020-02-01 · GlusterFS Challenges and Futures Jeff Darcy Storage Developer Conference September 17, 2012. What Is GlusterFS? ... use flags](https://reader031.vdocument.in/reader031/viewer/2022011815/5e5e524bcd753763a106c32a/html5/thumbnails/1.jpg)
GlusterFSChallenges and Futures
Jeff DarcyStorage Developer Conference
September 17, 2012
![Page 2: GlusterFS Challenges and Futures - SNIA · 2020-02-01 · GlusterFS Challenges and Futures Jeff Darcy Storage Developer Conference September 17, 2012. What Is GlusterFS? ... use flags](https://reader031.vdocument.in/reader031/viewer/2022011815/5e5e524bcd753763a106c32a/html5/thumbnails/2.jpg)
What Is GlusterFS?
● Just your basic distributed file system● sharding, replication, striping
● Decentralized● no central metadata server● core functionality on clients
● Modular● “translators”
![Page 3: GlusterFS Challenges and Futures - SNIA · 2020-02-01 · GlusterFS Challenges and Futures Jeff Darcy Storage Developer Conference September 17, 2012. What Is GlusterFS? ... use flags](https://reader031.vdocument.in/reader031/viewer/2022011815/5e5e524bcd753763a106c32a/html5/thumbnails/3.jpg)
Translator Stacking
FUSE
DHT
AFR-1
Client-3Client-1
AFR-0
Client-2Client-0
● One to one, one to many, one to zero (?)● Rearrange, move from client to server, ...
![Page 4: GlusterFS Challenges and Futures - SNIA · 2020-02-01 · GlusterFS Challenges and Futures Jeff Darcy Storage Developer Conference September 17, 2012. What Is GlusterFS? ... use flags](https://reader031.vdocument.in/reader031/viewer/2022011815/5e5e524bcd753763a106c32a/html5/thumbnails/4.jpg)
Distribution (now)
Server A
Server BServer C
File X
File Y
![Page 5: GlusterFS Challenges and Futures - SNIA · 2020-02-01 · GlusterFS Challenges and Futures Jeff Darcy Storage Developer Conference September 17, 2012. What Is GlusterFS? ... use flags](https://reader031.vdocument.in/reader031/viewer/2022011815/5e5e524bcd753763a106c32a/html5/thumbnails/5.jpg)
Replication (now)
● Based on changelog (“dirty flags”)● set flags, do operation, clear flags● use flags to determine repair (“self-heal”) after
failure
● Latency sensitive● 3+ network round trips per user request● implementation heavily optimized
![Page 6: GlusterFS Challenges and Futures - SNIA · 2020-02-01 · GlusterFS Challenges and Futures Jeff Darcy Storage Developer Conference September 17, 2012. What Is GlusterFS? ... use flags](https://reader031.vdocument.in/reader031/viewer/2022011815/5e5e524bcd753763a106c32a/html5/thumbnails/6.jpg)
Challenge: Multitenancy
● Focus of CloudFSHekaFS● to be merged with GlusterFS soon-ish
● Isolate name and ID spaces● Encrypt in flight and at rest● Auth*● Quota (next slide)● Performance isolation
● cgroups
![Page 7: GlusterFS Challenges and Futures - SNIA · 2020-02-01 · GlusterFS Challenges and Futures Jeff Darcy Storage Developer Conference September 17, 2012. What Is GlusterFS? ... use flags](https://reader031.vdocument.in/reader031/viewer/2022011815/5e5e524bcd753763a106c32a/html5/thumbnails/7.jpg)
Challenge: Distributed Quota
● Can't trust clients to enforce quota● Can't just divide equally among servers
● unequal usage (e.g. due to explicit placement)● EDQUOT on one while still space on another
● “Quota rebalancing daemon”● monitor/adjust continuously● interesting problem at high scale
![Page 8: GlusterFS Challenges and Futures - SNIA · 2020-02-01 · GlusterFS Challenges and Futures Jeff Darcy Storage Developer Conference September 17, 2012. What Is GlusterFS? ... use flags](https://reader031.vdocument.in/reader031/viewer/2022011815/5e5e524bcd753763a106c32a/html5/thumbnails/8.jpg)
Challenge: Better Rebalancing
● Optimal placement vs. minimal data movement● Different kinds of weighting
Node A
Node B
Node C
Node D
Node E
![Page 9: GlusterFS Challenges and Futures - SNIA · 2020-02-01 · GlusterFS Challenges and Futures Jeff Darcy Storage Developer Conference September 17, 2012. What Is GlusterFS? ... use flags](https://reader031.vdocument.in/reader031/viewer/2022011815/5e5e524bcd753763a106c32a/html5/thumbnails/9.jpg)
Challenge: Replication Latency
● Reaching limits of current approach● Have to go async?
● but still ordered● exploit compute/data locality (e.g. Hadoop)● journaling, conflict resolution
![Page 10: GlusterFS Challenges and Futures - SNIA · 2020-02-01 · GlusterFS Challenges and Futures Jeff Darcy Storage Developer Conference September 17, 2012. What Is GlusterFS? ... use flags](https://reader031.vdocument.in/reader031/viewer/2022011815/5e5e524bcd753763a106c32a/html5/thumbnails/10.jpg)
Challenge: Directory Traversal
● Piggyback attrs (and xattrs) on readdir● Even better: cursor approach
● read everything in opendir● zero network activity for readdir● less current, but more consistent
![Page 11: GlusterFS Challenges and Futures - SNIA · 2020-02-01 · GlusterFS Challenges and Futures Jeff Darcy Storage Developer Conference September 17, 2012. What Is GlusterFS? ... use flags](https://reader031.vdocument.in/reader031/viewer/2022011815/5e5e524bcd753763a106c32a/html5/thumbnails/11.jpg)
Challenge: Many Small Files
● Prefetch whole directories● if marked, below size threshold, ...
● Exploit async journal● only works if compute/data colocated
● Weaken consistency?● allow create/write/close to be buffered● directory-level fsync (magic xattr)
![Page 12: GlusterFS Challenges and Futures - SNIA · 2020-02-01 · GlusterFS Challenges and Futures Jeff Darcy Storage Developer Conference September 17, 2012. What Is GlusterFS? ... use flags](https://reader031.vdocument.in/reader031/viewer/2022011815/5e5e524bcd753763a106c32a/html5/thumbnails/12.jpg)
Conclusions
● Most of these challenges are not unique to GlusterFS
● Modularity and incremental progress are preferable to monolithic “solve all problems at once”
● GlusterFS provides a good environment in which to experiment with solutions