enabling space elasticity in storage systems · enabling space elasticity in storage systems helgi...
TRANSCRIPT
![Page 1: Enabling Space Elasticity in Storage Systems · Enabling Space Elasticity in Storage Systems Helgi Sigurbjarnarson Pétur Orri Ragnarsson Junchen Yang Ymir Vigfusson Mahesh Balakrishnan](https://reader033.vdocument.in/reader033/viewer/2022053003/5f06f9d07e708231d41aac9d/html5/thumbnails/1.jpg)
Enabling Space Elasticity in Storage Systems
Helgi SigurbjarnarsonPétur Orri Ragnarsson
Junchen YangYmir Vigfusson
Mahesh Balakrishnan
![Page 2: Enabling Space Elasticity in Storage Systems · Enabling Space Elasticity in Storage Systems Helgi Sigurbjarnarson Pétur Orri Ragnarsson Junchen Yang Ymir Vigfusson Mahesh Balakrishnan](https://reader033.vdocument.in/reader033/viewer/2022053003/5f06f9d07e708231d41aac9d/html5/thumbnails/2.jpg)
Elasticity for CPU and memory well known
Storage use typically hard to decrease
Motivation
![Page 3: Enabling Space Elasticity in Storage Systems · Enabling Space Elasticity in Storage Systems Helgi Sigurbjarnarson Pétur Orri Ragnarsson Junchen Yang Ymir Vigfusson Mahesh Balakrishnan](https://reader033.vdocument.in/reader033/viewer/2022053003/5f06f9d07e708231d41aac9d/html5/thumbnails/3.jpg)
00s:
● Single cores● 1 Gbps networks● Large HDDs
Motivation
![Page 4: Enabling Space Elasticity in Storage Systems · Enabling Space Elasticity in Storage Systems Helgi Sigurbjarnarson Pétur Orri Ragnarsson Junchen Yang Ymir Vigfusson Mahesh Balakrishnan](https://reader033.vdocument.in/reader033/viewer/2022053003/5f06f9d07e708231d41aac9d/html5/thumbnails/4.jpg)
A lot of data is volatile:
Swap files
Constructed from other data (thumbnails, indices, memoized computations)
Fetched over the network (browser and package manager caches)
Case in point: up to 55% of stored data on our dev VMs is ephemeral
Motivation
![Page 5: Enabling Space Elasticity in Storage Systems · Enabling Space Elasticity in Storage Systems Helgi Sigurbjarnarson Pétur Orri Ragnarsson Junchen Yang Ymir Vigfusson Mahesh Balakrishnan](https://reader033.vdocument.in/reader033/viewer/2022053003/5f06f9d07e708231d41aac9d/html5/thumbnails/5.jpg)
Today:
● Many cores● 40 Gbps networks● Smaller SSDs
Storage systems still promise never to lose data.
Motivation
![Page 6: Enabling Space Elasticity in Storage Systems · Enabling Space Elasticity in Storage Systems Helgi Sigurbjarnarson Pétur Orri Ragnarsson Junchen Yang Ymir Vigfusson Mahesh Balakrishnan](https://reader033.vdocument.in/reader033/viewer/2022053003/5f06f9d07e708231d41aac9d/html5/thumbnails/6.jpg)
Create a system that:
● Identifies data that isn’t really needed● Removes this data when space needs to be recovered● In case you do need some data, recover it
Our goal
![Page 7: Enabling Space Elasticity in Storage Systems · Enabling Space Elasticity in Storage Systems Helgi Sigurbjarnarson Pétur Orri Ragnarsson Junchen Yang Ymir Vigfusson Mahesh Balakrishnan](https://reader033.vdocument.in/reader033/viewer/2022053003/5f06f9d07e708231d41aac9d/html5/thumbnails/7.jpg)
Motif:A piece of code that knows how to create a file.
![Page 8: Enabling Space Elasticity in Storage Systems · Enabling Space Elasticity in Storage Systems Helgi Sigurbjarnarson Pétur Orri Ragnarsson Junchen Yang Ymir Vigfusson Mahesh Balakrishnan](https://reader033.vdocument.in/reader033/viewer/2022053003/5f06f9d07e708231d41aac9d/html5/thumbnails/8.jpg)
Motifs
More specifically: An expand function and metadata
Key properties:
● A motif is stateful● Motifs can be recursive● A single file can have multiple motifs● Can define circular dependencies● Can be invalidated● Support writes
○ Optional contract function
![Page 9: Enabling Space Elasticity in Storage Systems · Enabling Space Elasticity in Storage Systems Helgi Sigurbjarnarson Pétur Orri Ragnarsson Junchen Yang Ymir Vigfusson Mahesh Balakrishnan](https://reader033.vdocument.in/reader033/viewer/2022053003/5f06f9d07e708231d41aac9d/html5/thumbnails/9.jpg)
Carillon:A system that utilizes motifs to provide space elasticity
![Page 10: Enabling Space Elasticity in Storage Systems · Enabling Space Elasticity in Storage Systems Helgi Sigurbjarnarson Pétur Orri Ragnarsson Junchen Yang Ymir Vigfusson Mahesh Balakrishnan](https://reader033.vdocument.in/reader033/viewer/2022053003/5f06f9d07e708231d41aac9d/html5/thumbnails/10.jpg)
Two main components: Runtime and storage shim
Runtime is independent of the underlying storage layerShim is tailored to it
Operate in tandem to provide elasticity
Each different storage layer requires its own runtime/shim pair
Design goal: Add elasticity to existing storage with minimal effort
Carillon
![Page 11: Enabling Space Elasticity in Storage Systems · Enabling Space Elasticity in Storage Systems Helgi Sigurbjarnarson Pétur Orri Ragnarsson Junchen Yang Ymir Vigfusson Mahesh Balakrishnan](https://reader033.vdocument.in/reader033/viewer/2022053003/5f06f9d07e708231d41aac9d/html5/thumbnails/11.jpg)
The Carillon runtime is responsible for several things
● Managing motif metadata● Accept storage policies (eg. there is now less space available)● Track statistics● Execute motifs based on statistics and available space
Carillon
![Page 12: Enabling Space Elasticity in Storage Systems · Enabling Space Elasticity in Storage Systems Helgi Sigurbjarnarson Pétur Orri Ragnarsson Junchen Yang Ymir Vigfusson Mahesh Balakrishnan](https://reader033.vdocument.in/reader033/viewer/2022053003/5f06f9d07e708231d41aac9d/html5/thumbnails/12.jpg)
A Carillon shim, by contrast, does mostly one thing
● Intercept calls to the underlying storage layer and forward to runtime
Carillon
![Page 13: Enabling Space Elasticity in Storage Systems · Enabling Space Elasticity in Storage Systems Helgi Sigurbjarnarson Pétur Orri Ragnarsson Junchen Yang Ymir Vigfusson Mahesh Balakrishnan](https://reader033.vdocument.in/reader033/viewer/2022053003/5f06f9d07e708231d41aac9d/html5/thumbnails/13.jpg)
Overview
![Page 14: Enabling Space Elasticity in Storage Systems · Enabling Space Elasticity in Storage Systems Helgi Sigurbjarnarson Pétur Orri Ragnarsson Junchen Yang Ymir Vigfusson Mahesh Balakrishnan](https://reader033.vdocument.in/reader033/viewer/2022053003/5f06f9d07e708231d41aac9d/html5/thumbnails/14.jpg)
Ideal goal: Never wait for expansion
Can’t know the future
Actual goal: Minimize wait time
Model as a 0-1 knapsack problem; slow to solve
Cache algorithms!
What to delete?
![Page 15: Enabling Space Elasticity in Storage Systems · Enabling Space Elasticity in Storage Systems Helgi Sigurbjarnarson Pétur Orri Ragnarsson Junchen Yang Ymir Vigfusson Mahesh Balakrishnan](https://reader033.vdocument.in/reader033/viewer/2022053003/5f06f9d07e708231d41aac9d/html5/thumbnails/15.jpg)
Cache algorithms
![Page 16: Enabling Space Elasticity in Storage Systems · Enabling Space Elasticity in Storage Systems Helgi Sigurbjarnarson Pétur Orri Ragnarsson Junchen Yang Ymir Vigfusson Mahesh Balakrishnan](https://reader033.vdocument.in/reader033/viewer/2022053003/5f06f9d07e708231d41aac9d/html5/thumbnails/16.jpg)
Most operations forwarded without extra work.
Except: stat, open, unlink, rename, truncate, utime
CarillonFS
![Page 17: Enabling Space Elasticity in Storage Systems · Enabling Space Elasticity in Storage Systems Helgi Sigurbjarnarson Pétur Orri Ragnarsson Junchen Yang Ymir Vigfusson Mahesh Balakrishnan](https://reader033.vdocument.in/reader033/viewer/2022053003/5f06f9d07e708231d41aac9d/html5/thumbnails/17.jpg)
Key-value store
Graph database
Route planner
Dijkstra’s algorithm has a lot of internal state that’s usually discarded
Motif-ize some of it to speed up future runs
CarillonKV
![Page 18: Enabling Space Elasticity in Storage Systems · Enabling Space Elasticity in Storage Systems Helgi Sigurbjarnarson Pétur Orri Ragnarsson Junchen Yang Ymir Vigfusson Mahesh Balakrishnan](https://reader033.vdocument.in/reader033/viewer/2022053003/5f06f9d07e708231d41aac9d/html5/thumbnails/18.jpg)
Filebench performance
Evaluation
![Page 19: Enabling Space Elasticity in Storage Systems · Enabling Space Elasticity in Storage Systems Helgi Sigurbjarnarson Pétur Orri Ragnarsson Junchen Yang Ymir Vigfusson Mahesh Balakrishnan](https://reader033.vdocument.in/reader033/viewer/2022053003/5f06f9d07e708231d41aac9d/html5/thumbnails/19.jpg)
CarillonFS elasticity
![Page 20: Enabling Space Elasticity in Storage Systems · Enabling Space Elasticity in Storage Systems Helgi Sigurbjarnarson Pétur Orri Ragnarsson Junchen Yang Ymir Vigfusson Mahesh Balakrishnan](https://reader033.vdocument.in/reader033/viewer/2022053003/5f06f9d07e708231d41aac9d/html5/thumbnails/20.jpg)
CarillonKV elasticity
![Page 21: Enabling Space Elasticity in Storage Systems · Enabling Space Elasticity in Storage Systems Helgi Sigurbjarnarson Pétur Orri Ragnarsson Junchen Yang Ymir Vigfusson Mahesh Balakrishnan](https://reader033.vdocument.in/reader033/viewer/2022053003/5f06f9d07e708231d41aac9d/html5/thumbnails/21.jpg)
Questions?
![Page 22: Enabling Space Elasticity in Storage Systems · Enabling Space Elasticity in Storage Systems Helgi Sigurbjarnarson Pétur Orri Ragnarsson Junchen Yang Ymir Vigfusson Mahesh Balakrishnan](https://reader033.vdocument.in/reader033/viewer/2022053003/5f06f9d07e708231d41aac9d/html5/thumbnails/22.jpg)
Bonus slides!
![Page 23: Enabling Space Elasticity in Storage Systems · Enabling Space Elasticity in Storage Systems Helgi Sigurbjarnarson Pétur Orri Ragnarsson Junchen Yang Ymir Vigfusson Mahesh Balakrishnan](https://reader033.vdocument.in/reader033/viewer/2022053003/5f06f9d07e708231d41aac9d/html5/thumbnails/23.jpg)
Highly skewed trace
A vast majority of file accesses happens to a very small subset of files
![Page 24: Enabling Space Elasticity in Storage Systems · Enabling Space Elasticity in Storage Systems Helgi Sigurbjarnarson Pétur Orri Ragnarsson Junchen Yang Ymir Vigfusson Mahesh Balakrishnan](https://reader033.vdocument.in/reader033/viewer/2022053003/5f06f9d07e708231d41aac9d/html5/thumbnails/24.jpg)
Network storage motif
Contracts a file by copying it to a remote store
Expands by copying back
Very similar to the one used in our evaluations
Example motif