1 uflip: understanding flash io patterns luc bouganim, inria rocquencourt, france philippe bonnet,...
TRANSCRIPT
![Page 1: 1 uFLIP: Understanding Flash IO Patterns Luc Bouganim, INRIA Rocquencourt, France Philippe Bonnet, DIKU Copenhagen, Denmark Björn Þór Jónsson, RU Reykjavík,](https://reader035.vdocument.in/reader035/viewer/2022062417/5518bffa550346b31f8b5524/html5/thumbnails/1.jpg)
1
uFLIP: Understanding Flash IO Patterns
Luc Bouganim, INRIARocquencourt, France
Philippe Bonnet, DIKUCopenhagen, Denmark
Björn Þór Jónsson, RUReykjavík, Iceland
![Page 2: 1 uFLIP: Understanding Flash IO Patterns Luc Bouganim, INRIA Rocquencourt, France Philippe Bonnet, DIKU Copenhagen, Denmark Björn Þór Jónsson, RU Reykjavík,](https://reader035.vdocument.in/reader035/viewer/2022062417/5518bffa550346b31f8b5524/html5/thumbnails/2.jpg)
2
Flash cells
Why should we consider flash devices?
• NAND flash chip typical timings (SLC chip): Read a 2KB page:
RAM buffer
• A single flash chip could potentially deliver: Read throughput of 23 MB/s, write throughput of 6 MB/s
• And… Random access is potentially as fast as sequential access! An SSD contains many (e.g., 8, 16) flash chips. Potential parallelism!
Flash devices have a high potential
Write a 2KB page:
Transfer (60 µs), write page (200µs)
Erase before rewrite! (2ms for 128 KB)
Read page (25 µs), transfer (60 µs)
![Page 3: 1 uFLIP: Understanding Flash IO Patterns Luc Bouganim, INRIA Rocquencourt, France Philippe Bonnet, DIKU Copenhagen, Denmark Björn Þór Jónsson, RU Reykjavík,](https://reader035.vdocument.in/reader035/viewer/2022062417/5518bffa550346b31f8b5524/html5/thumbnails/3.jpg)
3
but …
• Flash chips have many constraints I/O granularity: a flash page (2 KB) No update: Erase before write; erase granularity: a block (64 pages) Writes must be sequential within a flash block Limited lifetime: max 105 – 106 erase Usually, a software layer (Flash Translation Layer) handle these constraints
• Flash devices are not flash chips Do not behave as the flash chip they contain No access to the flash chip API but only through the device API Complex architecture and software, proprietary and not documented Flash devices are black boxes !
• How can we model flash devices? First step: understand their performance
Need for a benchmark.Need for a benchmark.
![Page 4: 1 uFLIP: Understanding Flash IO Patterns Luc Bouganim, INRIA Rocquencourt, France Philippe Bonnet, DIKU Copenhagen, Denmark Björn Þór Jónsson, RU Reykjavík,](https://reader035.vdocument.in/reader035/viewer/2022062417/5518bffa550346b31f8b5524/html5/thumbnails/4.jpg)
4
Read
Write
Erase
RAM R/W
Free Block
Unfilled block
Filled block
When possible redirect writes to previously erased locations Update blocks
The Flash Translation Layer
• Emulates a normal block device, handling flash constraints
RA
MF
LA
SH
FTL Blocks Update BlocksData Blocks
FT
L M
AP
FT
L M
AP
Oth
er F
TL
O
ther
FT
L
stru
ctu
res
stru
ctu
res
Read(@100, …):
Read(@101, …):
Write(@900, …):
Write(@200, …):
D
R
R
R FF D
R FF D D D D D D D D D D D D D D D D
Read(@100, …)
Maps logical address (LBA) to physical locations Mapping information Distributes erase across the device (wear levelling) Other data structures
FF D D
Read(@101, …)Write(@900, …)Write(@200, …)
Flash DeviceFlash DeviceFlash Translation Layer API: Read(LBA, &data) / Write (LBA, data)
……
…………
……
…………
• IO Cost thus depends on The mode of the IO (i.e., read, write) Recent IOs (caching in the device RAM) The device state (i.e., flash state and data structures)
• Device state depends on Entire history of previous IO requests
![Page 5: 1 uFLIP: Understanding Flash IO Patterns Luc Bouganim, INRIA Rocquencourt, France Philippe Bonnet, DIKU Copenhagen, Denmark Björn Þór Jónsson, RU Reykjavík,](https://reader035.vdocument.in/reader035/viewer/2022062417/5518bffa550346b31f8b5524/html5/thumbnails/5.jpg)
5
• Why do we need to benchmark flash devices? DB technology relies on the HD characteristics …
… Flash devices will replace or complement HDs …… and we have a poor knowledge of flash devices.
Flash devices are black boxes (complex and undocumented FTLs)
Large range from USB flash drives to high performance flash board.
• Benchmarking flash devices is difficult: Need to design a sound benchmarking methodology
– IO cost is highly variable and depends on the whole device history! Need to define a broad benchmark
– No safe assumption can be made on the device behavior (black box)– Moreover, we do not want to restrict the benchmark usage!
Benchmarking flash devices: Goal and difficulties
![Page 6: 1 uFLIP: Understanding Flash IO Patterns Luc Bouganim, INRIA Rocquencourt, France Philippe Bonnet, DIKU Copenhagen, Denmark Björn Þór Jónsson, RU Reykjavík,](https://reader035.vdocument.in/reader035/viewer/2022062417/5518bffa550346b31f8b5524/html5/thumbnails/6.jpg)
6
Methodology (1): Device state
• Measuring Samsung SSD RW performance Out-of-the-box …
0.1
1
10
100
100 200 300 400 500 IO number
Re
spo
ns
e ti
me
(m
s)
rtAvg(rt)
0.1
1
10
100
100 200 300 400 500 IO number
Re
spo
ns
e ti
me
(m
s)
rtAvg(rt)
Random Writes – Samsung SSDOut of the box
![Page 7: 1 uFLIP: Understanding Flash IO Patterns Luc Bouganim, INRIA Rocquencourt, France Philippe Bonnet, DIKU Copenhagen, Denmark Björn Þór Jónsson, RU Reykjavík,](https://reader035.vdocument.in/reader035/viewer/2022062417/5518bffa550346b31f8b5524/html5/thumbnails/7.jpg)
7
Methodology (1): Device state
• Measuring Samsung SSD RW performance Out-of-the-box … and after filling the device!!! (similar behavior on Intel SSD)
0.1
1
10
100
100 200 300 400 500 IO number
Re
spo
ns
e ti
me
(m
s)
rtAvg(rt)
0.1
1
10
100
100 200 300 400 500 IO number
Re
spo
ns
e ti
me
(m
s)
rtAvg(rt)
Random Writes – Samsung SSDOut of the box
Random Writes – Samsung SSDAfter filling the device
0.1
1
10
100
100 200 300 400 500 IO number
Re
spo
ns
e ti
me
(m
s)
rtAvg(rt)
Avg(rt) o-o-b0.1
1
10
100
100 200 300 400 500 IO number
Re
spo
ns
e ti
me
(m
s)
rtAvg(rt)
Avg(rt) o-o-b
![Page 8: 1 uFLIP: Understanding Flash IO Patterns Luc Bouganim, INRIA Rocquencourt, France Philippe Bonnet, DIKU Copenhagen, Denmark Björn Þór Jónsson, RU Reykjavík,](https://reader035.vdocument.in/reader035/viewer/2022062417/5518bffa550346b31f8b5524/html5/thumbnails/8.jpg)
8
Methodology (2): Startup and running phases
• When do we reach a steady state? How long to run each test?
0,1
1
10
100
0 100 200 300IO number
Re
spo
ns
e ti
me
(m
s)
rtAvg(rt) incl.Avg(rt) excl.
0,1
1
10
100
0 100 200 300IO number
Re
spo
ns
e ti
me
(m
s)
rtAvg(rt) incl.Avg(rt) excl.
1
10
100
1000
0 100 200 300
IO number
Re
spo
ns
e ti
me
(m
s)
rtAvg(rt)
1
10
100
1000
0 100 200 300
IO number
Re
spo
ns
e ti
me
(m
s)
rtAvg(rt)
1
10
100
1000
0 100 200 300
IO number
Re
spo
ns
e ti
me
(m
s)
rtAvg(rt)
Startup and running phases for the Mtron SSD (RW)
Running phase for the Kingston DTI flash Drive (SW)
![Page 9: 1 uFLIP: Understanding Flash IO Patterns Luc Bouganim, INRIA Rocquencourt, France Philippe Bonnet, DIKU Copenhagen, Denmark Björn Þór Jónsson, RU Reykjavík,](https://reader035.vdocument.in/reader035/viewer/2022062417/5518bffa550346b31f8b5524/html5/thumbnails/9.jpg)
9
Methodology (3): Interferences between consecutive runs
0.1
1
10
100
0 5000 10000
IO number
Res
pon
se ti
me
(ms)
rt
13000
Pause length
Seq. Reads Random Writes Seq. Reads
0.1
1
10
100
0 5000 10000
IO number
Res
pon
se ti
me
(ms)
rt
13000
Pause length
Seq. Reads Random Writes Seq. Reads
Setup experiment for the Mtron SSD
![Page 10: 1 uFLIP: Understanding Flash IO Patterns Luc Bouganim, INRIA Rocquencourt, France Philippe Bonnet, DIKU Copenhagen, Denmark Björn Þór Jónsson, RU Reykjavík,](https://reader035.vdocument.in/reader035/viewer/2022062417/5518bffa550346b31f8b5524/html5/thumbnails/10.jpg)
10
Proposed methodology:
• Device state : Enforce a well-defined device state performing random write IOs of random size on the whole device The alternative, sequential IOs, is less stable, thus more difficult to enforce
• Startup and running phase: Run experiments to define IOIgnore: Number of IOs ignored when computing statistics IOCount: Number of measures to allow for convergence of those statistics.
• Interferences: Introduce a pause between runs Run the following experiment: SR, then RW, then SR (with a large IOCount) Measure the interferences. In previous experiment, 3000 IOs Overestimate the length of the Pause
![Page 11: 1 uFLIP: Understanding Flash IO Patterns Luc Bouganim, INRIA Rocquencourt, France Philippe Bonnet, DIKU Copenhagen, Denmark Björn Þór Jónsson, RU Reykjavík,](https://reader035.vdocument.in/reader035/viewer/2022062417/5518bffa550346b31f8b5524/html5/thumbnails/11.jpg)
11
uFLIP (1): Basic construct: IO Pattern
• An IO Pattern is a sequence of IOs An IO is defined by 4 attributes (time, size, LBA, mode) Baseline Patterns (Seq. Read, Random Read, Seq. Write, Random Write) More patterns by using parameterized functions for each attribute
ConsecutiveSequential
ConsecutiveRandom
PauseSequential
BurstSequential
Ordered Partitioned
timeConsecutivePause (Pause)Burst (Pause,Burst)
size size (Size) LBA
SequentialRandomOrdered (Incr)Partitioned (Partitions)
modeReadWrite
![Page 12: 1 uFLIP: Understanding Flash IO Patterns Luc Bouganim, INRIA Rocquencourt, France Philippe Bonnet, DIKU Copenhagen, Denmark Björn Þór Jónsson, RU Reykjavík,](https://reader035.vdocument.in/reader035/viewer/2022062417/5518bffa550346b31f8b5524/html5/thumbnails/12.jpg)
12
uFLIP (1): Basic construct: IO Pattern
• An IO Pattern is a sequence of IOs An IO is defined by 4 attributes (time, size, LBA, mode) Baseline Patterns (Seq. Read, Random Read, Seq. Write, Random Write) More patterns by using parameterized functions for each attribute
• Potentially relevant IO patterns Basic patterns: one function for each attribute Mixed patterns: combining basic patterns Parallel patterns: replicating a basic pattern or mixing in // basic patterns
• Problems: IO patterns space is too large! Mixed and parallel patterns may be too complex to analyze
timeConsecutivePause (Pause)Burst (Pause,Burst)
size size (Size) LBA
SequentialRandomOrdered (Incr)Partitioned (Partitions)
modeReadWrite
![Page 13: 1 uFLIP: Understanding Flash IO Patterns Luc Bouganim, INRIA Rocquencourt, France Philippe Bonnet, DIKU Copenhagen, Denmark Björn Þór Jónsson, RU Reykjavík,](https://reader035.vdocument.in/reader035/viewer/2022062417/5518bffa550346b31f8b5524/html5/thumbnails/13.jpg)
13
uFLIP (2): What is a uFLIP micro-benchmark
• An execution of a reference pattern is a run. Measure the response time for individual IOs Compute statistics (min, max, mean, standard deviation) to summarize it.
• A collection of runs of the same pattern is an experiment Restriction to a single varying parameter for sound analysis
• A collection of related experiments is a micro-benchmark Defined over the baseline patterns with the same varying parameter
• 9 varying parameters 9 micro-benchmarks Basic patterns: IOSize, IOShift, TargetSize, Partitions, Incr, Pause,
Burst Mixed patterns: Ratio (mix only two baseline patterns) Parallel patterns: ParallelDegree (replicate in // each baseline pattern)
{{{IO}}}{{run}}{experiment}Micro-benchmark
![Page 14: 1 uFLIP: Understanding Flash IO Patterns Luc Bouganim, INRIA Rocquencourt, France Philippe Bonnet, DIKU Copenhagen, Denmark Björn Þór Jónsson, RU Reykjavík,](https://reader035.vdocument.in/reader035/viewer/2022062417/5518bffa550346b31f8b5524/html5/thumbnails/14.jpg)
14
uFLIP (3): The 9 micro-benchmarks
1 Granularity IOSize Basic performance? Device latency?
2 Alignment IOShift Penalty for badly aligned IOs?
3 LocalityTargetSize
IOs focused on a reduced area?
4 Partitioning Partitions IOs in several partitions?
5 Order Incr Reverse pattern, In place pattern, IOs with gaps?
6 ParallelismParallelDegree
IOs in parallel?
7 Mix Ratio Mixing two baseline patterns?
8 Pause Pause Device capacity to benefit from idle periods?
9 Bursts Burst Asynchronous overhead accumulation in time?
![Page 15: 1 uFLIP: Understanding Flash IO Patterns Luc Bouganim, INRIA Rocquencourt, France Philippe Bonnet, DIKU Copenhagen, Denmark Björn Þór Jónsson, RU Reykjavík,](https://reader035.vdocument.in/reader035/viewer/2022062417/5518bffa550346b31f8b5524/html5/thumbnails/15.jpg)
15
1
2
3
4
5
6
7
8
9
1 2 4 8 16 32 64 128TargetSize (MB)
Re
spo
ns
e ti
me
(re
lati
ve t
o S
W)
SamsungMemorightMtron
1
2
3
4
5
6
7
8
9
1 2 4 8 16 32 64 128TargetSize (MB)
Re
spo
ns
e ti
me
(re
lati
ve t
o S
W)
SamsungMemorightMtron
Results
Locality for the Samsung, Memoright and Mtron SSDs
• When limited to a focused area, RW performs very well
• For SR, SW and RR, linear behavior, almost no latency good throughputs with large IO Size
• For RW, 5ms for a 16KB-128KB IO
Granularity for the Memoright SSD
0
1
2
3
4
5
6
7
8
0 100 200 300 400 500
IO size (KB)
Re
spo
nse
tim
e (
ms)
SRRRSWRW
0
1
2
3
4
5
6
7
8
0 100 200 300 400 500
IO size (KB)
Re
spo
nse
tim
e (
ms)
SRRRSWRW
![Page 16: 1 uFLIP: Understanding Flash IO Patterns Luc Bouganim, INRIA Rocquencourt, France Philippe Bonnet, DIKU Copenhagen, Denmark Björn Þór Jónsson, RU Reykjavík,](https://reader035.vdocument.in/reader035/viewer/2022062417/5518bffa550346b31f8b5524/html5/thumbnails/16.jpg)
16
Baseline patterns (32 KB) Locality Partitioning Ordered Parallelism
SR RR SW RW RW SW Reverse In-Place
(ms) (ms) (ms) (ms) (MB) (Partitions) (Incr = -1) (Incr = 0)
SSD Memoright 0.3 0.4 0.3 5 8 (=) 8 (=) = = no
SSD Mtron 0.4 0.5 0.4 9 8 (2) 4 (1.5) = = no
SSD Samsung 0.5 0,5 0.6 18 16 (1.5) 4 (2) 1.5 0.6 no
Module Transcend 1.2 1.3 1.7 18 4 (2) 4 (2) 3 2 no
SSD Transcend MLC 1.4 3.0 2.6 233 4 (=) 4 (2) 2 2 no
USB Kingston DTHX 1.3 1.5 1.8 270 16 (20) 8 (20) 7 6 no
USB Kingston DTI 1.9 2.2 2.9 256 No 4 (5) 8 40 no
Results: summary
• SR, RR and SW are very efficient
• Flash devices incur large latency for RW
• Random writes should be limited to a focused area
• Sequential writes should be limited to a few partitions
• Good support for reverse and in place patterns.
• Surprisingly, no device supports parallel IO submission
![Page 17: 1 uFLIP: Understanding Flash IO Patterns Luc Bouganim, INRIA Rocquencourt, France Philippe Bonnet, DIKU Copenhagen, Denmark Björn Þór Jónsson, RU Reykjavík,](https://reader035.vdocument.in/reader035/viewer/2022062417/5518bffa550346b31f8b5524/html5/thumbnails/17.jpg)
17
Conclusion
• The uFLIP benchmark Sound methodology: Device preparation & setup stable measurements
Broad: 9 micro benchmarks, 39 experiments
Detailed: 1400 runs, 1 to 5 million I/Os ... for a single device!
Simple: an experiment = a 2-dimensional graph
Publicly available: www.uflip.org
• First results: Flash devices exhibit similar behaviors Despite their differences in cost / complexity / interface
• Current & Future work Short term: Visualization tool, with several levels of summarization
Enhance the software: setup parameters, benchmark duration, ...
Exploit the benchmark results!
![Page 18: 1 uFLIP: Understanding Flash IO Patterns Luc Bouganim, INRIA Rocquencourt, France Philippe Bonnet, DIKU Copenhagen, Denmark Björn Þór Jónsson, RU Reykjavík,](https://reader035.vdocument.in/reader035/viewer/2022062417/5518bffa550346b31f8b5524/html5/thumbnails/18.jpg)
18
Experiment color analysis
Run
IO
![Page 19: 1 uFLIP: Understanding Flash IO Patterns Luc Bouganim, INRIA Rocquencourt, France Philippe Bonnet, DIKU Copenhagen, Denmark Björn Þór Jónsson, RU Reykjavík,](https://reader035.vdocument.in/reader035/viewer/2022062417/5518bffa550346b31f8b5524/html5/thumbnails/19.jpg)
19
Details
![Page 20: 1 uFLIP: Understanding Flash IO Patterns Luc Bouganim, INRIA Rocquencourt, France Philippe Bonnet, DIKU Copenhagen, Denmark Björn Þór Jónsson, RU Reykjavík,](https://reader035.vdocument.in/reader035/viewer/2022062417/5518bffa550346b31f8b5524/html5/thumbnails/20.jpg)
20
Conclusion
• The uFLIP benchmark Sound methodology: Device preparation & setup stable measurements
Broad: 9 micro benchmarks, 39 experiments
Detailed: 1400 runs, 1 to 5 million I/Os ... for a single device!
Simple: an experiment = a 2-dimensional graph
Publicly available: www.uflip.org
• First results: Flash devices exhibit similar behaviors Despite their differences in cost / complexity / interface
• Current & Future work Short term: Visualization tool, with several levels of summarization
Enhance the software: setup parameters, benchmark duration, ...
Exploit the benchmark results!
![Page 21: 1 uFLIP: Understanding Flash IO Patterns Luc Bouganim, INRIA Rocquencourt, France Philippe Bonnet, DIKU Copenhagen, Denmark Björn Þór Jónsson, RU Reykjavík,](https://reader035.vdocument.in/reader035/viewer/2022062417/5518bffa550346b31f8b5524/html5/thumbnails/21.jpg)
21
www.uflip.org
Questions?
![Page 22: 1 uFLIP: Understanding Flash IO Patterns Luc Bouganim, INRIA Rocquencourt, France Philippe Bonnet, DIKU Copenhagen, Denmark Björn Þór Jónsson, RU Reykjavík,](https://reader035.vdocument.in/reader035/viewer/2022062417/5518bffa550346b31f8b5524/html5/thumbnails/22.jpg)
22
![Page 23: 1 uFLIP: Understanding Flash IO Patterns Luc Bouganim, INRIA Rocquencourt, France Philippe Bonnet, DIKU Copenhagen, Denmark Björn Þór Jónsson, RU Reykjavík,](https://reader035.vdocument.in/reader035/viewer/2022062417/5518bffa550346b31f8b5524/html5/thumbnails/23.jpg)
23
Selecting a device
![Page 24: 1 uFLIP: Understanding Flash IO Patterns Luc Bouganim, INRIA Rocquencourt, France Philippe Bonnet, DIKU Copenhagen, Denmark Björn Þór Jónsson, RU Reykjavík,](https://reader035.vdocument.in/reader035/viewer/2022062417/5518bffa550346b31f8b5524/html5/thumbnails/24.jpg)
24
Device result summary
Partitioning R W
Order R W
RR/RW SW/RWMix SR/RR SR/SW SR/RW RR/SW
Granularity SR RR SW RW
Alignment SR RR SW RW
Locality SR RR SW RW
Parallelism SR RR SW RW
Pause SR RR SW RW
Bursts SR RR SW RW
Experiment marked as interesting
Experiment marked as not interesting
Not performed
![Page 25: 1 uFLIP: Understanding Flash IO Patterns Luc Bouganim, INRIA Rocquencourt, France Philippe Bonnet, DIKU Copenhagen, Denmark Björn Þór Jónsson, RU Reykjavík,](https://reader035.vdocument.in/reader035/viewer/2022062417/5518bffa550346b31f8b5524/html5/thumbnails/25.jpg)
25
Experiment analysis
![Page 26: 1 uFLIP: Understanding Flash IO Patterns Luc Bouganim, INRIA Rocquencourt, France Philippe Bonnet, DIKU Copenhagen, Denmark Björn Þór Jónsson, RU Reykjavík,](https://reader035.vdocument.in/reader035/viewer/2022062417/5518bffa550346b31f8b5524/html5/thumbnails/26.jpg)
26
Experiment color analysis
Run
IO
![Page 27: 1 uFLIP: Understanding Flash IO Patterns Luc Bouganim, INRIA Rocquencourt, France Philippe Bonnet, DIKU Copenhagen, Denmark Björn Þór Jónsson, RU Reykjavík,](https://reader035.vdocument.in/reader035/viewer/2022062417/5518bffa550346b31f8b5524/html5/thumbnails/27.jpg)
27
Details