nand flash solid state storage performance and … slide or slides used must be reproduced in ......

NAND Flash Solid State Storage Performance and Capability -- an In-depth Look

jonathan thatcher, Fusion-io

NAND Flash Solid State Storage Performance and Capability © 2009 Storage Networking Industry Association. All Rights Reserved.

2

SNIA Legal Notice

The material contained in this tutorial is copyrighted by the SNIA. Member companies and individual members may use this material in presentations and literature under the following conditions:

Any slide or slides used must be reproduced in their entirety without modificationThe SNIA must be acknowledged as the source of any material used in the body of any document containing material from these presentations.

This presentation is a project of the SNIA Education Committee.Neither the author nor the presenter is an attorney and nothing in this presentation is intended to be, or should be construed as legal advice or an opinion of counsel. If you need legal advice or a legal opinion please contact your attorney.The information presented herein represents the author's personal opinion and current understanding of the relevant issues involved. The author, the presenter, and the SNIA do not assume any responsibility or liability for damages arising out of any reliance on or use of this information.

NO WARRANTIES, EXPRESS OR IMPLIED. USE AT YOUR OWN RISK.


Abstract

NAND Flash Solid State Storage Performance and Capability"This tutorial provides an in-depth examination of the fundamental theoretical performance, capabilities, and limitations of NAND Flash-based Solid State Storage (SSS). The tutorial will explore the raw performance capabilities of NAND Flash, and limitations to performance imposed by mitigation of reliability issues, interfaces, protocols, and technology types. Best practices for system integration of SSS will be discussed. Performance achievements will be reviewed for various products and applications. Several examples of successful enterprise deployments with comparative performance and cost-performance will be presented."

3


Data Integrity + Performance

There can be no data integrity trade-off for performance

4

Balance

Reliability & Data Integrity


Media Reliability / Availability

The GOOD:No moving partsCatastrophic device failures are rare (post infant mortality)

The BAD:Relatively high bit error rate, increasing with wear

MLC wear rate (higher capacity density) worse than SLCHigher density NAND Flash will increase bit error rate

Program and Read Disturbs

The UGLY:Partial Page ProgrammingData retention is poor at high temperatureInfant mortality is high (large number of parts…)

5


Controller Reliability Management

Wear leveling & Spare Capacity (e.g. Spare Blocks)Read & Program Disturb ControlsData & Index Protection

ECC CorrectionInternal RAIDData Integrity Field (DIF)

Management

Note: Multipage Programming Should Not Be Done

6

Poor Media + Great Controller Great SSS Solution


Data Integrity .V. Performance

7

BAD GOODP

erfo

rman

ce

Data Integrity99.999%99.99%99.9%99% …


Lower CapExFewer CPUsLess RAMLess Network GearFewer SW LicensesLess Space

Lower OpExLess HW MaintenanceLess SW MaintenanceGreater UptimeLess Power/CoolingFewer Diverse Skills

8

Higher Productivity

Performance is about ROI


SSS ROI: Three Quick Examples

Case StudiesWine.comCloudmark

O&G

9


Company: Wine.com

On-line internet Retail

Representative Markets: Transaction ProcessingData Mining

ProblemSystems not capable Contract out data miningPerformance hitting 100% Loss of revenue (4Q07)Cost to meet growth: prohibitive

10


11

Back EndPayment

Processing

Front End

33% CapEx avoidance50% OpEx reduction33% Footprint reduction

12x improvement on write- Latency down from 4 to <1ms

14x improvement on read- Latency down from 12 to <1ms

Before (3Q2008) After (1Q2009)

“Enough capacity to cover 24 months of growth”Geoffrey Smalling - CTO

Wine.com

InternetInternet

Application level replication for redundancy


Company: Cloudmark

Email Security – Spam, Phishing, Viruses, etc.

Protecting 850 Million mailboxes globally100+ Service Provider customers (mostly Tier-1)High TPS at master – morning bursts are higher

Problem scaling MySQL & InnoDB worldwideRapid growth – both customers and new threatsI/O limited hit performance ceiling 50% of the timeSlave servers fall behind increasing risk windowHigh transaction rate constant disk failuresCost to meet growth: prohibitive

12


Cloudmark

Transformed slave clusters to Solid State Storage and retired disk arrays – results:

4 to 1 footprint reductionCapEx avoidance - $2 million10x performance increaseCluster peak utilization now only 15% of IOPS capability12 months growth secured with current hardware

“We never thought our problems would ever change from disk I/O problems to CPU bottlenecks!”

Ryan White – Director of Operations

13


Company: LLOG

Oil & Gas explorationMarket:

Natural resource discovery using seismic interpretation leading to productionSeismic analysis software used for 3D interpretations

Manipulation and analysis of large datasets (1Tb+)

Quality interpretations result in accurate auction bidsMinimize risk and make informed decisions

Geoscientist productivity challengesIdle time loading data into high-end workstationsLost productivity during basic analysis tasksTime wasted in running jobs in serial

14


LLOG

Use of Solid State Storage on workstations Load time reduced from 8 to 2 minutes3D time slice time reduced from 28 to 18 minutes

“Supercharged Virtualization” with Solid State StorageProjects took 30+ minutes causing 100% CPU utilization

Now 10 minutes with projects run in parallelIncreased geoscientist’s productivity 5x (projects in parallel)

“We are more competitive because we make better decisions and did it while reducing our costs”

John Hollins – Geophysicist

15


16


17


Media Performance

The GOOD:Performance is great (wrt HDDs)High performance/power (IOPS/Watt)Low pin count: shared command / data bus good balance

The BAD:Not really a random access device

Block orientedSlow effective write (erase/transfer/program) latencyR/W access speed imbalance

Performance changes with wearSome controllers do read/modify/writeOthers use inefficient garbage collection

The UGLY:Some controllers do read/erase/modify/write

18


Performance Drivers – SSS Design

InterconnectNumber of NAND Flash Chips (Die)Number of Buses (Real / Pipelined)Data Protection (internal/external RAID; DIF; ECC…)SLC / MLCEffective Block (LBA; Sector) SizeWrite AmplificationGC EfficiencyBandwidth ThrottlingBuffer Capacity & Mgmt

19


Performance Drivers - External Cond

Transfer SizeRead/Write RatiosTemporal Randomness of AccessReserve Capacity Setting (% of used capacity)System Limitations (especially wrt scalability)

External Controller (#, Type, Performance); # ThreadsCPU (#Cores, GHz)System BandwidthSoftware Stack; Interrupt Handler

External RAIDLife of device (change in device affects tErase & tProgram)

20


Simplified Theoretical Analysis

Bandwidth Only (Not IOPS)Large Transfers (Data length = Integer * # die)Infinite BufferReads/Writes queued for maximum bandwidthNo system latency

Read/Write Ratio %’s fixed100/0, 75/25, 50/50, 25/75, 0/100Steady State, 100% Efficient GC (EB erase / EB written = 1)

Maximum Total BW for SATA-II and PCI-e X4No overhead considered

SLC

21


Bandwidth Depends on # Die

22

-

200

400

600

800

1,000

1,200

16 32 48 64 80 96 112 128 144 160 176 192

Theoretical BW (MB/s) v Number of Die (SLC, MLC)

SLC BW (0/100, 50/50, 75/25) MB/s

MLC BW (50/50) MB/s

MLC BW (25/75) MB/s

MLC BW (0/100) MB/s

Presumes 8 die per bus& 4 CS per bus

SLC MLC

Transfer Rate (MB/s) tRC & tWC 400 400

Page Program (us) tProgram 200 600

EB Erase (us) tErase 3000 10,000

Load Page (us) tR (tRead) 25 60

Capacity per die 0.5 1.0


Single Layer Cell v Multi-Layer Cell

23

-

100

200

300

400

500

600

700

800

900

1,000

1 2 3 4 5 6 7 8

Bandwidth (MB/s) v #Chip Select @ R/W Ratio

SLC 100/0

SLC 75/25

SLC 50/50

SLC 25/75

SLC 0/100

MLC 100/0

MLC 75/25

MLC 50/50

MLC 25/75

MLC 0/100

Read / write performance imbalance closed with additional banksGreater R/W imblance in MLC requires more banks


Performance Data Acquisition

In this presentation, three commercially available SSS storage solutions were tested under a variety of conditions using a common system:Supermicro X7DWE Motherboard

Dual, Quad Core Intel X5460: 3.16 GHz16 GB DRAM

4x4GBKVR800D2D4F5 /4GI, 800MHz FBDIMM DDR2, CL5

PCI-E Gen 2 bus

SATA Controller LSI SAS3081E-R All tests except pathological write, which used on-board SATADriver version used was: 4.00.43.00; firmware rev: FwRev=011b0000h

24

Note: all data collected on devices at beginning of life


Performance Data Acquisition

Software / OSLinux/CentOS5/RHEL5 2.6.18-92.1.10.el5. “fio” program, version 1.21:

From http://freshmeat.net/projects/fio)

“IOMeter” V 2006.07.27 in Windows 2003/2008 for latency only

SettingsDirect I/O: o_direct used to bypass caching & bufferingI/O Scheduler (elevator algorithm) set to null

25

Note: all data collected on devices at beginning of life

http://freshmeat.net/projects/fio


Measured v Theoretical Bandwidth

26

Note: Theoretical Max BW with 24 channels (4 die per bus, 4 CS per bus) is identical to the PCI-C, 24 channel shown in these charts.

Capacity Multiplier: SATA-B: 1PCI-C: 2

0

500

1000

1500

2000

100/0 (X1)

75/25 (X1)

50/50 (X1)

25/75 (X1)

0/100 (X1)

Measured v Theoretical Max BW

24 channels (8 die per bus; 4 CS per bus)

PCI-C BW Measured

10 channels (SLC; 4 die per bus; 4 CS per bus)

SATA-B BW Measured

PCIe-X4 Max BW

SATA-II Max BW

40%50%60%70%80%90%

100%

100/0 (X1)

75/25 (X1)

50/50 (X1)

25/75 (X1)

0/100 (X1)

Measured BW as % of Theoretical Max

PCI-C BW Actual/Theoretical

SATA-B BW Actual/Theoretical


Product Specifications

27

SATA (A) SATA (B) PCI (C)

Capacity (GB) 32 32 160

Bus/Link SATA-II (3 Gb/s) SATA-II (3 Gb/s) PCI-E X4 1.1

MemoryType SLC SLC SLC

Adjustable Reserve Capacity No No Yes

SSS Internal RAID-- Running during test

NoN/A

NoN/A

YesYes

K-IOPS (RMS) 8 27 88

K-IOPS (RMS) / WATT 3 ? 7

Bandwidth (RMS, MB/s) 56 208 743

ECC correction 7 bits in 512B ? 11 bits in 240B

Features directly affecting performance measurements


Access Process (Physics Ignored)

Read AccessAddress Chip / EB / PageLoad Page into RegisterTransfer Data From Register 1-byte per cycle

Write AccessAddress Chip / EBErase EB

…some time later…

Address Chip / EB / PageTransfer Data To Register 1-byte per cycleProgram Register to Page

28

Typical NAND Flash Die:• 2000 Erase Blocks (EB)• 64 Pages per EB• 4000 Bytes per Page• 500 MByte Total Capacity


Example 1: Read/Erase/Modify/Write

29

Page Erase Block 1

0 b c -- --

1 j -- k l

2 m -- -- --

3 -- -- q r

Page Erase Block 1

0 b c W X

1 j Y k l

2 m

3 q r

Time = t2

Write Buffer & W,X,Y

Time = t1

Starting State

Time = t3

Write Buffer & Z,A,B’,C’,R’

Buffer holds data while EB-1 Erased

Page Erase Block 1

0 B’ C’ w x

1 j y k l

2 m Z A

3 q R’

Page Erase Block 1

0

1

2

3

Buffer holds data while EB-1 Erased

Page Erase Block 1

0

1

2

3


Explanation of Previous Slide

AssumptionsSimplified to show erase blocks with 4 pages, each page having 4 data blocksInvalid (erased or replaced) data is indicated by “—”Old data is indicated by lower case lettersNew data is indicated by CAPs; Replacement data is indicated by “prime” (e.g. c C’)

Detail T = t1 to T = t2 transitionData is read from EB-1EB-1 is erasedNew data {W,X,Y} modifies previous invalid dataData is written back to EB-1

Detail T = t2 to T = t3 transitionData is read from EB-1 into data bufferEB-1 is erasedNew data {B’, C’, Z, A, R’} modifies previous data in data bufferData is written back to EB-1

30

Note: backup material for those reviewing or looking at

presentation without audio/video


Example 2: Read/Modify/Write

31

Page Erase Block 1

0 b c -- --

1 j -- k l

2 m -- -- --

3 -- -- q r

Page Erase Block 2

0 b c W X

1 j Y k l

2 m

3 q r

Page Erase Block 3

0 B’ C’ w x

1 j y k l

2 m Z A

3 q R’

Time = t2

Data to Buffer (not shown)

Erase EB-1 (not shown)

Write Buffer & W,X,Y to EB-1

Time = t1

Starting State

Time = t3

Data to Buffer (not shown)

Erase EB-1 (not shown)

Write Z,A & Replace b,c,rwith B’,C’,R’ & Write EB-1

Implicit wear leveling; EB-1 EB-2 EB-3Presumes that destination EB-2 & EB-3 erased prior to transfer of data higher

performance (than previous “Read/Erase/Modify/Write” example)


“Write Amplification Impact”

In this example, Data written t1 to t2: 16 blocks

NEW DATA {W, X, Y} 3 blocks; Copied Data {b, c, j, k, l, m, q, r} 8 blocksNull Data: 5 blocks

Data written t2 to t3: 16 blocksNEW DATA {B’, C’, Z, A, R’} 5 blocks; Copied Data {w, x, j, y, k, l, m, q} 8 blocksNull Data: 3 blocks

(2) EB erasures25% (8 of 32) writes are user initiated75% (24 of 32) writes are internal data movement (overhead)

Important:Amount of valid or invalid data in EB-1 is irrelevant to performance impact“Write Amplification” is workload (access pattern) dependent (e.g., what if the write of R’ above was not coincident with B’ & C’)

32


SATA-B: IOPS vs Transfer Size

33

0

10000

20000

30000

40000

50000

60000

512 1k 2k 4k 8k 16k 32k 64k 128k 256k 512k 1m

SATA-B: IOPS v Transfer Size (X1)

1x Write

1x 25/75

1x 50/50

1x 75/25

1x Read


Example 3: Garbage Collection

34

Page Erase Block 1

0 b c -- --

1 j -- k l

2 m -- -- --

3 -- -- q r

Page Erase Block 1

0 b c -- --

1 j -- k l

2 m -- -- --

3 -- -- q r

Page Erase Block 1

0

1

2

3

Page Erase Block 2

0

1

2

3

Page Erase Block 2

0 W b c X

1 Y j k l

2 m q r

3

Page Erase Block 2

0 w -- -- x

1 y j k l

2 m q -- B’

3 C’ Z A R’

Time = t2

EB-1 GC’d to EB-2

W,X,Y added

Time = t3

EB-1 erase

b,c,r replaced by B’,C’,R’

Time = t1

Start Garbage Collect EB-1


Explanation of Previous Slide

AssumptionsNew data blocks and data blocks being garbage collected are interleaved

DetailsAt Time = t0, erase block 1 (EB-1) is identified for GCAt Time = t1, good data is moved from EB-1 to EB-2 (it is implicit that an index is updated accordingly); New data W, X, and Y are added while the GC is taking place. EB-1 is then ready to be erasedAt Time = t2, EB-2 is erased; Data for b, c, & r have been updated with B’, C’ & R’; b, c & r are indicated as “Invalid.”

35

Note: backup material for those reviewing or looking at

presentation without audio/video


GC Performance Impact

In this example, COPIED DATA: {b, c, j, k, l, m, q, r} 8 blocksNEW DATA {W, X, Y, B’, C’, Z, A, R’} 8 blocks50% (8 of 16) writes are user initiated50% (8 of 16) writes are internal movement (overhead)

Important:50% of EB-1 was “invalid data”What if only 10% had been “invalid data?”GC efficiency is dependent upon % of reserve capacity

36


Tower of Hanoi

37

Want to do this in fewer moves?Add more pegs!


GC: Pathological Write Conditions

IF high percentage of total storage capacity utilized

AND

High percentage of data has no correlation-in-time

AND

Continuous writing (no recovery time for GC)

THEN

Efficiency of GC greatly diminished

38


Pathological Write Condition

39

0

100

200

300

400

500

600

7000

540

1081

1622

2162

2703

3244

3785

4325

4866

5407

5948

6489

7030

7571

8112

8653

9194

MB

/s

Seconds

30GiB of 80G PCI-C

40GiB of 80G PCI-C

60GiB of 80G PCI-C

70GiB of 80G PCI-C

74GiB of 80G PCI-C

28GiB of 30G SATA-B

User Capacity Formatted of Total


Scalability

Following Slides Show Scalability of {1, 2, 4, 8} unitsOnly 1 SATA controller is used – limiting scalability

Only 1 thread running

Measurements taken at Read/Write Ratios of {100/0, 75/25, 50/50, 25/70, 0/100} RMS value is the “root mean square” of these five values

IOPS measurement taken at 512 Byte TransfersBandwidth taken at 128K Byte Transfers

Unless shown differentlyLinux has a 128K limit

40


Performance v R/W Ratio

41

0

20000

40000

60000

80000

100000

120000

100/

0 (X

1)

75/2

5 (X

1)

50/5

0 (X

1)

25/7

5 (X

1)

0/10

0 (X

1)

IOPS @ 512 B

0

100

200

300

400

500

600

700

800

900

100/

0 (X

1)

75/2

5 (X

1)

50/5

0 (X

1)

25/7

5 (X

1)

0/10

0 (X

1)

Bandwidth (MB/s) @ 128 KB

SATA-ASATA-BPCI-C

Read/Write Collisions Drop in Mixed Performance


Scalability v R/W Ratio

42

R/W Ratio (# Units in Parallel)

0

50000

100000

150000

200000

250000

300000

100/

0 (X

1)75

/25

(X1)

50/5

0 (X

1)25

/75

(X1)

0/10

0 (X

1)

100/

0 (X

2)75

/25

(X2)

50/5

0 (X

2)25

/75

(X2)

0/10

0 (X

2)

100/

0 (X

4)75

/25

(X4)

50/5

0 (X

4)25

/75

(X4)

0/10

0 (X

4)

100/

0 (X

8)75

/25

(X8)

50/5

0 (X

8)25

/75

(X8)

0/10

0 (X

8)

IOPS @ 512 B

X1 X2 X4 X8

0

1000

2000

3000

4000

5000

6000

100/

0 (X

1)75

/25

(X1)

50/5

0 (X

1)25

/75

(X1)

0/10

0 (X

1)

100/

0 (X

2)75

/25

(X2)

50/5

0 (X

2)25

/75

(X2)

0/10

0 (X

2)

100/

0 (X

4)75

/25

(X4)

50/5

0 (X

4)25

/75

(X4)

0/10

0 (X

4)

100/

0 (X

8)75

/25

(X8)

50/5

0 (X

8)25

/75

(X8)

0/10

0 (X

8)

Bandwidth (MB/s) @ 128 KB

SATA-A

SATA-B

PCI-C

X1 X2 X4 X8


RMS Scalability (# SSS Units)

43

0

1000

2000

3000

4000

5000

6000

1 2 4 8

RMS of Bandwidth v Scale

SATA-A

SATA-B

PCI-C

0

50000

100000

150000

200000

250000

1 2 4 8

RMS of IOPS v Scale

0.6

0.7

0.8

0.9

1

1.1

1 2 4 8

Normalized RMS Bandwidth v Scale

SATA-A

SATA-B

PCI-C

0

0.2

0.4

0.6

0.8

1

1.2

1 2 4 8

Normalized RMS IOPS v Scale


Performance v Block Size (75/25)

44

0

20000

40000

60000

80000

100000

120000

512 1k 2k 4k 8k 16k

32k

64k

128k

256k

512k 1m

Block Size

75/25 R/W IOPS

0

100

200

300

400

500

600

700

800

900

512 1k 2k 4k 8k 16k

32k

64k

128k

256k

512k 1m

Block Size

75/25 R/W Bandwidth (MB/s)

SATA-ASATA-BPCI-C


Scalability v RW Ratio v Block Size

45

512

32k0

250

500

750

1000

8x W

rite

8x 2

5/75

8x 5

0/50

8x 7

5/25

8x R

ead

4x W

rite

4x 2

5/75

4x 5

0/50

4x 7

5/25

4x R

ead

2x W

rite

2x 2

5/75

2x 5

0/50

2x 7

5/25

2x R

ead

1x W

rite

1x 2

5/75

1x 5

0/50

1x 7

5/25

1x R

ead

SATA-A Scalability Bandwidth (MB/s)

0

10000

20000

30000

40000

50000

60000

70000

80000

90000

100000

8x W

rite

8x 2

5/75

8x 5

0/50

8x 7

5/25

8x R

ead

4x W

rite

4x 2

5/75

4x 5

0/50

4x 7

5/25

4x R

ead

2x W

rite

2x 2

5/75

2x 5

0/50

2x 7

5/25

2x R

ead

1x W

rite

1x 2

5/75

1x 5

0/50

1x 7

5/25

1x R

ead

SATA-A ScalabilityIOPS


5128k

128k0

250

500

750

1000

1250

1500

8x W

rite

8x 2

5/75

8x 5

0/50

8x 7

5/25

8x R

ead

4x W

rite

4x 2

5/75

4x 5

0/50

4x 7

5/25

4x R

ead

2x W

rite

2x 2

5/75

2x 5

0/50

2x 7

5/25

2x R

ead

1x W

rite

1x 2

5/75

1x 5

0/50

1x 7

5/25

1x R

ead

SATA-B ScalabilityBandwidth (MB/s)


46

0

10000

20000

30000

40000

50000

60000

70000

80000

90000

100000

110000

120000

8x W

rite

8x 2

5/75

8x 5

0/50

8x 7

5/25

8x R

ead

4x W

rite

4x 2

5/75

4x 5

0/50

4x 7

5/25

4x R

ead

2x W

rite

2x 2

5/75

2x 5

0/50

2x 7

5/25

2x R

ead

1x W

rite

1x 2

5/75

1x 5

0/50

1x 7

5/25

1x R

ead

SATA-B ScalabilityIOPS



47

0100002000030000400005000060000700008000090000100000110000120000130000140000150000160000170000180000190000200000210000220000230000240000250000260000270000

8x W

rite

8x 2

5/75

8x 5

0/50

8x 7

5/25

8x R

ead

4x W

rite

4x 2

5/75

4x 5

0/50

4x 7

5/25

4x R

ead

2x W

rite

2x 2

5/75

2x 5

0/50

2x 7

5/25

2x R

ead

1x W

rite

1x 2

5/75

1x 5

0/50

1x 7

5/25

1x R

ead

PCI-C Scalability IOPS

5128k

128k02505007501000125015001750200022502500275030003250350037504000425045004750500052505500575060006250

8x W

rite

8x 2

5/75

8x 5

0/50

8x 7

5/25

8x R

ead

4x W

rite

4x 2

5/75

4x 5

0/50

4x 7

5/25

4x R

ead

2x W

rite

2x 2

5/75

2x 5

0/50

2x 7

5/25

2x R

ead

1x W

rite

1x 2

5/75

1x 5

0/50

1x 7

5/25

1x R

ead

PCI-C ScalabilityBandwidth (MB/s)


Media Reliability / Availability

48

Problem SSS Solution System Solution

High Infant Mortality RAID: NAND Flash RAID: SSS

High Error Rate & Wearout

RAID: NAND Flash RAID: SSS

Robust ECC No Defragmentation

DIF DIF

Wear Leveling Access Tuning

Never Multi-page PGM

Thermal Management Temperature & Air Flow

Non-random Address Indirection N/A

R/W not symmetric More Banks (CS); esp for MLC Reduce write thrashing


System Level Considerations

Data / Index Protection (DIF ; RAID)

Scalability

Compare system- or data-center-level; not device

Best case: test on real application, not benchmarkPlan to do tuning to reach top performance / objectivesApplications may have contra-indicated optimizations

Example: keeping data in close physical proximity (short stroking)Example: caching algorithms

49


Questions to Ask : Things to Know

Bandwidth / IOPS atBlock size(s) you needR/W ratio you useSteady State / BurstData’s temporal relationshipScalabilityRAIDingReserve capacity usedBOL / EOL

50


Questions to Ask : Things to Know

Design impacts on data integrity; life; failures & performanceECC robustnessWrite amplification / GC efficiencyInternal RAIDBandwidth throttlingPartial Page Programming

Test ConditionsRAID On/Off during testing?Caching On/Off during testing?WorkloadTemporal RelationshipsUser capacity / reserve capacity

51


52

Q&A / Feedback

Please send any questions or comments on this presentation to SNIA: [email protected]

Many thanks to the following individuals for their contributions to this tutorial.

- SNIA Education Committee

Khaled AmerPhil Mills

Rob PeglarMarius Tudor

mailto:[email protected]

nand flash solid state storage performance and … slide or slides used must be reproduced in ......

Documents