an in-depth look at snia's enterprise solid state storage … · 2020. 9. 16. · pts-e 1.0...

An In-Depth Look at SNIA's Enterprise Solid State Storage Performance Test Specification (PTS-E V1.1)

Easen Ho, CTO, Calypso Systems

An In-Depth Look at SNIA’s SSS PTS-E V 1.1 © 2012 Storage Networking Industry Association. All Rights Reserved. 2 2

SNIA Legal Notice

The material contained in this tutorial is copyrighted by the SNIA. Member companies and individual members may use this material in presentations and literature under the following conditions:

Any slide or slides used must be reproduced in their entirety without modification The SNIA must be acknowledged as the source of any material used in the body of any document containing material from these presentations.

This presentation is a project of the SNIA Education Committee. Neither the author nor the presenter is an attorney and nothing in this presentation is intended to be, or should be construed as legal advice or an opinion of counsel. If you need legal advice or a legal opinion please contact your attorney. The information presented herein represents the author's personal opinion and current understanding of the relevant issues involved. The author, the presenter, and the SNIA do not assume any responsibility or liability for damages arising out of any reliance on or use of this information. NO WARRANTIES, EXPRESS OR IMPLIED. USE AT YOUR OWN RISK.

An In-Depth Look at SNIA’s SSS PTS-E V 1.1 © 2012 Storage Networking Industry Association. All Rights Reserved.

Abstract

Many things influence the performance of a SSD: current and previous workloads, block size, R/W mix, queue depth, just to name a few… SNIA’s Solid State Storage (SSS) Performance Test Specification (PTS) provides methodology that ensures fair and consistent performance test results. This tutorial will provide an overview of the PTS-E V1.1 for enterprise SSDs as well as usage examples

3


Some Definitions and Clarifications

SNIA’s Solid State Storage Performance Test Specification (short hand) = PTS

Enterprise = PTS-E Client= PTS-C PTS-E 1.0 released 2011-04-26 PTS-C 1.0 released 2011-08-06

Solid State Storage (SSS) & Solid State Drive (SSD) in this presentation refers to NAND flash-based storage devices in all form factors and interfaces

4


Outline

Why a SSD Testing Standard is Needed Review of What’s in the PTS-E V1.0 What’s New in the PTS-E V1.1

Host Idle Recovery Test (HIR) Cross Stimulus Response Test(XSR) Demand Intensity/Response Time Histogram Test (DIRTH) Enterprise Composite Workload Test (ECW)

Other Work In Progress

5


SSD Testing - Need For Standards

No standard test methodology Everyone has a favorite benchmark program Pre-conditioning? Steady State? Demand Intensity? Active Range? Data Pattern? ….

No standard testing platform Platforms (software+hardware) becomes more relevant as SSDs gets faster & faster Can tests results be compared?

No standard performance metrics IOPS? TP? Block Size? R/W Mix? Response Time? ….

6


Where Do We Report Performance?

7

PCIe SLC MLC HDD

Log Scale!

An In-Depth Look at SNIA’s SSS PTS-E V 1.1 © 2012 Storage Networking Industry Association. All Rights Reserved. 8

Factoring Influencing Performance

Write History

• What was previously written

PC Active Range

• Where data was previously written

• Trim effects

Test Active Range

• Where data is currently written

• Temporal or spatial coherency?

Data Content and Size

• What is the nature of the data

Access Pattern

• Manner in which data is being accessed

Demand Intensity

• How hard the application is driving the device

Throttling

• How fast is data being written vs warranty

Others

• ?


Two specifications released

9


SSSI Reference Test Platform (RTP)

10

Intel S5520HC

Single Intel W5580, 3.2GHz, Quad-core CPU

12GB, 1333MHz, ECC DDR3 RAM

LSI 9212-4e4i 6Gb/s SAS HBA

Intel ICH10R 3Gb/s SATA

8X Gen-II PCI-e

CentOS 6.X

Calypso RTP Backend V1.5

Calypso Test Suite (CTS) V6.5


Outline




11


Basic Test Flow

12

• Security Erase, Sanitize, Format Unit, other proprietary methods where indicated 1. Purge

• Set user selectable test parameters, such as Active Range, Data Pattern, Demand Intensity 2. Set Conditions

• Workload independent (WIPC) • Workload dependent (WDPC) 3. Pre-Condition

• Reiterate loops until Steady State is reached, or run to a prescribed maximum number of loops 4. Run Until SS

• Collect data from Steady State Measurement Window 5. Collect Data

• Use standard report formats and include required and optional elements 6. Generate Reports

!

!


Tests Contained In PTS-E 1.0 SPEC

Enterprise Performance Test Specification (PTS-E) V1.0 encompasses:

A suite of basic SSS performance tests

Preconditioning and Steady State requirements

Standard test procedures and reporting requirements

13

Write Saturation

• Random Access • R/W: • 100% Writes

• BS: • 4KiB

Enterprise IOPS

• Random Access • R/W: • 100/0, 95/5, 65/35,

50/50, 35/65, 5/95, 0/100

• BS: • 1024KiB, 128KiB,

64KiB, 32KiB, 16KiB, 8KiB, 4KiB, 0.5KiB

Enterprise TP

• Sequential Access

• R/W: • 100/0, 0/100 • BS: • 1024KiB, 128KiB

Enterprise Latency

• Random Access • R/W: • 100/0, 65/35, 0/100

• BS: • 8KiB, 4KiB, 0.5KiB


Tests Contained In PTS-C 1.0 SPEC

14

Client IOPS

• Random Access • R/W: • 100/0, 95/5, 65/35, 50/50, 35/65,

5/95, 0/100 • BS: • 1024KiB, 128KiB, 64KiB, 32KiB,

16KiB, 8KiB, 4KiB, 0.5KiB • Range Restriction: • 100% & 75% LBA • 2048 Segments • Active Footprint

Restriction: • 8 & 16 GiB

Client TP

• Sequential Access • R/W: • 100/0, 0/100 • BS: • 1024KiB • Range Restriction: • 100% & 75% LBA • 2048 Segments • Active Footprint


Client Latency

• Random Access • R/W: • 100/0, 65/35, 0/100

• BS: • 8KiB, 4KiB, 0.5KiB • Range Restriction: • 100% & 75% LBA • 2048 Segments • Active Footprint



Key Concept: Pre-Conditioning

Transition

F O B

Steady State

Pre-Conditioning


Key Concept: Steady State

Premise is that reported data should be take only AFTER the test results show the drive has reached and MAINTAINED “Steady State” The Measurement Window is the interval, measured in Rounds, when the test results have entered and maintained Steady State for 5 Rounds

16


Key Concept: Steady State

Steady State is reached only if BOTH of the following conditions are satisfied (assuming “y” is the variable being tracked): 1. Variation of y within the Measurement Windows is

within 20% of the Average “ Max(y)-Min(y) within the Measurement Window is no more than 20% of

the Ave(y) within the Measurement Window; and ”

2. Trending of y within the Measurement Windows is within 10% of the Average

“ [Max(y) as defined by the linear curve fit of the data within the Measurement Window] – [Min(y) as defined by the best linear curve fit of the data within the Measurement Window] is within 10% of Ave(y) within the Measurement Window. “

17

…not too much data variation…

…not too much

trending…


SS Measurement Window

18


SS Measurement Window

19


Common Reporting Format (IOPS - Test Set-Up Pages 1-3 of 6)

Steady State Convergence Steady State Window Steady State Measurement Calculations

MLC-A


Common Reporting Format (IOPS - Results Pages 4-6 of 6)

Tabular Data 2D Plot 3D Bar Plot

MLC-A


Outline




22


Tests Contained In PTS-E 1.1

PTS-E 1.1 adds:

23

Host Idle Recovery

• Examines effect of idle (no IO) on small block RND writes

• RND/4KiB Writes

Cross Stimulus Response

• Examines switching between large block SEQ and small block RND writes

• SEQ/1024KiB &

RND/4KiB Writes

Demand Intensity – Response Time

Histograms

• Performance and detailed response time statistics under various workload types

• R/W=65/35 %, RND/8K

• R/W=90/10 %, RND/128K

• Response Time Histograms at various operating points

Enterprise Composite Workload

• Performance and detailed response time in a mixed IO Enterprise environment

• R/W=60/40 % • BS from 0.5-

64KiB • Three LBA

Groups


Host Idle Recovery Test (HIR) Purpose Examines Effect of Host Idle Period (No Host IO) On

The Performance of RND Small Block Writes

Test Setup

Preconditioning RND/4KiB Writes to Steady State

Test Insert various amount of idle time (no IO from host) between periods of 5 second RND/4KiB writes: Segment 1 (Wait State 1): 360 x (5S Write + 5S Idle) + 360 x (5S Write) Segment 2 (Wait State 2): 360 x (5S Write + 10S Idle) + 360 x (5S Write) Segment 3 (Wait State 3): 360 x (5S Write + 15S Idle) + 360 x (5S Write) Segment 4 (Wait State 5): 360 x (5S Write + 25S Idle) + 360 x (5S Write) Segment 5 (Wait State 10): 360 x (5S Write + 50S Idle) + 360 x (5S Write)


HIR Example: MLC/SATA

25


HIR Example: SLC/SAS

26


Cross Stimulus Response Test (XSR) Purpose Examines Switching Between Sustained Large Block SEQ

and Sustained Small Block RND Writes

Test Setup

Preconditioning None

Test Apply three Access Groups: Access Group 1 (Large Block SEQ): 100% SEQ Write, Block Size=1024 KiB Access Group 2 (Small Block RND): 100% RND Write, Block Size=4 KiB Access Group 3 (Large Block SEQ): 100% SEQ Write, Block Size=1024 KiB


Example: MLC/SATA

28

1

2 3


Example: MLC/SAS

29

1

2 3


Demand Intensity, Response Time Histogram Test (DIRTH)

Purpose Examines IOPS and Response Time Characteristics of Various Enterprise Workloads

Test Setup

Preconditioning Access Pattern, 100% Writes, until Steady State

Test 1. Using TC=[1-32] and OIO/Thread= [1-32], apply ECW using order of decreasing total OIO, until Steady State is reached for (32,32)

2. Manually determine the following operating points:

MaxIOPS: operating point with maximum IOPS while maintaining an ART < 5 mS MinIOPS: operating point with minimum measured IOPS MidIOPS: a minimum of one or more operating point(s) that has IOPS values between and equally divides the IOPS value spanned by MaxIOPS and MinIOPS 3. Perform Response Time Histograms, capturing all IO completion times for 10 Min at each operating points.


DIRTH Test

Currently there are two Access Patterns specified for the DIRTH test:

OLTP-Like: BS= 8 KiB R/W= 65/35 % Random Access, Random Data Full Drive Access

Video-Server-Like BS= 128 KiB R/W= 90/10 % Random Access, Random Data Full Drive Access

31


DIRTH (OLTP): Demand Intensity

32


DIRTH (OLTP): Demand Intensity

33


DIRTH (OLTP): MaxIOPS Point

34


DIRTH (OLTP): MidIOPS Point

35


DIRTH (OLTP): MinIOPS Point

36


Enterprise Composite Workload (ECW) Test

Purpose Examines IOPS and Response Time Characteristics Using a Mixed IO Workload

Test Setup

Preconditioning ECW, 100% Write, to Steady State

Test 1. Using TC=[1-32] and OIO/Thread= [1-32], apply ECW using order of decreasing total OIO, until Steady State is reached for (32,32)

2. Manually determine the following operating points:

MaxIOPS: operating point with maximum IOPS while maintaining an ART < 5 mS MinIOPS: operating point with minimum measured IOPS MidIOPS: a minimum of one or more operating point(s) that has IOPS values between and equally divides the IOPS value spanned by MaxIOPS and MinIOPS 3. Perform Response Time Histograms, capturing all IO completion times for 10 Min at each operating points.


The Enterprise Composite Workload

The ECW is a R/W=40/60%, random access pattern with a distribution of Block Sizes, each with a pre-defined Access Probability, plus restrictions on Access Range Probability Distribution

38

Block Size in Bytes (KiB) Access Probability Within Each Measurement Period

512 bytes (0.5 KiB) 4% 1024 bytes (1 KiB) 1% 1536 bytes (1.5 KiB) 1% 2048 bytes (2 KiB) 1% 2560 bytes (2.5 KiB) 1% 3072 bytes (3 KiB) 1% 3584 bytes (3.5 KiB) 1% 4096 bytes (4 KiB) 67% 8192 bytes (8 KiB) 10% 16,384 bytes (16 KiB) 7% 32,768 bytes (32 KiB) 3% 65,536 bytes (64 KiB) 3%

Total 100%

% of Access within

1 Measurement Period

Active Range Restriction

Label

50%

First 5% LBA Group A

30%

Next 15% LBA Group B

20%

Remaining 80% LBA Group C


ECW: Demand Intensity (PCIe, MLC)

39


ECW: TRSP Histograms (PCIe,MLC)

40

A

B

C

A


ECW: Demand Intensity (SAS, MLC)

41


ECW: TRSP Histograms (SAS,MLC)

42

A

B

C


Released SNIA SSS PTS is available for download at:

http://www.snia.org/tech_activities/sta

ndards/curr_standards/pts

43


Outline




44


PTS Follow-On Work (PTS-E/C)

45

• Conduct workload summary capture and see if different common use- cases can be found

Workload Capture/Distillation

• Test and metrics related to power usage and efficiency Power Efficiency


46

SSD Client & Enterprise PTS Comparison Data, Whitepapers, and Webcasts

Available at

www.snia.org/forums/sssi/pts

An In-Depth Look at SNIA’s SSS PTS-E V 1.1 © 2012 Storage Networking Industry Association. All Rights Reserved. 47 47

Q&A / Feedback

Please send any questions or comments on this presentation to SNIA: [email protected]

Many thanks to the following individuals

for their contributions to this tutorial.

- SNIA Education Committee

Eden Kim Easen Ho

mailto:[email protected]�

an in-depth look at snia's enterprise solid state storage … · 2020. 9. 16. · pts-e 1.0...

Documents