how iot, analytics and ml unfolds in storage fabric · 2019 storage developer conference india ©...

2019 Storage Developer Conference India © All Rights Reserved. 1

How IOT, Analytics and ML unfolds in Storage Fabric

Sharath T SMicrochip


Our Map!

Know your sources

Collect the data

Prepare data

Machine Learning

ML based Storage Mgmt.


Our itinerary

r Effect of IoT to Data Centersr Effect of IoT in Data Centersr Collection of data from sourcesr Prepare datar Applying ML for different uses casesr Data visualization


Effect of IoT to Data Centers


Effect of IoT to Data Centers

r 26 B sensors by 2020 and 50 B connected devicesr 5G IoTr Edge computingr Detailed analysisr IoT impact on data-center management


Effect of IoT in Data CentersKnow your

sources


End points in Data Centers

r EMS (Environmental Monitoring Systems) & ASHRAE (American Society of Heating, Refrigeration, and Air-Conditioning Engineers)r Temperature (18 to 27 °C)r Humidity and water (RH 45% to 60%)r Air flow sensorsr Static electric sensorsr Server room and rack entryr Aisle conditions


End points in Data Centers

r Server r Storage controllerr Physical Drives (S.M.A.R.T) r Chassis


Collection of data from sources

Know your sources

Collect the data


What data to collect?

r SensorrTemperature, humidity, static electric charges,

intrusionr Storage

rSystem, Storage pools, Storage volumes, Drives and Chassis


Example Data Collection

r System Informationr CPU Utilizationr Network Utilizationr Memory Utilizationr OS detailsr Uptime

r Storage Poolr Statusr Interfacer Total Sizer Unused Sizer Spare Rebuild moder Volume countr Drive countr Type

r Storage Volumer Statusr Interfacer Total Sizer Unused Sizer Block Sizer RAID Levelr Drive countr Protected by Hot-

Sparer Write-cacher Read-cacher Acceleration method

r Driver Manufacturerr Typer Statusr Interfacer Total Sizer Unused Sizer Reserved Sizer Block Sizer Transfer speedr SMART statsr Bad Blocks

r Storage Controller Information

r Statusr Moder Interfacer Temperature


When to collect data?

r Periodic intervalrTime Series analysis and Forecasting

r Event BasedrUser initiated, System initiated


How to collect data?r Push mechanism

r Source / System generatedr Breach of any threshold valuesr Listener is required to read and store value

r Pull mechanismr Application / User requestedr On demand / Periodicr Programmatically


Prepare data

Know your sources

Collect the data

Prepare data


r Types of source datar Unstructuredr Semi-structuredr Structured

r On-Line Analytical Processing (OLAP) of prepared datar Cubesr Dimensions

Data


Introduction to Machine Learning

Know your sources

Collect the data

Prepare data

Machine Learning


Introduction to Machine Learningr Introduction r Flow

Collecting data

Preparing data

Training the model

Evaluating the model

Improve performanceEvolution

EvolutionDigital Journal

http://www.digitaljournal.com/science/incredible-one-eyed-microscopic-creature-discovered/article/438974


Data Feature Selection


Data Feature Selection

r Filter method

r Ex: Chi-Square, LDA, ANOVA

Set of all Features

Selecting the best subset

Learning Algorithm Performance


Data Feature Selection…

r Wrapper method

r Ex: Forward selection, Backward elimination, Recursive feature elimination, etc.

Set of all Features

Generate subset


Selecting best of Subset


Data Feature Selection…

r Embedded method

r Ex: LASSOS, Ridge Regression

Set of all Features

Generate subset


Selecting best of Subset


Apply ML for multiple use cases

Know your sources

Collect the data

Prepare data

Machine Learning



Apply ML for multiple use cases…

r Case study - Data center managementr Drive failure predictionr Storage tiering suggestionr Storage usage trendr Storage requirement prediction


Drive Failure Prediction


Workflow

r What is S.M.A.R.T?r Data setr Periodic collectionr Selection features / attributes r Applying Support-vector-machine to predict drive failurer Outcome


Data set (Features)r List of S.M.A.R.T attributes

r Read Error Rater Reallocated Sectors

Countr Spin Retry Countr End-to-End errorr Temperaturer Command Timeoutr Reallocation Event Countr Uncorrectable Sector

Countr Soft ECC Correctionr G-Sense Error Rater Loaded Hours

q Sampling data (Periodic collection)Hours Temp1 . . . . . . ReadErr Servo4

2633 58 6 2944

2635 57 13 2688

2637 56 36 5189

2639 57 0 4032

2641 56 0 8384

. . . .

. . . .

2855 58 14 3322

2857 59 20 2624


Prepared Datar Feature selection

Attribute % Good % Failed

Temp1 11.8 48.2

Temp3 35.2 45.3

Temp4 8.8 59.2

Glist 0.5 8.8

ReadError1 0.4 0.8

WriteError 0.8 2.3

Reallocated sector 5.8 30.2

Uncorrectable sector 4.8 34.5

Spin-up time 5.2 14.2

Command timeout 6.2 29.8


Machine Learning Modelr Support-vector-machine (supervised learning)

r A Support Vector Machine (SVM) is a discriminative classifier formally defined by a separating hyperplane. In other words, given labeled training data, the algorithm outputs an optimal hyperplane which categorizes new examples.


X X

Y Y


X

ZY Y

X X


X X X

Y Y Y


X

Y

X

Y


Machine Learningr Train the model (80%)

Good Drive

Failed DriveSVM Model

r Test the model (20%)

SVM Model

Unlabeled Drive

Performance

Tune parameters


Outcome

r Reduce false alarm of failurer Automated policy implementation


Storage Tiering Suggestion


Storage Tiering

r Physically partitioned into multiple distinct classes based on price, performance or other attributesr Swordfish – ClassOfService

r Data may be dynamically moved among classes within a tiered storage implementation


Storage Class (SNIA)r Media class

r High performance SSD/Cache

r High performance HDDr High capacity HDDr Tape

r Data classr Mission criticalr Hotr Cold

r Pricing classr Networked storager DASr Cloud


Feature Collectionr Data/Block access frequencyr Last accessedr Last modification timer Size of objectr Encryptionr Drive type (HDD, SSD)r Drive interface typer Drive temperaturer Caching informationr …


Machine Learning Model

r k-means (unsupervised learning)r clusteringr centroidr aggregation


Machine Learning

Block

Acce

ss fr

eq.

On Cache On SSD On HDD

BlockAc

cess

freq

.Block

Acce

ss fr

eq.


Storage Usage Trendr How Storage is used in a data center over a time period.r Time Series Algorithmsr Ex of trends

r Which class of service is more used in futurer Which media class will be more used in future


Data Visualization

Know your sources

Collect the data

Prepare data

Machine Learning



Data Visualizationr Tools available

r Power BI, Tableau, Data Dog – commercial r Dash (personal favorite!) – opensource

r Dashr Pure python based frameworkr Abstracts away all technology and protocolr Ideal for building data visualization appsr https://plot.ly/products/dash/

https://plot.ly/products/dash/


Fraunhofer lines, in astronomical spectroscopy


Thank [email protected]

https://www.linkedin.com/in/sharath-ts-5720a520

mailto:[email protected]

https://www.linkedin.com/in/sharath-ts-5720a520

how iot, analytics and ml unfolds in storage fabric · 2019 storage developer conference india ©...

Documents